Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-05
19:56:48 mriedem stvnoyes: i think i what might be an issue with boot from volume with multiple instances and the new cinder volume attach flow...
19:57:36 sean-k-mooney mriedem: i have 3 patches i need to rebase for rocky later this week so ill add it to the list if it wont effect future backport or ill try to make time to fix it specificlly tomrow if it more urgent. anyway got to run enjoy your evening.
19:59:23 mriedem stvnoyes: we call this for each instance we create in a multi-create request https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1253
19:59:46 mriedem and _validate_bdm will loop over the volumes passed in, and if everything is new enough, create an attachment per volume and store that on the bdm record
20:00:05 mriedem we then make a copy of that list per instance https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1267
20:00:27 mriedem i'm wondering if we're overwriting or losing any of the bdm.attachment_id records when creating multiple instances
20:00:31 mriedem maybe it's ok
20:01:29 mriedem might be a decent test though, create >1 instances in the same request using the same volume and make sure each instance has unique attachments to that volume
20:02:11 mriedem i suppose if you aren't using a multiattach volume, the 2nd instance in the loop would fail to create an attachment to the volume because the volume's attach_status will be 'attaching'
20:04:29 ameeda mriedem: now the commit message have those 2 lines
20:04:30 ameeda (cherry picked from commit eba33a0e2d8aace6fbc889026fb80dc2c208bc8d) (cherry picked from commit c0aca1d333628f9174e8db53674787e281e7e0d7)
20:05:51 mriedem ameeda: those should go on separate lines
20:05:52 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug nova-api service version check during boot from volume https://review.openstack.org/540992
20:05:53 mriedem but that's ine
20:05:55 mriedem fine
20:08:13 mriedem dansmith: we haven't been doing proper service version checking in nova-api since pike https://review.openstack.org/#/c/539755/ - should be an easy +W
20:08:21 ameeda mriedem: yes they are , so I can do git review now ?
20:08:27 mriedem ameeda: yes
20:08:42 ameeda mriedem: Thank you !!
20:09:15 openstackgerrit Ameed Ashour proposed openstack/nova stable/ocata: add "--until-complete" option for nova-manage db archive_deleted_rows. https://review.openstack.org/540274
20:09:51 ameeda mriedem: could you please give me +2 here https://review.openstack.org/#/c/528385/ :$
20:10:10 melwitt jaypipes: replied
20:12:11 mriedem ameeda: your ocata backport is wrong
20:14:05 ameeda I see
20:23:27 openstackgerrit Ameed Ashour proposed openstack/nova stable/ocata: add "--until-complete" option for nova-manage db archive_deleted_rows. https://review.openstack.org/540274
20:27:42 ameeda mriedem: I don't know if I did the correct thing, please check it ^
20:28:33 mriedem ameeda: yes that's correct, thanks
20:29:53 ameeda Thank you !!
20:30:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a nova-caching-scheduler job to the experimental queue https://review.openstack.org/539260
20:32:18 ameeda mriedem: please don't forget to review this https://review.openstack.org/#/c/528385/
20:56:55 mriedem efried: i think this might have broken the nova-next job but i'm not sure how https://github.com/openstack-dev/devstack/commit/49144e627120f9ec1cf72dfc84e02b9b81e669b6
20:56:59 mriedem https://bugs.launchpad.net/nova/+bug/1747511
20:56:59 openstack Launchpad bug 1747511 in OpenStack Compute (nova) "legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find requested image" since 1/31" [Undecided,New]
20:57:51 mriedem the only thing unique about the nova-next job at this point is it sets NOVA_USE_SERVICE_TOKEN=True in devstack
20:59:45 openstackgerrit Dan Smith proposed openstack/nova master: Bump compute RPC API to version 5.0 https://review.openstack.org/541005
20:59:49 dansmith gawd
21:00:00 mriedem ufta
21:06:58 efried mriedem: Is the bustage consistent?
21:07:15 mriedem efried: yeah, 100% fail
21:07:48 efried mriedem: I can propose reversion of the above, then a blank nova patch that Depends-On it, and see if that at least makes it pass. Shall I?
21:08:03 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug nova-next image not found failures https://review.openstack.org/541008
21:08:18 mriedem i think we could also just ignore the service_user config in nova as a test
21:08:43 efried I have to jet for a doc appt right now, will catch up with you in an hour or so.
21:12:18 jaypipes thx melwitt
21:15:05 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Add release note for Aggregate[Core|Ram|Disk]Filter change https://review.openstack.org/541013
21:16:25 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug nova-next image not found failures https://review.openstack.org/541008
21:30:42 openstackgerrit melanie witt proposed openstack/nova master: Add release note for Aggregate[Core|Ram|Disk]Filter change https://review.openstack.org/541018
21:48:14 mriedem maciejjozefczyk: not sure why you'd +1 this :) https://review.openstack.org/#/c/540992/
21:50:32 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug nova-api service version check during boot from volume https://review.openstack.org/540992
21:56:19 openstackgerrit Matthew Edmonds proposed openstack/nova master: Clarify CONF.scheduler.max_attempts https://review.openstack.org/541031
21:56:46 edmondsw_ edleafe noticed this after reading your blog ^
21:59:31 edmondsw mriedem do we have a more general term that encompasses create, resize, and migrate if "build" doesn't do that?
22:02:38 mriedem scheduling?
22:02:41 mriedem like it said
22:02:42 edmondsw we can't say "schedule", because we only talk to the scheduler once.
22:03:26 melwitt it's still scheduling though, isn't it? trying alternate hosts *shrug*
22:03:42 edmondsw I guess
22:03:58 edmondsw at least the "for a chosen host" bit from the old help text is wrong, right?
22:04:09 edleafe mriedem: I think the problematic wording is "for a chosen host". But that was wrong before, too.
22:04:14 edleafe jinx
22:04:28 edmondsw edleafe yeah that is what first caught my eye
22:04:42 mriedem yes 'for a chosen host' is wrong
22:04:46 edmondsw I can change it back to scheduling and correct that part
22:05:27 edleafe "for a given VM build/move operation"
22:06:07 edmondsw I like that...
22:06:09 mriedem can we somehow make this about ironic and say, 'for a given bm movement'
22:06:32 edleafe nah, that sounds kinda crappy
22:06:33 edmondsw lol
22:07:43 openstackgerrit Matthew Edmonds proposed openstack/nova master: Clarify CONF.scheduler.max_attempts https://review.openstack.org/541031
22:08:15 efried mriedem: I have returned. Any update on the glance image snafu?
22:08:45 mriedem efried: debug patch is still running
22:08:53 jroll edleafe: "for a given VM b/m operation" :P
22:09:49 mriedem i can't wait to be in the vm bm room for 2 days in dublin
22:10:12 mriedem talking about...keystone
22:10:21 mriedem i came for the bm, stayed for the keystone
22:10:35 openstackgerrit Matthew Edmonds proposed openstack/nova master: Clarify CONF.scheduler.max_attempts https://review.openstack.org/541031
22:11:23 edleafe I'm SO glad I made it out of third grade
22:11:56 openstackgerrit Dan Smith proposed openstack/nova master: Bump compute RPC API to version 5.0 https://review.openstack.org/541005
22:11:57 openstackgerrit Dan Smith proposed openstack/nova master: Compute RPC client bump to 5.0 https://review.openstack.org/541035
22:12:06 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Query all cells for service version in _validate_bdm https://review.openstack.org/541036
22:13:23 mriedem efried: results http://logs.openstack.org/08/541008/2/check/legacy-tempest-dsvm-neutron-nova-next-full/f24c9d2/logs/screen-n-api.txt.gz?level=TRACE
22:15:02 efried mriedem: Looks to me like neither of your glance.py messages hit.
22:15:09 efried which is... weird.
22:15:53 openstackgerrit Matthew Edmonds proposed openstack/nova master: Clarify CONF.scheduler.max_attempts https://review.openstack.org/541031
22:16:49 mriedem http://logs.openstack.org/08/541008/2/check/legacy-tempest-dsvm-neutron-nova-next-full/f24c9d2/logs/screen-n-api.txt.gz#_Feb_05_21_46_51_455934
22:16:53 mriedem Using glance API servers: [u'http://158.69.92.100/image/v2/']
22:17:49 efried mriedem: http://logs.openstack.org/08/541008/2/check/legacy-tempest-dsvm-neutron-nova-next-full/f24c9d2/logs/screen-g-api.txt.gz#_Feb_05_21_46_51_651885
22:17:55 mriedem efried: maybe the image exists but the context doesn't have auth to GET it?
22:18:06 efried mriedem: This is https://bugs.launchpad.net/python-glanceclient/+bug/1707995
22:18:07 openstack Launchpad bug 1707995 in Glance Client "Can't use versioned endpoints" [Undecided,New]
22:18:18 mriedem shazbot!
22:18:27 efried Though I thought we weren't subject to that with this set of changes.
22:18:58 mriedem not sure why the other jobs wouldn't be failing though
22:19:23 efried Also checked this through the CI six ways from sunday before we merged it.
22:19:46 mriedem devstack doesn't run nova-next
22:19:52 efried So some weird combo of environments - or possibly a recent "fix" in ksa? - is exposing this.
22:20:13 mriedem i can try to nuke the usage of the service_user in this debug patch and see if things start magically passing
22:20:20 efried mriedem: No, but I made DNM nova patches with Depends-On the devstack change, exactly to validate this.
22:20:28 mriedem nova-next is also non-voting
22:20:37 mriedem so might not have noticed

Earlier   Later