Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-02
13:28:07 openstackgerrit Eric Fried proposed openstack/nova master: Process requested_resources in ResourceRequest init https://review.opendev.org/696354
13:28:07 openstackgerrit Eric Fried proposed openstack/nova master: Reusable RequestGroup.add_{resource|trait} https://review.opendev.org/696380
13:28:08 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use string suffixes and provider mappings https://review.opendev.org/696418
13:28:16 efried stephenfin, gibi: fixed the unit test on the bottom one ^
13:34:26 openstackgerrit Merged openstack/os-traits master: Add 'TYPE_PLOOP' image type. https://review.opendev.org/696435
14:23:17 gibi efried: I'm +2 on the bottom patch
14:23:25 efried thanks gibi
14:23:34 gibi thanks for starting this work
14:23:42 gibi you are solving some of my todos :)
14:23:47 efried does the strategy seem sane to you wrt the requester_id <=> group ID?
14:24:12 gibi efried: yeah, that step is what I had in my mind
14:24:17 efried great
14:25:30 gibi efried: I'm a bit affraid the replacement of fill_provider_mapping with the mapping from the a_c response. Theortically it should work I just hope there will not be too much complication
14:25:42 efried we'll see :)
14:26:03 efried gibi: I think the most complicated bit might be revert_resize
14:26:12 gibi yes, that is the type of surgery that better do that plan
14:26:19 efried By that time the original mappings might be long gone
14:26:33 efried ...unless we persist them in the RequestSpec...?
14:26:49 gibi efried: in case of revert_resize we can either start using multiple port binding for resize which is hard but nice, or persist the mapping somewhere
14:27:01 gibi if we go to the later then I would persist it to the Migration object
14:27:29 gibi as the migration object has the proper lifecycle for this type of data
14:27:37 gibi the RequestSpec lives longer
14:41:25 sean-k-mooney gibi: i would like to move to useing multiple port bindings for resize for what its worth
14:41:38 sean-k-mooney gibi: we will be using it for cross cell resize
14:41:58 sean-k-mooney so i think it makes sense to just use it always for resize/cold migrate
14:46:02 sean-k-mooney efried: gibi when ye have time could yet take a look at https://review.opendev.org/#/c/674072/ and maybe https://review.opendev.org/#/c/695118/
14:47:31 sean-k-mooney ill be going on vacation in 2 weeks but i have a 4 patches i really want to get merged if i can before i do those are two of them and im resping the other two now
14:49:56 efried stephenfin, lyarwood: thoughts on bug tagging here? https://review.opendev.org/#/c/682594/
14:51:41 efried sean-k-mooney: that might be a bit out of my bailiwick, but I'll give it a try.
14:53:01 stephenfin looking
14:53:32 lyarwood efried: yeah that's fair, I can also address the nits before we merge this afternoon
14:53:41 efried okay
14:53:48 lyarwood efried: I think I was trying to avoid any suggestion that this was the only fix required here
14:53:58 stephenfin Your call. I could go either way. I'm guessing it was left that way since mriedem has a better albeit not backportable fix
14:54:04 stephenfin yup
14:54:28 efried right, I'm looking at it from the perspective of stable, where if we don't Closes-Bug on this patch, there will never be a "fix".
14:54:58 efried I guess if you really want, we could keep this one Related-Bug and switch it to Closes-Bug on the backports
14:55:07 efried can't believe there's not a precedent for something like this.
14:56:03 lyarwood Yeah not that I recall but I'd be okay with that from a stable point of view
14:56:15 lyarwood related on master, closes on stable that is
14:56:23 sean-k-mooney well does it actully fix the bug
14:56:30 sean-k-mooney or just make it less bad
14:56:58 sean-k-mooney ye are talking about https://bugs.launchpad.net/nova/+bug/1844296 right
14:56:58 openstack Launchpad bug 1844296 in OpenStack Compute (nova) "Stale BDM records remain in the DB after n-api to n-cpu RPC timeouts during reserve_block_device_name" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood)
14:57:15 sean-k-mooney and this patch https://review.opendev.org/#/c/682594/
14:57:25 lyarwood sean-k-mooney: yeah, it cleans up after one type of failure, it doesn't stop the failure from happening.
14:57:38 sean-k-mooney ah i see
14:57:51 sean-k-mooney so its treating the symtoms not the cause
14:58:11 lyarwood sean-k-mooney: yeah and very specific symptoms at that.
14:58:47 sean-k-mooney you could make it partial-bug if you plan to try and fix it someother way on master in a follow up patch
14:59:19 lyarwood ah yeah I forgot about partial
14:59:21 lyarwood that might do
14:59:37 sean-k-mooney related bug makes sense too but i can see why you might what cloeses bug for stabel if you dont think any complete fix woudl be backportable
15:00:41 sean-k-mooney Partial-Bug i think is the most correct but that wont mark it as "fixed" on stable
15:00:45 mriedem see my comments from PS4
15:00:50 mriedem 1. Can we sort out the various bugs I've mentioned to see which ones this fixes and then duplicate whatever is a duplicate and decide on one (probably the oldest) bug for the same issue.
15:00:55 mriedem 2. I don't trust unit tests for this type of stuff since it involves (a) more than one service and (b) the database. Can we write a functional test to recreate the bug and then assert the fix resolves it?
15:01:26 mriedem i believe i sorted out the bazillion related bugs here https://review.opendev.org/#/c/692940/
15:02:35 lyarwood gah, I'm not sure how I missed that, sorry mriedem
15:03:03 mriedem probably distracted by all of the comments inline about races and locks and such that might mean this doesn't fix a bug
15:04:14 openstackgerrit Dan Smith proposed openstack/nova master: Add a way to exit early from a wait_for_instance_event() https://review.opendev.org/695985
15:10:00 mriedem if you want i could write the functional test if you don't want to bother with that
15:11:40 mriedem lyarwood: since the bdm.create is the last thing that happens on the compute, in what case would this cleanup catch and actually clean anything up?
15:11:55 mriedem like just the split second that we actually created the record and then got a messaging timeout?
15:13:03 lyarwood mriedem: happy to write the func test
15:13:26 lyarwood mriedem: and yeah but that's asssuming that is a split second and n-cpu isn't stuck somehow
15:13:38 lyarwood mriedem: as mdbooth suggested
15:14:02 lyarwood mriedem: FWIW we have seen this downstream at least, so either way it's possible to hit and reasonable to cleanup
15:14:16 mriedem sure, so if n-cpu is stuck on let's say getting the instance uuid lock, or doing something in the driver like _get_device_name_for_instance, we wouldn't create the bdm in that case anyway
15:14:32 mriedem where/how was it triggered downstream?
15:14:48 mriedem slow compute and the api timed out?
15:15:05 mriedem because if so, the change to make reserve_block_device_name using the long_rpc_timeout will fix that
15:15:54 lyarwood mriedem: https://bugzilla.redhat.com/show_bug.cgi?id=1752734 - I didn't get a root cause out of this as the env wasn't logging in DEBUG unfortunatley
15:15:55 openstack bugzilla.redhat.com bug 1752734 in openstack-nova "Invalid bdm record remains when reserve_block_device_name rpc call times out" [High,On_dev] - Assigned to lyarwood
15:16:16 lyarwood mriedem: but we did end up with stale BDM records in the DB after seeing timeouts
15:16:53 sean-k-mooney lyarwood: ok so there is a customer case attached to that so its not a ci issue
15:17:38 mriedem 3. nova-api again requests to nova-compute to do attaching operation 4. nova-compute updates bdm with retrieved connection info 5. if some error like timeout detected between 3-4, remove bdm record created at 2
15:17:43 mriedem so that's not the same thing
15:18:35 mriedem that's (1) reserve_block_device_name worked and created a bdm and then (2) when the api cast to attach_volume something timed out and the bdm was orphaned
15:19:37 lyarwood mriedem: yeah I think that description is wrong, there's reserve_block_device_name failures documented in the next private comment of the bug
15:20:26 lyarwood sean-k-mooney: ^ can confirm that, however I don't think we can share the actual logs here.
15:20:49 mriedem even so, the change to make reserve_block_device_name using the long_rpc_timeout will fix that
15:20:58 mriedem but i don't think that long rpc timeout goes back to queens
15:21:21 sean-k-mooney am i was reading it and im not sure the comment need to be private. we cant share the raw logs but i do not see antynig in comment 1 that is really an issue
15:21:36 mriedem if it's just a messagingtimeout i can understand that
15:21:48 sean-k-mooney MessagingTimeout: Timed out waiting for a reply to message ID fa3d55d2d8c248ba82a13c940350ea9c
15:21:49 lyarwood super secret volume uuids
15:21:52 lyarwood ;)
15:22:10 sean-k-mooney so the long rpc might fix it ya
15:22:14 mriedem it's a 60 second timeout by default with a blocking api - if the compute is slow or we're waiting on the instance uuid lock (maybe because a periodic is doing something with the instance at the time of attach), we could reasonably timeout
15:22:37 mriedem https://review.opendev.org/#/c/566696/ only goes back to rocky though, so you can't get the long_rpc_timeout fix to queens
15:22:43 mriedem so no use in saying that's the backportable fix
15:23:33 mriedem anyway, it just seems like a really small window where the fix in the api would handle it since the bdm create is the last thing to happen - if we timeout before that we still orphan the bdm record
15:23:54 mriedem and i know i'm contradicting my "what's the harm?" replies to mdbooth in the comments :)
15:24:07 mriedem the better longer term fix is the wip i posted, but that's not backportable
15:24:45 lyarwood yeah agreed on that part, we could backport long_rpc_timeout downstream *cough*
15:24:51 lyarwood but yeah
15:25:07 mriedem also your bug is definitely a duplicate of https://bugs.launchpad.net/nova/+bug/1425352
15:25:07 openstack Launchpad bug 1425352 in OpenStack Compute (nova) "A volume remains attached and cannot be detached after attaching it fails" [Low,Confirmed]
15:25:59 sean-k-mooney lyarwood: the issue would be the oslo.messaging min version bump

Earlier   Later