| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-02 | |||
| 12:00:00 | sean-k-mooney | * i can't use | |
| 12:31:23 | stephenfin | sean-k-mooney: Yup, but I don't think it's a massive deal that '_wait_for_state_change' behaves oddly on stable branches, given that there's plenty of tests using it | |
| 12:33:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523 | |
| 12:33:55 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with https://review.opendev.org/693337 | |
| 12:39:44 | sean-k-mooney | stephenfin: ya im using the fake notifier too so i can also use the notifer to wait for the completeion events too so its fine ill rework them later today | |
| 12:49:34 | sean-k-mooney | gibi: johnthetubaguy do either of ye feel comfrotable reviewing pci code | |
| 12:59:45 | stephenfin | sean-k-mooney: Can I say no to that? ^ :P | |
| 13:07:31 | sean-k-mooney | you can but you already reviewed the patch and said yes :) | |
| 13:07:50 | sean-k-mooney | also i said comfortabley not do you like reviewing pci stuff :) | |
| 13:11:03 | sean-k-mooney | i also feel like i missed a chance to use "no take backsies" but that might date me a little | |
| 13:14:38 | openstackgerrit | Eric Fried proposed openstack/os-traits master: Add 'TYPE_PLOOP' image type. https://review.opendev.org/696435 | |
| 13:15:17 | efried | stephenfin: I fixed that link ^ -- do you want to confirm real quick before I send it? | |
| 13:24:21 | sean-k-mooney | the ploop image type is only used by parallel/openvz containers rights | |
| 13:25:10 | efried | No idea, but it's on the list, so we should have a trait for it. | |
| 13:25:41 | efried | (and by "no idea", I mean your sentence has at least three words I don't know) | |
| 13:25:42 | sean-k-mooney | ya it is https://wiki.openvz.org/Ploop/format | |
| 13:26:04 | sean-k-mooney | and yes we should have a trait for it | |
| 13:26:18 | efried | sounds like a euphemism for a particular type of turd to me <shrug> | |
| 13:28:07 | openstackgerrit | Eric Fried proposed openstack/nova master: Reusable RequestGroup.add_{resource|trait} https://review.opendev.org/696380 | |
| 13:28:07 | openstackgerrit | Eric Fried proposed openstack/nova master: Process requested_resources in ResourceRequest init https://review.opendev.org/696354 | |
| 13:28:08 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use string suffixes and provider mappings https://review.opendev.org/696418 | |
| 13:28:16 | efried | stephenfin, gibi: fixed the unit test on the bottom one ^ | |
| 13:34:26 | openstackgerrit | Merged openstack/os-traits master: Add 'TYPE_PLOOP' image type. https://review.opendev.org/696435 | |
| 14:23:17 | gibi | efried: I'm +2 on the bottom patch | |
| 14:23:25 | efried | thanks gibi | |
| 14:23:34 | gibi | thanks for starting this work | |
| 14:23:42 | gibi | you are solving some of my todos :) | |
| 14:23:47 | efried | does the strategy seem sane to you wrt the requester_id <=> group ID? | |
| 14:24:12 | gibi | efried: yeah, that step is what I had in my mind | |
| 14:24:17 | efried | great | |
| 14:25:30 | gibi | efried: I'm a bit affraid the replacement of fill_provider_mapping with the mapping from the a_c response. Theortically it should work I just hope there will not be too much complication | |
| 14:25:42 | efried | we'll see :) | |
| 14:26:03 | efried | gibi: I think the most complicated bit might be revert_resize | |
| 14:26:12 | gibi | yes, that is the type of surgery that better do that plan | |
| 14:26:19 | efried | By that time the original mappings might be long gone | |
| 14:26:33 | efried | ...unless we persist them in the RequestSpec...? | |
| 14:26:49 | gibi | efried: in case of revert_resize we can either start using multiple port binding for resize which is hard but nice, or persist the mapping somewhere | |
| 14:27:01 | gibi | if we go to the later then I would persist it to the Migration object | |
| 14:27:29 | gibi | as the migration object has the proper lifecycle for this type of data | |
| 14:27:37 | gibi | the RequestSpec lives longer | |
| 14:41:25 | sean-k-mooney | gibi: i would like to move to useing multiple port bindings for resize for what its worth | |
| 14:41:38 | sean-k-mooney | gibi: we will be using it for cross cell resize | |
| 14:41:58 | sean-k-mooney | so i think it makes sense to just use it always for resize/cold migrate | |
| 14:46:02 | sean-k-mooney | efried: gibi when ye have time could yet take a look at https://review.opendev.org/#/c/674072/ and maybe https://review.opendev.org/#/c/695118/ | |
| 14:47:31 | sean-k-mooney | ill be going on vacation in 2 weeks but i have a 4 patches i really want to get merged if i can before i do those are two of them and im resping the other two now | |
| 14:49:56 | efried | stephenfin, lyarwood: thoughts on bug tagging here? https://review.opendev.org/#/c/682594/ | |
| 14:51:41 | efried | sean-k-mooney: that might be a bit out of my bailiwick, but I'll give it a try. | |
| 14:53:01 | stephenfin | looking | |
| 14:53:32 | lyarwood | efried: yeah that's fair, I can also address the nits before we merge this afternoon | |
| 14:53:41 | efried | okay | |
| 14:53:48 | lyarwood | efried: I think I was trying to avoid any suggestion that this was the only fix required here | |
| 14:53:58 | stephenfin | Your call. I could go either way. I'm guessing it was left that way since mriedem has a better albeit not backportable fix | |
| 14:54:04 | stephenfin | yup | |
| 14:54:28 | efried | right, I'm looking at it from the perspective of stable, where if we don't Closes-Bug on this patch, there will never be a "fix". | |
| 14:54:58 | efried | I guess if you really want, we could keep this one Related-Bug and switch it to Closes-Bug on the backports | |
| 14:55:07 | efried | can't believe there's not a precedent for something like this. | |
| 14:56:03 | lyarwood | Yeah not that I recall but I'd be okay with that from a stable point of view | |
| 14:56:15 | lyarwood | related on master, closes on stable that is | |
| 14:56:23 | sean-k-mooney | well does it actully fix the bug | |
| 14:56:30 | sean-k-mooney | or just make it less bad | |
| 14:56:58 | openstack | Launchpad bug 1844296 in OpenStack Compute (nova) "Stale BDM records remain in the DB after n-api to n-cpu RPC timeouts during reserve_block_device_name" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 14:56:58 | sean-k-mooney | ye are talking about https://bugs.launchpad.net/nova/+bug/1844296 right | |
| 14:57:15 | sean-k-mooney | and this patch https://review.opendev.org/#/c/682594/ | |
| 14:57:25 | lyarwood | sean-k-mooney: yeah, it cleans up after one type of failure, it doesn't stop the failure from happening. | |
| 14:57:38 | sean-k-mooney | ah i see | |
| 14:57:51 | sean-k-mooney | so its treating the symtoms not the cause | |
| 14:58:11 | lyarwood | sean-k-mooney: yeah and very specific symptoms at that. | |
| 14:58:47 | sean-k-mooney | you could make it partial-bug if you plan to try and fix it someother way on master in a follow up patch | |
| 14:59:19 | lyarwood | ah yeah I forgot about partial | |
| 14:59:21 | lyarwood | that might do | |
| 14:59:37 | sean-k-mooney | related bug makes sense too but i can see why you might what cloeses bug for stabel if you dont think any complete fix woudl be backportable | |
| 15:00:41 | sean-k-mooney | Partial-Bug i think is the most correct but that wont mark it as "fixed" on stable | |
| 15:00:45 | mriedem | see my comments from PS4 | |
| 15:00:50 | mriedem | 1. Can we sort out the various bugs I've mentioned to see which ones this fixes and then duplicate whatever is a duplicate and decide on one (probably the oldest) bug for the same issue. | |
| 15:00:55 | mriedem | 2. I don't trust unit tests for this type of stuff since it involves (a) more than one service and (b) the database. Can we write a functional test to recreate the bug and then assert the fix resolves it? | |
| 15:01:26 | mriedem | i believe i sorted out the bazillion related bugs here https://review.opendev.org/#/c/692940/ | |
| 15:02:35 | lyarwood | gah, I'm not sure how I missed that, sorry mriedem | |
| 15:03:03 | mriedem | probably distracted by all of the comments inline about races and locks and such that might mean this doesn't fix a bug | |
| 15:04:14 | openstackgerrit | Dan Smith proposed openstack/nova master: Add a way to exit early from a wait_for_instance_event() https://review.opendev.org/695985 | |
| 15:10:00 | mriedem | if you want i could write the functional test if you don't want to bother with that | |
| 15:11:40 | mriedem | lyarwood: since the bdm.create is the last thing that happens on the compute, in what case would this cleanup catch and actually clean anything up? | |
| 15:11:55 | mriedem | like just the split second that we actually created the record and then got a messaging timeout? | |
| 15:13:03 | lyarwood | mriedem: happy to write the func test | |
| 15:13:26 | lyarwood | mriedem: and yeah but that's asssuming that is a split second and n-cpu isn't stuck somehow | |
| 15:13:38 | lyarwood | mriedem: as mdbooth suggested | |
| 15:14:02 | lyarwood | mriedem: FWIW we have seen this downstream at least, so either way it's possible to hit and reasonable to cleanup | |
| 15:14:16 | mriedem | sure, so if n-cpu is stuck on let's say getting the instance uuid lock, or doing something in the driver like _get_device_name_for_instance, we wouldn't create the bdm in that case anyway | |
| 15:14:32 | mriedem | where/how was it triggered downstream? | |
| 15:14:48 | mriedem | slow compute and the api timed out? | |
| 15:15:05 | mriedem | because if so, the change to make reserve_block_device_name using the long_rpc_timeout will fix that | |
| 15:15:54 | lyarwood | mriedem: https://bugzilla.redhat.com/show_bug.cgi?id=1752734 - I didn't get a root cause out of this as the env wasn't logging in DEBUG unfortunatley | |
| 15:15:55 | openstack | bugzilla.redhat.com bug 1752734 in openstack-nova "Invalid bdm record remains when reserve_block_device_name rpc call times out" [High,On_dev] - Assigned to lyarwood | |
| 15:16:16 | lyarwood | mriedem: but we did end up with stale BDM records in the DB after seeing timeouts | |
| 15:16:53 | sean-k-mooney | lyarwood: ok so there is a customer case attached to that so its not a ci issue | |
| 15:17:38 | mriedem | 3. nova-api again requests to nova-compute to do attaching operation 4. nova-compute updates bdm with retrieved connection info 5. if some error like timeout detected between 3-4, remove bdm record created at 2 | |
| 15:17:43 | mriedem | so that's not the same thing | |
| 15:18:35 | mriedem | that's (1) reserve_block_device_name worked and created a bdm and then (2) when the api cast to attach_volume something timed out and the bdm was orphaned | |
| 15:19:37 | lyarwood | mriedem: yeah I think that description is wrong, there's reserve_block_device_name failures documented in the next private comment of the bug | |
| 15:20:26 | lyarwood | sean-k-mooney: ^ can confirm that, however I don't think we can share the actual logs here. | |
| 15:20:49 | mriedem | even so, the change to make reserve_block_device_name using the long_rpc_timeout will fix that | |