| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-08-26 | |||
| 17:49:32 | gibi | OK, that is fine | |
| 17:49:34 | sean-k-mooney | as you suggest no reboot needed | |
| 17:49:45 | sean-k-mooney | actully | |
| 17:49:51 | sean-k-mooney | if we get instance not found form our own db | |
| 17:50:02 | sean-k-mooney | well no form libvirt | |
| 17:50:11 | sean-k-mooney | prefumably that means we raced with a delete? | |
| 17:50:17 | dansmith | on bdm save, yeah | |
| 17:50:27 | sean-k-mooney | ya so nothing to try and recover | |
| 17:51:50 | sean-k-mooney | dansmith: you can do git review -R by the way to avoid a rebase | |
| 17:52:13 | dansmith | sean-k-mooney: no, I've already rebased user data locally so I could get this rebased and ready | |
| 17:52:51 | sean-k-mooney | ah ok well it would not be the end fo the world if it was rebased. it needs to be updated for the api sample issue anyway | |
| 17:53:30 | dansmith | I just don't want to step on the other author if they already have things in the middle | |
| 17:54:15 | sean-k-mooney | ack, but yes it sounds like you can just proceed and leak the attacment and may just log it so an op could clean it up later | |
| 17:56:39 | sean-k-mooney | so whats left is gettin gth image form galnce. using that to calualte the size, setting up the wait for the external event and then calling cinder to do the reimage | |
| 17:56:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations https://review.opendev.org/c/openstack/nova/+/852397 | |
| 17:56:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/853835 | |
| 17:56:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468 | |
| 17:56:53 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias https://review.opendev.org/c/openstack/nova/+/853316 | |
| 17:56:53 | opendevreview | Balazs Gibizer proposed openstack/nova master: Create RequestGroups from InstancePCIRequests https://review.opendev.org/c/openstack/nova/+/852771 | |
| 17:56:54 | opendevreview | Balazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs https://review.opendev.org/c/openstack/nova/+/854118 | |
| 17:56:54 | opendevreview | Balazs Gibizer proposed openstack/nova master: Split PCI pools per PF https://review.opendev.org/c/openstack/nova/+/854440 | |
| 17:56:55 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters https://review.opendev.org/c/openstack/nova/+/854119 | |
| 17:56:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/854121 | |
| 17:56:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation https://review.opendev.org/c/openstack/nova/+/854120 | |
| 17:56:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support cold migrate and resize with PCI tracking in placement https://review.opendev.org/c/openstack/nova/+/854247 | |
| 17:56:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling https://review.opendev.org/c/openstack/nova/+/854122 | |
| 17:57:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support unshelve with PCI in placement https://review.opendev.org/c/openstack/nova/+/854616 | |
| 17:57:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support evacuate with PCI in placement https://review.opendev.org/c/openstack/nova/+/854615 | |
| 17:57:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Test reschedule with PCI in placement https://review.opendev.org/c/openstack/nova/+/854626 | |
| 17:57:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support same host resize with PCI in placement https://review.opendev.org/c/openstack/nova/+/854441 | |
| 17:57:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Heal allocation for same host resize https://review.opendev.org/c/openstack/nova/+/854822 | |
| 17:57:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support multi create with PCI in placement https://review.opendev.org/c/openstack/nova/+/854663 | |
| 17:58:06 | dansmith | ugh, that causes another failure later, which may be a quirk of our cinder fixture | |
| 17:58:19 | dansmith | the attachment isn't deleted, so much later it complains that the instance is double-attached | |
| 17:58:46 | sean-k-mooney | well form the cinder side i guess it is | |
| 17:59:04 | sean-k-mooney | we could try and recover at that point and delete the old one again | |
| 17:59:16 | sean-k-mooney | which would possisble stop the leak | |
| 17:59:18 | dansmith | that's what I'm trying to avoid because it ends up a pretty nested mess | |
| 17:59:24 | sean-k-mooney | ah ok | |
| 17:59:38 | dansmith | and it's likely to fail in reality if we just failed to delete | |
| 18:00:27 | dansmith | I'm also not sure I know why we're detaching and re-attaching here, just to do the reimage | |
| 18:00:27 | sean-k-mooney | we are not doing the attchment delete before the save to prevent the other case right | |
| 18:00:38 | sean-k-mooney | where our db is now out of sync if we fail to save | |
| 18:00:41 | dansmith | must be some cinder reason why that happens, to reset the state while the instance is powered off or something | |
| 18:00:58 | dansmith | yeah I think we have to update our db before the delete for races | |
| 18:01:11 | sean-k-mooney | ya i think so too. | |
| 18:01:58 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db) https://review.opendev.org/c/openstack/nova/+/831193 | |
| 18:01:59 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 18:01:59 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects) https://review.opendev.org/c/openstack/nova/+/839401 | |
| 18:02:00 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (api) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 18:02:00 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 18:02:01 | opendevreview | ribaudr proposed openstack/nova master: Bump compute version and check shares support https://review.opendev.org/c/openstack/nova/+/850499 | |
| 18:02:02 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 18:02:02 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 18:02:03 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 18:02:04 | opendevreview | ribaudr proposed openstack/nova master: Add instance.power_on_error notification https://review.opendev.org/c/openstack/nova/+/852084 | |
| 18:02:04 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 18:02:06 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/852085 | |
| 18:02:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.power_off_error notification https://review.opendev.org/c/openstack/nova/+/852278 | |
| 18:02:08 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 18:02:08 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 18:02:10 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 18:02:10 | opendevreview | ribaudr proposed openstack/nova master: Change microversion to 2.93 https://review.opendev.org/c/openstack/nova/+/852088 | |
| 18:02:12 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute and API part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 18:02:19 | sean-k-mooney | i dont recall why but i rememebr lee disucsing this at some point | |
| 18:06:42 | sean-k-mooney | i dont see this dicussed in the ptg | |
| 18:07:14 | sean-k-mooney | ah | |
| 18:07:16 | sean-k-mooney | https://review.opendev.org/c/openstack/nova-specs/+/840155/5/specs/zed/approved/volume-backed-server-rebuild.rst#58 | |
| 18:07:31 | sean-k-mooney | #. Create an empty (no connector) volume attachment for the volume and | |
| 18:07:33 | sean-k-mooney | server. This ensures the volume remains ``reserved`` through the next | |
| 18:07:35 | sean-k-mooney | step. | |
| 18:07:37 | sean-k-mooney | #. Delete the existing volume attachment (the old one). | |
| 18:07:39 | sean-k-mooney | #. Save the new attachment UUID to the BDM. | |
| 18:07:41 | sean-k-mooney | #. The above two steps are needed to keep the volume in ``reserved`` state | |
| 18:07:43 | sean-k-mooney | as a management state which is required by cinder to perform re-image | |
| 18:07:45 | sean-k-mooney | operation on it. | |
| 18:07:47 | sean-k-mooney | #. Call the new ``os-reimage`` cinder API. | |
| 18:07:55 | dansmith | yeah, to reset the state basically I assume | |
| 18:08:24 | sean-k-mooney | we need to move it form in-use to reserved to reimage | |
| 18:11:20 | sean-k-mooney | honestly i feel like the way out of this if everything explodes is just call rebuild again | |
| 18:11:28 | sean-k-mooney | like wait for cinder to be back | |
| 18:11:43 | sean-k-mooney | and then the vm that is precumsble in error if we coudl not comptle the reimage later | |
| 18:11:52 | sean-k-mooney | could just be rebuilt again | |
| 18:12:18 | dansmith | yeah I assume that will work, as long as it's still in a reasonable state | |
| 18:12:19 | sean-k-mooney | kind of like what we do with a failed evacuate we just evacuate again | |
| 18:13:06 | dansmith | so I guess I'm just going to put it somewhat back the way it was, let it fail for all cases, but only call delete if we created the new attachment | |
| 18:13:14 | dansmith | right now it'll call delete(None) which I assume will fail | |
| 18:13:55 | sean-k-mooney | ya presumabley with a type/atibute error | |
| 18:14:14 | sean-k-mooney | None type has no attirbute uuid or something like that | |
| 18:15:00 | gibi | rebuild after a failed rebuild make sense to me too | |
| 18:16:01 | sean-k-mooney | i dont know if the precondtions on rebuild allow that | |
| 18:16:06 | dansmith | yeah, I'm not so concerned about that, I just want to make sure we haven't left things in too bad of a state where that's not possible | |
| 18:16:08 | sean-k-mooney | or if you would have to do reset state first | |
| 18:16:16 | dansmith | sounds to me like we're going to have leaked an attachment regardless | |
| 18:16:28 | dansmith | because we have nowhere to stash it and try to delete it later | |
| 18:16:28 | sean-k-mooney | yes | |
| 18:16:38 | sean-k-mooney | although the user can actully delete that if they wanted too | |
| 18:17:42 | sean-k-mooney | it would be a bit painful however | |
| 18:17:52 | sean-k-mooney | since they have no way to lsit teh bdms today | |
| 18:18:02 | sean-k-mooney | its not part of server detail show | |