Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-31
19:28:14 melwitt right. the repro steps also say to create 2 or more snapshots
19:28:35 sean-k-mooney im not entirly sure why that woudl matter but it woudl likely explain why it was missed
19:29:03 sean-k-mooney beyond the fact we dont test nfs i mean :)
19:29:29 melwitt heh
20:02:12 opendevreview melanie witt proposed openstack/nova master: DNM test volume snapshot NFS https://review.opendev.org/c/openstack/nova/+/956282
20:46:06 mikal sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/950643 is ready for a quick re-review whenever you have a minute.
21:05:13 opendevreview melanie witt proposed openstack/nova master: WIP libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/955039
21:32:26 melwitt ok cool the use of two snapshots did repro the bug without needing to go to multinode
21:36:26 sean-k-mooney mikal: well i just got back from a dog walk and put food in the oven so sure i have 20 mins
21:39:28 sean-k-mooney mikal: looks good to me
21:40:15 sean-k-mooney melwitt: i wonder why it needs two, do you know?
21:47:55 melwitt sean-k-mooney: something to do with nova rebasing the image. there's something where when you delete a volume snapshot while the instance is stopped, nova will end up rebasing the image which corrupts the metadata and then attempt to delete the last snapshot fails on the cinder side
21:49:27 melwitt so the repro is, boot from volume, create two volume snapshots, stop instance, delete the newest volume snapshot (works), delete the oldest volume snapshot (fails)
21:49:50 melwitt this is what I uploaded to do ^ https://review.opendev.org/c/openstack/tempest/+/956302/1/tempest/scenario/test_volume_boot_pattern.py
21:54:51 sean-k-mooney melwitt: so nova is not ment to allow that
21:55:07 sean-k-mooney when the instnace is stopped nova returns a 400 to cinder
21:55:30 sean-k-mooney when it calls the assisted voluem snapshot delete api
21:56:47 sean-k-mooney ya that tesss looks resonable
21:57:11 gmaan sean-k-mooney: that is only case (nova reject based on instance state )when merge_target_file is None https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5926C34-L5926C51
21:57:39 gmaan NOT None
21:57:59 gmaan when i looked into this bug, i also got confused with when nova reject based on instance state and when not
21:58:02 sean-k-mooney hum
21:58:25 sean-k-mooney well that is proably too narrorw
21:58:42 gmaan yeah
21:58:45 sean-k-mooney i mean we could make it work offline
21:58:57 sean-k-mooney but for right now there are obviously other cases where it can fail
21:59:29 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1919487
21:59:36 sean-k-mooney that was the bug lee was tryign to mitigate
22:00:22 sean-k-mooney ok so thinking about this
22:00:57 sean-k-mooney if (delete_info.get('merge_target_file') is not None and
22:00:59 sean-k-mooney instance.vm_state != vm_states.ACTIVE
22:01:01 sean-k-mooney ):
22:01:26 melwitt not supposed to allow it? I did not know that
22:01:26 sean-k-mooney so if there is a merge_target_file
22:01:32 sean-k-mooney and the vm is stopped
22:01:34 melwitt coincidence I guess that it works fine with relative paths
22:01:35 sean-k-mooney then we raise
22:02:01 melwitt well, I guess let me wait for my test first. maybe it won't pass
22:02:05 sean-k-mooney melwitt: well i was thinkign ti was not inteded
22:02:19 sean-k-mooney becuase the logs i was looking at had a 400 saying it was not supported
22:02:28 melwitt oh, huh.
22:02:31 sean-k-mooney and if we added it then it was not inteded
22:02:39 sean-k-mooney so i think that was hitting this check
22:04:03 sean-k-mooney there are too bugs downstream for the issue your looking at fixing
22:05:03 sean-k-mooney this is in the cinder log of the other one
22:05:05 sean-k-mooney Call to Nova delete snapshot failed: novaclient.exceptions.BadRequest: Instance 9b578630-6c54-4481-8c49-6e0dd5d7e1fa in vm_state stopped. Cannot volume_snapshot_delete while the instance is in this state. (HTTP 400)
22:05:33 melwitt oh 🫤
22:07:08 sean-k-mooney in the othe rbug they had 3 snapshot
22:07:22 sean-k-mooney the first one deleted fine
22:08:49 melwitt it just looked like such an easy simple thing to put it back to using relative path for the qemu-img rebase
22:09:35 sean-k-mooney the scond one the invalid backing file
22:10:20 sean-k-mooney the third once got that error above
22:10:36 sean-k-mooney melwitt: well it is
22:10:41 sean-k-mooney we shoud fix that first
22:11:17 sean-k-mooney when you have 3 snapshot deliting the most recnet breaked deletin the next oldest becuase of the backing file
22:11:18 melwitt I had no idea the depths of this issue 🙂
22:11:42 melwitt yeah. that seems to be what happens with the 2 snapshots
22:11:44 sean-k-mooney melwitt: do you remember the last time you fix an nfs snapshot issue....
22:11:53 melwitt lol
22:12:09 melwitt twas a journey
22:12:47 sean-k-mooney so i dont know if the backing file issue is having sidefect
22:13:20 sean-k-mooney i.e. if that is fixed i dont knwo if the other isssue will happen
22:13:35 melwitt oh, I see what you mean
22:13:50 melwitt hells yeah the test passes
22:14:01 sean-k-mooney oh with your chage
22:14:03 sean-k-mooney cool
22:14:20 sean-k-mooney can you try 3 snapshot?
22:14:25 melwitt yeah. I thought it would but I wanted to have proof
22:14:51 melwitt sean-k-mooney: yeah I can try that. in what order do you want to delete? newest then middle then oldest or?
22:16:21 sean-k-mooney yep that what they did
22:16:27 melwitt k
22:16:54 sean-k-mooney although i dont know if they waited. in the down stream bug they are suing the cli and issue the 3 delete back to back then list the snapshot
23:30:47 melwitt sean-k-mooney: for your tomorrow - test just passed with 3 snapshots also
#openstack-nova - 2025-08-01
00:00:04 opendevreview Merged openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643
00:35:12 gmaan melwitt nice, I can see same error in cinder log also https://zuul.opendev.org/t/openstack/build/c68c0479054f47d08d47535f2625a7b8/log/controller/logs/screen-c-vol.txt#9067
00:35:25 gmaan invalid backing file. and with your fix it poass
00:35:51 gmaan let me know once you are ready with tempest test if anything more you want to update and I can revierw
03:47:33 mikal sean-k-mooney: thank you.
09:38:24 sean-k-mooney Uggla: when you arbout can you sett implemention to complete https://blueprints.launchpad.net/nova/+spec/libvirt-spice-vdi
09:39:00 Uggla sean-k-mooney ah cool ! gonna do that.
09:39:00 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/950643 was the last patch
09:39:34 mikal sean-k-mooney: turns out as the author I can do that, so I've done it just now... Hopefully that's not a paddling.
09:39:54 sean-k-mooney mikal: not that fine
09:40:08 sean-k-mooney iw as goign to do it for you but im not in the corerct group in launch pad
09:40:33 sean-k-mooney *no thats fine
09:41:01 sean-k-mooney mikal: you will have to let use know how all this works out for kerbside once folks that using it
09:41:16 sean-k-mooney i.e. are there more vdi enhancments that are needed ectra
09:41:25 sean-k-mooney mikal: i assume kolla-ansible is your next priority
09:42:00 mikal sean-k-mooney: yeah, I've spent the last few weeks working on a kolla-ansible role. I figure if I do one example that will make it easier for others. At some point I should figure out how to get onto that team's radar, although I assume that's a G Release problem at this point.
09:42:38 mikal I can't think of any further VDI patches for now, but I think you're right that there will likely be more as people play with it.
10:00:11 sean-k-mooney mikal: well they used to use spec and they alwasy have ptg sessions, so i woudl see if they can give you a slot at a not horible time
10:00:54 sean-k-mooney they used to be pretty active on irc in the past too but its been a while since i did anything with kolla directly
10:10:21 mikal sean-k-mooney: I figure I'll finish the role and upload it, and then that will give something tangle to talk about. That said, the PTG does sound like a good idea. I have a mate who thinks I should shoot some demo videos as well, but baby steps and all that.
10:17:57 opendevreview Balazs Gibizer proposed openstack/nova master: Document native threading mode and tuneables https://review.opendev.org/c/openstack/nova/+/949364
10:17:58 opendevreview Balazs Gibizer proposed openstack/nova master: Allow services to start with threading https://review.opendev.org/c/openstack/nova/+/948311
10:17:59 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450
10:17:59 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode https://review.opendev.org/c/openstack/nova/+/951957
10:18:00 opendevreview Balazs Gibizer proposed openstack/nova master: Do not yield in threading mode https://review.opendev.org/c/openstack/nova/+/950994
10:18:00 opendevreview Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436
10:18:01 opendevreview Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475

Earlier   Later