| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-31 | |||
| 19:25:32 | sean-k-mooney | so the relitive path bug happens because thing are mounted at diffent locations based on cidner volume vs nova presecitve right | |
| 19:26:06 | sean-k-mooney | so i think to see it you will need a 2 node jobs wherne you run nova compute on a serepte ndoe form cidner | |
| 19:26:09 | melwitt | yes | |
| 19:26:42 | sean-k-mooney | ya so you will need to tweak the job to do that by usign the multi node node set and disbale n-cpu on the contoller | |
| 19:27:03 | melwitt | I wonder if the lack of a second snapshot is related | |
| 19:27:20 | sean-k-mooney | yes perhaps try that first | |
| 19:27:29 | melwitt | ++ | |
| 19:27:52 | sean-k-mooney | im not sure if it iss but that was what war reported right multiple snaphost and it was the last oen that failed to be removed | |
| 19:28:14 | melwitt | right. the repro steps also say to create 2 or more snapshots | |
| 19:28:35 | sean-k-mooney | im not entirly sure why that woudl matter but it woudl likely explain why it was missed | |
| 19:29:03 | sean-k-mooney | beyond the fact we dont test nfs i mean :) | |
| 19:29:29 | melwitt | heh | |
| 20:02:12 | opendevreview | melanie witt proposed openstack/nova master: DNM test volume snapshot NFS https://review.opendev.org/c/openstack/nova/+/956282 | |
| 20:46:06 | mikal | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/950643 is ready for a quick re-review whenever you have a minute. | |
| 21:05:13 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/955039 | |
| 21:32:26 | melwitt | ok cool the use of two snapshots did repro the bug without needing to go to multinode | |
| 21:36:26 | sean-k-mooney | mikal: well i just got back from a dog walk and put food in the oven so sure i have 20 mins | |
| 21:39:28 | sean-k-mooney | mikal: looks good to me | |
| 21:40:15 | sean-k-mooney | melwitt: i wonder why it needs two, do you know? | |
| 21:47:55 | melwitt | sean-k-mooney: something to do with nova rebasing the image. there's something where when you delete a volume snapshot while the instance is stopped, nova will end up rebasing the image which corrupts the metadata and then attempt to delete the last snapshot fails on the cinder side | |
| 21:49:27 | melwitt | so the repro is, boot from volume, create two volume snapshots, stop instance, delete the newest volume snapshot (works), delete the oldest volume snapshot (fails) | |
| 21:49:50 | melwitt | this is what I uploaded to do ^ https://review.opendev.org/c/openstack/tempest/+/956302/1/tempest/scenario/test_volume_boot_pattern.py | |
| 21:54:51 | sean-k-mooney | melwitt: so nova is not ment to allow that | |
| 21:55:07 | sean-k-mooney | when the instnace is stopped nova returns a 400 to cinder | |
| 21:55:30 | sean-k-mooney | when it calls the assisted voluem snapshot delete api | |
| 21:56:47 | sean-k-mooney | ya that tesss looks resonable | |
| 21:57:11 | gmaan | sean-k-mooney: that is only case (nova reject based on instance state )when merge_target_file is None https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5926C34-L5926C51 | |
| 21:57:39 | gmaan | NOT None | |
| 21:57:59 | gmaan | when i looked into this bug, i also got confused with when nova reject based on instance state and when not | |
| 21:58:02 | sean-k-mooney | hum | |
| 21:58:25 | sean-k-mooney | well that is proably too narrorw | |
| 21:58:42 | gmaan | yeah | |
| 21:58:45 | sean-k-mooney | i mean we could make it work offline | |
| 21:58:57 | sean-k-mooney | but for right now there are obviously other cases where it can fail | |
| 21:59:29 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1919487 | |
| 21:59:36 | sean-k-mooney | that was the bug lee was tryign to mitigate | |
| 22:00:22 | sean-k-mooney | ok so thinking about this | |
| 22:00:57 | sean-k-mooney | if (delete_info.get('merge_target_file') is not None and | |
| 22:00:59 | sean-k-mooney | instance.vm_state != vm_states.ACTIVE | |
| 22:01:01 | sean-k-mooney | ): | |
| 22:01:26 | melwitt | not supposed to allow it? I did not know that | |
| 22:01:26 | sean-k-mooney | so if there is a merge_target_file | |
| 22:01:32 | sean-k-mooney | and the vm is stopped | |
| 22:01:34 | melwitt | coincidence I guess that it works fine with relative paths | |
| 22:01:35 | sean-k-mooney | then we raise | |
| 22:02:01 | melwitt | well, I guess let me wait for my test first. maybe it won't pass | |
| 22:02:05 | sean-k-mooney | melwitt: well i was thinkign ti was not inteded | |
| 22:02:19 | sean-k-mooney | becuase the logs i was looking at had a 400 saying it was not supported | |
| 22:02:28 | melwitt | oh, huh. | |
| 22:02:31 | sean-k-mooney | and if we added it then it was not inteded | |
| 22:02:39 | sean-k-mooney | so i think that was hitting this check | |
| 22:04:03 | sean-k-mooney | there are too bugs downstream for the issue your looking at fixing | |
| 22:05:03 | sean-k-mooney | this is in the cinder log of the other one | |
| 22:05:05 | sean-k-mooney | Call to Nova delete snapshot failed: novaclient.exceptions.BadRequest: Instance 9b578630-6c54-4481-8c49-6e0dd5d7e1fa in vm_state stopped. Cannot volume_snapshot_delete while the instance is in this state. (HTTP 400) | |
| 22:05:33 | melwitt | oh 🫤 | |
| 22:07:08 | sean-k-mooney | in the othe rbug they had 3 snapshot | |
| 22:07:22 | sean-k-mooney | the first one deleted fine | |
| 22:08:49 | melwitt | it just looked like such an easy simple thing to put it back to using relative path for the qemu-img rebase | |
| 22:09:35 | sean-k-mooney | the scond one the invalid backing file | |
| 22:10:20 | sean-k-mooney | the third once got that error above | |
| 22:10:36 | sean-k-mooney | melwitt: well it is | |
| 22:10:41 | sean-k-mooney | we shoud fix that first | |
| 22:11:17 | sean-k-mooney | when you have 3 snapshot deliting the most recnet breaked deletin the next oldest becuase of the backing file | |
| 22:11:18 | melwitt | I had no idea the depths of this issue 🙂 | |
| 22:11:42 | melwitt | yeah. that seems to be what happens with the 2 snapshots | |
| 22:11:44 | sean-k-mooney | melwitt: do you remember the last time you fix an nfs snapshot issue.... | |
| 22:11:53 | melwitt | lol | |
| 22:12:09 | melwitt | twas a journey | |
| 22:12:47 | sean-k-mooney | so i dont know if the backing file issue is having sidefect | |
| 22:13:20 | sean-k-mooney | i.e. if that is fixed i dont knwo if the other isssue will happen | |
| 22:13:35 | melwitt | oh, I see what you mean | |
| 22:13:50 | melwitt | hells yeah the test passes | |
| 22:14:01 | sean-k-mooney | oh with your chage | |
| 22:14:03 | sean-k-mooney | cool | |
| 22:14:20 | sean-k-mooney | can you try 3 snapshot? | |
| 22:14:25 | melwitt | yeah. I thought it would but I wanted to have proof | |
| 22:14:51 | melwitt | sean-k-mooney: yeah I can try that. in what order do you want to delete? newest then middle then oldest or? | |
| 22:16:21 | sean-k-mooney | yep that what they did | |
| 22:16:27 | melwitt | k | |
| 22:16:54 | sean-k-mooney | although i dont know if they waited. in the down stream bug they are suing the cli and issue the 3 delete back to back then list the snapshot | |
| 23:30:47 | melwitt | sean-k-mooney: for your tomorrow - test just passed with 3 snapshots also | |
| #openstack-nova - 2025-08-01 | |||
| 00:00:04 | opendevreview | Merged openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643 | |
| 00:35:12 | gmaan | melwitt nice, I can see same error in cinder log also https://zuul.opendev.org/t/openstack/build/c68c0479054f47d08d47535f2625a7b8/log/controller/logs/screen-c-vol.txt#9067 | |
| 00:35:25 | gmaan | invalid backing file. and with your fix it poass | |
| 00:35:51 | gmaan | let me know once you are ready with tempest test if anything more you want to update and I can revierw | |
| 03:47:33 | mikal | sean-k-mooney: thank you. | |
| 09:38:24 | sean-k-mooney | Uggla: when you arbout can you sett implemention to complete https://blueprints.launchpad.net/nova/+spec/libvirt-spice-vdi | |
| 09:39:00 | Uggla | sean-k-mooney ah cool ! gonna do that. | |
| 09:39:00 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/950643 was the last patch | |
| 09:39:34 | mikal | sean-k-mooney: turns out as the author I can do that, so I've done it just now... Hopefully that's not a paddling. | |
| 09:39:54 | sean-k-mooney | mikal: not that fine | |
| 09:40:08 | sean-k-mooney | iw as goign to do it for you but im not in the corerct group in launch pad | |
| 09:40:33 | sean-k-mooney | *no thats fine | |
| 09:41:01 | sean-k-mooney | mikal: you will have to let use know how all this works out for kerbside once folks that using it | |
| 09:41:16 | sean-k-mooney | i.e. are there more vdi enhancments that are needed ectra | |
| 09:41:25 | sean-k-mooney | mikal: i assume kolla-ansible is your next priority | |
| 09:42:00 | mikal | sean-k-mooney: yeah, I've spent the last few weeks working on a kolla-ansible role. I figure if I do one example that will make it easier for others. At some point I should figure out how to get onto that team's radar, although I assume that's a G Release problem at this point. | |
| 09:42:38 | mikal | I can't think of any further VDI patches for now, but I think you're right that there will likely be more as people play with it. | |
| 10:00:11 | sean-k-mooney | mikal: well they used to use spec and they alwasy have ptg sessions, so i woudl see if they can give you a slot at a not horible time | |
| 10:00:54 | sean-k-mooney | they used to be pretty active on irc in the past too but its been a while since i did anything with kolla directly | |