| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-31 | |||
| 18:00:05 | melwitt | thanks for mentioning it. my brain is just 😑 | |
| 18:01:01 | sean-k-mooney | ya so its cinder nfs and that will verify the issue your tryign to fix | |
| 18:01:03 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ae12c161604b49e49e05b8ccef739785/log/controller/logs/etc/cinder/cinder_conf.txt | |
| 18:03:16 | sean-k-mooney | so the nova-ovs-hybrid plug job currently test well ml2/ovs with hybrid plug for move ops but also spice on debian 12 | |
| 18:03:27 | sean-k-mooney | i want to rename that to nova-alt | |
| 18:03:51 | sean-k-mooney | so like nova-next but no implciation that the alternitive config it is testing are intended to become new defaults | |
| 18:04:11 | sean-k-mooney | i think we could slot nfs testing into that too | |
| 18:05:49 | melwitt | ah ok | |
| 18:07:23 | melwitt | I'd at least name it nova-alt-config because nova-alt sounds too vague to me | |
| 19:10:37 | sean-k-mooney | but want if i want a dark mode emo job :) | |
| 19:11:09 | sean-k-mooney | but ya good idea i just want to name it somehting better then it is currenlty if we extend the focus of the job | |
| 19:12:30 | opendevreview | Merged openstack/nova master: Fix doc comment on manager role change https://review.opendev.org/c/openstack/nova/+/956189 | |
| 19:16:04 | melwitt | yep agreed | |
| 19:16:58 | melwitt | nova-goth | |
| 19:17:10 | melwitt | the dark practice of using nfs | |
| 19:18:26 | melwitt | ok so the existing tempest test doesn't trigger the bug so I'll try adding something to it | |
| 19:24:51 | sean-k-mooney | oh so you wont see it in single node | |
| 19:25:32 | sean-k-mooney | so the relitive path bug happens because thing are mounted at diffent locations based on cidner volume vs nova presecitve right | |
| 19:26:06 | sean-k-mooney | so i think to see it you will need a 2 node jobs wherne you run nova compute on a serepte ndoe form cidner | |
| 19:26:09 | melwitt | yes | |
| 19:26:42 | sean-k-mooney | ya so you will need to tweak the job to do that by usign the multi node node set and disbale n-cpu on the contoller | |
| 19:27:03 | melwitt | I wonder if the lack of a second snapshot is related | |
| 19:27:20 | sean-k-mooney | yes perhaps try that first | |
| 19:27:29 | melwitt | ++ | |
| 19:27:52 | sean-k-mooney | im not sure if it iss but that was what war reported right multiple snaphost and it was the last oen that failed to be removed | |
| 19:28:14 | melwitt | right. the repro steps also say to create 2 or more snapshots | |
| 19:28:35 | sean-k-mooney | im not entirly sure why that woudl matter but it woudl likely explain why it was missed | |
| 19:29:03 | sean-k-mooney | beyond the fact we dont test nfs i mean :) | |
| 19:29:29 | melwitt | heh | |
| 20:02:12 | opendevreview | melanie witt proposed openstack/nova master: DNM test volume snapshot NFS https://review.opendev.org/c/openstack/nova/+/956282 | |
| 20:46:06 | mikal | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/950643 is ready for a quick re-review whenever you have a minute. | |
| 21:05:13 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/955039 | |
| 21:32:26 | melwitt | ok cool the use of two snapshots did repro the bug without needing to go to multinode | |
| 21:36:26 | sean-k-mooney | mikal: well i just got back from a dog walk and put food in the oven so sure i have 20 mins | |
| 21:39:28 | sean-k-mooney | mikal: looks good to me | |
| 21:40:15 | sean-k-mooney | melwitt: i wonder why it needs two, do you know? | |
| 21:47:55 | melwitt | sean-k-mooney: something to do with nova rebasing the image. there's something where when you delete a volume snapshot while the instance is stopped, nova will end up rebasing the image which corrupts the metadata and then attempt to delete the last snapshot fails on the cinder side | |
| 21:49:27 | melwitt | so the repro is, boot from volume, create two volume snapshots, stop instance, delete the newest volume snapshot (works), delete the oldest volume snapshot (fails) | |
| 21:49:50 | melwitt | this is what I uploaded to do ^ https://review.opendev.org/c/openstack/tempest/+/956302/1/tempest/scenario/test_volume_boot_pattern.py | |
| 21:54:51 | sean-k-mooney | melwitt: so nova is not ment to allow that | |
| 21:55:07 | sean-k-mooney | when the instnace is stopped nova returns a 400 to cinder | |
| 21:55:30 | sean-k-mooney | when it calls the assisted voluem snapshot delete api | |
| 21:56:47 | sean-k-mooney | ya that tesss looks resonable | |
| 21:57:11 | gmaan | sean-k-mooney: that is only case (nova reject based on instance state )when merge_target_file is None https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5926C34-L5926C51 | |
| 21:57:39 | gmaan | NOT None | |
| 21:57:59 | gmaan | when i looked into this bug, i also got confused with when nova reject based on instance state and when not | |
| 21:58:02 | sean-k-mooney | hum | |
| 21:58:25 | sean-k-mooney | well that is proably too narrorw | |
| 21:58:42 | gmaan | yeah | |
| 21:58:45 | sean-k-mooney | i mean we could make it work offline | |
| 21:58:57 | sean-k-mooney | but for right now there are obviously other cases where it can fail | |
| 21:59:29 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1919487 | |
| 21:59:36 | sean-k-mooney | that was the bug lee was tryign to mitigate | |
| 22:00:22 | sean-k-mooney | ok so thinking about this | |
| 22:00:57 | sean-k-mooney | if (delete_info.get('merge_target_file') is not None and | |
| 22:00:59 | sean-k-mooney | instance.vm_state != vm_states.ACTIVE | |
| 22:01:01 | sean-k-mooney | ): | |
| 22:01:26 | melwitt | not supposed to allow it? I did not know that | |
| 22:01:26 | sean-k-mooney | so if there is a merge_target_file | |
| 22:01:32 | sean-k-mooney | and the vm is stopped | |
| 22:01:34 | melwitt | coincidence I guess that it works fine with relative paths | |
| 22:01:35 | sean-k-mooney | then we raise | |
| 22:02:01 | melwitt | well, I guess let me wait for my test first. maybe it won't pass | |
| 22:02:05 | sean-k-mooney | melwitt: well i was thinkign ti was not inteded | |
| 22:02:19 | sean-k-mooney | becuase the logs i was looking at had a 400 saying it was not supported | |
| 22:02:28 | melwitt | oh, huh. | |
| 22:02:31 | sean-k-mooney | and if we added it then it was not inteded | |
| 22:02:39 | sean-k-mooney | so i think that was hitting this check | |
| 22:04:03 | sean-k-mooney | there are too bugs downstream for the issue your looking at fixing | |
| 22:05:03 | sean-k-mooney | this is in the cinder log of the other one | |
| 22:05:05 | sean-k-mooney | Call to Nova delete snapshot failed: novaclient.exceptions.BadRequest: Instance 9b578630-6c54-4481-8c49-6e0dd5d7e1fa in vm_state stopped. Cannot volume_snapshot_delete while the instance is in this state. (HTTP 400) | |
| 22:05:33 | melwitt | oh 🫤 | |
| 22:07:08 | sean-k-mooney | in the othe rbug they had 3 snapshot | |
| 22:07:22 | sean-k-mooney | the first one deleted fine | |
| 22:08:49 | melwitt | it just looked like such an easy simple thing to put it back to using relative path for the qemu-img rebase | |
| 22:09:35 | sean-k-mooney | the scond one the invalid backing file | |
| 22:10:20 | sean-k-mooney | the third once got that error above | |
| 22:10:36 | sean-k-mooney | melwitt: well it is | |
| 22:10:41 | sean-k-mooney | we shoud fix that first | |
| 22:11:17 | sean-k-mooney | when you have 3 snapshot deliting the most recnet breaked deletin the next oldest becuase of the backing file | |
| 22:11:18 | melwitt | I had no idea the depths of this issue 🙂 | |
| 22:11:42 | melwitt | yeah. that seems to be what happens with the 2 snapshots | |
| 22:11:44 | sean-k-mooney | melwitt: do you remember the last time you fix an nfs snapshot issue.... | |
| 22:11:53 | melwitt | lol | |
| 22:12:09 | melwitt | twas a journey | |
| 22:12:47 | sean-k-mooney | so i dont know if the backing file issue is having sidefect | |
| 22:13:20 | sean-k-mooney | i.e. if that is fixed i dont knwo if the other isssue will happen | |
| 22:13:35 | melwitt | oh, I see what you mean | |
| 22:13:50 | melwitt | hells yeah the test passes | |
| 22:14:01 | sean-k-mooney | oh with your chage | |
| 22:14:03 | sean-k-mooney | cool | |
| 22:14:20 | sean-k-mooney | can you try 3 snapshot? | |
| 22:14:25 | melwitt | yeah. I thought it would but I wanted to have proof | |
| 22:14:51 | melwitt | sean-k-mooney: yeah I can try that. in what order do you want to delete? newest then middle then oldest or? | |
| 22:16:21 | sean-k-mooney | yep that what they did | |
| 22:16:27 | melwitt | k | |
| 22:16:54 | sean-k-mooney | although i dont know if they waited. in the down stream bug they are suing the cli and issue the 3 delete back to back then list the snapshot | |
| 23:30:47 | melwitt | sean-k-mooney: for your tomorrow - test just passed with 3 snapshots also | |
| #openstack-nova - 2025-08-01 | |||
| 00:00:04 | opendevreview | Merged openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643 | |
| 00:35:12 | gmaan | melwitt nice, I can see same error in cinder log also https://zuul.opendev.org/t/openstack/build/c68c0479054f47d08d47535f2625a7b8/log/controller/logs/screen-c-vol.txt#9067 | |