| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-30 | |||
| 12:11:37 | sean-k-mooney | with ceph in this case you were using lvm right | |
| 12:11:52 | sean-k-mooney | the backend should not really matter | |
| 12:12:02 | stephenfin | yes (pretty much a "stock" DevStack configuration) | |
| 12:12:04 | sean-k-mooney | but that is both local block and network type covered | |
| 12:12:10 | sean-k-mooney | ack | |
| 12:12:19 | sean-k-mooney | let me see if i still have a c9s devstack somewhere | |
| 12:14:32 | sean-k-mooney | that woudl be no but i can check our watcher ci job | |
| 12:14:46 | sean-k-mooney | actully no | |
| 12:15:04 | sean-k-mooney | we have master jobs that trigger with our new installer on cenetos butthsoe are disbaling cinder right now | |
| 12:15:14 | sean-k-mooney | i think neutorn and maybe cinder have centos josb | |
| 12:15:34 | sean-k-mooney | we might have an experminatal one too | |
| 12:16:07 | stephenfin | I'll just create one locally | |
| 12:16:38 | stephenfin | However, I see a few related changed that I'm going to try reverting out of caracal first | |
| 12:22:52 | gibi | I did the last round of fixes in that codepath so I remember it was working at that time to wait for the events. If the event does not come we still have code to check if the actual detach happened and move on. But as you noted it might be excessive waiting time | |
| 12:23:53 | gibi | if this is visible in upstream CI then please file an upstream bug with the links and I will try to look at it | |
| 12:27:14 | gibi | In https://d6e5655f0b1ee2f8df74-e35c415f7cc7f78f33337c81de121e72.ssl.cf2.rackcdn.com/openstack/52b57c77f805400888762a87e08196f1/controller/logs/screen-n-cpu.txt I see cases when it is on 20 sec wait and we move one | |
| 12:27:32 | gibi | Jul 30 01:36:30.229367 npa183b57ac85f4 nova-compute[75854]: DEBUG nova.virt.libvirt.guest [None req-5859b1ac-1198-4297-845b-2c4ef82e635b tempest-VolumesAssistedSnapshotsTest-257024579 tempest-VolumesAssistedSnapshotsTest-257024579-project-admin] detach device xml: <disk type="block" device="disk"> | |
| 12:27:47 | gibi | Jul 30 01:36:50.849840 npa183b57ac85f4 nova-compute[75854]: WARNING nova.virt.libvirt.driver [None req-5859b1ac-1198-4297-845b-2c4ef82e635b tempest-VolumesAssistedSnapshotsTest-257024579 tempest-VolumesAssistedSnapshotsTest-257024579-project-admin] Waiting for libvirt event about the detach of device vdb with device alias ua-50261be4-1b93-406a-ad22-095d53ab27bb from instance | |
| 12:27:53 | gibi | 8b434db2-72e7-45a7-a946-a43b7e5b04de is timed out. | |
| 12:27:57 | gibi | Jul 30 01:36:50.862330 npa183b57ac85f4 nova-compute[75854]: INFO nova.virt.libvirt.driver [None req-5859b1ac-1198-4297-845b-2c4ef82e635b tempest-VolumesAssistedSnapshotsTest-257024579 tempest-VolumesAssistedSnapshotsTest-257024579-project-admin] Successfully detached device vdb from instance 8b434db2-72e7-45a7-a946-a43b7e5b04de from the live domain config. | |
| 12:29:22 | gibi | obviously it is fishy that the event is never recevied even though the detach happened successfully on libvirt side | |
| 12:29:41 | gibi | so either libvirt does not send the event or nova does not receive it | |
| 12:33:27 | sean-k-mooney | gibi: stephen and i were just lookign at it inhis env | |
| 12:33:38 | sean-k-mooney | the is a virsh command to pretty print the events | |
| 12:33:52 | sean-k-mooney | and we are seeing the evnet form the qemu monitor | |
| 12:34:00 | sean-k-mooney | but nova does not for some reason | |
| 12:34:40 | sean-k-mooney | gibi: stephen goign to head to lunch in a bit but it looks liek somethign broke along the way | |
| 12:34:50 | gibi | cool nice troubleshooting | |
| 12:35:32 | sean-k-mooney | stephens theory was maybe the alisa did not match or it was someithng related to how we were mathcing the event | |
| 12:35:50 | sean-k-mooney | but i think we likely need to add extra loging to print all the libvirt events and confirm if nova even gets it | |
| 12:36:21 | sean-k-mooney | it could be something funky with the sepeate tread we have for that but its definly and upstream bug | |
| 12:39:33 | stephenfin | Yeah, what sean-k-mooney said. The reason it's such a big deal for cloud-provider-openstack is that the tests (a) create and delete a lot of PVCs (which translates to a lot of attach/detach calls), and (b) run serially, so things compound | |
| 12:40:13 | gibi | https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L312-L319 | |
| 12:40:34 | gibi | this is the code that finds which event waiter to unblock based on incoming events | |
| 12:41:01 | gibi | called from here https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L385 | |
| 12:41:32 | gibi | https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L2512-L2542 and this is the place where we get the events from libvirt | |
| 12:41:39 | gibi | it has logs | |
| 12:41:49 | gibi | so if these logs are not visible it means nova is not receiving any events | |
| 12:47:37 | sean-k-mooney | im not sure if that loggin actully works, which thrad is that runign on | |
| 12:47:40 | sean-k-mooney | specificly https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L2512-L2542 | |
| 12:47:58 | sean-k-mooney | is that on the libvirt pthread or on the nova main thread | |
| 12:48:17 | sean-k-mooney | there is some issue with mixing real thread and oslo.log | |
| 12:50:43 | sean-k-mooney | for what its worht the matching code looks corect | |
| 12:50:44 | gibi | yeah that is likely running on the libvirt pthread | |
| 12:50:45 | sean-k-mooney | https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L316-L319 | |
| 12:51:28 | gibi | I thought we can still log from the libvirt pthread | |
| 12:51:49 | sean-k-mooney | it was "fixed" by adding a pipemutex in oslo logs | |
| 12:52:01 | sean-k-mooney | but im not sure i really trust that entilry | |
| 12:52:07 | gibi | :) | |
| 12:52:38 | sean-k-mooney | im thinkging we add some good old prints or perhaps writes to a file that we can tail | |
| 12:52:55 | sean-k-mooney | although not right now. | |
| 12:56:28 | gibi | yeah that is always an option locally to troubleshoot it further | |
| 12:58:37 | stephenfin | https://bugs.launchpad.net/nova/+bug/2119114 | |
| 12:59:15 | stephenfin | I marked it as confirmed since eharney also took a look at this and I've reproduced it across multiple environments. I did not set a priority though | |
| 12:59:32 | stephenfin | eharney and sean-k-mooney, naturally | |
| 13:00:16 | gibi | stephenfin: thank you! | |
| 13:00:21 | gibi | Uggla: ^^ for visibility | |
| 13:01:30 | stephenfin | I will dump all this is the bug, but fwiw, I see no entries for `Received event` messages in Caracal or later. I do see them on Bobcat | |
| 13:01:43 | stephenfin | (in the n-cpu logs, that is) | |
| 13:04:05 | Uggla | gibi, thanks, I'm not sure. Does it relates to the "gate" bug we discussed in the meeting yesterday ? | |
| 13:08:44 | sean-k-mooney | stephenfin: so its not actully breaking anything its just slow but i think tis in the medium to high range | |
| 13:09:16 | sean-k-mooney | i woudl proably set it to high but its been 3 release so medium woudl work for me too | |
| 13:19:07 | gibi | Uggla: no it is a new one, it does not block the gate | |
| 13:19:45 | Uggla | gibi, oh ok. | |
| 13:20:45 | opendevreview | Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/956166 | |
| 13:22:23 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/944603 | |
| 13:42:21 | stephenfin | So this is interesting: restarting the n-cpu services fixes things... 🤔 | |
| 14:11:47 | gibi | stephenfin: maybe something blocks the libvirt thread receiving the notifications? | |
| 14:12:13 | gibi | and it gets unblocked by the restart until that something hits again and blocks it again | |
| 14:13:16 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.1: Add ServersViewBuilderTestV296 unit test class https://review.opendev.org/c/openstack/nova/+/955304 | |
| 14:13:17 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.1: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955305 | |
| 14:17:36 | dansmith | bauzas: did you see my comments on the tpm series about the confirm flag? | |
| 14:18:10 | dansmith | you seemed to ask why they were changed.. I haven't looked at the change yet, but I asked to do it differently to avoid creating a mess we have to clean up or live with forever | |
| 14:18:41 | opendevreview | Merged openstack/nova master: Update contributor guide for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/944603 | |
| 14:21:36 | andrewbonney | hi all. if there are any cores around I'd appreciate a review of https://review.opendev.org/c/openstack/nova/+/919961 | |
| 14:21:51 | andrewbonney | that's a small bug fix which has been hanging around for 1 year+ | |
| 14:26:23 | gibi | eventlet removal sync in 4 minutes https://meet.google.com/bcy-uqoz-hje | |
| 14:28:33 | gibi | andrewbonney: you are today's lucky one :) I approved the fix. Thanks for fixing that bug. | |
| 14:28:42 | andrewbonney | Thank you! | |
| 14:32:20 | stephenfin | gibi: Yeah, sean-k-mooney and I were thinking it might be a race, but it's weird that its happening consistently on every branch since bobcat | |
| 14:32:33 | stephenfin | restarting libvirtd didn't help either | |
| 15:03:49 | opendevreview | Ian Watson proposed openstack/nova master: Add migration conditional for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 15:09:23 | dansmith | gibi: andrewbonney I had commented about some missing test coverage | |
| 15:09:40 | dansmith | looks like we raced and I lost | |
| 15:14:22 | gibi | I'm fine pulling it from the gate or looking at a fup with the test coverage | |
| 15:15:42 | dansmith | maybe andrewbonney can at least respond to my question and do a FUP if we just need an extra assert in an adjacent test | |
| 15:16:23 | andrewbonney | I'll take a look, but I've been away for a few months so need to context switch back in | |
| 15:18:03 | sean-k-mooney | oh the dns patch | |
| 15:19:17 | sean-k-mooney | dansmith: your asking is the a testcase where we do resest the dns name? | |
| 15:20:04 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for quota sets API https://review.opendev.org/c/openstack/nova/+/956141 | |
| 15:20:05 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for migrations API https://review.opendev.org/c/openstack/nova/+/956142 | |
| 15:20:05 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server migrations API https://review.opendev.org/c/openstack/nova/+/956143 | |
| 15:20:06 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server tags API https://review.opendev.org/c/openstack/nova/+/956144 | |
| 15:20:06 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server metadata APIs https://review.opendev.org/c/openstack/nova/+/956145 | |
| 15:22:55 | dansmith | sean-k-mooney: no, | |
| 15:23:12 | sean-k-mooney | oh then i did not understand your question in the review | |
| 15:23:40 | dansmith | sean-k-mooney: the new test case added an assertion that we don't call reset when we're not supposed to, but I don't see an adjacent similar test case that asserts that we do call it when expected | |
| 15:23:59 | dansmith | I'm assuming there just needs to be an assert_called_once() somewhere in one of the nearby tests | |
| 15:24:26 | dansmith | point being, the patch is disabling calling that function in some situations, but I don't know that anything would break if it incorrectly disables the call in _all_ situations | |
| 15:26:48 | sean-k-mooney | oh ok am its only requrie if a prot is detached so the next test i think could have that mocked and assert its called | |