Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-30
12:11:09 sean-k-mooney so its happenign on debian too
12:11:37 sean-k-mooney with ceph in this case you were using lvm right
12:11:52 sean-k-mooney the backend should not really matter
12:12:02 stephenfin yes (pretty much a "stock" DevStack configuration)
12:12:04 sean-k-mooney but that is both local block and network type covered
12:12:10 sean-k-mooney ack
12:12:19 sean-k-mooney let me see if i still have a c9s devstack somewhere
12:14:32 sean-k-mooney that woudl be no but i can check our watcher ci job
12:14:46 sean-k-mooney actully no
12:15:04 sean-k-mooney we have master jobs that trigger with our new installer on cenetos butthsoe are disbaling cinder right now
12:15:14 sean-k-mooney i think neutorn and maybe cinder have centos josb
12:15:34 sean-k-mooney we might have an experminatal one too
12:16:07 stephenfin I'll just create one locally
12:16:38 stephenfin However, I see a few related changed that I'm going to try reverting out of caracal first
12:22:52 gibi I did the last round of fixes in that codepath so I remember it was working at that time to wait for the events. If the event does not come we still have code to check if the actual detach happened and move on. But as you noted it might be excessive waiting time
12:23:53 gibi if this is visible in upstream CI then please file an upstream bug with the links and I will try to look at it
12:27:14 gibi In https://d6e5655f0b1ee2f8df74-e35c415f7cc7f78f33337c81de121e72.ssl.cf2.rackcdn.com/openstack/52b57c77f805400888762a87e08196f1/controller/logs/screen-n-cpu.txt I see cases when it is on 20 sec wait and we move one
12:27:32 gibi Jul 30 01:36:30.229367 npa183b57ac85f4 nova-compute[75854]: DEBUG nova.virt.libvirt.guest [None req-5859b1ac-1198-4297-845b-2c4ef82e635b tempest-VolumesAssistedSnapshotsTest-257024579 tempest-VolumesAssistedSnapshotsTest-257024579-project-admin] detach device xml: <disk type="block" device="disk">
12:27:47 gibi Jul 30 01:36:50.849840 npa183b57ac85f4 nova-compute[75854]: WARNING nova.virt.libvirt.driver [None req-5859b1ac-1198-4297-845b-2c4ef82e635b tempest-VolumesAssistedSnapshotsTest-257024579 tempest-VolumesAssistedSnapshotsTest-257024579-project-admin] Waiting for libvirt event about the detach of device vdb with device alias ua-50261be4-1b93-406a-ad22-095d53ab27bb from instance
12:27:53 gibi 8b434db2-72e7-45a7-a946-a43b7e5b04de is timed out.
12:27:57 gibi Jul 30 01:36:50.862330 npa183b57ac85f4 nova-compute[75854]: INFO nova.virt.libvirt.driver [None req-5859b1ac-1198-4297-845b-2c4ef82e635b tempest-VolumesAssistedSnapshotsTest-257024579 tempest-VolumesAssistedSnapshotsTest-257024579-project-admin] Successfully detached device vdb from instance 8b434db2-72e7-45a7-a946-a43b7e5b04de from the live domain config.
12:29:22 gibi obviously it is fishy that the event is never recevied even though the detach happened successfully on libvirt side
12:29:41 gibi so either libvirt does not send the event or nova does not receive it
12:33:27 sean-k-mooney gibi: stephen and i were just lookign at it inhis env
12:33:38 sean-k-mooney the is a virsh command to pretty print the events
12:33:52 sean-k-mooney and we are seeing the evnet form the qemu monitor
12:34:00 sean-k-mooney but nova does not for some reason
12:34:40 sean-k-mooney gibi: stephen goign to head to lunch in a bit but it looks liek somethign broke along the way
12:34:50 gibi cool nice troubleshooting
12:35:32 sean-k-mooney stephens theory was maybe the alisa did not match or it was someithng related to how we were mathcing the event
12:35:50 sean-k-mooney but i think we likely need to add extra loging to print all the libvirt events and confirm if nova even gets it
12:36:21 sean-k-mooney it could be something funky with the sepeate tread we have for that but its definly and upstream bug
12:39:33 stephenfin Yeah, what sean-k-mooney said. The reason it's such a big deal for cloud-provider-openstack is that the tests (a) create and delete a lot of PVCs (which translates to a lot of attach/detach calls), and (b) run serially, so things compound
12:40:13 gibi https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L312-L319
12:40:34 gibi this is the code that finds which event waiter to unblock based on incoming events
12:41:01 gibi called from here https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L385
12:41:32 gibi https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L2512-L2542 and this is the place where we get the events from libvirt
12:41:39 gibi it has logs
12:41:49 gibi so if these logs are not visible it means nova is not receiving any events
12:47:37 sean-k-mooney im not sure if that loggin actully works, which thrad is that runign on
12:47:40 sean-k-mooney specificly https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L2512-L2542
12:47:58 sean-k-mooney is that on the libvirt pthread or on the nova main thread
12:48:17 sean-k-mooney there is some issue with mixing real thread and oslo.log
12:50:43 sean-k-mooney for what its worht the matching code looks corect
12:50:44 gibi yeah that is likely running on the libvirt pthread
12:50:45 sean-k-mooney https://github.com/openstack/nova/blob/3a08a557df7b9a52c8faefe010df0ba9994c63a9/nova/virt/libvirt/driver.py#L316-L319
12:51:28 gibi I thought we can still log from the libvirt pthread
12:51:49 sean-k-mooney it was "fixed" by adding a pipemutex in oslo logs
12:52:01 sean-k-mooney but im not sure i really trust that entilry
12:52:07 gibi :)
12:52:38 sean-k-mooney im thinkging we add some good old prints or perhaps writes to a file that we can tail
12:52:55 sean-k-mooney although not right now.
12:56:28 gibi yeah that is always an option locally to troubleshoot it further
12:58:37 stephenfin https://bugs.launchpad.net/nova/+bug/2119114
12:59:15 stephenfin I marked it as confirmed since eharney also took a look at this and I've reproduced it across multiple environments. I did not set a priority though
12:59:32 stephenfin eharney and sean-k-mooney, naturally
13:00:16 gibi stephenfin: thank you!
13:00:21 gibi Uggla: ^^ for visibility
13:01:30 stephenfin I will dump all this is the bug, but fwiw, I see no entries for `Received event` messages in Caracal or later. I do see them on Bobcat
13:01:43 stephenfin (in the n-cpu logs, that is)
13:04:05 Uggla gibi, thanks, I'm not sure. Does it relates to the "gate" bug we discussed in the meeting yesterday ?
13:08:44 sean-k-mooney stephenfin: so its not actully breaking anything its just slow but i think tis in the medium to high range
13:09:16 sean-k-mooney i woudl proably set it to high but its been 3 release so medium woudl work for me too
13:19:07 gibi Uggla: no it is a new one, it does not block the gate
13:19:45 Uggla gibi, oh ok.
13:20:45 opendevreview Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/956166
13:22:23 opendevreview Takashi Natsume proposed openstack/nova master: Update contributor guide for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/944603
13:42:21 stephenfin So this is interesting: restarting the n-cpu services fixes things... 🤔
14:11:47 gibi stephenfin: maybe something blocks the libvirt thread receiving the notifications?
14:12:13 gibi and it gets unblocked by the restart until that something hits again and blocks it again
14:13:16 opendevreview Wesley Hershberger proposed openstack/nova stable/2024.1: Add ServersViewBuilderTestV296 unit test class https://review.opendev.org/c/openstack/nova/+/955304
14:13:17 opendevreview Wesley Hershberger proposed openstack/nova stable/2024.1: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955305
14:17:36 dansmith bauzas: did you see my comments on the tpm series about the confirm flag?
14:18:10 dansmith you seemed to ask why they were changed.. I haven't looked at the change yet, but I asked to do it differently to avoid creating a mess we have to clean up or live with forever
14:18:41 opendevreview Merged openstack/nova master: Update contributor guide for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/944603
14:21:36 andrewbonney hi all. if there are any cores around I'd appreciate a review of https://review.opendev.org/c/openstack/nova/+/919961
14:21:51 andrewbonney that's a small bug fix which has been hanging around for 1 year+
14:26:23 gibi eventlet removal sync in 4 minutes https://meet.google.com/bcy-uqoz-hje
14:28:33 gibi andrewbonney: you are today's lucky one :) I approved the fix. Thanks for fixing that bug.
14:28:42 andrewbonney Thank you!
14:32:20 stephenfin gibi: Yeah, sean-k-mooney and I were thinking it might be a race, but it's weird that its happening consistently on every branch since bobcat
14:32:33 stephenfin restarting libvirtd didn't help either
15:03:49 opendevreview Ian Watson proposed openstack/nova master: Add migration conditional for resource tracker https://review.opendev.org/c/openstack/nova/+/955690
15:09:23 dansmith gibi: andrewbonney I had commented about some missing test coverage
15:09:40 dansmith looks like we raced and I lost
15:14:22 gibi I'm fine pulling it from the gate or looking at a fup with the test coverage
15:15:42 dansmith maybe andrewbonney can at least respond to my question and do a FUP if we just need an extra assert in an adjacent test
15:16:23 andrewbonney I'll take a look, but I've been away for a few months so need to context switch back in
15:18:03 sean-k-mooney oh the dns patch
15:19:17 sean-k-mooney dansmith: your asking is the a testcase where we do resest the dns name?
15:20:04 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for quota sets API https://review.opendev.org/c/openstack/nova/+/956141
15:20:05 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for migrations API https://review.opendev.org/c/openstack/nova/+/956142
15:20:05 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server migrations API https://review.opendev.org/c/openstack/nova/+/956143
15:20:06 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server tags API https://review.opendev.org/c/openstack/nova/+/956144
15:20:06 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server metadata APIs https://review.opendev.org/c/openstack/nova/+/956145
15:22:55 dansmith sean-k-mooney: no,
15:23:12 sean-k-mooney oh then i did not understand your question in the review
15:23:40 dansmith sean-k-mooney: the new test case added an assertion that we don't call reset when we're not supposed to, but I don't see an adjacent similar test case that asserts that we do call it when expected
15:23:59 dansmith I'm assuming there just needs to be an assert_called_once() somewhere in one of the nearby tests
15:24:26 dansmith point being, the patch is disabling calling that function in some situations, but I don't know that anything would break if it incorrectly disables the call in _all_ situations

Earlier   Later