Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-02-06
15:47:52 ykarel ohkk no idea then where the issue is, i will start with nova bug and other points could be checked as part of it
15:48:01 ykarel thx sean-k-mooney for sharing all the insights
15:48:13 sean-k-mooney i dont think you shoudl start with a nova bug
15:48:43 sean-k-mooney i also think we sthil have some rety logic because _destory does call itesl https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:48:49 ykarel nova people may know more about libvirt/qemu bits so can judge more then me so i thought to start with it :)
15:51:11 sean-k-mooney i would not personaly consider it a nova bug but other might disagree
15:52:45 sean-k-mooney the orgianl issue was not a nova bug it was a workaround for a libvirt/qemu one
15:55:03 sean-k-mooney ykarel: as far as i can see we actuly retry infinetly https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1603
15:56:56 sean-k-mooney im not a huge fan of who this is written but in destroy we are using loopingcall form oslo.service to execute _wait_for_destroy which checks if the instnace is stil runing. if it sets kwargs['is_running'] = True
15:57:26 sean-k-mooney which asues use to recursivly call destroy https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:58:37 sean-k-mooney that loop will break however if we get an excption form libvirt
16:02:02 sean-k-mooney so if we get EBUSY or other libvirt exction in _wait_for_destroy it wont retry
16:02:15 ykarel but that is_running = true seems not reached as i don't see it in logs
16:02:19 ykarel it's raising at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1566
16:02:46 ykarel i.e not seeing in logs "Going to destroy instance again"
16:02:57 sean-k-mooney that means the libvirt could not destory the instnace
16:03:41 ykarel but you said it will retry indefinitely
16:03:48 sean-k-mooney yes if there is no error
16:04:00 sean-k-mooney so if we ask libvirt to destory the instance and there is no error
16:04:14 sean-k-mooney but it does not complete in half a secodn we ask it to do it again
16:04:36 sean-k-mooney so we retry only when tehre isnt an error
16:05:59 sean-k-mooney if you retired the delete or hard reboot ectra it may work if libvirt has correct its internal error or qemu finally died
16:07:16 sean-k-mooney if qemu is locked up badly enough to service a sig kill form libvirt its possibel the only way to kill it woudl be a host reboot. trying it again might work but there is no garentee
16:07:55 sean-k-mooney we coudl readd a limite retry in error but it woudl really just eb papering over a potital libvirt or qemu bug
16:08:37 sean-k-mooney we have done that in the past but its kind of tech debt every time we do which is why we removed this
16:09:18 ykarel removal seems was done as part of cleanup of old libvirt version, but okk got everything what you mean
16:10:12 ykarel but it's good if it could be worked around atleast for the cases where it's not a persistent issue on qemu side
16:10:38 sean-k-mooney if you want to file it as a bug do, if we readd this i think we woudl put it behind a workaround flag
16:10:59 ykarel ok that should also work
16:11:12 ykarel thx again
16:11:14 sean-k-mooney it would also look at the failing job
16:11:35 sean-k-mooney this happens because of very high load
16:11:58 sean-k-mooney so the job may be usign too high a concurancy or have other isseus that is triggering it
16:12:25 ykarel hmm underlying hypervisor where it ran might also be overloaded
16:13:44 sean-k-mooney i belive libvirt is now going to wait at least 40s instead of 15 before retuing EBUSY
16:22:38 ykarel yes as per logs i see 40s b/w requesting destroy and returned error
17:50:04 opendevreview Doug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during live migration https://review.opendev.org/c/openstack/nova/+/940900
17:53:52 dougszu Any opinions on that would be great ^. I've got another fix for cold migration using block pull to share if it's of interest. And of course, the final bit to close the bug would be not making file systems on the ephemerals at all.
19:04:10 sean-k-mooney dougszu: i see you tried removing the backing file
19:04:36 sean-k-mooney longterm definlty the direction we shoudl go
19:05:08 sean-k-mooney we likely would need to cahnge intal boot/reboot and cold migration or evacuate too
19:05:12 sean-k-mooney if we were to do this.
19:06:02 sean-k-mooney i.e procive a way for nova, on the next lifeccyle operation on the guest to move or redifien the domain, to remove the backing file for ephmeral and swap disk
19:07:23 mikal sean-k-mooney: yeah, I am using Debian containers on Kolla-Ansible, so that's why I didn't notice the Ubuntu behaviour.
19:07:49 sean-k-mooney i need to push my pathc to try the job with debian
19:08:05 sean-k-mooney i have been gettign distracted for like the last 3 hours
19:08:21 mikal Heh, its ok, I didn't manage to get to tempest for similar reasons yesterday.
19:09:03 mikal I find people dropping SPICE quite frustrating to be entirely honest. The public excuse seems to be H.264 requirements, but RDP has the same requirements and there is a patent license from Cisco for the codec.
19:09:28 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
19:09:28 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
19:09:52 sean-k-mooney mikal: spice is the better console. vnc is catching up but spice is still better
19:10:15 sean-k-mooney mikal: i dont think it h.264 related honestly
19:10:25 mikal sean-k-mooney: yeah, RDP is probably about as good as SPICE in terms of features, but qemu doesn't support it at all and the protocol is much more complicated as best as I can tell.
19:13:24 sean-k-mooney whats funney is as far as i know its the default in gnome boxes and a few other things
19:13:36 sean-k-mooney so its not like spice was unsued
19:14:03 mikal oVirt used it a bit too, as did proxmox.
19:14:23 mikal Also... its pretty stable. It hasn't changed much if at all in years.
19:14:30 sean-k-mooney i think its still an option in proxmox given that derived form rhel
19:14:32 mikal So there's no feature chasing etc.
19:14:34 sean-k-mooney *debian
19:15:20 sean-k-mooney speaking of stable i have updated the patch to the latest debian stable (12/bookworm)
19:16:37 sean-k-mooney actully we do not need grenade and some of the other jobs ill quickly drop those out whiel we are testign
19:17:37 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
19:17:37 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
19:18:46 sean-k-mooney ok better its just runnign one job now
19:18:54 opendevreview Michael Still proposed openstack/nova master: Don't calculate the minimum compute version repeatedly. https://review.opendev.org/c/openstack/nova/+/940848
19:18:55 opendevreview Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126
19:18:55 opendevreview Michael Still proposed openstack/nova master: Protect older compute managers from sound model requests. https://review.opendev.org/c/openstack/nova/+/940770
19:18:56 opendevreview Michael Still proposed openstack/nova master: libvirt: Add extra specs for USB redirection. https://review.opendev.org/c/openstack/nova/+/927354
19:19:07 mikal ^--- fixes a one character error in the first patch in that series
19:20:42 sean-k-mooney ah you have split out the sound/usb ones and the performance improvment, nice
19:22:55 mikal Yeah, I wrote up a decoder ring on the etherpad because its getting complicated -- https://etherpad.opendev.org/p/nova-2025.1-status
19:23:20 mikal But there's basically two independent series now -- the API changes, and the "VDI changes" (sound, usb, compute version bumps etc).
19:23:53 mikal I agree we should focus on the API changes first, but that's blocking right now on me getting some quality time with tempest.
19:27:27 sean-k-mooney in its current form both can be merged in either order and in parallel so that will help i think.
20:14:21 sean-k-mooney mikal: https://zuul.opendev.org/t/openstack/build/842a66f2cfe54eb39d856a3bf311bb5c
20:14:34 sean-k-mooney mikal: debian works and the job passes with it
20:15:04 sean-k-mooney so ill revert the commetned out jobs but we should be able to test in that job
20:27:39 opendevreview sean mooney proposed openstack/nova master: move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
20:27:39 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
20:34:21 mikal sean-k-mooney: cool, thank you. I assume we still want to do a tempest test, which I will try to actually get around to attempting today.
20:35:21 sean-k-mooney i think so but those should help with a job we can run the test in
20:35:56 mikal Agreed
20:36:05 mikal Thanks for chasing this bit for me.
20:36:09 sean-k-mooney i just added those to the etherpad too with some explanations
20:36:28 mikal Cool
20:36:45 sean-k-mooney mikal: if nothing else, we have not deprecated spice supprot and there is litrally no spice job anywhere so just for that i consider those to be a win
#openstack-nova - 2025-02-07
06:11:58 mikal sean-k-mooney / dansmith / melwitt: https://review.opendev.org/c/openstack/tempest/+/940943 is an attempt at a tempest test for the spice-direct stuff. I think it likely needs to be entangled with sean-k-mooney's devstack and zuul config changes to actually work though.
06:13:14 mikal I've put it on the etherpad as well at the end of the API series.
07:37:58 opendevreview benlei proposed openstack/nova master: Abort live migration task when stop nova compute service https://review.opendev.org/c/openstack/nova/+/938223
10:34:48 opendevreview Balazs Gibizer proposed openstack/nova master: Bump InstanceNUMACell version after data migration https://review.opendev.org/c/openstack/nova/+/940604
10:34:49 opendevreview Balazs Gibizer proposed openstack/nova master: Bump InstanceNUMACell version after half migration https://review.opendev.org/c/openstack/nova/+/940953
10:55:37 Uggla Tentative vGPU vfio discussing latest open points. Friday, February 7 · 14:00 – 14:45 UTC link: https://meet.google.com/hia-iodh-fcz
11:10:35 opendevreview Taketani Ryo proposed openstack/nova-specs master: Add a spec for libvirt driver launching Arm CCA-encrypted instances https://review.opendev.org/c/openstack/nova-specs/+/938276
11:20:13 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version after data migration https://review.opendev.org/c/openstack/nova/+/940604
11:20:13 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version in more cases https://review.opendev.org/c/openstack/nova/+/940953
11:20:14 opendevreview Balazs Gibizer proposed openstack/nova master: Refactor InstanceNUMA* data migrations https://review.opendev.org/c/openstack/nova/+/940958
11:30:01 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version in more cases https://review.opendev.org/c/openstack/nova/+/940953
12:40:58 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version in more cases https://review.opendev.org/c/openstack/nova/+/940953
12:41:25 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing image properties in server show response https://review.opendev.org/c/openstack/nova/+/939649

Earlier   Later