Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-02-06
15:56:56 sean-k-mooney im not a huge fan of who this is written but in destroy we are using loopingcall form oslo.service to execute _wait_for_destroy which checks if the instnace is stil runing. if it sets kwargs['is_running'] = True
15:57:26 sean-k-mooney which asues use to recursivly call destroy https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:58:37 sean-k-mooney that loop will break however if we get an excption form libvirt
16:02:02 sean-k-mooney so if we get EBUSY or other libvirt exction in _wait_for_destroy it wont retry
16:02:15 ykarel but that is_running = true seems not reached as i don't see it in logs
16:02:19 ykarel it's raising at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1566
16:02:46 ykarel i.e not seeing in logs "Going to destroy instance again"
16:02:57 sean-k-mooney that means the libvirt could not destory the instnace
16:03:41 ykarel but you said it will retry indefinitely
16:03:48 sean-k-mooney yes if there is no error
16:04:00 sean-k-mooney so if we ask libvirt to destory the instance and there is no error
16:04:14 sean-k-mooney but it does not complete in half a secodn we ask it to do it again
16:04:36 sean-k-mooney so we retry only when tehre isnt an error
16:05:59 sean-k-mooney if you retired the delete or hard reboot ectra it may work if libvirt has correct its internal error or qemu finally died
16:07:16 sean-k-mooney if qemu is locked up badly enough to service a sig kill form libvirt its possibel the only way to kill it woudl be a host reboot. trying it again might work but there is no garentee
16:07:55 sean-k-mooney we coudl readd a limite retry in error but it woudl really just eb papering over a potital libvirt or qemu bug
16:08:37 sean-k-mooney we have done that in the past but its kind of tech debt every time we do which is why we removed this
16:09:18 ykarel removal seems was done as part of cleanup of old libvirt version, but okk got everything what you mean
16:10:12 ykarel but it's good if it could be worked around atleast for the cases where it's not a persistent issue on qemu side
16:10:38 sean-k-mooney if you want to file it as a bug do, if we readd this i think we woudl put it behind a workaround flag
16:10:59 ykarel ok that should also work
16:11:12 ykarel thx again
16:11:14 sean-k-mooney it would also look at the failing job
16:11:35 sean-k-mooney this happens because of very high load
16:11:58 sean-k-mooney so the job may be usign too high a concurancy or have other isseus that is triggering it
16:12:25 ykarel hmm underlying hypervisor where it ran might also be overloaded
16:13:44 sean-k-mooney i belive libvirt is now going to wait at least 40s instead of 15 before retuing EBUSY
16:22:38 ykarel yes as per logs i see 40s b/w requesting destroy and returned error
17:50:04 opendevreview Doug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during live migration https://review.opendev.org/c/openstack/nova/+/940900
17:53:52 dougszu Any opinions on that would be great ^. I've got another fix for cold migration using block pull to share if it's of interest. And of course, the final bit to close the bug would be not making file systems on the ephemerals at all.
19:04:10 sean-k-mooney dougszu: i see you tried removing the backing file
19:04:36 sean-k-mooney longterm definlty the direction we shoudl go
19:05:08 sean-k-mooney we likely would need to cahnge intal boot/reboot and cold migration or evacuate too
19:05:12 sean-k-mooney if we were to do this.
19:06:02 sean-k-mooney i.e procive a way for nova, on the next lifeccyle operation on the guest to move or redifien the domain, to remove the backing file for ephmeral and swap disk
19:07:23 mikal sean-k-mooney: yeah, I am using Debian containers on Kolla-Ansible, so that's why I didn't notice the Ubuntu behaviour.
19:07:49 sean-k-mooney i need to push my pathc to try the job with debian
19:08:05 sean-k-mooney i have been gettign distracted for like the last 3 hours
19:08:21 mikal Heh, its ok, I didn't manage to get to tempest for similar reasons yesterday.
19:09:03 mikal I find people dropping SPICE quite frustrating to be entirely honest. The public excuse seems to be H.264 requirements, but RDP has the same requirements and there is a patent license from Cisco for the codec.
19:09:28 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
19:09:28 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
19:09:52 sean-k-mooney mikal: spice is the better console. vnc is catching up but spice is still better
19:10:15 sean-k-mooney mikal: i dont think it h.264 related honestly
19:10:25 mikal sean-k-mooney: yeah, RDP is probably about as good as SPICE in terms of features, but qemu doesn't support it at all and the protocol is much more complicated as best as I can tell.
19:13:24 sean-k-mooney whats funney is as far as i know its the default in gnome boxes and a few other things
19:13:36 sean-k-mooney so its not like spice was unsued
19:14:03 mikal oVirt used it a bit too, as did proxmox.
19:14:23 mikal Also... its pretty stable. It hasn't changed much if at all in years.
19:14:30 sean-k-mooney i think its still an option in proxmox given that derived form rhel
19:14:32 mikal So there's no feature chasing etc.
19:14:34 sean-k-mooney *debian
19:15:20 sean-k-mooney speaking of stable i have updated the patch to the latest debian stable (12/bookworm)
19:16:37 sean-k-mooney actully we do not need grenade and some of the other jobs ill quickly drop those out whiel we are testign
19:17:37 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
19:17:37 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
19:18:46 sean-k-mooney ok better its just runnign one job now
19:18:54 opendevreview Michael Still proposed openstack/nova master: Don't calculate the minimum compute version repeatedly. https://review.opendev.org/c/openstack/nova/+/940848
19:18:55 opendevreview Michael Still proposed openstack/nova master: Protect older compute managers from sound model requests. https://review.opendev.org/c/openstack/nova/+/940770
19:18:55 opendevreview Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126
19:18:56 opendevreview Michael Still proposed openstack/nova master: libvirt: Add extra specs for USB redirection. https://review.opendev.org/c/openstack/nova/+/927354
19:19:07 mikal ^--- fixes a one character error in the first patch in that series
19:20:42 sean-k-mooney ah you have split out the sound/usb ones and the performance improvment, nice
19:22:55 mikal Yeah, I wrote up a decoder ring on the etherpad because its getting complicated -- https://etherpad.opendev.org/p/nova-2025.1-status
19:23:20 mikal But there's basically two independent series now -- the API changes, and the "VDI changes" (sound, usb, compute version bumps etc).
19:23:53 mikal I agree we should focus on the API changes first, but that's blocking right now on me getting some quality time with tempest.
19:27:27 sean-k-mooney in its current form both can be merged in either order and in parallel so that will help i think.
20:14:21 sean-k-mooney mikal: https://zuul.opendev.org/t/openstack/build/842a66f2cfe54eb39d856a3bf311bb5c
20:14:34 sean-k-mooney mikal: debian works and the job passes with it
20:15:04 sean-k-mooney so ill revert the commetned out jobs but we should be able to test in that job
20:27:39 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
20:27:39 opendevreview sean mooney proposed openstack/nova master: move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
20:34:21 mikal sean-k-mooney: cool, thank you. I assume we still want to do a tempest test, which I will try to actually get around to attempting today.
20:35:21 sean-k-mooney i think so but those should help with a job we can run the test in
20:35:56 mikal Agreed
20:36:05 mikal Thanks for chasing this bit for me.
20:36:09 sean-k-mooney i just added those to the etherpad too with some explanations
20:36:28 mikal Cool
20:36:45 sean-k-mooney mikal: if nothing else, we have not deprecated spice supprot and there is litrally no spice job anywhere so just for that i consider those to be a win
#openstack-nova - 2025-02-07
06:11:58 mikal sean-k-mooney / dansmith / melwitt: https://review.opendev.org/c/openstack/tempest/+/940943 is an attempt at a tempest test for the spice-direct stuff. I think it likely needs to be entangled with sean-k-mooney's devstack and zuul config changes to actually work though.
06:13:14 mikal I've put it on the etherpad as well at the end of the API series.
07:37:58 opendevreview benlei proposed openstack/nova master: Abort live migration task when stop nova compute service https://review.opendev.org/c/openstack/nova/+/938223
10:34:48 opendevreview Balazs Gibizer proposed openstack/nova master: Bump InstanceNUMACell version after data migration https://review.opendev.org/c/openstack/nova/+/940604
10:34:49 opendevreview Balazs Gibizer proposed openstack/nova master: Bump InstanceNUMACell version after half migration https://review.opendev.org/c/openstack/nova/+/940953
10:55:37 Uggla Tentative vGPU vfio discussing latest open points. Friday, February 7 · 14:00 – 14:45 UTC link: https://meet.google.com/hia-iodh-fcz
11:10:35 opendevreview Taketani Ryo proposed openstack/nova-specs master: Add a spec for libvirt driver launching Arm CCA-encrypted instances https://review.opendev.org/c/openstack/nova-specs/+/938276
11:20:13 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version in more cases https://review.opendev.org/c/openstack/nova/+/940953
11:20:13 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version after data migration https://review.opendev.org/c/openstack/nova/+/940604
11:20:14 opendevreview Balazs Gibizer proposed openstack/nova master: Refactor InstanceNUMA* data migrations https://review.opendev.org/c/openstack/nova/+/940958
11:30:01 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version in more cases https://review.opendev.org/c/openstack/nova/+/940953
12:40:58 opendevreview Balazs Gibizer proposed openstack/nova master: Update InstanceNUMACell version in more cases https://review.opendev.org/c/openstack/nova/+/940953
12:41:25 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing image properties in server show response https://review.opendev.org/c/openstack/nova/+/939649
12:47:49 gibi dansmith: the pcpuset fix is now split and hopefully satisfies CI https://review.opendev.org/q/topic:%22bug/2097359%22
12:52:29 sean-k-mooney mikal: ill be back on monday, im on pto today but we should be able to add a patch to nova on top of my series that depend on that and your api change and that will demonstrate it workign end to end, you can have depneds on in the same repo we just normally dont becuase its harder to debug if there is a merge conflcit, but it will mean we do not need to stack the patches.
15:09:44 opendevreview ribaudr proposed openstack/nova master: Bump os-traits to 3.3.0 in requirements https://review.opendev.org/c/openstack/nova/+/940975
15:16:27 opendevreview ribaudr proposed openstack/placement master: Bump os-traits to 3.3.0 in requirements https://review.opendev.org/c/openstack/placement/+/940976
16:05:36 opendevreview Balazs Gibizer proposed openstack/nova master: [test]generic OVO version update tests https://review.opendev.org/c/openstack/nova/+/940984
16:07:23 gibi dansmith: ^^ this is my fist stab about the generic ovo version update testing
16:08:04 dansmith ack
16:08:09 gibi it already able to catch the pcpuset bug while showing that the same bug is not reproducible via request_spec.numa_topology just via instance.numa_topology

Earlier   Later