Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-02-06
11:05:36 bauzas indeed, but you haven't asked for it, that's my point
11:05:39 sean-k-mooney based on the host config. we will be adding the vtpm secrete type
11:05:53 sean-k-mooney you have not but we shoudl not exclude them
11:06:15 bauzas anyway, I need to go off
11:06:19 sean-k-mooney ack
12:20:41 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
12:20:41 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
13:15:49 opendevreview suiong ng proposed openstack/nova master: Fix parameter order in add_instance_info_to_node https://review.opendev.org/c/openstack/nova/+/939411
13:53:54 opendevreview sean mooney proposed openstack/nova master: allow discover host to be enabled in multiple schedulers https://review.opendev.org/c/openstack/nova/+/938523
13:54:31 sean-k-mooney gibi: i just dropped the job change form ^
13:55:48 sean-k-mooney if you have time to look at https://review.opendev.org/c/openstack/nova/+/939476 as well it would be nice to land that
13:59:20 sean-k-mooney ...
13:59:27 sean-k-mooney mikal: "libvirt.libvirtError: unsupported configuration: spice graphics are not supported with this QEMU"
13:59:58 sean-k-mooney it looks like ubuntu 24.04 didnt just remove qxl they fully removed spice supprot like rhel9
14:00:32 sean-k-mooney i think, debian still supprots it
14:01:21 sean-k-mooney so i could move that job to debian but if centos/rhel 9 and ubuntu 24.04 have fully drop spice that proably our last option to test it
14:01:58 sean-k-mooney kolla uses debian continer on all distos now so i woudl guess that is why its workign for you in kolla-ansible
14:02:24 sean-k-mooney that our your still using ubuntu 22.04?
15:04:13 ykarel Hi is the issue a known one where test fails randomly with libvirt.libvirtError: Failed to terminate process 70698 with SIGKILL: Device or resource busy ?
15:04:46 ykarel like seen in
15:04:46 ykarel https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
15:04:51 ykarel https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
15:06:23 ykarel found a quite old related patch https://review.opendev.org/c/openstack/nova/+/639091 but that was targetting a libvirt version and it got removed from nova now as that old versions no longer supported
15:35:42 sean-k-mooney ykarel: rerely teh qemu monitor process that is responisble for respondind to the howdown of the guest can become unresponsive
15:36:38 sean-k-mooney but htis is not realy a nova issue its more of a libvirt/qemu one
15:36:55 sean-k-mooney nova should not need to actuly retry the destroy at all
15:38:52 ykarel sean-k-mooney, ohk so it's a known issue? what is the mitigation for this then?
15:39:05 ykarel the operator/user have to retry the operation?
15:39:10 sean-k-mooney not really
15:39:32 sean-k-mooney its a very rare failure mode in qemu to exit gracefully when ask
15:39:51 sean-k-mooney libvirt tried to fix this by sendign sigkill instead of just sigterm
15:39:59 sean-k-mooney so the os would evenutally reap the process
15:40:45 sean-k-mooney if sigkill does not work there nothing nova can do to force it to exit really
15:42:00 ykarel ohkk got it, but why nova cannot do retry like done in above patch if response is ebusy ?
15:43:51 sean-k-mooney libvirt is ment to do it internally
15:45:31 ykarel ohkk but as per logs thats not helping as there are failures in CI
15:46:05 ykarel ohh or you mean libvirt should do it but it's not doing it and libvirt have to be fixed for that?
15:46:27 sean-k-mooney its either not doing it or it is doing it and qemu is still not exiting
15:47:52 ykarel ohkk no idea then where the issue is, i will start with nova bug and other points could be checked as part of it
15:48:01 ykarel thx sean-k-mooney for sharing all the insights
15:48:13 sean-k-mooney i dont think you shoudl start with a nova bug
15:48:43 sean-k-mooney i also think we sthil have some rety logic because _destory does call itesl https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:48:49 ykarel nova people may know more about libvirt/qemu bits so can judge more then me so i thought to start with it :)
15:51:11 sean-k-mooney i would not personaly consider it a nova bug but other might disagree
15:52:45 sean-k-mooney the orgianl issue was not a nova bug it was a workaround for a libvirt/qemu one
15:55:03 sean-k-mooney ykarel: as far as i can see we actuly retry infinetly https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1603
15:56:56 sean-k-mooney im not a huge fan of who this is written but in destroy we are using loopingcall form oslo.service to execute _wait_for_destroy which checks if the instnace is stil runing. if it sets kwargs['is_running'] = True
15:57:26 sean-k-mooney which asues use to recursivly call destroy https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:58:37 sean-k-mooney that loop will break however if we get an excption form libvirt
16:02:02 sean-k-mooney so if we get EBUSY or other libvirt exction in _wait_for_destroy it wont retry
16:02:15 ykarel but that is_running = true seems not reached as i don't see it in logs
16:02:19 ykarel it's raising at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1566
16:02:46 ykarel i.e not seeing in logs "Going to destroy instance again"
16:02:57 sean-k-mooney that means the libvirt could not destory the instnace
16:03:41 ykarel but you said it will retry indefinitely
16:03:48 sean-k-mooney yes if there is no error
16:04:00 sean-k-mooney so if we ask libvirt to destory the instance and there is no error
16:04:14 sean-k-mooney but it does not complete in half a secodn we ask it to do it again
16:04:36 sean-k-mooney so we retry only when tehre isnt an error
16:05:59 sean-k-mooney if you retired the delete or hard reboot ectra it may work if libvirt has correct its internal error or qemu finally died
16:07:16 sean-k-mooney if qemu is locked up badly enough to service a sig kill form libvirt its possibel the only way to kill it woudl be a host reboot. trying it again might work but there is no garentee
16:07:55 sean-k-mooney we coudl readd a limite retry in error but it woudl really just eb papering over a potital libvirt or qemu bug
16:08:37 sean-k-mooney we have done that in the past but its kind of tech debt every time we do which is why we removed this
16:09:18 ykarel removal seems was done as part of cleanup of old libvirt version, but okk got everything what you mean
16:10:12 ykarel but it's good if it could be worked around atleast for the cases where it's not a persistent issue on qemu side
16:10:38 sean-k-mooney if you want to file it as a bug do, if we readd this i think we woudl put it behind a workaround flag
16:10:59 ykarel ok that should also work
16:11:12 ykarel thx again
16:11:14 sean-k-mooney it would also look at the failing job
16:11:35 sean-k-mooney this happens because of very high load
16:11:58 sean-k-mooney so the job may be usign too high a concurancy or have other isseus that is triggering it
16:12:25 ykarel hmm underlying hypervisor where it ran might also be overloaded
16:13:44 sean-k-mooney i belive libvirt is now going to wait at least 40s instead of 15 before retuing EBUSY
16:22:38 ykarel yes as per logs i see 40s b/w requesting destroy and returned error
17:50:04 opendevreview Doug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during live migration https://review.opendev.org/c/openstack/nova/+/940900
17:53:52 dougszu Any opinions on that would be great ^. I've got another fix for cold migration using block pull to share if it's of interest. And of course, the final bit to close the bug would be not making file systems on the ephemerals at all.
19:04:10 sean-k-mooney dougszu: i see you tried removing the backing file
19:04:36 sean-k-mooney longterm definlty the direction we shoudl go
19:05:08 sean-k-mooney we likely would need to cahnge intal boot/reboot and cold migration or evacuate too
19:05:12 sean-k-mooney if we were to do this.
19:06:02 sean-k-mooney i.e procive a way for nova, on the next lifeccyle operation on the guest to move or redifien the domain, to remove the backing file for ephmeral and swap disk
19:07:23 mikal sean-k-mooney: yeah, I am using Debian containers on Kolla-Ansible, so that's why I didn't notice the Ubuntu behaviour.
19:07:49 sean-k-mooney i need to push my pathc to try the job with debian
19:08:05 sean-k-mooney i have been gettign distracted for like the last 3 hours
19:08:21 mikal Heh, its ok, I didn't manage to get to tempest for similar reasons yesterday.
19:09:03 mikal I find people dropping SPICE quite frustrating to be entirely honest. The public excuse seems to be H.264 requirements, but RDP has the same requirements and there is a patent license from Cisco for the codec.
19:09:28 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
19:09:28 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
19:09:52 sean-k-mooney mikal: spice is the better console. vnc is catching up but spice is still better
19:10:15 sean-k-mooney mikal: i dont think it h.264 related honestly
19:10:25 mikal sean-k-mooney: yeah, RDP is probably about as good as SPICE in terms of features, but qemu doesn't support it at all and the protocol is much more complicated as best as I can tell.
19:13:24 sean-k-mooney whats funney is as far as i know its the default in gnome boxes and a few other things
19:13:36 sean-k-mooney so its not like spice was unsued
19:14:03 mikal oVirt used it a bit too, as did proxmox.
19:14:23 mikal Also... its pretty stable. It hasn't changed much if at all in years.
19:14:30 sean-k-mooney i think its still an option in proxmox given that derived form rhel
19:14:32 mikal So there's no feature chasing etc.
19:14:34 sean-k-mooney *debian
19:15:20 sean-k-mooney speaking of stable i have updated the patch to the latest debian stable (12/bookworm)
19:16:37 sean-k-mooney actully we do not need grenade and some of the other jobs ill quickly drop those out whiel we are testign

Earlier   Later