Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-02-06
11:01:47 sean-k-mooney i.e. if 2 isntance request the image property on one host adn one instnace request it on anohter shoudl not affect the weight
11:02:00 sean-k-mooney i think you are trying to also include that by using counter right?
11:02:58 sean-k-mooney we can chat later but i was intentionly trying not to have the number of vms on a host affect teh weighing of the host
11:03:21 bauzas I tested it, and the problem is that for example, if you have a host that has already 2 instances and one other that only has 1 using the prop, then it would prefer the second
11:04:08 bauzas so I preferred to just return the number of how many asked properties that are alrady used in the host
11:04:20 bauzas this is simplier to calculate and understand
11:04:45 sean-k-mooney i think that is more complex
11:04:54 bauzas also, I saw with devstack that when you create an instance, you get more props that the one you asked
11:04:56 sean-k-mooney and harder ot understand
11:05:14 sean-k-mooney you do yes we set some like the machine type on boot
11:05:19 sean-k-mooney or config drive
11:05:36 bauzas indeed, but you haven't asked for it, that's my point
11:05:39 sean-k-mooney based on the host config. we will be adding the vtpm secrete type
11:05:53 sean-k-mooney you have not but we shoudl not exclude them
11:06:15 bauzas anyway, I need to go off
11:06:19 sean-k-mooney ack
12:20:41 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
12:20:41 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
13:15:49 opendevreview suiong ng proposed openstack/nova master: Fix parameter order in add_instance_info_to_node https://review.opendev.org/c/openstack/nova/+/939411
13:53:54 opendevreview sean mooney proposed openstack/nova master: allow discover host to be enabled in multiple schedulers https://review.opendev.org/c/openstack/nova/+/938523
13:54:31 sean-k-mooney gibi: i just dropped the job change form ^
13:55:48 sean-k-mooney if you have time to look at https://review.opendev.org/c/openstack/nova/+/939476 as well it would be nice to land that
13:59:20 sean-k-mooney ...
13:59:27 sean-k-mooney mikal: "libvirt.libvirtError: unsupported configuration: spice graphics are not supported with this QEMU"
13:59:58 sean-k-mooney it looks like ubuntu 24.04 didnt just remove qxl they fully removed spice supprot like rhel9
14:00:32 sean-k-mooney i think, debian still supprots it
14:01:21 sean-k-mooney so i could move that job to debian but if centos/rhel 9 and ubuntu 24.04 have fully drop spice that proably our last option to test it
14:01:58 sean-k-mooney kolla uses debian continer on all distos now so i woudl guess that is why its workign for you in kolla-ansible
14:02:24 sean-k-mooney that our your still using ubuntu 22.04?
15:04:13 ykarel Hi is the issue a known one where test fails randomly with libvirt.libvirtError: Failed to terminate process 70698 with SIGKILL: Device or resource busy ?
15:04:46 ykarel like seen in
15:04:46 ykarel https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
15:04:51 ykarel https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
15:06:23 ykarel found a quite old related patch https://review.opendev.org/c/openstack/nova/+/639091 but that was targetting a libvirt version and it got removed from nova now as that old versions no longer supported
15:35:42 sean-k-mooney ykarel: rerely teh qemu monitor process that is responisble for respondind to the howdown of the guest can become unresponsive
15:36:38 sean-k-mooney but htis is not realy a nova issue its more of a libvirt/qemu one
15:36:55 sean-k-mooney nova should not need to actuly retry the destroy at all
15:38:52 ykarel sean-k-mooney, ohk so it's a known issue? what is the mitigation for this then?
15:39:05 ykarel the operator/user have to retry the operation?
15:39:10 sean-k-mooney not really
15:39:32 sean-k-mooney its a very rare failure mode in qemu to exit gracefully when ask
15:39:51 sean-k-mooney libvirt tried to fix this by sendign sigkill instead of just sigterm
15:39:59 sean-k-mooney so the os would evenutally reap the process
15:40:45 sean-k-mooney if sigkill does not work there nothing nova can do to force it to exit really
15:42:00 ykarel ohkk got it, but why nova cannot do retry like done in above patch if response is ebusy ?
15:43:51 sean-k-mooney libvirt is ment to do it internally
15:45:31 ykarel ohkk but as per logs thats not helping as there are failures in CI
15:46:05 ykarel ohh or you mean libvirt should do it but it's not doing it and libvirt have to be fixed for that?
15:46:27 sean-k-mooney its either not doing it or it is doing it and qemu is still not exiting
15:47:52 ykarel ohkk no idea then where the issue is, i will start with nova bug and other points could be checked as part of it
15:48:01 ykarel thx sean-k-mooney for sharing all the insights
15:48:13 sean-k-mooney i dont think you shoudl start with a nova bug
15:48:43 sean-k-mooney i also think we sthil have some rety logic because _destory does call itesl https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:48:49 ykarel nova people may know more about libvirt/qemu bits so can judge more then me so i thought to start with it :)
15:51:11 sean-k-mooney i would not personaly consider it a nova bug but other might disagree
15:52:45 sean-k-mooney the orgianl issue was not a nova bug it was a workaround for a libvirt/qemu one
15:55:03 sean-k-mooney ykarel: as far as i can see we actuly retry infinetly https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1603
15:56:56 sean-k-mooney im not a huge fan of who this is written but in destroy we are using loopingcall form oslo.service to execute _wait_for_destroy which checks if the instnace is stil runing. if it sets kwargs['is_running'] = True
15:57:26 sean-k-mooney which asues use to recursivly call destroy https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:58:37 sean-k-mooney that loop will break however if we get an excption form libvirt
16:02:02 sean-k-mooney so if we get EBUSY or other libvirt exction in _wait_for_destroy it wont retry
16:02:15 ykarel but that is_running = true seems not reached as i don't see it in logs
16:02:19 ykarel it's raising at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1566
16:02:46 ykarel i.e not seeing in logs "Going to destroy instance again"
16:02:57 sean-k-mooney that means the libvirt could not destory the instnace
16:03:41 ykarel but you said it will retry indefinitely
16:03:48 sean-k-mooney yes if there is no error
16:04:00 sean-k-mooney so if we ask libvirt to destory the instance and there is no error
16:04:14 sean-k-mooney but it does not complete in half a secodn we ask it to do it again
16:04:36 sean-k-mooney so we retry only when tehre isnt an error
16:05:59 sean-k-mooney if you retired the delete or hard reboot ectra it may work if libvirt has correct its internal error or qemu finally died
16:07:16 sean-k-mooney if qemu is locked up badly enough to service a sig kill form libvirt its possibel the only way to kill it woudl be a host reboot. trying it again might work but there is no garentee
16:07:55 sean-k-mooney we coudl readd a limite retry in error but it woudl really just eb papering over a potital libvirt or qemu bug
16:08:37 sean-k-mooney we have done that in the past but its kind of tech debt every time we do which is why we removed this
16:09:18 ykarel removal seems was done as part of cleanup of old libvirt version, but okk got everything what you mean
16:10:12 ykarel but it's good if it could be worked around atleast for the cases where it's not a persistent issue on qemu side
16:10:38 sean-k-mooney if you want to file it as a bug do, if we readd this i think we woudl put it behind a workaround flag
16:10:59 ykarel ok that should also work
16:11:12 ykarel thx again
16:11:14 sean-k-mooney it would also look at the failing job
16:11:35 sean-k-mooney this happens because of very high load
16:11:58 sean-k-mooney so the job may be usign too high a concurancy or have other isseus that is triggering it
16:12:25 ykarel hmm underlying hypervisor where it ran might also be overloaded
16:13:44 sean-k-mooney i belive libvirt is now going to wait at least 40s instead of 15 before retuing EBUSY
16:22:38 ykarel yes as per logs i see 40s b/w requesting destroy and returned error
17:50:04 opendevreview Doug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during live migration https://review.opendev.org/c/openstack/nova/+/940900
17:53:52 dougszu Any opinions on that would be great ^. I've got another fix for cold migration using block pull to share if it's of interest. And of course, the final bit to close the bug would be not making file systems on the ephemerals at all.
19:04:10 sean-k-mooney dougszu: i see you tried removing the backing file
19:04:36 sean-k-mooney longterm definlty the direction we shoudl go
19:05:08 sean-k-mooney we likely would need to cahnge intal boot/reboot and cold migration or evacuate too
19:05:12 sean-k-mooney if we were to do this.
19:06:02 sean-k-mooney i.e procive a way for nova, on the next lifeccyle operation on the guest to move or redifien the domain, to remove the backing file for ephmeral and swap disk
19:07:23 mikal sean-k-mooney: yeah, I am using Debian containers on Kolla-Ansible, so that's why I didn't notice the Ubuntu behaviour.
19:07:49 sean-k-mooney i need to push my pathc to try the job with debian
19:08:05 sean-k-mooney i have been gettign distracted for like the last 3 hours
19:08:21 mikal Heh, its ok, I didn't manage to get to tempest for similar reasons yesterday.
19:09:03 mikal I find people dropping SPICE quite frustrating to be entirely honest. The public excuse seems to be H.264 requirements, but RDP has the same requirements and there is a patent license from Cisco for the codec.
19:09:28 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
19:09:28 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
19:09:52 sean-k-mooney mikal: spice is the better console. vnc is catching up but spice is still better

Earlier   Later