Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-02-06
06:46:09 opendevreview Michael Still proposed openstack/nova master: libvirt: direct SPICE console database changes https://review.opendev.org/c/openstack/nova/+/926877
06:59:09 mikal I have updated the tracking etherpad at https://etherpad.opendev.org/p/nova-2025.1-status with a brain dump of the current state of the various SPICE VDI patches.
08:25:37 opendevreview benlei proposed openstack/nova master: Abort live migration task when stop nova compute service https://review.opendev.org/c/openstack/nova/+/938223
08:50:56 artem_vasilyev Hey, could you review a small linter and test fixes for macOS support, please https://review.opendev.org/c/openstack/nova/+/937727
10:20:15 bauzas dansmith: Uggla: thanks for the reviews on https://review.opendev.org/c/openstack/nova/+/940642
10:27:26 Uggla bauzas, you are welcome.
10:48:26 sean-k-mooney bauzas: i just left my comment too i dont partically like the approch you took
10:48:43 sean-k-mooney it seems complex for what your trying to do
10:57:21 bauzas sean-k-mooney: thanks, I'll look over them after lunch
10:58:07 bauzas sean-k-mooney: your idea of getting all the props by a object property is actually a good one
10:59:43 sean-k-mooney i ocationally have them
11:00:21 bauzas for your algorithm with union/intersection, let's discuss this later, as I need to explain correctly why it wasn't working when I tested them
11:00:28 sean-k-mooney i check ovo and it does not have a funciton for that but they do it often internally. i woudl consider adding set_filed ot novas base object class
11:01:14 sean-k-mooney sure. form my perspecive the cardinatly shoudl not matter
11:01:47 sean-k-mooney i.e. if 2 isntance request the image property on one host adn one instnace request it on anohter shoudl not affect the weight
11:02:00 sean-k-mooney i think you are trying to also include that by using counter right?
11:02:58 sean-k-mooney we can chat later but i was intentionly trying not to have the number of vms on a host affect teh weighing of the host
11:03:21 bauzas I tested it, and the problem is that for example, if you have a host that has already 2 instances and one other that only has 1 using the prop, then it would prefer the second
11:04:08 bauzas so I preferred to just return the number of how many asked properties that are alrady used in the host
11:04:20 bauzas this is simplier to calculate and understand
11:04:45 sean-k-mooney i think that is more complex
11:04:54 bauzas also, I saw with devstack that when you create an instance, you get more props that the one you asked
11:04:56 sean-k-mooney and harder ot understand
11:05:14 sean-k-mooney you do yes we set some like the machine type on boot
11:05:19 sean-k-mooney or config drive
11:05:36 bauzas indeed, but you haven't asked for it, that's my point
11:05:39 sean-k-mooney based on the host config. we will be adding the vtpm secrete type
11:05:53 sean-k-mooney you have not but we shoudl not exclude them
11:06:15 bauzas anyway, I need to go off
11:06:19 sean-k-mooney ack
12:20:41 opendevreview sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873
12:20:41 opendevreview sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835
13:15:49 opendevreview suiong ng proposed openstack/nova master: Fix parameter order in add_instance_info_to_node https://review.opendev.org/c/openstack/nova/+/939411
13:53:54 opendevreview sean mooney proposed openstack/nova master: allow discover host to be enabled in multiple schedulers https://review.opendev.org/c/openstack/nova/+/938523
13:54:31 sean-k-mooney gibi: i just dropped the job change form ^
13:55:48 sean-k-mooney if you have time to look at https://review.opendev.org/c/openstack/nova/+/939476 as well it would be nice to land that
13:59:20 sean-k-mooney ...
13:59:27 sean-k-mooney mikal: "libvirt.libvirtError: unsupported configuration: spice graphics are not supported with this QEMU"
13:59:58 sean-k-mooney it looks like ubuntu 24.04 didnt just remove qxl they fully removed spice supprot like rhel9
14:00:32 sean-k-mooney i think, debian still supprots it
14:01:21 sean-k-mooney so i could move that job to debian but if centos/rhel 9 and ubuntu 24.04 have fully drop spice that proably our last option to test it
14:01:58 sean-k-mooney kolla uses debian continer on all distos now so i woudl guess that is why its workign for you in kolla-ansible
14:02:24 sean-k-mooney that our your still using ubuntu 22.04?
15:04:13 ykarel Hi is the issue a known one where test fails randomly with libvirt.libvirtError: Failed to terminate process 70698 with SIGKILL: Device or resource busy ?
15:04:46 ykarel https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
15:04:46 ykarel like seen in
15:04:51 ykarel https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
15:06:23 ykarel found a quite old related patch https://review.opendev.org/c/openstack/nova/+/639091 but that was targetting a libvirt version and it got removed from nova now as that old versions no longer supported
15:35:42 sean-k-mooney ykarel: rerely teh qemu monitor process that is responisble for respondind to the howdown of the guest can become unresponsive
15:36:38 sean-k-mooney but htis is not realy a nova issue its more of a libvirt/qemu one
15:36:55 sean-k-mooney nova should not need to actuly retry the destroy at all
15:38:52 ykarel sean-k-mooney, ohk so it's a known issue? what is the mitigation for this then?
15:39:05 ykarel the operator/user have to retry the operation?
15:39:10 sean-k-mooney not really
15:39:32 sean-k-mooney its a very rare failure mode in qemu to exit gracefully when ask
15:39:51 sean-k-mooney libvirt tried to fix this by sendign sigkill instead of just sigterm
15:39:59 sean-k-mooney so the os would evenutally reap the process
15:40:45 sean-k-mooney if sigkill does not work there nothing nova can do to force it to exit really
15:42:00 ykarel ohkk got it, but why nova cannot do retry like done in above patch if response is ebusy ?
15:43:51 sean-k-mooney libvirt is ment to do it internally
15:45:31 ykarel ohkk but as per logs thats not helping as there are failures in CI
15:46:05 ykarel ohh or you mean libvirt should do it but it's not doing it and libvirt have to be fixed for that?
15:46:27 sean-k-mooney its either not doing it or it is doing it and qemu is still not exiting
15:47:52 ykarel ohkk no idea then where the issue is, i will start with nova bug and other points could be checked as part of it
15:48:01 ykarel thx sean-k-mooney for sharing all the insights
15:48:13 sean-k-mooney i dont think you shoudl start with a nova bug
15:48:43 sean-k-mooney i also think we sthil have some rety logic because _destory does call itesl https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:48:49 ykarel nova people may know more about libvirt/qemu bits so can judge more then me so i thought to start with it :)
15:51:11 sean-k-mooney i would not personaly consider it a nova bug but other might disagree
15:52:45 sean-k-mooney the orgianl issue was not a nova bug it was a workaround for a libvirt/qemu one
15:55:03 sean-k-mooney ykarel: as far as i can see we actuly retry infinetly https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1603
15:56:56 sean-k-mooney im not a huge fan of who this is written but in destroy we are using loopingcall form oslo.service to execute _wait_for_destroy which checks if the instnace is stil runing. if it sets kwargs['is_running'] = True
15:57:26 sean-k-mooney which asues use to recursivly call destroy https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612
15:58:37 sean-k-mooney that loop will break however if we get an excption form libvirt
16:02:02 sean-k-mooney so if we get EBUSY or other libvirt exction in _wait_for_destroy it wont retry
16:02:15 ykarel but that is_running = true seems not reached as i don't see it in logs
16:02:19 ykarel it's raising at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1566
16:02:46 ykarel i.e not seeing in logs "Going to destroy instance again"
16:02:57 sean-k-mooney that means the libvirt could not destory the instnace
16:03:41 ykarel but you said it will retry indefinitely
16:03:48 sean-k-mooney yes if there is no error
16:04:00 sean-k-mooney so if we ask libvirt to destory the instance and there is no error
16:04:14 sean-k-mooney but it does not complete in half a secodn we ask it to do it again
16:04:36 sean-k-mooney so we retry only when tehre isnt an error
16:05:59 sean-k-mooney if you retired the delete or hard reboot ectra it may work if libvirt has correct its internal error or qemu finally died
16:07:16 sean-k-mooney if qemu is locked up badly enough to service a sig kill form libvirt its possibel the only way to kill it woudl be a host reboot. trying it again might work but there is no garentee
16:07:55 sean-k-mooney we coudl readd a limite retry in error but it woudl really just eb papering over a potital libvirt or qemu bug
16:08:37 sean-k-mooney we have done that in the past but its kind of tech debt every time we do which is why we removed this
16:09:18 ykarel removal seems was done as part of cleanup of old libvirt version, but okk got everything what you mean
16:10:12 ykarel but it's good if it could be worked around atleast for the cases where it's not a persistent issue on qemu side
16:10:38 sean-k-mooney if you want to file it as a bug do, if we readd this i think we woudl put it behind a workaround flag
16:10:59 ykarel ok that should also work
16:11:12 ykarel thx again
16:11:14 sean-k-mooney it would also look at the failing job
16:11:35 sean-k-mooney this happens because of very high load
16:11:58 sean-k-mooney so the job may be usign too high a concurancy or have other isseus that is triggering it
16:12:25 ykarel hmm underlying hypervisor where it ran might also be overloaded
16:13:44 sean-k-mooney i belive libvirt is now going to wait at least 40s instead of 15 before retuing EBUSY
16:22:38 ykarel yes as per logs i see 40s b/w requesting destroy and returned error
17:50:04 opendevreview Doug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during live migration https://review.opendev.org/c/openstack/nova/+/940900

Earlier   Later