| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-02-05 | |||
| 22:10:31 | mikal | sean-k-mooney: thanks, I'll take a look soon. | |
| #openstack-nova - 2025-02-06 | |||
| 04:49:04 | chungwon | Hello, can you review this? | |
| 04:49:57 | chungwon | review.opendev.org/c/openstack/nova/+/939929 | |
| 05:21:32 | opendevreview | suiong ng proposed openstack/nova stable/2024.1: ironic: Fix ConflictException when deleting server https://review.opendev.org/c/openstack/nova/+/940846 | |
| 06:42:46 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 06:42:47 | opendevreview | Michael Still proposed openstack/nova master: Protect older compute managers from sound model requests. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 06:42:47 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra specs for USB redirection. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 06:42:48 | opendevreview | Michael Still proposed openstack/nova master: Don't calculate the minimum compute version repeatedly. https://review.opendev.org/c/openstack/nova/+/940848 | |
| 06:46:08 | opendevreview | Michael Still proposed openstack/nova master: libvirt: direct SPICE console object changes https://review.opendev.org/c/openstack/nova/+/926876 | |
| 06:46:09 | opendevreview | Michael Still proposed openstack/nova master: libvirt: direct SPICE console database changes https://review.opendev.org/c/openstack/nova/+/926877 | |
| 06:46:09 | opendevreview | Michael Still proposed openstack/nova master: libvirt: allow direct SPICE connections to qemu https://review.opendev.org/c/openstack/nova/+/924844 | |
| 06:59:09 | mikal | I have updated the tracking etherpad at https://etherpad.opendev.org/p/nova-2025.1-status with a brain dump of the current state of the various SPICE VDI patches. | |
| 08:25:37 | opendevreview | benlei proposed openstack/nova master: Abort live migration task when stop nova compute service https://review.opendev.org/c/openstack/nova/+/938223 | |
| 08:50:56 | artem_vasilyev | Hey, could you review a small linter and test fixes for macOS support, please https://review.opendev.org/c/openstack/nova/+/937727 | |
| 10:20:15 | bauzas | dansmith: Uggla: thanks for the reviews on https://review.opendev.org/c/openstack/nova/+/940642 | |
| 10:27:26 | Uggla | bauzas, you are welcome. | |
| 10:48:26 | sean-k-mooney | bauzas: i just left my comment too i dont partically like the approch you took | |
| 10:48:43 | sean-k-mooney | it seems complex for what your trying to do | |
| 10:57:21 | bauzas | sean-k-mooney: thanks, I'll look over them after lunch | |
| 10:58:07 | bauzas | sean-k-mooney: your idea of getting all the props by a object property is actually a good one | |
| 10:59:43 | sean-k-mooney | i ocationally have them | |
| 11:00:21 | bauzas | for your algorithm with union/intersection, let's discuss this later, as I need to explain correctly why it wasn't working when I tested them | |
| 11:00:28 | sean-k-mooney | i check ovo and it does not have a funciton for that but they do it often internally. i woudl consider adding set_filed ot novas base object class | |
| 11:01:14 | sean-k-mooney | sure. form my perspecive the cardinatly shoudl not matter | |
| 11:01:47 | sean-k-mooney | i.e. if 2 isntance request the image property on one host adn one instnace request it on anohter shoudl not affect the weight | |
| 11:02:00 | sean-k-mooney | i think you are trying to also include that by using counter right? | |
| 11:02:58 | sean-k-mooney | we can chat later but i was intentionly trying not to have the number of vms on a host affect teh weighing of the host | |
| 11:03:21 | bauzas | I tested it, and the problem is that for example, if you have a host that has already 2 instances and one other that only has 1 using the prop, then it would prefer the second | |
| 11:04:08 | bauzas | so I preferred to just return the number of how many asked properties that are alrady used in the host | |
| 11:04:20 | bauzas | this is simplier to calculate and understand | |
| 11:04:45 | sean-k-mooney | i think that is more complex | |
| 11:04:54 | bauzas | also, I saw with devstack that when you create an instance, you get more props that the one you asked | |
| 11:04:56 | sean-k-mooney | and harder ot understand | |
| 11:05:14 | sean-k-mooney | you do yes we set some like the machine type on boot | |
| 11:05:19 | sean-k-mooney | or config drive | |
| 11:05:36 | bauzas | indeed, but you haven't asked for it, that's my point | |
| 11:05:39 | sean-k-mooney | based on the host config. we will be adding the vtpm secrete type | |
| 11:05:53 | sean-k-mooney | you have not but we shoudl not exclude them | |
| 11:06:15 | bauzas | anyway, I need to go off | |
| 11:06:19 | sean-k-mooney | ack | |
| 12:20:41 | opendevreview | sean mooney proposed openstack/nova master: [WIP] move nova-ovs-hybrid-plug to deploy with spice and fix qxl default https://review.opendev.org/c/openstack/nova/+/940835 | |
| 12:20:41 | opendevreview | sean mooney proposed openstack/nova master: Dont deploy n-spice on compute nodes. https://review.opendev.org/c/openstack/nova/+/940873 | |
| 13:15:49 | opendevreview | suiong ng proposed openstack/nova master: Fix parameter order in add_instance_info_to_node https://review.opendev.org/c/openstack/nova/+/939411 | |
| 13:53:54 | opendevreview | sean mooney proposed openstack/nova master: allow discover host to be enabled in multiple schedulers https://review.opendev.org/c/openstack/nova/+/938523 | |
| 13:54:31 | sean-k-mooney | gibi: i just dropped the job change form ^ | |
| 13:55:48 | sean-k-mooney | if you have time to look at https://review.opendev.org/c/openstack/nova/+/939476 as well it would be nice to land that | |
| 13:59:20 | sean-k-mooney | ... | |
| 13:59:27 | sean-k-mooney | mikal: "libvirt.libvirtError: unsupported configuration: spice graphics are not supported with this QEMU" | |
| 13:59:58 | sean-k-mooney | it looks like ubuntu 24.04 didnt just remove qxl they fully removed spice supprot like rhel9 | |
| 14:00:32 | sean-k-mooney | i think, debian still supprots it | |
| 14:01:21 | sean-k-mooney | so i could move that job to debian but if centos/rhel 9 and ubuntu 24.04 have fully drop spice that proably our last option to test it | |
| 14:01:58 | sean-k-mooney | kolla uses debian continer on all distos now so i woudl guess that is why its workign for you in kolla-ansible | |
| 14:02:24 | sean-k-mooney | that our your still using ubuntu 22.04? | |
| 15:04:13 | ykarel | Hi is the issue a known one where test fails randomly with libvirt.libvirtError: Failed to terminate process 70698 with SIGKILL: Device or resource busy ? | |
| 15:04:46 | ykarel | like seen in | |
| 15:04:46 | ykarel | https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html | |
| 15:04:51 | ykarel | https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html | |
| 15:06:23 | ykarel | found a quite old related patch https://review.opendev.org/c/openstack/nova/+/639091 but that was targetting a libvirt version and it got removed from nova now as that old versions no longer supported | |
| 15:35:42 | sean-k-mooney | ykarel: rerely teh qemu monitor process that is responisble for respondind to the howdown of the guest can become unresponsive | |
| 15:36:38 | sean-k-mooney | but htis is not realy a nova issue its more of a libvirt/qemu one | |
| 15:36:55 | sean-k-mooney | nova should not need to actuly retry the destroy at all | |
| 15:38:52 | ykarel | sean-k-mooney, ohk so it's a known issue? what is the mitigation for this then? | |
| 15:39:05 | ykarel | the operator/user have to retry the operation? | |
| 15:39:10 | sean-k-mooney | not really | |
| 15:39:32 | sean-k-mooney | its a very rare failure mode in qemu to exit gracefully when ask | |
| 15:39:51 | sean-k-mooney | libvirt tried to fix this by sendign sigkill instead of just sigterm | |
| 15:39:59 | sean-k-mooney | so the os would evenutally reap the process | |
| 15:40:45 | sean-k-mooney | if sigkill does not work there nothing nova can do to force it to exit really | |
| 15:42:00 | ykarel | ohkk got it, but why nova cannot do retry like done in above patch if response is ebusy ? | |
| 15:43:51 | sean-k-mooney | libvirt is ment to do it internally | |
| 15:45:31 | ykarel | ohkk but as per logs thats not helping as there are failures in CI | |
| 15:46:05 | ykarel | ohh or you mean libvirt should do it but it's not doing it and libvirt have to be fixed for that? | |
| 15:46:27 | sean-k-mooney | its either not doing it or it is doing it and qemu is still not exiting | |
| 15:47:52 | ykarel | ohkk no idea then where the issue is, i will start with nova bug and other points could be checked as part of it | |
| 15:48:01 | ykarel | thx sean-k-mooney for sharing all the insights | |
| 15:48:13 | sean-k-mooney | i dont think you shoudl start with a nova bug | |
| 15:48:43 | sean-k-mooney | i also think we sthil have some rety logic because _destory does call itesl https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612 | |
| 15:48:49 | ykarel | nova people may know more about libvirt/qemu bits so can judge more then me so i thought to start with it :) | |
| 15:51:11 | sean-k-mooney | i would not personaly consider it a nova bug but other might disagree | |
| 15:52:45 | sean-k-mooney | the orgianl issue was not a nova bug it was a workaround for a libvirt/qemu one | |
| 15:55:03 | sean-k-mooney | ykarel: as far as i can see we actuly retry infinetly https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1603 | |
| 15:56:56 | sean-k-mooney | im not a huge fan of who this is written but in destroy we are using loopingcall form oslo.service to execute _wait_for_destroy which checks if the instnace is stil runing. if it sets kwargs['is_running'] = True | |
| 15:57:26 | sean-k-mooney | which asues use to recursivly call destroy https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1612 | |
| 15:58:37 | sean-k-mooney | that loop will break however if we get an excption form libvirt | |
| 16:02:02 | sean-k-mooney | so if we get EBUSY or other libvirt exction in _wait_for_destroy it wont retry | |
| 16:02:15 | ykarel | but that is_running = true seems not reached as i don't see it in logs | |
| 16:02:19 | ykarel | it's raising at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1566 | |
| 16:02:46 | ykarel | i.e not seeing in logs "Going to destroy instance again" | |
| 16:02:57 | sean-k-mooney | that means the libvirt could not destory the instnace | |
| 16:03:41 | ykarel | but you said it will retry indefinitely | |
| 16:03:48 | sean-k-mooney | yes if there is no error | |
| 16:04:00 | sean-k-mooney | so if we ask libvirt to destory the instance and there is no error | |
| 16:04:14 | sean-k-mooney | but it does not complete in half a secodn we ask it to do it again | |
| 16:04:36 | sean-k-mooney | so we retry only when tehre isnt an error | |
| 16:05:59 | sean-k-mooney | if you retired the delete or hard reboot ectra it may work if libvirt has correct its internal error or qemu finally died | |
| 16:07:16 | sean-k-mooney | if qemu is locked up badly enough to service a sig kill form libvirt its possibel the only way to kill it woudl be a host reboot. trying it again might work but there is no garentee | |
| 16:07:55 | sean-k-mooney | we coudl readd a limite retry in error but it woudl really just eb papering over a potital libvirt or qemu bug | |
| 16:08:37 | sean-k-mooney | we have done that in the past but its kind of tech debt every time we do which is why we removed this | |
| 16:09:18 | ykarel | removal seems was done as part of cleanup of old libvirt version, but okk got everything what you mean | |
| 16:10:12 | ykarel | but it's good if it could be worked around atleast for the cases where it's not a persistent issue on qemu side | |