| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-03 | |||
| 16:33:32 | sean-k-mooney | i think that is a no | |
| 16:33:40 | gibi | ∅ | |
| 16:33:46 | sean-k-mooney | do we want to do some bug triage or give everyone back some time | |
| 16:34:19 | Uggla | I think we gonna keep this one short. Especially after last long week. | |
| 16:34:35 | sean-k-mooney | +1 | |
| 16:35:00 | Uggla | I will restart the rooster stuff for triage, along the week. it should be ready next week. | |
| 16:35:47 | fwiesel | Then, thanks everyone | |
| 16:35:49 | Uggla | anyway, time to close. Thanks for joining this meeting. Have a nice day/evening. | |
| 16:36:03 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-11-03-16.02.log.html | |
| 16:36:03 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-11-03-16.02.txt | |
| 16:36:03 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-11-03-16.02.html | |
| 16:36:03 | opendevmeet | Meeting ended Mon Nov 3 16:36:03 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:36:03 | Uggla | #endmeeting | |
| 16:36:06 | gibi | thakmks | |
| 16:36:11 | gibi | thanks | |
| 16:51:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 18:36:49 | dansmith | melwitt: so... I redeployed the vTPM series at the host patch this morning | |
| 18:37:39 | dansmith | and I'm able to create an instance in host mode as a user, stop it, and then restart it as the admin | |
| 18:37:59 | dansmith | which shows that we've actually improved things before we even get to the deployment mode and migration part of the series | |
| 18:38:02 | dansmith | that's expected right? | |
| 18:38:22 | melwitt | dansmith: yes that is expected | |
| 18:38:43 | melwitt | it was a small change in the 'host' support patch to make restart as admin work | |
| 18:38:50 | dansmith | confirming because I'm not really testing the tpm part, but the instance does have a tpm and it's not complaining when the admin does it | |
| 18:38:55 | dansmith | yeah, cool | |
| 18:39:39 | dansmith | so I'm thinking it would be good, since we have several +2s (some of which are stale) to just go ahead and merge up to the host patch to reduce the size of the series, claim some success and focus on the later bits | |
| 18:39:44 | dansmith | any problem with that? | |
| 18:39:51 | dansmith | I wish bauzas was here but I think he's already EOD | |
| 18:40:48 | melwitt | sounds cool to me | |
| 18:41:25 | dansmith | okay I'll make sure my +2s are good down the line from there and then we can pounce on him tomorrow | |
| 18:43:43 | melwitt | ++ | |
| #openstack-nova - 2025-11-04 | |||
| 06:47:01 | tonyb | review.o.o is down. When I check it was in shutoff state. That's the second time recently. One guess is that the hypervisor host's OOM killer is ... well killing the VM (it is a memory hog). | |
| 06:48:08 | tonyb | If that happened, and nova notices does nova take any action? Like maybe logging a fake/minimal server event | |
| 07:12:11 | tonyb | Maybe it looks like a "power-update" to nova.compute.api::external_instance_event and emits and event here: https://opendev.org/openstack/nova/src/commit/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/api.py#L5994 ? | |
| 08:00:59 | gibi | tonyb: nova logs the following from a periodic tasks if detects that the power state of the VM is different than the power state of the domain on the hypervisor https://github.com/openstack/nova/blob/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/manager.py#L11134 | |
| 08:01:11 | gibi | then aligns the power state in the DB | |
| 08:02:03 | tonyb | gibi: In this case i don't have access to the nova logs | |
| 08:02:49 | gibi | I assume after the OOM kill libvirt will report that the VM is stopped. If so then nova will use the normal VM stop api to stop the VM properly https://github.com/openstack/nova/blob/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/manager.py#L11190 | |
| 08:02:50 | tonyb | gibi: I do see a server event: https://paste.opendev.org/show/bvD4k4YfaLIc7qYkc3dc/ | |
| 08:04:17 | gibi | during the normal stop we record events to the DB and send notification on the message bus https://github.com/openstack/nova/blob/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/manager.py#L3465-L3470 | |
| 08:04:49 | gibi | tonyb: yepp that server event seems like the one happening if nova detects that the VM is in stopped state on the hypervisor | |
| 08:04:54 | gibi | but active in the DB | |
| 08:04:58 | gibi | and aligns the DB | |
| 08:05:03 | tonyb | gibi: Thanks | |
| 08:06:04 | gibi | note that the same would happen if somebody logs into the VM and shutdowns it from within the guest | |
| 08:06:27 | gibi | nova does not really know why the VM is suddenly stopped on the hypervisor | |
| 08:06:34 | tonyb | Ah okay. I can check if that was it but given it's review.o.o I doubt it | |
| 08:07:36 | tonyb | Yup. I was just wanting to verify that the event "mostly agrees" with the theory the the OOM-killer fired | |
| 08:09:50 | gibi | OK | |
| 08:09:58 | gibi | sounds like it agrees | |
| 08:36:04 | tonyb | gibi: perfect thanks | |
| 09:26:06 | LarsErikP | Hi. I reported this, after talking to melwitt back in september. Could anyone take a look? https://bugs.launchpad.net/nova/+bug/2125730 | |
| 09:46:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 09:46:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Compute manager to use native thread pools https://review.opendev.org/c/openstack/nova/+/966016 | |
| 09:46:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 09:49:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 12:43:20 | sean-k-mooney | LarsErikP: i think dansmith fixed that but i dont knwo in which release | |
| 12:43:45 | sean-k-mooney | my guess is this si because nova in carical does not have the fix | |
| 12:44:53 | sean-k-mooney | caracal is now unmaintaiend so this bug is liekly not valid form a core team point of view but the fix might be backprotable | |
| 12:45:19 | sean-k-mooney | assuimg the cahge we merged for mast resolve the issue | |
| 12:46:07 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/930754 | |
| 12:46:31 | sean-k-mooney | that in epoxy but not in older release | |
| 13:16:30 | LarsErikP | sean-k-mooney: ah ok. that makes sense. thanks | |
| 13:22:11 | jlejeune | hello guys, can I have some reviews please for my topics: https://review.opendev.org/q/topic:%22bug/2085135%22 and https://review.opendev.org/q/topic:%22bug-2044235%22 ? | |
| 13:22:34 | jlejeune | just matter of backports | |
| 14:25:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 14:25:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 14:27:15 | dansmith | bauzas: in case you didn't see the mention, can you see my proposal here please? https://review.opendev.org/c/openstack/nova/+/941795/comments/fc49c07d_c06afef1 | |
| 14:51:21 | bauzas | dansmith: nope sorry, working hard on something else, f*** agile :) | |
| 14:51:42 | bauzas | dansmith: I definitely want to review again the vtpm series, will do it hopefully today | |
| 14:52:59 | dansmith | bauzas: alright, well, that link is to a specific proposal for for merging the bottom part now, which I think you've mostly reviewed already | |
| 14:54:53 | gibi | dansmith: thanks for the review on switching the default to native threading. Did you missed that it has two pre-req patch in the series? | |
| 14:55:32 | dansmith | gibi: oh, no I saw it, but I got distracted before I followed that down, sorry about that | |
| 14:55:42 | gibi | no worries. Take you time :) | |
| 14:57:16 | dansmith | I had skimmed them looking to see if the api/__init__ was being added in this series or was already in place | |
| 14:58:54 | gibi | it was in place. Actually I needed to improve and move from from cmd/__init__ to cmd/<entrypoint2025-11-03 14:29:01.144038 | ubuntu-noble | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading'> | |
| 14:59:00 | gibi | oops | |
| 14:59:08 | gibi | copy past buffer grrr | |
| 14:59:26 | gibi | it was in place. Actually I needed to improve and move from from cmd/__init__ to cmd/<entrypoint>.py | |
| 14:59:53 | gibi | so I can make the default different per entry point | |
| 15:00:04 | dansmith | you're planning to do that for api? | |
| 15:00:09 | gibi | but the wsgi entry point is differnt than the cmd one | |
| 15:00:37 | dansmith | right, but we _could_ have an entry point somewhere else for api/wsgi that does the patching before it imports... | |
| 15:00:46 | gibi | as both wsgi service we have are switching defaults at the same time I did not needed to touch the api part | |
| 15:01:31 | dansmith | yeah, I'm more concerned about just something else importing something from api to get a constant or something and screwing up | |
| 15:04:16 | gibi | I cannot just delay monkey_patch call in api wsgi as other imports might start using primitives that should have been monkey patched | |
| 15:05:26 | gibi | if another service imports from api then we assume that the another service already went through monkey patching. And oslo.service does not allow changing the backend type once it set so we will catch it | |
| 15:05:49 | dansmith | you can if you control the order of the importing in the entry point | |
| 15:05:51 | gibi | when the second monkey patch call tries to change the backend due to differning default | |
| 15:06:30 | gibi | dansmith: that is what we do, we added the monkey patch as early as possible in the import list | |
| 15:06:37 | gibi | that is our control | |
| 15:07:44 | dansmith | I'm not sure if you're not understanding what I'm saying, or don't think it's possible, or think it's a bad idea.. either way, I'm not arguing...just commenting about my opinion | |
| 15:07:52 | gibi | import wsgi from api first execute __init__ so we need to be in __init__ if __init__ has other imports like oslo.log that creates a lock | |
| 15:08:22 | gibi | I guess I'm missing what would you like to see in case of api/__init__ | |
| 15:08:44 | gibi | s/import wsgi from api/from api import wsgi/ | |
| 15:10:05 | gibi | I think it is not possible to move the monkey_patch to api/wsgi from api/__init__ | |
| 15:10:32 | gibi | as api/__init__ has other imports | |
| 15:11:56 | sean-k-mooney | you could monkey patch before the other import | |
| 15:12:08 | sean-k-mooney | you might need to add a noqa or similar comment | |
| 15:12:08 | dansmith | this ^ | |
| 15:12:11 | dansmith | but I don't want to argue | |