| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-03 | |||
| 18:39:39 | dansmith | so I'm thinking it would be good, since we have several +2s (some of which are stale) to just go ahead and merge up to the host patch to reduce the size of the series, claim some success and focus on the later bits | |
| 18:39:44 | dansmith | any problem with that? | |
| 18:39:51 | dansmith | I wish bauzas was here but I think he's already EOD | |
| 18:40:48 | melwitt | sounds cool to me | |
| 18:41:25 | dansmith | okay I'll make sure my +2s are good down the line from there and then we can pounce on him tomorrow | |
| 18:43:43 | melwitt | ++ | |
| #openstack-nova - 2025-11-04 | |||
| 06:47:01 | tonyb | review.o.o is down. When I check it was in shutoff state. That's the second time recently. One guess is that the hypervisor host's OOM killer is ... well killing the VM (it is a memory hog). | |
| 06:48:08 | tonyb | If that happened, and nova notices does nova take any action? Like maybe logging a fake/minimal server event | |
| 07:12:11 | tonyb | Maybe it looks like a "power-update" to nova.compute.api::external_instance_event and emits and event here: https://opendev.org/openstack/nova/src/commit/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/api.py#L5994 ? | |
| 08:00:59 | gibi | tonyb: nova logs the following from a periodic tasks if detects that the power state of the VM is different than the power state of the domain on the hypervisor https://github.com/openstack/nova/blob/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/manager.py#L11134 | |
| 08:01:11 | gibi | then aligns the power state in the DB | |
| 08:02:03 | tonyb | gibi: In this case i don't have access to the nova logs | |
| 08:02:49 | gibi | I assume after the OOM kill libvirt will report that the VM is stopped. If so then nova will use the normal VM stop api to stop the VM properly https://github.com/openstack/nova/blob/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/manager.py#L11190 | |
| 08:02:50 | tonyb | gibi: I do see a server event: https://paste.opendev.org/show/bvD4k4YfaLIc7qYkc3dc/ | |
| 08:04:17 | gibi | during the normal stop we record events to the DB and send notification on the message bus https://github.com/openstack/nova/blob/32f58e8ad6a7ff896cc6ae8a361e3a18f5b35c9a/nova/compute/manager.py#L3465-L3470 | |
| 08:04:49 | gibi | tonyb: yepp that server event seems like the one happening if nova detects that the VM is in stopped state on the hypervisor | |
| 08:04:54 | gibi | but active in the DB | |
| 08:04:58 | gibi | and aligns the DB | |
| 08:05:03 | tonyb | gibi: Thanks | |
| 08:06:04 | gibi | note that the same would happen if somebody logs into the VM and shutdowns it from within the guest | |
| 08:06:27 | gibi | nova does not really know why the VM is suddenly stopped on the hypervisor | |
| 08:06:34 | tonyb | Ah okay. I can check if that was it but given it's review.o.o I doubt it | |
| 08:07:36 | tonyb | Yup. I was just wanting to verify that the event "mostly agrees" with the theory the the OOM-killer fired | |
| 08:09:50 | gibi | OK | |
| 08:09:58 | gibi | sounds like it agrees | |
| 08:36:04 | tonyb | gibi: perfect thanks | |
| 09:26:06 | LarsErikP | Hi. I reported this, after talking to melwitt back in september. Could anyone take a look? https://bugs.launchpad.net/nova/+bug/2125730 | |
| 09:46:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 09:46:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 09:46:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Compute manager to use native thread pools https://review.opendev.org/c/openstack/nova/+/966016 | |
| 09:49:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 12:43:20 | sean-k-mooney | LarsErikP: i think dansmith fixed that but i dont knwo in which release | |
| 12:43:45 | sean-k-mooney | my guess is this si because nova in carical does not have the fix | |
| 12:44:53 | sean-k-mooney | caracal is now unmaintaiend so this bug is liekly not valid form a core team point of view but the fix might be backprotable | |
| 12:45:19 | sean-k-mooney | assuimg the cahge we merged for mast resolve the issue | |
| 12:46:07 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/930754 | |
| 12:46:31 | sean-k-mooney | that in epoxy but not in older release | |
| 13:16:30 | LarsErikP | sean-k-mooney: ah ok. that makes sense. thanks | |
| 13:22:11 | jlejeune | hello guys, can I have some reviews please for my topics: https://review.opendev.org/q/topic:%22bug/2085135%22 and https://review.opendev.org/q/topic:%22bug-2044235%22 ? | |
| 13:22:34 | jlejeune | just matter of backports | |
| 14:25:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 14:25:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 14:27:15 | dansmith | bauzas: in case you didn't see the mention, can you see my proposal here please? https://review.opendev.org/c/openstack/nova/+/941795/comments/fc49c07d_c06afef1 | |
| 14:51:21 | bauzas | dansmith: nope sorry, working hard on something else, f*** agile :) | |
| 14:51:42 | bauzas | dansmith: I definitely want to review again the vtpm series, will do it hopefully today | |
| 14:52:59 | dansmith | bauzas: alright, well, that link is to a specific proposal for for merging the bottom part now, which I think you've mostly reviewed already | |
| 14:54:53 | gibi | dansmith: thanks for the review on switching the default to native threading. Did you missed that it has two pre-req patch in the series? | |
| 14:55:32 | dansmith | gibi: oh, no I saw it, but I got distracted before I followed that down, sorry about that | |
| 14:55:42 | gibi | no worries. Take you time :) | |
| 14:57:16 | dansmith | I had skimmed them looking to see if the api/__init__ was being added in this series or was already in place | |
| 14:58:54 | gibi | it was in place. Actually I needed to improve and move from from cmd/__init__ to cmd/<entrypoint2025-11-03 14:29:01.144038 | ubuntu-noble | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading'> | |
| 14:59:00 | gibi | oops | |
| 14:59:08 | gibi | copy past buffer grrr | |
| 14:59:26 | gibi | it was in place. Actually I needed to improve and move from from cmd/__init__ to cmd/<entrypoint>.py | |
| 14:59:53 | gibi | so I can make the default different per entry point | |
| 15:00:04 | dansmith | you're planning to do that for api? | |
| 15:00:09 | gibi | but the wsgi entry point is differnt than the cmd one | |
| 15:00:37 | dansmith | right, but we _could_ have an entry point somewhere else for api/wsgi that does the patching before it imports... | |
| 15:00:46 | gibi | as both wsgi service we have are switching defaults at the same time I did not needed to touch the api part | |
| 15:01:31 | dansmith | yeah, I'm more concerned about just something else importing something from api to get a constant or something and screwing up | |
| 15:04:16 | gibi | I cannot just delay monkey_patch call in api wsgi as other imports might start using primitives that should have been monkey patched | |
| 15:05:26 | gibi | if another service imports from api then we assume that the another service already went through monkey patching. And oslo.service does not allow changing the backend type once it set so we will catch it | |
| 15:05:49 | dansmith | you can if you control the order of the importing in the entry point | |
| 15:05:51 | gibi | when the second monkey patch call tries to change the backend due to differning default | |
| 15:06:30 | gibi | dansmith: that is what we do, we added the monkey patch as early as possible in the import list | |
| 15:06:37 | gibi | that is our control | |
| 15:07:44 | dansmith | I'm not sure if you're not understanding what I'm saying, or don't think it's possible, or think it's a bad idea.. either way, I'm not arguing...just commenting about my opinion | |
| 15:07:52 | gibi | import wsgi from api first execute __init__ so we need to be in __init__ if __init__ has other imports like oslo.log that creates a lock | |
| 15:08:22 | gibi | I guess I'm missing what would you like to see in case of api/__init__ | |
| 15:08:44 | gibi | s/import wsgi from api/from api import wsgi/ | |
| 15:10:05 | gibi | I think it is not possible to move the monkey_patch to api/wsgi from api/__init__ | |
| 15:10:32 | gibi | as api/__init__ has other imports | |
| 15:11:56 | sean-k-mooney | you could monkey patch before the other import | |
| 15:12:08 | dansmith | this ^ | |
| 15:12:08 | sean-k-mooney | you might need to add a noqa or similar comment | |
| 15:12:11 | dansmith | but I don't want to argue | |
| 15:12:17 | sean-k-mooney | but we can if we need too | |
| 15:12:35 | gibi | but I monkey patch before all the imports | |
| 15:12:36 | gibi | :) | |
| 15:13:05 | sean-k-mooney | i have not been folloing the converstaion by they way | |
| 15:13:21 | gibi | the first line of nova/api/openstack/__init__.py is the monkey patch | |
| 15:13:47 | gibi | I think dansmith would like that to be moved somewhere else | |
| 15:13:51 | sean-k-mooney | ah yes i remoember dicussign the #noqa comment for that | |
| 15:14:38 | gibi | at the moment I'm not sure where to move our nova.api.openstack.compute.wsgi:init_application to be able to move the monkey_patch | |
| 15:14:41 | sean-k-mooney | so ya https://github.com/openstack/nova/blob/master/nova/api/__init__.py and https://github.com/openstack/nova/blob/master/nova/__init__.py are empty today | |
| 15:15:24 | sean-k-mooney | well the entry point ot that is https://github.com/openstack/nova/blob/master/nova/wsgi/osapi_compute.py | |
| 15:15:55 | sean-k-mooney | across openstack we are ment ot have <package>.wsgi.<api endpoint) | |
| 15:16:23 | sean-k-mooney | so you could monkey patch here https://github.com/openstack/nova/blob/master/nova/wsgi/__init__.py but im not sure it is required | |
| 15:16:25 | gibi | hm but our setup.cfg ponts to nova-api-wsgi = nova.api.openstack.compute.wsgi:init_application | |
| 15:16:49 | sean-k-mooney | that in wsgi script | |
| 15:16:52 | sean-k-mooney | but that is being deleted | |
| 15:17:05 | sean-k-mooney | it will generate the same script effectivly | |
| 15:17:42 | sean-k-mooney | we agreed at the ptg to finally delete https://github.com/openstack/nova/blob/master/setup.cfg#L87-L89 | |
| 15:17:51 | gibi | but until that setup.cfg is a thing I need to make sure that works. So today I cannot move the monkey patch to nova/wsgi because that will not be executed if the setup.cfg defined entry point is used | |
| 15:18:35 | gibi | I agree that once the setup.cfg does not have such entry point defined and only the nova/wsgi is used then I can move the monkey_patch | |
| 15:18:37 | sean-k-mooney | right but lest just find and merge stephesn patch | |
| 15:18:44 | sean-k-mooney | then you can move it if needed | |
| 15:18:50 | gibi | OK | |
| 15:19:14 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/902688 | |
| 15:19:40 | sean-k-mooney | he updated the docs too | |