| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-23 | |||
| 13:48:29 | stephenfin | sean-k-mooney: gmaan: Could you re-add your +2s to https://review.opendev.org/c/openstack/nova/+/956140/? | |
| 13:48:56 | stephenfin | sean-k-mooney: sorry https://review.opendev.org/c/openstack/nova/+/956141/ for you (need gmaan's +2 on the other one) | |
| 13:49:19 | sean-k-mooney | sure once im off this call ill take a look | |
| 13:49:32 | stephenfin | ta | |
| 13:52:01 | opendevreview | Merged openstack/nova stable/2024.2: Replace removed os-vif-ovs job https://review.opendev.org/c/openstack/nova/+/972665 | |
| 14:04:51 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 14:30:34 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 14:30:34 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 14:30:35 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 14:30:35 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 14:30:36 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 15:21:10 | opendevreview | Dominik proposed openstack/nova master: NUMA Topology with Resource Providers: Scheduler changes https://review.opendev.org/c/openstack/nova/+/971176 | |
| 15:21:10 | opendevreview | Dominik proposed openstack/nova master: NUMA Topology with Resource Providers: Libvirt NUMA Migrate https://review.opendev.org/c/openstack/nova/+/971177 | |
| 15:36:04 | sean-k-mooney | DominikDanelski[m]: so melwitt and i chated about this breifly a few weeks ago and we were not configed the repoduce actully repoduce things properly. or that the proposed fix didnt create ohter problems | |
| 15:37:10 | sean-k-mooney | but to be honestl i have lost all context on it since then | |
| 15:45:12 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 15:45:12 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 15:45:13 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 15:45:13 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 15:45:14 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 15:45:14 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 15:45:15 | opendevreview | Stephen Finucane proposed openstack/nova master: Disable flake8 rules https://review.opendev.org/c/openstack/nova/+/974442 | |
| 15:55:36 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 15:55:37 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 15:55:37 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 17:20:41 | gmaan | stephenfin: thanks for followup for 'network' field. I +w on quota ones now and re-approvd other one which were rebased | |
| 17:20:53 | stephenfin | ty! | |
| 18:52:50 | opendevreview | sean mooney proposed openstack/nova master: [WIP] Add PCI device groups for atomic allocation https://review.opendev.org/c/openstack/nova/+/973604 | |
| 19:26:57 | melwitt | dansmith, gmaan: thank you both for reviews :) but I regret to inform you there is a prereq refactor patch underneath it for the service_auth.py commonizing https://review.opendev.org/c/openstack/nova/+/974267 | |
| 19:32:04 | rm_work[m] | anyone offhand have any ideas why placement might take a long time to show a node within openstack allocation candidate list after it goes available in ironic? it's usually about 5 minutes in normal operation and sometimes we see it take up to 20 minutes :( | |
| 19:33:57 | rm_work[m] | is there some setting we can tweak? is this a periodic task that is running too slowly? | |
| 19:34:26 | sean-k-mooney | melwitt: +1 for now ill try and take a look properly on monday but it looks ok at first glance | |
| 19:34:52 | melwitt | thanks sean-k-mooney :) | |
| 19:35:05 | sean-k-mooney | rm_work[m]: is the RP not there or is it there but disabled? | |
| 19:35:54 | sean-k-mooney | rm_work[m]: my guess of the top of my head is that its related to the update_aviable_resouces periodic task tiem of 5 mins and the delay is just in the virt driver runnign that and noticing the new node | |
| 19:38:07 | rm_work[m] | ah ok so we could maybe tune that task down to 1m or something? | |
| 19:38:08 | gmaan | melwitt: yeah, I was reviewing that also, lgtm. thanks for refactoring. | |
| 19:38:23 | melwitt | thanks gmaan | |
| 19:39:26 | sean-k-mooney | rm_work[m]: perhaps this is the optioni mentioned https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.update_resources_interval | |
| 19:39:35 | rm_work[m] | i think the BAD case (the 20m I talked about) might be when we had other issues going on slowing things down, so maybe it's as simple as setting the update_aviable_resouces periodic to run faster | |
| 19:39:44 | sean-k-mooney | rm_work[m]: what exactly do you mean by aviable by the way | |
| 19:39:52 | sean-k-mooney | do you meain aviable in the ironic node list | |
| 19:40:02 | rm_work[m] | the ironic state yeah | |
| 19:40:52 | sean-k-mooney | ack i dont recall if there is a specifric addtioanl dely in the ironic case but maybe JayF woudl know | |
| 19:40:59 | rm_work[m] | he said he didn't :D | |
| 19:41:01 | JayF | I sent rm_work[m] to you :) | |
| 19:41:18 | sean-k-mooney | ack i was onderign if it could be relate the the lease stuff | |
| 19:41:34 | rm_work[m] | we'll try tweaking that periodic | |
| 19:41:35 | rm_work[m] | or, that config value | |
| 19:41:35 | JayF | I honestly have no idea where to even start from the Ironic side; we just fetch nodes and pass them in | |
| 19:41:44 | JayF | *Ironic driver side | |
| 19:41:51 | JayF | and I don't know much about the mechanisms on the other side of that API interface | |
| 19:42:07 | rm_work[m] | the default being "0" to "run at the default interval" does not tell me what the default interval is, lol | |
| 19:42:17 | rm_work[m] | why would the default not just BE the default interval 😅 | |
| 19:42:31 | sean-k-mooney | the default interval for perodics in oslo and as a result nova is 500 seconds | |
| 19:42:37 | sean-k-mooney | sory 300 | |
| 19:42:38 | rm_work[m] | ah ok | |
| 19:42:42 | sean-k-mooney | so 5 mins | |
| 19:42:46 | rm_work[m] | yeah | |
| 19:42:55 | rm_work[m] | alright, thanks we will poke at it | |
| 19:43:15 | JayF | rm_work[m]: please make sure if you figure out a set of good configs for your scale, please document it -- even if just in a mailing list half-blogpost | |
| 19:44:22 | sean-k-mooney | so oen thing to look at would be the logs in the relevent nova-comptue between the previous start of that perodic and then first one after the node appeard | |
| 19:44:42 | sean-k-mooney | nova-comnpute when its managing a lot fo ironic nodes can take a long time for that perodic to complete | |
| 19:45:02 | rm_work[m] | we have 240ish nodes so i don't think it's considered "a lot" | |
| 19:45:04 | sean-k-mooney | we shoudl eventually improve that but ironic shardign might help | |
| 19:45:20 | sean-k-mooney | 240 nodes manged by a singel nova compute agent? | |
| 19:45:22 | rm_work[m] | our clouds are relatively small ATM | |
| 19:46:08 | sean-k-mooney | or are you still using the deprecated peers parmaters with the hashring that loadblances them between multiple agents? | |
| 19:46:18 | rm_work[m] | just one agent | |
| 19:46:34 | rm_work[m] | also FWIW we're on 2024.2 | |
| 19:47:28 | sean-k-mooney | ya so 250 ironics on 1 nova compute is both a lot and not a lot. for perodics that loop over all the nodes they manage and update placement that can be slow since itd getign the infor one ironic node at a time | |
| 19:47:43 | sean-k-mooney | for actully triging the provisioning ectra its fine | |
| 19:47:54 | rm_work[m] | hmm | |
| 19:48:22 | rm_work[m] | but it sounds like you're saying we CAN'T have multiple? because whatever way they spread out is deprecated? or is that just one of multiple methods | |
| 19:48:42 | rm_work[m] | surprised it's sequential for the node data gathering | |
| 19:48:46 | sean-k-mooney | there is a repalcement know as ironic sharding | |
| 19:48:59 | sean-k-mooney | that allow you to map subset of ironic nodes to diffent nova compute agents | |
| 19:49:08 | sean-k-mooney | im not sure at what scale using that is recommended | |
| 19:49:23 | sean-k-mooney | rm_work[m]: i would try seeign how long the perodic takes to run | |
| 19:49:33 | sean-k-mooney | if its like 90 seconds your fine | |
| 19:49:42 | sean-k-mooney | if its 15 mins then that the issue | |
| 19:50:39 | rm_work[m] | hmm k | |
| 19:50:46 | rm_work[m] | will do some timing and log diving | |
| 19:51:13 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/2024.1/implemented/ironic-shards.html is the new way of scaling nova-compute with ironic | |
| 19:51:33 | sean-k-mooney | the peer_list is deprecated but still in master because we never got aroudn to deleting it | |
| 19:51:39 | sean-k-mooney | but we really should at some point | |
| 19:53:37 | sean-k-mooney | rm_work[m]: for what its worht it woudl be posible for us to optimize this in nova in the future by say delegting the node update to a thread pool on each perodic but we have not been inveting time in that space with the eventlet removal ectra that has been happening | |
| 19:58:08 | rm_work[m] | right yeah, no worries. thanks for taking the time to explain! | |
| 19:58:43 | rm_work[m] | yeah my thought would just be to parallelize the node checks in the job | |
| 19:58:50 | rm_work[m] | I wonder if that would actually be that complicated... | |
| 19:59:38 | sean-k-mooney | in the past yes with all the work we are doing to make the compute agent work with thread pool executors instead of eventlet i suspect its simpler then before | |
| 20:00:26 | sean-k-mooney | i would focus on confiming this is the issue first before going much further however | |
| 20:00:44 | sean-k-mooney | if we can confirm this as a problem it would be a good topic for a ptg session with the ironic folks | |
| 22:30:31 | opendevreview | Merged openstack/nova master: Refactor service user authentication https://review.opendev.org/c/openstack/nova/+/974267 | |
| 22:30:47 | opendevreview | Merged openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | |
| #openstack-nova - 2026-01-24 | |||
| 07:28:39 | opendevreview | Merged openstack/nova master: api: Add response body schemas for quota class sets API https://review.opendev.org/c/openstack/nova/+/956140 | |
| 08:29:08 | opendevreview | Merged openstack/nova master: api: Add response body schemas for quota sets API https://review.opendev.org/c/openstack/nova/+/956141 | |
| 08:29:22 | opendevreview | Merged openstack/nova master: api: Add response body schemas for migrations API https://review.opendev.org/c/openstack/nova/+/956142 | |
| 14:47:00 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 14:47:00 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |