| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-23 | |||
| 11:24:10 | sean-k-mooney | so yes it for uefi guest | |
| 11:24:27 | sean-k-mooney | we are planning to delegate this logic back to libvirt now that it supprots it | |
| 11:24:44 | sean-k-mooney | i.e. allowign libvirt to selct the correct firmware based on the funcitonaltiy you have requested | |
| 11:26:17 | sean-k-mooney | for example if you request secure boot we have to select the secure boot enabeld ovmf uefi firmware or it wont work i belvie the opiste it ture as well if you have it disabled then we need to use the non secure boot version | |
| 11:27:20 | sean-k-mooney | when you enabel stateless vms for confidental compute you similarly need to select the correct firmware file. this is fixing that in nova before we adtop the libvirt feature so it can potitrally be backported | |
| 11:28:06 | sean-k-mooney | well its already being backported but you get the point we exended nova logic before riping it out beacuse we are not backporting the use of the new libvirt feature | |
| 11:29:22 | zigo | Ok, thanks for explaining. | |
| 11:56:31 | opendevreview | Merged openstack/nova stable/2024.2: Fix bug 2000069 https://review.opendev.org/c/openstack/nova/+/970634 | |
| 12:01:57 | DominikDanelski[m] | sean-k-mooney: Sorry, could you go through my replies to your comments in https://review.opendev.org/c/openstack/nova/+/969251?tab=comments ? I think it's very close to being merged after I addressed your questions there, I hope. | |
| 13:46:45 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove networks key from quota schemas https://review.opendev.org/c/openstack/nova/+/974417 | |
| 13:48:29 | stephenfin | sean-k-mooney: gmaan: Could you re-add your +2s to https://review.opendev.org/c/openstack/nova/+/956140/? | |
| 13:48:56 | stephenfin | sean-k-mooney: sorry https://review.opendev.org/c/openstack/nova/+/956141/ for you (need gmaan's +2 on the other one) | |
| 13:49:19 | sean-k-mooney | sure once im off this call ill take a look | |
| 13:49:32 | stephenfin | ta | |
| 13:52:01 | opendevreview | Merged openstack/nova stable/2024.2: Replace removed os-vif-ovs job https://review.opendev.org/c/openstack/nova/+/972665 | |
| 14:04:51 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 14:30:34 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 14:30:34 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 14:30:35 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 14:30:35 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 14:30:36 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 15:21:10 | opendevreview | Dominik proposed openstack/nova master: NUMA Topology with Resource Providers: Scheduler changes https://review.opendev.org/c/openstack/nova/+/971176 | |
| 15:21:10 | opendevreview | Dominik proposed openstack/nova master: NUMA Topology with Resource Providers: Libvirt NUMA Migrate https://review.opendev.org/c/openstack/nova/+/971177 | |
| 15:36:04 | sean-k-mooney | DominikDanelski[m]: so melwitt and i chated about this breifly a few weeks ago and we were not configed the repoduce actully repoduce things properly. or that the proposed fix didnt create ohter problems | |
| 15:37:10 | sean-k-mooney | but to be honestl i have lost all context on it since then | |
| 15:45:12 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 15:45:12 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 15:45:13 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 15:45:13 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 15:45:14 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 15:45:14 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 15:45:15 | opendevreview | Stephen Finucane proposed openstack/nova master: Disable flake8 rules https://review.opendev.org/c/openstack/nova/+/974442 | |
| 15:55:36 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 15:55:37 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 15:55:37 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 17:20:41 | gmaan | stephenfin: thanks for followup for 'network' field. I +w on quota ones now and re-approvd other one which were rebased | |
| 17:20:53 | stephenfin | ty! | |
| 18:52:50 | opendevreview | sean mooney proposed openstack/nova master: [WIP] Add PCI device groups for atomic allocation https://review.opendev.org/c/openstack/nova/+/973604 | |
| 19:26:57 | melwitt | dansmith, gmaan: thank you both for reviews :) but I regret to inform you there is a prereq refactor patch underneath it for the service_auth.py commonizing https://review.opendev.org/c/openstack/nova/+/974267 | |
| 19:32:04 | rm_work[m] | anyone offhand have any ideas why placement might take a long time to show a node within openstack allocation candidate list after it goes available in ironic? it's usually about 5 minutes in normal operation and sometimes we see it take up to 20 minutes :( | |
| 19:33:57 | rm_work[m] | is there some setting we can tweak? is this a periodic task that is running too slowly? | |
| 19:34:26 | sean-k-mooney | melwitt: +1 for now ill try and take a look properly on monday but it looks ok at first glance | |
| 19:34:52 | melwitt | thanks sean-k-mooney :) | |
| 19:35:05 | sean-k-mooney | rm_work[m]: is the RP not there or is it there but disabled? | |
| 19:35:54 | sean-k-mooney | rm_work[m]: my guess of the top of my head is that its related to the update_aviable_resouces periodic task tiem of 5 mins and the delay is just in the virt driver runnign that and noticing the new node | |
| 19:38:07 | rm_work[m] | ah ok so we could maybe tune that task down to 1m or something? | |
| 19:38:08 | gmaan | melwitt: yeah, I was reviewing that also, lgtm. thanks for refactoring. | |
| 19:38:23 | melwitt | thanks gmaan | |
| 19:39:26 | sean-k-mooney | rm_work[m]: perhaps this is the optioni mentioned https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.update_resources_interval | |
| 19:39:35 | rm_work[m] | i think the BAD case (the 20m I talked about) might be when we had other issues going on slowing things down, so maybe it's as simple as setting the update_aviable_resouces periodic to run faster | |
| 19:39:44 | sean-k-mooney | rm_work[m]: what exactly do you mean by aviable by the way | |
| 19:39:52 | sean-k-mooney | do you meain aviable in the ironic node list | |
| 19:40:02 | rm_work[m] | the ironic state yeah | |
| 19:40:52 | sean-k-mooney | ack i dont recall if there is a specifric addtioanl dely in the ironic case but maybe JayF woudl know | |
| 19:40:59 | rm_work[m] | he said he didn't :D | |
| 19:41:01 | JayF | I sent rm_work[m] to you :) | |
| 19:41:18 | sean-k-mooney | ack i was onderign if it could be relate the the lease stuff | |
| 19:41:34 | rm_work[m] | we'll try tweaking that periodic | |
| 19:41:35 | rm_work[m] | or, that config value | |
| 19:41:35 | JayF | I honestly have no idea where to even start from the Ironic side; we just fetch nodes and pass them in | |
| 19:41:44 | JayF | *Ironic driver side | |
| 19:41:51 | JayF | and I don't know much about the mechanisms on the other side of that API interface | |
| 19:42:07 | rm_work[m] | the default being "0" to "run at the default interval" does not tell me what the default interval is, lol | |
| 19:42:17 | rm_work[m] | why would the default not just BE the default interval 😅 | |
| 19:42:31 | sean-k-mooney | the default interval for perodics in oslo and as a result nova is 500 seconds | |
| 19:42:37 | sean-k-mooney | sory 300 | |
| 19:42:38 | rm_work[m] | ah ok | |
| 19:42:42 | sean-k-mooney | so 5 mins | |
| 19:42:46 | rm_work[m] | yeah | |
| 19:42:55 | rm_work[m] | alright, thanks we will poke at it | |
| 19:43:15 | JayF | rm_work[m]: please make sure if you figure out a set of good configs for your scale, please document it -- even if just in a mailing list half-blogpost | |
| 19:44:22 | sean-k-mooney | so oen thing to look at would be the logs in the relevent nova-comptue between the previous start of that perodic and then first one after the node appeard | |
| 19:44:42 | sean-k-mooney | nova-comnpute when its managing a lot fo ironic nodes can take a long time for that perodic to complete | |
| 19:45:02 | rm_work[m] | we have 240ish nodes so i don't think it's considered "a lot" | |
| 19:45:04 | sean-k-mooney | we shoudl eventually improve that but ironic shardign might help | |
| 19:45:20 | sean-k-mooney | 240 nodes manged by a singel nova compute agent? | |
| 19:45:22 | rm_work[m] | our clouds are relatively small ATM | |
| 19:46:08 | sean-k-mooney | or are you still using the deprecated peers parmaters with the hashring that loadblances them between multiple agents? | |
| 19:46:18 | rm_work[m] | just one agent | |
| 19:46:34 | rm_work[m] | also FWIW we're on 2024.2 | |
| 19:47:28 | sean-k-mooney | ya so 250 ironics on 1 nova compute is both a lot and not a lot. for perodics that loop over all the nodes they manage and update placement that can be slow since itd getign the infor one ironic node at a time | |
| 19:47:43 | sean-k-mooney | for actully triging the provisioning ectra its fine | |
| 19:47:54 | rm_work[m] | hmm | |
| 19:48:22 | rm_work[m] | but it sounds like you're saying we CAN'T have multiple? because whatever way they spread out is deprecated? or is that just one of multiple methods | |
| 19:48:42 | rm_work[m] | surprised it's sequential for the node data gathering | |
| 19:48:46 | sean-k-mooney | there is a repalcement know as ironic sharding | |
| 19:48:59 | sean-k-mooney | that allow you to map subset of ironic nodes to diffent nova compute agents | |
| 19:49:08 | sean-k-mooney | im not sure at what scale using that is recommended | |
| 19:49:23 | sean-k-mooney | rm_work[m]: i would try seeign how long the perodic takes to run | |
| 19:49:33 | sean-k-mooney | if its like 90 seconds your fine | |
| 19:49:42 | sean-k-mooney | if its 15 mins then that the issue | |
| 19:50:39 | rm_work[m] | hmm k | |
| 19:50:46 | rm_work[m] | will do some timing and log diving | |
| 19:51:13 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/2024.1/implemented/ironic-shards.html is the new way of scaling nova-compute with ironic | |
| 19:51:33 | sean-k-mooney | the peer_list is deprecated but still in master because we never got aroudn to deleting it | |
| 19:51:39 | sean-k-mooney | but we really should at some point | |
| 19:53:37 | sean-k-mooney | rm_work[m]: for what its worht it woudl be posible for us to optimize this in nova in the future by say delegting the node update to a thread pool on each perodic but we have not been inveting time in that space with the eventlet removal ectra that has been happening | |
| 19:58:08 | rm_work[m] | right yeah, no worries. thanks for taking the time to explain! | |
| 19:58:43 | rm_work[m] | yeah my thought would just be to parallelize the node checks in the job | |
| 19:58:50 | rm_work[m] | I wonder if that would actually be that complicated... | |