| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-23 | |||
| 19:38:23 | melwitt | thanks gmaan | |
| 19:39:26 | sean-k-mooney | rm_work[m]: perhaps this is the optioni mentioned https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.update_resources_interval | |
| 19:39:35 | rm_work[m] | i think the BAD case (the 20m I talked about) might be when we had other issues going on slowing things down, so maybe it's as simple as setting the update_aviable_resouces periodic to run faster | |
| 19:39:44 | sean-k-mooney | rm_work[m]: what exactly do you mean by aviable by the way | |
| 19:39:52 | sean-k-mooney | do you meain aviable in the ironic node list | |
| 19:40:02 | rm_work[m] | the ironic state yeah | |
| 19:40:52 | sean-k-mooney | ack i dont recall if there is a specifric addtioanl dely in the ironic case but maybe JayF woudl know | |
| 19:40:59 | rm_work[m] | he said he didn't :D | |
| 19:41:01 | JayF | I sent rm_work[m] to you :) | |
| 19:41:18 | sean-k-mooney | ack i was onderign if it could be relate the the lease stuff | |
| 19:41:34 | rm_work[m] | we'll try tweaking that periodic | |
| 19:41:35 | JayF | I honestly have no idea where to even start from the Ironic side; we just fetch nodes and pass them in | |
| 19:41:35 | rm_work[m] | or, that config value | |
| 19:41:44 | JayF | *Ironic driver side | |
| 19:41:51 | JayF | and I don't know much about the mechanisms on the other side of that API interface | |
| 19:42:07 | rm_work[m] | the default being "0" to "run at the default interval" does not tell me what the default interval is, lol | |
| 19:42:17 | rm_work[m] | why would the default not just BE the default interval 😅 | |
| 19:42:31 | sean-k-mooney | the default interval for perodics in oslo and as a result nova is 500 seconds | |
| 19:42:37 | sean-k-mooney | sory 300 | |
| 19:42:38 | rm_work[m] | ah ok | |
| 19:42:42 | sean-k-mooney | so 5 mins | |
| 19:42:46 | rm_work[m] | yeah | |
| 19:42:55 | rm_work[m] | alright, thanks we will poke at it | |
| 19:43:15 | JayF | rm_work[m]: please make sure if you figure out a set of good configs for your scale, please document it -- even if just in a mailing list half-blogpost | |
| 19:44:22 | sean-k-mooney | so oen thing to look at would be the logs in the relevent nova-comptue between the previous start of that perodic and then first one after the node appeard | |
| 19:44:42 | sean-k-mooney | nova-comnpute when its managing a lot fo ironic nodes can take a long time for that perodic to complete | |
| 19:45:02 | rm_work[m] | we have 240ish nodes so i don't think it's considered "a lot" | |
| 19:45:04 | sean-k-mooney | we shoudl eventually improve that but ironic shardign might help | |
| 19:45:20 | sean-k-mooney | 240 nodes manged by a singel nova compute agent? | |
| 19:45:22 | rm_work[m] | our clouds are relatively small ATM | |
| 19:46:08 | sean-k-mooney | or are you still using the deprecated peers parmaters with the hashring that loadblances them between multiple agents? | |
| 19:46:18 | rm_work[m] | just one agent | |
| 19:46:34 | rm_work[m] | also FWIW we're on 2024.2 | |
| 19:47:28 | sean-k-mooney | ya so 250 ironics on 1 nova compute is both a lot and not a lot. for perodics that loop over all the nodes they manage and update placement that can be slow since itd getign the infor one ironic node at a time | |
| 19:47:43 | sean-k-mooney | for actully triging the provisioning ectra its fine | |
| 19:47:54 | rm_work[m] | hmm | |
| 19:48:22 | rm_work[m] | but it sounds like you're saying we CAN'T have multiple? because whatever way they spread out is deprecated? or is that just one of multiple methods | |
| 19:48:42 | rm_work[m] | surprised it's sequential for the node data gathering | |
| 19:48:46 | sean-k-mooney | there is a repalcement know as ironic sharding | |
| 19:48:59 | sean-k-mooney | that allow you to map subset of ironic nodes to diffent nova compute agents | |
| 19:49:08 | sean-k-mooney | im not sure at what scale using that is recommended | |
| 19:49:23 | sean-k-mooney | rm_work[m]: i would try seeign how long the perodic takes to run | |
| 19:49:33 | sean-k-mooney | if its like 90 seconds your fine | |
| 19:49:42 | sean-k-mooney | if its 15 mins then that the issue | |
| 19:50:39 | rm_work[m] | hmm k | |
| 19:50:46 | rm_work[m] | will do some timing and log diving | |
| 19:51:13 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/2024.1/implemented/ironic-shards.html is the new way of scaling nova-compute with ironic | |
| 19:51:33 | sean-k-mooney | the peer_list is deprecated but still in master because we never got aroudn to deleting it | |
| 19:51:39 | sean-k-mooney | but we really should at some point | |
| 19:53:37 | sean-k-mooney | rm_work[m]: for what its worht it woudl be posible for us to optimize this in nova in the future by say delegting the node update to a thread pool on each perodic but we have not been inveting time in that space with the eventlet removal ectra that has been happening | |
| 19:58:08 | rm_work[m] | right yeah, no worries. thanks for taking the time to explain! | |
| 19:58:43 | rm_work[m] | yeah my thought would just be to parallelize the node checks in the job | |
| 19:58:50 | rm_work[m] | I wonder if that would actually be that complicated... | |
| 19:59:38 | sean-k-mooney | in the past yes with all the work we are doing to make the compute agent work with thread pool executors instead of eventlet i suspect its simpler then before | |
| 20:00:26 | sean-k-mooney | i would focus on confiming this is the issue first before going much further however | |
| 20:00:44 | sean-k-mooney | if we can confirm this as a problem it would be a good topic for a ptg session with the ironic folks | |
| 22:30:31 | opendevreview | Merged openstack/nova master: Refactor service user authentication https://review.opendev.org/c/openstack/nova/+/974267 | |
| 22:30:47 | opendevreview | Merged openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | |
| #openstack-nova - 2026-01-24 | |||
| 07:28:39 | opendevreview | Merged openstack/nova master: api: Add response body schemas for quota class sets API https://review.opendev.org/c/openstack/nova/+/956140 | |
| 08:29:08 | opendevreview | Merged openstack/nova master: api: Add response body schemas for quota sets API https://review.opendev.org/c/openstack/nova/+/956141 | |
| 08:29:22 | opendevreview | Merged openstack/nova master: api: Add response body schemas for migrations API https://review.opendev.org/c/openstack/nova/+/956142 | |
| 14:47:00 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 14:47:00 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 14:47:01 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 14:47:01 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 14:47:02 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 14:47:02 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 14:47:03 | opendevreview | Stephen Finucane proposed openstack/nova master: Disable flake8 rules https://review.opendev.org/c/openstack/nova/+/974442 | |
| 19:22:34 | frickler | nova-ceph-multistore seems broken https://zuul.opendev.org/t/openstack/build/fc9475a1a9864d2994e8fd24456ce564 | |
| 20:42:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 20:42:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 20:42:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| #openstack-nova - 2026-01-25 | |||
| 03:50:34 | opendevreview | Merged openstack/nova master: api: Add response body schemas for server migrations API https://review.opendev.org/c/openstack/nova/+/956143 | |
| 03:50:52 | opendevreview | Merged openstack/nova master: api: Add response body schemas for server tags API https://review.opendev.org/c/openstack/nova/+/956144 | |
| 12:11:12 | yuval | I see this patch: https://review.opendev.org/c/openstack/nova/+/384871 | |
| 12:11:37 | yuval | why it was never merged markus is still around? | |
| 20:05:26 | gibi | yuval: it was abandoned 9 years ago, we probably need a historian to figure out the reasons :) But there is recent effort implementing iothread that landed couple of days ago https://review.opendev.org/c/openstack/nova/+/939254 | |
| 22:43:15 | opendevreview | Steve Baker proposed openstack/nova master: Add VNC console support for the Ironic driver https://review.opendev.org/c/openstack/nova/+/942528 | |
| #openstack-nova - 2026-01-26 | |||
| 02:57:39 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/974667 | |
| 04:25:22 | mkuroha | Hi gibi, nova-team. I’ve addressed the comments and would appreciate it if you could take another look when you have time. https://review.opendev.org/c/openstack/nova/+/948304 | |
| 05:07:41 | opendevreview | Seyeong Kim proposed openstack/nova master: libvirt: Support boot_index for multiple block devices https://review.opendev.org/c/openstack/nova/+/963665 | |
| 08:08:32 | opendevreview | Johannes Kulik proposed openstack/nova master: Attaching a volume returns HTTP 202 https://review.opendev.org/c/openstack/nova/+/971068 | |
| 08:26:25 | nicolairuckel | tkajinam: Any updates on the issue regarding the reset of the NVRAM? | |
| 09:43:19 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd https://review.opendev.org/c/openstack/nova/+/705657 | |
| 09:43:19 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 09:43:20 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 09:43:20 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 09:43:21 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 09:43:21 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 09:43:22 | opendevreview | Stephen Finucane proposed openstack/nova master: Disable flake8 rules https://review.opendev.org/c/openstack/nova/+/974442 | |
| 09:43:23 | opendevreview | Stephen Finucane proposed openstack/nova master: console: Fix type error https://review.opendev.org/c/openstack/nova/+/974687 | |
| 09:43:23 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Filter out (more) eventlet deprecation warnings https://review.opendev.org/c/openstack/nova/+/974686 | |
| 09:43:25 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.console https://review.opendev.org/c/openstack/nova/+/974689 | |
| 09:43:25 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.conf https://review.opendev.org/c/openstack/nova/+/974688 | |
| 09:43:27 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Disallow incomplete defs https://review.opendev.org/c/openstack/nova/+/974690 | |
| 10:54:15 | opendevreview | Nisha Brahmankar proposed openstack/nova master: Fix exception handling in nova cache https://review.opendev.org/c/openstack/nova/+/974695 | |
| 11:31:20 | jkulik | gibi: did you read my reply on https://review.opendev.org/c/openstack/nova/+/971068 regarding ServiceTooOld and old microversion? Do we need to rework the patch for that? | |
| 12:19:47 | gibi | jkulik: thanks for the ping, I read it now. I'm a bit hesitant about the way forward but I summoned dansmith to help :) | |
| 13:08:04 | opendevreview | Nisha Brahmankar proposed openstack/nova master: Fix exception handling in nova cache https://review.opendev.org/c/openstack/nova/+/974695 | |
| 13:45:38 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd https://review.opendev.org/c/openstack/nova/+/705657 | |