| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-28 | |||
| 09:05:37 | opendevreview | huanhongda proposed openstack/nova master: Fix scheduler aggregate cache race on concurrent add_host https://review.opendev.org/c/openstack/nova/+/990437 | |
| 09:16:24 | sean-k-mooney | im wondering if we coudl run a little experiment in the nova gate for a week or two to see if we could finally get rid of the kernel paincs | |
| 09:17:11 | sean-k-mooney | i was chating to frickler in the infra channel and they had the insight that we may have been thinking about the root cause incorrectly and that the panic was actully cause by file system currption | |
| 09:18:25 | sean-k-mooney | they pointed out that the current behiovor looks as if the inital copy of the filesystem form the initram was interupted | |
| 09:18:47 | sean-k-mooney | cirros ships with a mostly empty rootfs (jsut a partion table) | |
| 09:18:57 | sean-k-mooney | and copies the root on first boot | |
| 09:19:15 | sean-k-mooney | if we interupt that then we will get a panic when we resize | |
| 09:19:26 | sean-k-mooney | so we inheringly have a race in the ci right now | |
| 09:19:40 | sean-k-mooney | once we normally win because the root fs is like 10 MB or so | |
| 09:20:09 | sean-k-mooney | based on there suggestion i built a test image https://github.com/SeanMooney/cirros/releases/tag/cirros-d260527-x86_64-5a75ef2-test | |
| 09:20:26 | sean-k-mooney | where the root is prepopulated and tested it with a quick devstack change | |
| 09:20:36 | sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/990348 | |
| 09:20:44 | sean-k-mooney | so the experimetn i would like to run in nova gate | |
| 09:21:04 | sean-k-mooney | is override the cirrors image we use in teh 2-3 jobs we normally see the issue in | |
| 09:21:15 | sean-k-mooney | and poitn it at my image | |
| 09:21:44 | sean-k-mooney | while i speraly chat to frickler and the other cirros folks about how to do this upstram in the cirros project | |
| 09:22:29 | sean-k-mooney | we typically see 10-20 kernel panics a day on nova jobs | |
| 09:22:48 | sean-k-mooney | so what im hopign is if we run that experimetn for a week or two that woudl drop to 0 | |
| 09:23:48 | sean-k-mooney | im going to create a patch to override the cirros image path for the relevent jobs so we can dicsuss mroe but i would be intersted in what folks think | |
| 09:24:43 | sean-k-mooney | are we willing to run the experiment for 2 weeks to get the data? we can obvioulsy quickly revert if needed | |
| 09:28:50 | sean-k-mooney | my corrent workign thory is we actully had 2 diffent kernel paninc, the first was resovled with the waiting ofr sshable changes for volume attach/detach. and the second was this file system currptionissue that we see with BFV + resize or BFV + any reboot i guess | |
| 09:29:32 | gibi | sean-k-mooney: your plan sounds sane to me | |
| 09:39:49 | sean-k-mooney | what i can do is prepare the url overied patch and add the topic to the irc meeting adjenda for monday. if we are happy to proceed we can merge it after the meeting adn then monitor it and see | |
| 11:06:37 | opendevreview | Joan Gilabert proposed openstack/nova-specs master: Repropose and update cyborg vGPU (mdev) support https://review.opendev.org/c/openstack/nova-specs/+/967515 | |
| 12:05:48 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 12:37:41 | kklimaszewski | Hello, I've recently proposed a new nova spec for SPDK-based nova-provisioned storage backend: https://review.opendev.org/c/openstack/nova-specs/+/985676. Would anyone here be willing to review it/be its liaison? Also: Dominik (known here as Etua) has reproposed a spec from Ussuri for NUMA topology with resource providers: https://review.opendev.org/c/openstack/nova-specs/+/978570. Could someone here please review it? | |
| 14:49:47 | JayF | sean-k-mooney: I think I need to find time, regardless of interruptions and urgency, to get your patches cleaned up and landed upstream. That's a radical improvement. https://usercontent.irccloud-cdn.com/file/i2G0mxRP/image.png | |
| 14:50:45 | JayF | tl;dr: G-Research's (somewhat patched) 2024.1, one nova-compute that supported 1000 Ironic nodes was taking 65 minutes to restart. With three changes, that was reduced to 2 minutes. | |
| 14:51:11 | sean-k-mooney | damb | |
| 14:51:13 | sean-k-mooney | ok | |
| 14:51:18 | JayF | (NCI = "nova-compute-ironic" -- the internal GR moniker for nova-computes w/Ironic driver) | |
| 14:51:46 | JayF | given step #1 is also the first thing we wanna remove -- moving the resource updates to Ironic -- gives me hope as well that we're on the right track | |
| 14:52:56 | sean-k-mooney | so step 1 is the dispatching of the udpate to the thread pool right | |
| 14:53:12 | sean-k-mooney | so you can run N threads to do the work | |
| 14:53:23 | sean-k-mooney | you mentioned 32 workser but were they all used | |
| 14:53:55 | JayF | passed on the ask, I don't know | |
| 14:53:59 | sean-k-mooney | i assume you have not had time to test if say 8 or 16 has a linear effect? | |
| 14:54:02 | JayF | to be clear: I don't have operational access to this cluster at all | |
| 14:54:09 | sean-k-mooney | no worries | |
| 14:54:18 | sean-k-mooney | jsut wondering | |
| 14:54:45 | JayF | Honestly though, if we really get it down to 2 minutes | |
| 14:54:57 | JayF | something like a graceful shutdown starts looking really good for HA updates | |
| 14:55:03 | sean-k-mooney | JayF: the vtpm thing is also interesting i think we may have fixed that laze loadign thing on master recently | |
| 14:55:24 | JayF | if we could push Ironic queries to a secondary nova-compute while letting the first one complete actions in progress (or as discussed; hand off those in-progress actions from checkpoints) | |
| 14:57:27 | sean-k-mooney | JayF: https://review.opendev.org/c/openstack/nova/+/977037 | |
| 14:57:32 | sean-k-mooney | so not merged yet | |
| 14:57:47 | sean-k-mooney | JayF: but htat will fix the vtpm issue | |
| 14:57:56 | sean-k-mooney | its an issue for libivrt as well | |
| 14:58:29 | sean-k-mooney | added you to that so you shoudl get a notificaion whn that lands | |
| 14:59:46 | sean-k-mooney | i kind of forgot about that patch if im being honest but looks like i called out the ironic inpact the last tiem i looked :) | |
| 15:07:38 | sean-k-mooney | JayF: so the 3 patchs ye really need are https://review.opendev.org/c/openstack/nova/+/977037 https://review.opendev.org/c/openstack/nova/+/980676 and https://review.opendev.org/c/openstack/nova/+/980679/1 | |
| 15:16:15 | sean-k-mooney | on other thing that we coudl perhaps look into in the futre is allow nova-compute too run in a active stadby mode or a better way to have muliple nova-computes per shared then the previous hashring approch | |
| 15:16:21 | JayF | sean-k-mooney: I'm not convinced https://review.opendev.org/c/openstack/nova/+/977037 does the same as the downstream does | |
| 15:17:02 | sean-k-mooney | it does not skip the check but it but it remove the laze load overhead | |
| 15:17:05 | JayF | if not self.driver.capabilities.get('can_have_vtpm_instances', True):; return <--- he essentially just added this to the top of _validate_vtpm_configuration | |
| 15:17:35 | JayF | and added that as a driver capability (Which ironic has set to false) | |
| 15:17:38 | sean-k-mooney | ya that replace the expensive lazeload with an if | |
| 15:17:41 | sean-k-mooney | which is also valid | |
| 15:17:58 | sean-k-mooney | yep that will also work | |
| 15:18:19 | sean-k-mooney | but the libvirt driver or any that supprt it in the futre woudl still be needless slwo withouthte other fix | |
| 15:18:32 | sean-k-mooney | we can do both | |
| 15:18:36 | sean-k-mooney | those are not competing | |
| 15:19:02 | JayF | perfect | |
| 15:19:30 | JayF | I should have time this afternoon/tomorrow to look at my normal todo list, which has been cast aside with "do the startup benchmark on nova-compute" at the top for a few weeks | |
| 15:19:44 | sean-k-mooney | hum | |
| 15:19:51 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/977037/6/nova/compute/manager.py#1151 | |
| 15:20:06 | sean-k-mooney | so that if was ment to skip this on ironic already | |
| 15:21:42 | sean-k-mooney | ... | |
| 15:21:46 | sean-k-mooney | if self.driver.capabilities.get('supports_vtpm', False): | |
| 15:21:48 | sean-k-mooney | return | |
| 15:21:54 | sean-k-mooney | do you see the bug.... | |
| 15:23:14 | sean-k-mooney | JayF: a not woudl really help that do the right ting dont you think | |
| 15:24:02 | sean-k-mooney | oh well no | |
| 15:24:22 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1150-L1168 | |
| 15:24:34 | sean-k-mooney | the full funtion is | |
| 15:25:11 | sean-k-mooney | workign slightly difetn so if the driver supprot vtpu we return early | |
| 15:25:35 | sean-k-mooney | if if the dreiver does not reprot supprot we check fi there are any isntace that request it | |
| 15:25:49 | sean-k-mooney | and treate that as an error bceaus you for migrated a vm or similar | |
| 15:25:56 | JayF | https://opendev.org/openstack/nova/src/branch/master/nova/virt/ironic/driver.py#L156 | |
| 15:26:11 | sean-k-mooney | the problme with the logic is that | |
| 15:26:42 | sean-k-mooney | for ironic is say it does not supprot it | |
| 15:26:58 | sean-k-mooney | and then we validate all of the instance anyway | |
| 15:27:12 | sean-k-mooney | even though we knwo none of them shoud ever use it | |
| 15:27:23 | JayF | the code you linked is an early return? | |
| 15:27:38 | JayF | if Ironic says we don't support it? | |
| 15:28:07 | sean-k-mooney | if ironic said you did supprot it it would early exit | |
| 15:28:15 | sean-k-mooney | the logic in the validat fuction is incorrect | |
| 15:28:30 | JayF | OH | |
| 15:28:33 | JayF | IT'S REVERSED | |
| 15:28:38 | sean-k-mooney | kind of ya | |
| 15:28:56 | sean-k-mooney | that why i orginall said a not would help | |
| 15:29:02 | sean-k-mooney | what the fucntion is for | |
| 15:29:20 | sean-k-mooney | was to detch instnace that requested vtpm on host where it was disable | |
| 15:29:21 | JayF | Yeah but this flows both ways; this also means you haven't been running this on drivers that *support* it | |
| 15:29:32 | JayF | which seems like a more concerning bug, no? | |
| 15:29:42 | sean-k-mooney | so yes and no | |
| 15:29:55 | sean-k-mooney | this was kind of a tepmey upgrade check of a sort | |
| 15:30:10 | sean-k-mooney | the code is buggy and we shoudl fix it | |
| 15:30:17 | sean-k-mooney | but it s not just buggy for ironic | |
| 15:30:22 | sean-k-mooney | it buggy in general | |