| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-28 | |||
| 09:29:32 | gibi | sean-k-mooney: your plan sounds sane to me | |
| 09:39:49 | sean-k-mooney | what i can do is prepare the url overied patch and add the topic to the irc meeting adjenda for monday. if we are happy to proceed we can merge it after the meeting adn then monitor it and see | |
| 11:06:37 | opendevreview | Joan Gilabert proposed openstack/nova-specs master: Repropose and update cyborg vGPU (mdev) support https://review.opendev.org/c/openstack/nova-specs/+/967515 | |
| 12:05:48 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 12:37:41 | kklimaszewski | Hello, I've recently proposed a new nova spec for SPDK-based nova-provisioned storage backend: https://review.opendev.org/c/openstack/nova-specs/+/985676. Would anyone here be willing to review it/be its liaison? Also: Dominik (known here as Etua) has reproposed a spec from Ussuri for NUMA topology with resource providers: https://review.opendev.org/c/openstack/nova-specs/+/978570. Could someone here please review it? | |
| 14:49:47 | JayF | sean-k-mooney: I think I need to find time, regardless of interruptions and urgency, to get your patches cleaned up and landed upstream. That's a radical improvement. https://usercontent.irccloud-cdn.com/file/i2G0mxRP/image.png | |
| 14:50:45 | JayF | tl;dr: G-Research's (somewhat patched) 2024.1, one nova-compute that supported 1000 Ironic nodes was taking 65 minutes to restart. With three changes, that was reduced to 2 minutes. | |
| 14:51:11 | sean-k-mooney | damb | |
| 14:51:13 | sean-k-mooney | ok | |
| 14:51:18 | JayF | (NCI = "nova-compute-ironic" -- the internal GR moniker for nova-computes w/Ironic driver) | |
| 14:51:46 | JayF | given step #1 is also the first thing we wanna remove -- moving the resource updates to Ironic -- gives me hope as well that we're on the right track | |
| 14:52:56 | sean-k-mooney | so step 1 is the dispatching of the udpate to the thread pool right | |
| 14:53:12 | sean-k-mooney | so you can run N threads to do the work | |
| 14:53:23 | sean-k-mooney | you mentioned 32 workser but were they all used | |
| 14:53:55 | JayF | passed on the ask, I don't know | |
| 14:53:59 | sean-k-mooney | i assume you have not had time to test if say 8 or 16 has a linear effect? | |
| 14:54:02 | JayF | to be clear: I don't have operational access to this cluster at all | |
| 14:54:09 | sean-k-mooney | no worries | |
| 14:54:18 | sean-k-mooney | jsut wondering | |
| 14:54:45 | JayF | Honestly though, if we really get it down to 2 minutes | |
| 14:54:57 | JayF | something like a graceful shutdown starts looking really good for HA updates | |
| 14:55:03 | sean-k-mooney | JayF: the vtpm thing is also interesting i think we may have fixed that laze loadign thing on master recently | |
| 14:55:24 | JayF | if we could push Ironic queries to a secondary nova-compute while letting the first one complete actions in progress (or as discussed; hand off those in-progress actions from checkpoints) | |
| 14:57:27 | sean-k-mooney | JayF: https://review.opendev.org/c/openstack/nova/+/977037 | |
| 14:57:32 | sean-k-mooney | so not merged yet | |
| 14:57:47 | sean-k-mooney | JayF: but htat will fix the vtpm issue | |
| 14:57:56 | sean-k-mooney | its an issue for libivrt as well | |
| 14:58:29 | sean-k-mooney | added you to that so you shoudl get a notificaion whn that lands | |
| 14:59:46 | sean-k-mooney | i kind of forgot about that patch if im being honest but looks like i called out the ironic inpact the last tiem i looked :) | |
| 15:07:38 | sean-k-mooney | JayF: so the 3 patchs ye really need are https://review.opendev.org/c/openstack/nova/+/977037 https://review.opendev.org/c/openstack/nova/+/980676 and https://review.opendev.org/c/openstack/nova/+/980679/1 | |
| 15:16:15 | sean-k-mooney | on other thing that we coudl perhaps look into in the futre is allow nova-compute too run in a active stadby mode or a better way to have muliple nova-computes per shared then the previous hashring approch | |
| 15:16:21 | JayF | sean-k-mooney: I'm not convinced https://review.opendev.org/c/openstack/nova/+/977037 does the same as the downstream does | |
| 15:17:02 | sean-k-mooney | it does not skip the check but it but it remove the laze load overhead | |
| 15:17:05 | JayF | if not self.driver.capabilities.get('can_have_vtpm_instances', True):; return <--- he essentially just added this to the top of _validate_vtpm_configuration | |
| 15:17:35 | JayF | and added that as a driver capability (Which ironic has set to false) | |
| 15:17:38 | sean-k-mooney | ya that replace the expensive lazeload with an if | |
| 15:17:41 | sean-k-mooney | which is also valid | |
| 15:17:58 | sean-k-mooney | yep that will also work | |
| 15:18:19 | sean-k-mooney | but the libvirt driver or any that supprt it in the futre woudl still be needless slwo withouthte other fix | |
| 15:18:32 | sean-k-mooney | we can do both | |
| 15:18:36 | sean-k-mooney | those are not competing | |
| 15:19:02 | JayF | perfect | |
| 15:19:30 | JayF | I should have time this afternoon/tomorrow to look at my normal todo list, which has been cast aside with "do the startup benchmark on nova-compute" at the top for a few weeks | |
| 15:19:44 | sean-k-mooney | hum | |
| 15:19:51 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/977037/6/nova/compute/manager.py#1151 | |
| 15:20:06 | sean-k-mooney | so that if was ment to skip this on ironic already | |
| 15:21:42 | sean-k-mooney | ... | |
| 15:21:46 | sean-k-mooney | if self.driver.capabilities.get('supports_vtpm', False): | |
| 15:21:48 | sean-k-mooney | return | |
| 15:21:54 | sean-k-mooney | do you see the bug.... | |
| 15:23:14 | sean-k-mooney | JayF: a not woudl really help that do the right ting dont you think | |
| 15:24:02 | sean-k-mooney | oh well no | |
| 15:24:22 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1150-L1168 | |
| 15:24:34 | sean-k-mooney | the full funtion is | |
| 15:25:11 | sean-k-mooney | workign slightly difetn so if the driver supprot vtpu we return early | |
| 15:25:35 | sean-k-mooney | if if the dreiver does not reprot supprot we check fi there are any isntace that request it | |
| 15:25:49 | sean-k-mooney | and treate that as an error bceaus you for migrated a vm or similar | |
| 15:25:56 | JayF | https://opendev.org/openstack/nova/src/branch/master/nova/virt/ironic/driver.py#L156 | |
| 15:26:11 | sean-k-mooney | the problme with the logic is that | |
| 15:26:42 | sean-k-mooney | for ironic is say it does not supprot it | |
| 15:26:58 | sean-k-mooney | and then we validate all of the instance anyway | |
| 15:27:12 | sean-k-mooney | even though we knwo none of them shoud ever use it | |
| 15:27:23 | JayF | the code you linked is an early return? | |
| 15:27:38 | JayF | if Ironic says we don't support it? | |
| 15:28:07 | sean-k-mooney | if ironic said you did supprot it it would early exit | |
| 15:28:15 | sean-k-mooney | the logic in the validat fuction is incorrect | |
| 15:28:30 | JayF | OH | |
| 15:28:33 | JayF | IT'S REVERSED | |
| 15:28:38 | sean-k-mooney | kind of ya | |
| 15:28:56 | sean-k-mooney | that why i orginall said a not would help | |
| 15:29:02 | sean-k-mooney | what the fucntion is for | |
| 15:29:20 | sean-k-mooney | was to detch instnace that requested vtpm on host where it was disable | |
| 15:29:21 | JayF | Yeah but this flows both ways; this also means you haven't been running this on drivers that *support* it | |
| 15:29:32 | JayF | which seems like a more concerning bug, no? | |
| 15:29:42 | sean-k-mooney | so yes and no | |
| 15:29:55 | sean-k-mooney | this was kind of a tepmey upgrade check of a sort | |
| 15:30:10 | sean-k-mooney | the code is buggy and we shoudl fix it | |
| 15:30:17 | sean-k-mooney | but it s not just buggy for ironic | |
| 15:30:22 | sean-k-mooney | it buggy in general | |
| 15:32:28 | sean-k-mooney | JayF: so the vtpm supprot in the libvirt driver is configurbale | |
| 15:32:41 | sean-k-mooney | JayF: it depen on an addtion software depency called swtpm | |
| 15:32:51 | sean-k-mooney | which when we added the feature was only shiped in some distos | |
| 15:32:58 | sean-k-mooney | so we made thie a boolean flag | |
| 15:33:22 | sean-k-mooney | and this was ment ot prevent start an agent when you have instnace that need vtpm supprot but you have it turn off | |
| 15:34:00 | sean-k-mooney | JayF: ill try and write this up as a bug noting the ironic impact | |
| 15:34:14 | sean-k-mooney | and see if i can hack somethign togheter for this. | |
| 15:34:38 | sean-k-mooney | JayF: but also feel free to push any patches ye have if ye get time | |
| 15:42:06 | opendevreview | sean mooney proposed openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | |
| 17:38:33 | sean-k-mooney | JayF: https://bugs.launchpad.net/nova/+bug/2154495 is the bug for that by the way | |
| 17:39:06 | sean-k-mooney | adding the new capablity trait is one of the ways to fix that but i am goign to see if i can first create a repoducer and then we can see what the best way forward is | |
| 17:39:40 | sean-k-mooney | im condiering if i just want to wrap the body of the function in a check for the verit driver that is in use or if this check shoudl move into the driver instead | |
| 17:40:09 | sean-k-mooney | all 3 approchs woudl work its just a question fo what is the better long term approch | |
| 17:41:40 | sean-k-mooney | im really not sure why this is not part of self.driver.init_host() | |
| 17:41:50 | sean-k-mooney | given its a driver sepcific check in the first palce | |
| 18:16:30 | isaacvicente[m] | there's a patch for this but seems to be abandoned, what you guys think? | |
| 18:16:30 | isaacvicente[m] | hi o/ is this bug still being tracked? https://bugs.launchpad.net/nova/+bug/2065599 | |
| 18:30:18 | sean-k-mooney | isaacvicente[m]: having both imageRef": "f2285517-a996-40d3-b331-c8214ec66b77", and destion_type: volume is not invlid nessisarly | |
| 18:30:37 | sean-k-mooney | ppofied the bdm and the imageref are the same | |
| 18:31:51 | sean-k-mooney | the metadata in the domain xml is a debug interface so wether tie image is incldue is mostly cosmetic | |
| 18:32:41 | sean-k-mooney | the real pain poitn woudl be if you were rlying on the presence or absance of the image refe to determin if its a BFV guest | |