| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-06-15 | |||
| 12:53:50 | sean-k-mooney | to a host numa cell | |
| 12:54:22 | sean-k-mooney | we do not expect any numa blancing to happen with that config unless you also request hw:mem_page_size in teh falvor or image | |
| 12:58:12 | sean-k-mooney | bauzas: so im sorry to say but thet bug is technially inviald and its a psudo feature request. | |
| 13:00:00 | bauzas | but the issue still remains | |
| 13:00:10 | sean-k-mooney | the flavor is not valid | |
| 13:00:39 | sean-k-mooney | we can add supprot for numa blancin on shared cpus only but that was never supproted by nova | |
| 13:00:47 | sean-k-mooney | we supprot it for pinned cpus | |
| 13:00:50 | sean-k-mooney | and for guest memory | |
| 13:01:01 | sean-k-mooney | but not for shared cpus | |
| 13:01:08 | sean-k-mooney | so im not agains supproting that | |
| 13:01:33 | sean-k-mooney | but the expecation today is that we wont consider the cpu allcation ration at the numa node level today | |
| 13:09:51 | bauzas | honestly, I don't know what to say :( | |
| 13:10:31 | sean-k-mooney | bauzas: im not agaisnt building thei fucntionlatiy into nova but its really a new feature | |
| 13:11:07 | sean-k-mooney | and if we are to do it we need to do it for spawn and cold migrate and unshleve ectra | |
| 13:12:47 | sean-k-mooney | thisis not the first tirm that i have had this converatoion with operators or custoemrs and pointed out that simply requesting hw:numa_node and nothing else is only vaild if your usign file backed memory as we do not do numa aware shared cpu placement so we will not blance between numa node | |
| 13:13:05 | sean-k-mooney | to be clear even with file backed memory it not a good idea | |
| 13:13:51 | sean-k-mooney | if other really want to tack this as a bug we need to add supprot for this on spwan and other move operaiotn and more fucntional tests to cover that | |
| 13:23:10 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1792985 and https://bugs.launchpad.net/nova/+bug/1439247 are relevent context for why the current flavor is invlaid | |
| 13:24:22 | sean-k-mooney | usign file backed memory prevetn the OOM kill but without also susign cpu pinnign it does nto adress the cpu blancing part | |
| 13:24:47 | sean-k-mooney | which is what ye were tryign to address with that new bug | |
| 13:50:12 | opendevreview | Joan Gilabert proposed openstack/nova master: Rename vtpm job and add mtty support for vpgu test https://review.opendev.org/c/openstack/nova/+/922140 | |
| 14:00:02 | opendevreview | Merged openstack/nova stable/2025.1: Add reproducer test for bug 2105896 https://review.opendev.org/c/openstack/nova/+/989535 | |
| 15:26:48 | dansmith | sean-k-mooney: unrelated to the numa aspect, why is it not right to set self.limits before we call that compute-via-RPC check? it seems that the code expects limits to be set,. | |
| 15:26:58 | dansmith | but I'm not sure when else they would be, but I'm sure I'm missing something | |
| 15:27:46 | sean-k-mooney | dansmith: that part isned nessarly wrong but the expecation that nova shoudl blance based on that is. also nova shoudl not be multiplying the core or memory on a given numa node by the ratios | |
| 15:27:55 | Uggla | reminder: upstream meeting in ~30mn | |
| 15:28:45 | sean-k-mooney | well the memofy shoudl not be multipled the cores could be viable | |
| 15:29:36 | dansmith | sean-k-mooney: yeah okay I was going to say.. the assumption in the patch being wrong makes sense, not arguing with that | |
| 15:30:06 | dansmith | it just looks to me like in the force case we probably need to be setting self.limits | |
| 15:30:35 | sean-k-mooney | dansmith: so there is a seprate bug in teh conductor | |
| 15:30:43 | sean-k-mooney | where the limt are nto passed to the compute | |
| 15:30:50 | dansmith | yeah | |
| 15:30:56 | sean-k-mooney | so the cshcdule does nto actully select the placmenet on the host | |
| 15:30:58 | sean-k-mooney | the compute does | |
| 15:31:09 | sean-k-mooney | so untile we etenbr recreate them there ro pass them | |
| 15:31:28 | sean-k-mooney | the two could still disagree | |
| 15:31:30 | dansmith | "select the placement" meaning the numa placement? | |
| 15:31:39 | dansmith | we're still calling scheduler even in the force case right? | |
| 15:32:01 | sean-k-mooney | if you mean force live migration i belive if you use the old api where that exist | |
| 15:32:14 | sean-k-mooney | we only check that the host exist and skipp all the filters | |
| 15:32:27 | sean-k-mooney | but if you use the non force microverion where you can pass a host | |
| 15:32:33 | sean-k-mooney | then we run true the filters | |
| 15:33:03 | dansmith | sean-k-mooney: L100 here https://review.opendev.org/c/openstack/nova/+/990212/5/nova/conductor/tasks/live_migrate.py# | |
| 15:33:07 | sean-k-mooney | so if you use the old foce api then we just trust you that it fits and send it to the compute | |
| 15:33:48 | dansmith | oh I guess in that case we don't even have limits to set | |
| 15:34:16 | sean-k-mooney | right because we are currently relying on a sideffect fo the numa topltoy filter to create them | |
| 15:34:25 | dansmith | yeah, okay | |
| 15:34:28 | sean-k-mooney | which is the bug with numa vswthces | |
| 15:36:27 | sean-k-mooney | without checking i woudl have to assume we are effectivly applying our defaults or no multipler at all when the limits are not pass today? | |
| 15:37:08 | sean-k-mooney | which while not ideal for memory and disk its at least <=1.0 now | |
| 15:37:52 | bauzas | sean-k-mooney: the problem is that we say 'no' to something is a tribal knowledge miss | |
| 15:37:59 | sean-k-mooney | in practice we are relying on placment to enfoce the multipler globally on the compute node | |
| 15:38:18 | bauzas | sean-k-mooney: if the scheduler provides limits and then the conductor doesn't use it, then it's a bug | |
| 15:38:27 | sean-k-mooney | bauzas: this is somethign we have discussed about changign a few time and when i have suggested it we wer econcerd about upgrade impact | |
| 15:38:36 | bauzas | but I understand your point, the scheduler shouldn't accept this | |
| 15:38:43 | sean-k-mooney | bauzas: dansmith just to be clear | |
| 15:38:53 | sean-k-mooney | im not agaisnt fixign the fact we shoudl pass limit when we dont | |
| 15:39:22 | sean-k-mooney | but im not ok with impliy hw:numa_nodes=1 with nothign else is a vlaid numa flavor that shoudl be balanced | |
| 15:39:36 | sean-k-mooney | at elast not until that works for all code paths | |
| 15:39:46 | bauzas | so we should provide a HTTP400 if an operator creates a wrong flavor then | |
| 15:39:56 | bauzas | at least when creating the instance | |
| 15:40:00 | sean-k-mooney | we coudl btu we rejected that in the past | |
| 15:40:15 | sean-k-mooney | we dicuss this when stephen was addign the falvor metadata validation | |
| 15:40:39 | sean-k-mooney | the minitum valid numa flavor to day has hw:mem_page_size set to anything | |
| 15:41:01 | bauzas | but that's a tribal knowledge, right? | |
| 15:41:02 | sean-k-mooney | everythign else builds form that baseline otherwise your vm will get OOM killed eventually | |
| 15:41:32 | sean-k-mooney | bauzas: sort of | |
| 15:41:34 | bauzas | because I wonder why we should just use this flavor extraspec (mem_page_size) just for NUMA usage ? | |
| 15:41:53 | sean-k-mooney | because without it we do non numa aware memroy tracking | |
| 15:42:17 | sean-k-mooney | and we pin you to a numa ndoe without chekcign if the sum of vms on that numa ndoe fit | |
| 15:42:27 | bauzas | so that's a tech debt | |
| 15:42:41 | sean-k-mooney | yep from day one of numa supprot in tree | |
| 15:42:52 | bauzas | operators should be able to ask for NUMA nodes without needing to ask page sizes | |
| 15:42:55 | sean-k-mooney | and its tech debt i hav eproposed fixing multipel time | |
| 15:43:08 | sean-k-mooney | but we have rejected because it breaks upgrades | |
| 15:43:15 | bauzas | but at least for the bug report, I think we can still accept it | |
| 15:43:34 | sean-k-mooney | we can accpt the bug for limits | |
| 15:43:36 | bauzas | if the scheduler provides limits, we could provide them to the conductor and then the compute | |
| 15:43:40 | sean-k-mooney | not for numa without pagesize | |
| 15:44:02 | bauzas | then the scheduler shouldn't provide the limits | |
| 15:44:06 | bauzas | right? | |
| 15:46:41 | sean-k-mooney | sorry im not following the question | |
| 15:47:12 | bauzas | lemme explain it better my question | |
| 15:48:09 | bauzas | if we don't want nova to accept flavors asking numa cores with no page sizes, then we should not pass limits down to the conductor | |
| 15:48:46 | sean-k-mooney | https://etherpad.opendev.org/p/nova-wallaby-ptg#L712 | |
| 15:48:52 | bauzas | if we're passing scheduler limits to the conductor, this means "oh, yeah, I accept this instance, please check those limits when you run the claims" | |
| 15:48:56 | sean-k-mooney | that was the last time i brought up chagnitn this at the ptg | |
| 15:49:17 | bauzas | sorry, my brain is bad about remembering things in general :-( | |
| 15:49:21 | sean-k-mooney | bauzas: the limits we are checkign are cpu ram and disk | |
| 15:49:29 | sean-k-mooney | those are enfocced by placment at the host level | |
| 15:49:42 | sean-k-mooney | the have never applied with the host at the numa level | |
| 15:49:45 | bauzas | like, I wasn't even to recollate the bug with my own numa vswitches | |
| 15:50:36 | sean-k-mooney | that whwere the diconnect is. we have never considerd the allcation tratios when it comes to numa affinity | |
| 15:52:19 | sean-k-mooney | bauzas: back in wallaby https://etherpad.opendev.org/p/r.321f34cf3eb9caa9d87a9ec8349c3d29#L712 we agreed an aproch to adress this so that any numa instance woudl be give a hw:mem_page_size if not set | |
| 15:52:42 | sean-k-mooney | that didnt happen because it was internally depriorised by our pm so i didnt end up workign on that | |
| 15:53:12 | bauzas | thanks, again, I wasn't able to remember this, sorry | |
| 15:54:05 | bauzas | but now we stora kind of accepting those flavors and we can't say 'well, doh, we don't support this, thanks' | |
| 15:54:10 | sean-k-mooney | no worries. so to have a path forward, i have no issues with passing limit when they shoudl be passed, and i have no issue with closign this numa footgun | |
| 15:54:17 | bauzas | I'm looking at the upstream docs to see whether we call it | |
| 15:54:49 | sean-k-mooney | bauzas: ya that has been the main concurn how do we fix this without breakign upgrades | |