| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-06 | |||
| 16:03:19 | opendevreview | Elod Illes proposed openstack/nova stable/2025.1: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/951968 | |
| 16:04:36 | opendevreview | Elod Illes proposed openstack/nova stable/2024.2: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/951969 | |
| 16:05:14 | opendevreview | Elod Illes proposed openstack/nova stable/2024.1: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/951970 | |
| 16:29:22 | noonedeadpunk | I'm trying here to setup hugepages for one of my computes, but instance gets filtered out at NUMATopologyFilter with no good candidates https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L115-L121 | |
| 16:31:31 | noonedeadpunk | compute object seems to be having mempages https://paste.openstack.org/show/bRHPMmumA43t0KHk186K/ | |
| 16:33:02 | noonedeadpunk | I wonder if that's might be also due to PCI Passthrough attempt, as that's I guess what might not work with numa placement nicely? | |
| 16:34:55 | noonedeadpunk | flavor defined as `hw:cpu_policy='dedicated', hw:mem_page_size='2MB', hw:vif_multiqueue_enabled='true', trait:CUSTOM_GPU='required', pci_passthrough:alias='A10_FULL:1'` | |
| 16:35:17 | noonedeadpunk | and obviously once I unset hw:mem_page_size - scheduling passing nicely | |
| 16:35:58 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 16:57:47 | sean-k-mooney | you can have both enabled | |
| 16:58:20 | sean-k-mooney | however if you have a numa vm by default we require the pci device request by the guest to come form the same numa node as the guest cpus/memory | |
| 16:58:30 | sean-k-mooney | noonedeadpunk: you can diabel that in your pci config | |
| 17:00:37 | sean-k-mooney | noonedeadpunk: in your pci alias you can set https://docs.openstack.org/nova/latest/configuration/config.html#pci.alias numa_policy=preferred | |
| 17:00:52 | sean-k-mooney | noonedeadpunk: you can also set that in the flavor which is generally simpler | |
| 17:02:04 | sean-k-mooney | i would recomend setting hw:pci_numa_affinity_policy=preferred or hw:pci_numa_affinity_policy=socket | |
| 17:02:25 | sean-k-mooney | preferred is the most relaxed policy | |
| 17:02:54 | sean-k-mooney | i woudl also suggest using hw:mem_page_size=large | |
| 17:05:36 | noonedeadpunk | large is 1G? | |
| 17:06:11 | noonedeadpunk | let me try setting policies then... | |
| 17:06:52 | noonedeadpunk | as I tried couple of options, like hw:numa_nodes=1 | |
| 17:12:37 | noonedeadpunk | nah, didn't fly somehow... I guess best thing to do for me now - go for a weekend and check on that with fresh eyes after it :) | |
| 17:12:43 | gmaan | about 2.0 removal. I am not sure what is benefits if we keep v2.1 as min version. 2.0 is just a endpoint without any change from v2.1 so no overhead of maintenance. This will just break user using 2.0 endpoint without giving any gain | |
| 17:21:33 | sean-k-mooney | noonedeadpunk: large is anything that isnt small i.e. anything other then 4k default pagesize | |
| 17:22:10 | noonedeadpunk | aha, ok | |
| 17:22:27 | sean-k-mooney | noonedeadpunk: if you have not turned on debug yet do that on monday in the schduler | |
| 17:22:39 | sean-k-mooney | you will need that to discover why | |
| 17:22:45 | noonedeadpunk | I did that on scheduler | |
| 17:22:51 | sean-k-mooney | ack | |
| 17:23:04 | noonedeadpunk | that's where I grabbed cpu capabilities | |
| 17:23:10 | noonedeadpunk | *compute | |
| 17:23:37 | sean-k-mooney | so your flavor was askign for pinned cpus and hugepages | |
| 17:23:43 | noonedeadpunk | yeah | |
| 17:23:48 | sean-k-mooney | did you actully configure cpu_dedicated_set ? | |
| 17:23:49 | noonedeadpunk | and PCI device | |
| 17:24:10 | sean-k-mooney | in the comptue section of the nova.conf | |
| 17:24:14 | sean-k-mooney | on the compute node | |
| 17:24:27 | noonedeadpunk | nope, I did not, I just set reserved_host_cpus = 2 and that's it | |
| 17:24:41 | noonedeadpunk | but it has no VMs running, as is in the isolated aggregate | |
| 17:24:47 | sean-k-mooney | ok well that probaly the issue | |
| 17:24:55 | sean-k-mooney | what release are you testing with | |
| 17:25:01 | noonedeadpunk | Caracal | |
| 17:25:42 | noonedeadpunk | when I drop hugepages from properties - it does allocate pinned cpus though | |
| 17:26:06 | sean-k-mooney | do you have https://docs.openstack.org/nova/2024.1/configuration/config.html#workarounds.disable_fallback_pcpu_query defiend | |
| 17:26:10 | noonedeadpunk | or you saying that without hugepages it does not go through the numa filter... | |
| 17:26:30 | noonedeadpunk | nope | |
| 17:26:38 | sean-k-mooney | ok so its falling abck | |
| 17:27:16 | sean-k-mooney | so any of hugpeages, pinning or an explcit numa topolgoy will result in the schduelr usign the numa filter to validate the host | |
| 17:27:52 | sean-k-mooney | how much hugepages did you allocate on the host? and how much ram are you asking for in the vm | |
| 17:28:14 | noonedeadpunk | ok, here where the problem can actually be.... | |
| 17:28:28 | noonedeadpunk | as I kind of took example from docs for now for some POC | |
| 17:28:39 | noonedeadpunk | so it's 1024*2MB | |
| 17:28:59 | noonedeadpunk | and I'm asking for *really* big instance of 200Gb RAM | |
| 17:29:55 | noonedeadpunk | So what I should be doing - is to split available ram into blocks and that num_pages * page_size == ram? | |
| 17:30:38 | sean-k-mooney | would you mind pasting the output of virsh cabalities | |
| 17:30:49 | sean-k-mooney | i can advise based on the size of the host with that | |
| 17:31:09 | noonedeadpunk | https://paste.openstack.org/show/bYi3OvxXJbP6Grx275O7/ | |
| 17:31:22 | noonedeadpunk | but we plan to run there like... 4 VMs max? | |
| 17:32:44 | sean-k-mooney | ok so you have 2 numa nodes | |
| 17:33:21 | noonedeadpunk | yup | |
| 17:33:23 | noonedeadpunk | no HT | |
| 17:33:39 | sean-k-mooney | and you allcoate 2 GB of 2MB hugepages per numa node | |
| 17:34:00 | sean-k-mooney | <pages unit='KiB' size='2048'>1024</pages> | |
| 17:34:25 | noonedeadpunk | Right... And I guess that's where the problem is | |
| 17:34:25 | sean-k-mooney | noonedeadpunk: try creating a 1G vm that asks for 1 cpu with hw:mem_page_size=large | |
| 17:34:44 | sean-k-mooney | if that works all you need ot do is decied how you want to partion the host ram | |
| 17:34:53 | sean-k-mooney | i.e. how much to reserve exclusivly for hugepages | |
| 17:36:14 | noonedeadpunk | so just to double check where I was wrong. When I set `hw:mem_page_size='2MB'` in flavor - what needs to happen is that flavor ram / hw:mem_page_size = amount of available on compute hugepages? | |
| 17:36:26 | sean-k-mooney | looks like you have 512GB total which 500 is usable | |
| 17:36:44 | noonedeadpunk | yeah, sounds about right | |
| 17:36:59 | noonedeadpunk | so 256 on single numa node | |
| 17:37:47 | sean-k-mooney | ya i woudl start conservity ly and allocage 480GB of hugepages on that system total | |
| 17:37:48 | noonedeadpunk | but actually seeing virsh capabiltities I'm really understanding where my logic went off now | |
| 17:37:58 | noonedeadpunk | ++ | |
| 17:38:07 | sean-k-mooney | that leave 32GB for the host | |
| 17:38:25 | sean-k-mooney | whcih with only 4 vms total of 120GB each should be plent | |
| 17:39:05 | sean-k-mooney | you can obvioulsy slice that up and run more or less vms | |
| 17:39:32 | noonedeadpunk | there's a limit on pci deviuces... but also it's a poc more or less | |
| 17:39:59 | sean-k-mooney | so you need to allcoate 245760 2MB hugepages i think | |
| 17:40:37 | sean-k-mooney | hugepagesz=2M hugepages=245760 | |
| 17:41:03 | sean-k-mooney | with that said the server is large enough that you might consider using 1G pages just to have faster boot times | |
| 17:41:36 | sean-k-mooney | in which case the kernel command line would be `hugepagesz=1G hugepages=480` | |
| 17:41:51 | sean-k-mooney | you can allcoate these at runtim too but its recommend to do it at boot | |
| 17:41:58 | sean-k-mooney | doign it at runtime is best effort | |
| 17:42:12 | noonedeadpunk | oh, how can I allocate at runtime? | |
| 17:42:30 | noonedeadpunk | as that's part for osa I'm missing now :) | |
| 17:42:52 | noonedeadpunk | and I don't want to reboot hardware on grub changes there | |
| 17:42:52 | sean-k-mooney | you echo a value in sys fs | |
| 17:42:54 | noonedeadpunk | aha | |
| 17:42:56 | noonedeadpunk | right | |
| 17:42:58 | sean-k-mooney | let me get you an example | |
| 17:43:05 | sean-k-mooney | you can do it with systemd or udev rules too | |
| 17:43:42 | noonedeadpunk | I'm changing dev-hugepages.mount as well | |
| 17:44:16 | sean-k-mooney | echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:44:53 | sean-k-mooney | nova and libvirt have caches so you need to restart nova-compute and libvirt to make sure they see the change | |
| 17:45:05 | sean-k-mooney | so you echo the desired ammount | |
| 17:45:21 | sean-k-mooney | cat that value and you can see how many are there | |
| 17:45:38 | sean-k-mooney | wehn you write to it it will block until its done i think | |
| 17:47:47 | noonedeadpunk | ok, trying that | |
| 17:47:52 | sean-k-mooney | allocating 1G pages at runtim normally does nto work because there often isnt enouch free contigous 1G adress spaces | |
| 17:47:53 | noonedeadpunk | friday evening - perfect time :) | |