| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-06 | |||
| 17:22:45 | noonedeadpunk | I did that on scheduler | |
| 17:22:51 | sean-k-mooney | ack | |
| 17:23:04 | noonedeadpunk | that's where I grabbed cpu capabilities | |
| 17:23:10 | noonedeadpunk | *compute | |
| 17:23:37 | sean-k-mooney | so your flavor was askign for pinned cpus and hugepages | |
| 17:23:43 | noonedeadpunk | yeah | |
| 17:23:48 | sean-k-mooney | did you actully configure cpu_dedicated_set ? | |
| 17:23:49 | noonedeadpunk | and PCI device | |
| 17:24:10 | sean-k-mooney | in the comptue section of the nova.conf | |
| 17:24:14 | sean-k-mooney | on the compute node | |
| 17:24:27 | noonedeadpunk | nope, I did not, I just set reserved_host_cpus = 2 and that's it | |
| 17:24:41 | noonedeadpunk | but it has no VMs running, as is in the isolated aggregate | |
| 17:24:47 | sean-k-mooney | ok well that probaly the issue | |
| 17:24:55 | sean-k-mooney | what release are you testing with | |
| 17:25:01 | noonedeadpunk | Caracal | |
| 17:25:42 | noonedeadpunk | when I drop hugepages from properties - it does allocate pinned cpus though | |
| 17:26:06 | sean-k-mooney | do you have https://docs.openstack.org/nova/2024.1/configuration/config.html#workarounds.disable_fallback_pcpu_query defiend | |
| 17:26:10 | noonedeadpunk | or you saying that without hugepages it does not go through the numa filter... | |
| 17:26:30 | noonedeadpunk | nope | |
| 17:26:38 | sean-k-mooney | ok so its falling abck | |
| 17:27:16 | sean-k-mooney | so any of hugpeages, pinning or an explcit numa topolgoy will result in the schduelr usign the numa filter to validate the host | |
| 17:27:52 | sean-k-mooney | how much hugepages did you allocate on the host? and how much ram are you asking for in the vm | |
| 17:28:14 | noonedeadpunk | ok, here where the problem can actually be.... | |
| 17:28:28 | noonedeadpunk | as I kind of took example from docs for now for some POC | |
| 17:28:39 | noonedeadpunk | so it's 1024*2MB | |
| 17:28:59 | noonedeadpunk | and I'm asking for *really* big instance of 200Gb RAM | |
| 17:29:55 | noonedeadpunk | So what I should be doing - is to split available ram into blocks and that num_pages * page_size == ram? | |
| 17:30:38 | sean-k-mooney | would you mind pasting the output of virsh cabalities | |
| 17:30:49 | sean-k-mooney | i can advise based on the size of the host with that | |
| 17:31:09 | noonedeadpunk | https://paste.openstack.org/show/bYi3OvxXJbP6Grx275O7/ | |
| 17:31:22 | noonedeadpunk | but we plan to run there like... 4 VMs max? | |
| 17:32:44 | sean-k-mooney | ok so you have 2 numa nodes | |
| 17:33:21 | noonedeadpunk | yup | |
| 17:33:23 | noonedeadpunk | no HT | |
| 17:33:39 | sean-k-mooney | and you allcoate 2 GB of 2MB hugepages per numa node | |
| 17:34:00 | sean-k-mooney | <pages unit='KiB' size='2048'>1024</pages> | |
| 17:34:25 | noonedeadpunk | Right... And I guess that's where the problem is | |
| 17:34:25 | sean-k-mooney | noonedeadpunk: try creating a 1G vm that asks for 1 cpu with hw:mem_page_size=large | |
| 17:34:44 | sean-k-mooney | if that works all you need ot do is decied how you want to partion the host ram | |
| 17:34:53 | sean-k-mooney | i.e. how much to reserve exclusivly for hugepages | |
| 17:36:14 | noonedeadpunk | so just to double check where I was wrong. When I set `hw:mem_page_size='2MB'` in flavor - what needs to happen is that flavor ram / hw:mem_page_size = amount of available on compute hugepages? | |
| 17:36:26 | sean-k-mooney | looks like you have 512GB total which 500 is usable | |
| 17:36:44 | noonedeadpunk | yeah, sounds about right | |
| 17:36:59 | noonedeadpunk | so 256 on single numa node | |
| 17:37:47 | sean-k-mooney | ya i woudl start conservity ly and allocage 480GB of hugepages on that system total | |
| 17:37:48 | noonedeadpunk | but actually seeing virsh capabiltities I'm really understanding where my logic went off now | |
| 17:37:58 | noonedeadpunk | ++ | |
| 17:38:07 | sean-k-mooney | that leave 32GB for the host | |
| 17:38:25 | sean-k-mooney | whcih with only 4 vms total of 120GB each should be plent | |
| 17:39:05 | sean-k-mooney | you can obvioulsy slice that up and run more or less vms | |
| 17:39:32 | noonedeadpunk | there's a limit on pci deviuces... but also it's a poc more or less | |
| 17:39:59 | sean-k-mooney | so you need to allcoate 245760 2MB hugepages i think | |
| 17:40:37 | sean-k-mooney | hugepagesz=2M hugepages=245760 | |
| 17:41:03 | sean-k-mooney | with that said the server is large enough that you might consider using 1G pages just to have faster boot times | |
| 17:41:36 | sean-k-mooney | in which case the kernel command line would be `hugepagesz=1G hugepages=480` | |
| 17:41:51 | sean-k-mooney | you can allcoate these at runtim too but its recommend to do it at boot | |
| 17:41:58 | sean-k-mooney | doign it at runtime is best effort | |
| 17:42:12 | noonedeadpunk | oh, how can I allocate at runtime? | |
| 17:42:30 | noonedeadpunk | as that's part for osa I'm missing now :) | |
| 17:42:52 | noonedeadpunk | and I don't want to reboot hardware on grub changes there | |
| 17:42:52 | sean-k-mooney | you echo a value in sys fs | |
| 17:42:54 | noonedeadpunk | aha | |
| 17:42:56 | noonedeadpunk | right | |
| 17:42:58 | sean-k-mooney | let me get you an example | |
| 17:43:05 | sean-k-mooney | you can do it with systemd or udev rules too | |
| 17:43:42 | noonedeadpunk | I'm changing dev-hugepages.mount as well | |
| 17:44:16 | sean-k-mooney | echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:44:53 | sean-k-mooney | nova and libvirt have caches so you need to restart nova-compute and libvirt to make sure they see the change | |
| 17:45:05 | sean-k-mooney | so you echo the desired ammount | |
| 17:45:21 | sean-k-mooney | cat that value and you can see how many are there | |
| 17:45:38 | sean-k-mooney | wehn you write to it it will block until its done i think | |
| 17:47:47 | noonedeadpunk | ok, trying that | |
| 17:47:52 | sean-k-mooney | allocating 1G pages at runtim normally does nto work because there often isnt enouch free contigous 1G adress spaces | |
| 17:47:53 | noonedeadpunk | friday evening - perfect time :) | |
| 17:48:19 | sean-k-mooney | noonedeadpunk: well we can pick it back up on monday too. i was just leaving anyway | |
| 17:48:38 | noonedeadpunk | I think I will try to go with 2MB still, as I can recall also your advice about live migrations | |
| 17:48:38 | sean-k-mooney | but ya i think nova was workign fine you just asked for more resocues then were aviable | |
| 17:48:57 | noonedeadpunk | that qemu would operate on page size for migration | |
| 17:49:05 | noonedeadpunk | so I'm not sure I wanna do it that big | |
| 17:49:07 | sean-k-mooney | ya it jsut tends to work betterr for vms in general | |
| 17:49:26 | noonedeadpunk | Yeah, I think this what was the issue | |
| 17:49:59 | noonedeadpunk | fwiw, I also got nowhere with migrations crashing on qemu, when more then 80% of memory is used by stress... | |
| 17:50:18 | sean-k-mooney | oh anything interssting | |
| 17:50:27 | sean-k-mooney | on you said no where | |
| 17:50:29 | sean-k-mooney | ok | |
| 17:50:53 | noonedeadpunk | not really.. except that 20GB for stress out of 32GB VM is working, but 14GB stress out of 16GB VM crashing | |
| 17:51:15 | noonedeadpunk | so it's not amout of memory or intensity - it's somehow percentage of VM memory which matters | |
| 17:51:23 | noonedeadpunk | which makes no sense to me at least | |
| 17:51:36 | sean-k-mooney | ya that supper odd to me too | |
| 17:51:58 | noonedeadpunk | I also tried post copy with no difference | |
| 17:51:59 | sean-k-mooney | and it was qemu that was rashing not the guest kernel OOMing right | |
| 17:52:12 | noonedeadpunk | yeah | |
| 17:52:27 | noonedeadpunk | migration failed, reason=crashed in qemu logs | |
| 17:52:53 | sean-k-mooney | ya i havnt seen that or herd other reports | |
| 17:53:04 | sean-k-mooney | that to me say qemu bug | |
| 17:53:05 | noonedeadpunk | and tried on different generations of intels | |
| 17:53:08 | sean-k-mooney | but i dont knwo what it could be | |
| 17:55:26 | noonedeadpunk | was very easy to reproduce - get 16gb VM on hypervisor with ubuntu 22.04 or 24.04, on VM run `screen stress-ng --vm 4 --vm-bytes 14G --vm-keep`; live migrate -> profit | |
| 17:56:26 | noonedeadpunk | almost 100% reproduction rate for me | |
| 17:56:36 | noonedeadpunk | maybe 98... not sure | |