| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-06 | |||
| 17:24:10 | sean-k-mooney | in the comptue section of the nova.conf | |
| 17:24:14 | sean-k-mooney | on the compute node | |
| 17:24:27 | noonedeadpunk | nope, I did not, I just set reserved_host_cpus = 2 and that's it | |
| 17:24:41 | noonedeadpunk | but it has no VMs running, as is in the isolated aggregate | |
| 17:24:47 | sean-k-mooney | ok well that probaly the issue | |
| 17:24:55 | sean-k-mooney | what release are you testing with | |
| 17:25:01 | noonedeadpunk | Caracal | |
| 17:25:42 | noonedeadpunk | when I drop hugepages from properties - it does allocate pinned cpus though | |
| 17:26:06 | sean-k-mooney | do you have https://docs.openstack.org/nova/2024.1/configuration/config.html#workarounds.disable_fallback_pcpu_query defiend | |
| 17:26:10 | noonedeadpunk | or you saying that without hugepages it does not go through the numa filter... | |
| 17:26:30 | noonedeadpunk | nope | |
| 17:26:38 | sean-k-mooney | ok so its falling abck | |
| 17:27:16 | sean-k-mooney | so any of hugpeages, pinning or an explcit numa topolgoy will result in the schduelr usign the numa filter to validate the host | |
| 17:27:52 | sean-k-mooney | how much hugepages did you allocate on the host? and how much ram are you asking for in the vm | |
| 17:28:14 | noonedeadpunk | ok, here where the problem can actually be.... | |
| 17:28:28 | noonedeadpunk | as I kind of took example from docs for now for some POC | |
| 17:28:39 | noonedeadpunk | so it's 1024*2MB | |
| 17:28:59 | noonedeadpunk | and I'm asking for *really* big instance of 200Gb RAM | |
| 17:29:55 | noonedeadpunk | So what I should be doing - is to split available ram into blocks and that num_pages * page_size == ram? | |
| 17:30:38 | sean-k-mooney | would you mind pasting the output of virsh cabalities | |
| 17:30:49 | sean-k-mooney | i can advise based on the size of the host with that | |
| 17:31:09 | noonedeadpunk | https://paste.openstack.org/show/bYi3OvxXJbP6Grx275O7/ | |
| 17:31:22 | noonedeadpunk | but we plan to run there like... 4 VMs max? | |
| 17:32:44 | sean-k-mooney | ok so you have 2 numa nodes | |
| 17:33:21 | noonedeadpunk | yup | |
| 17:33:23 | noonedeadpunk | no HT | |
| 17:33:39 | sean-k-mooney | and you allcoate 2 GB of 2MB hugepages per numa node | |
| 17:34:00 | sean-k-mooney | <pages unit='KiB' size='2048'>1024</pages> | |
| 17:34:25 | sean-k-mooney | noonedeadpunk: try creating a 1G vm that asks for 1 cpu with hw:mem_page_size=large | |
| 17:34:25 | noonedeadpunk | Right... And I guess that's where the problem is | |
| 17:34:44 | sean-k-mooney | if that works all you need ot do is decied how you want to partion the host ram | |
| 17:34:53 | sean-k-mooney | i.e. how much to reserve exclusivly for hugepages | |
| 17:36:14 | noonedeadpunk | so just to double check where I was wrong. When I set `hw:mem_page_size='2MB'` in flavor - what needs to happen is that flavor ram / hw:mem_page_size = amount of available on compute hugepages? | |
| 17:36:26 | sean-k-mooney | looks like you have 512GB total which 500 is usable | |
| 17:36:44 | noonedeadpunk | yeah, sounds about right | |
| 17:36:59 | noonedeadpunk | so 256 on single numa node | |
| 17:37:47 | sean-k-mooney | ya i woudl start conservity ly and allocage 480GB of hugepages on that system total | |
| 17:37:48 | noonedeadpunk | but actually seeing virsh capabiltities I'm really understanding where my logic went off now | |
| 17:37:58 | noonedeadpunk | ++ | |
| 17:38:07 | sean-k-mooney | that leave 32GB for the host | |
| 17:38:25 | sean-k-mooney | whcih with only 4 vms total of 120GB each should be plent | |
| 17:39:05 | sean-k-mooney | you can obvioulsy slice that up and run more or less vms | |
| 17:39:32 | noonedeadpunk | there's a limit on pci deviuces... but also it's a poc more or less | |
| 17:39:59 | sean-k-mooney | so you need to allcoate 245760 2MB hugepages i think | |
| 17:40:37 | sean-k-mooney | hugepagesz=2M hugepages=245760 | |
| 17:41:03 | sean-k-mooney | with that said the server is large enough that you might consider using 1G pages just to have faster boot times | |
| 17:41:36 | sean-k-mooney | in which case the kernel command line would be `hugepagesz=1G hugepages=480` | |
| 17:41:51 | sean-k-mooney | you can allcoate these at runtim too but its recommend to do it at boot | |
| 17:41:58 | sean-k-mooney | doign it at runtime is best effort | |
| 17:42:12 | noonedeadpunk | oh, how can I allocate at runtime? | |
| 17:42:30 | noonedeadpunk | as that's part for osa I'm missing now :) | |
| 17:42:52 | sean-k-mooney | you echo a value in sys fs | |
| 17:42:52 | noonedeadpunk | and I don't want to reboot hardware on grub changes there | |
| 17:42:54 | noonedeadpunk | aha | |
| 17:42:56 | noonedeadpunk | right | |
| 17:42:58 | sean-k-mooney | let me get you an example | |
| 17:43:05 | sean-k-mooney | you can do it with systemd or udev rules too | |
| 17:43:42 | noonedeadpunk | I'm changing dev-hugepages.mount as well | |
| 17:44:16 | sean-k-mooney | echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:44:53 | sean-k-mooney | nova and libvirt have caches so you need to restart nova-compute and libvirt to make sure they see the change | |
| 17:45:05 | sean-k-mooney | so you echo the desired ammount | |
| 17:45:21 | sean-k-mooney | cat that value and you can see how many are there | |
| 17:45:38 | sean-k-mooney | wehn you write to it it will block until its done i think | |
| 17:47:47 | noonedeadpunk | ok, trying that | |
| 17:47:52 | sean-k-mooney | allocating 1G pages at runtim normally does nto work because there often isnt enouch free contigous 1G adress spaces | |
| 17:47:53 | noonedeadpunk | friday evening - perfect time :) | |
| 17:48:19 | sean-k-mooney | noonedeadpunk: well we can pick it back up on monday too. i was just leaving anyway | |
| 17:48:38 | sean-k-mooney | but ya i think nova was workign fine you just asked for more resocues then were aviable | |
| 17:48:38 | noonedeadpunk | I think I will try to go with 2MB still, as I can recall also your advice about live migrations | |
| 17:48:57 | noonedeadpunk | that qemu would operate on page size for migration | |
| 17:49:05 | noonedeadpunk | so I'm not sure I wanna do it that big | |
| 17:49:07 | sean-k-mooney | ya it jsut tends to work betterr for vms in general | |
| 17:49:26 | noonedeadpunk | Yeah, I think this what was the issue | |
| 17:49:59 | noonedeadpunk | fwiw, I also got nowhere with migrations crashing on qemu, when more then 80% of memory is used by stress... | |
| 17:50:18 | sean-k-mooney | oh anything interssting | |
| 17:50:27 | sean-k-mooney | on you said no where | |
| 17:50:29 | sean-k-mooney | ok | |
| 17:50:53 | noonedeadpunk | not really.. except that 20GB for stress out of 32GB VM is working, but 14GB stress out of 16GB VM crashing | |
| 17:51:15 | noonedeadpunk | so it's not amout of memory or intensity - it's somehow percentage of VM memory which matters | |
| 17:51:23 | noonedeadpunk | which makes no sense to me at least | |
| 17:51:36 | sean-k-mooney | ya that supper odd to me too | |
| 17:51:58 | noonedeadpunk | I also tried post copy with no difference | |
| 17:51:59 | sean-k-mooney | and it was qemu that was rashing not the guest kernel OOMing right | |
| 17:52:12 | noonedeadpunk | yeah | |
| 17:52:27 | noonedeadpunk | migration failed, reason=crashed in qemu logs | |
| 17:52:53 | sean-k-mooney | ya i havnt seen that or herd other reports | |
| 17:53:04 | sean-k-mooney | that to me say qemu bug | |
| 17:53:05 | noonedeadpunk | and tried on different generations of intels | |
| 17:53:08 | sean-k-mooney | but i dont knwo what it could be | |
| 17:55:26 | noonedeadpunk | was very easy to reproduce - get 16gb VM on hypervisor with ubuntu 22.04 or 24.04, on VM run `screen stress-ng --vm 4 --vm-bytes 14G --vm-keep`; live migrate -> profit | |
| 17:56:26 | noonedeadpunk | almost 100% reproduction rate for me | |
| 17:56:36 | noonedeadpunk | maybe 98... not sure | |
| 18:04:00 | noonedeadpunk | yeah, once I incvreazed amount of pages, instance expectedly spawning | |
| 18:06:06 | noonedeadpunk | so thanks a ton for help! | |
| 18:06:21 | noonedeadpunk | these just somehow did not add up for me | |
| 18:08:38 | gmaan | stephenfin: stephenfin gibi I replied on the v2.0 removal spec. It will break users as v2.1 is not same as v2.0 from user perspective - https://review.opendev.org/c/openstack/nova-specs/+/951949 | |
| 18:08:59 | gmaan | if somehow we can know no one use v2.0 then I will be happy to remove | |
| 18:09:50 | gmaan | and my take that time will be to bump the min microversion also but we have to be very careful about this change as it will be big impact on users | |
| 18:16:21 | sean-k-mooney | i guess installer can choose not to deploy 2.0 today we dotn in our new installer and other tools already dont deploy it too and have not for a while | |
| 19:08:04 | noonedeadpunk | In OSA v2.1 seems to be deployed like always | |