| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-06 | |||
| 17:34:00 | sean-k-mooney | <pages unit='KiB' size='2048'>1024</pages> | |
| 17:34:25 | sean-k-mooney | noonedeadpunk: try creating a 1G vm that asks for 1 cpu with hw:mem_page_size=large | |
| 17:34:25 | noonedeadpunk | Right... And I guess that's where the problem is | |
| 17:34:44 | sean-k-mooney | if that works all you need ot do is decied how you want to partion the host ram | |
| 17:34:53 | sean-k-mooney | i.e. how much to reserve exclusivly for hugepages | |
| 17:36:14 | noonedeadpunk | so just to double check where I was wrong. When I set `hw:mem_page_size='2MB'` in flavor - what needs to happen is that flavor ram / hw:mem_page_size = amount of available on compute hugepages? | |
| 17:36:26 | sean-k-mooney | looks like you have 512GB total which 500 is usable | |
| 17:36:44 | noonedeadpunk | yeah, sounds about right | |
| 17:36:59 | noonedeadpunk | so 256 on single numa node | |
| 17:37:47 | sean-k-mooney | ya i woudl start conservity ly and allocage 480GB of hugepages on that system total | |
| 17:37:48 | noonedeadpunk | but actually seeing virsh capabiltities I'm really understanding where my logic went off now | |
| 17:37:58 | noonedeadpunk | ++ | |
| 17:38:07 | sean-k-mooney | that leave 32GB for the host | |
| 17:38:25 | sean-k-mooney | whcih with only 4 vms total of 120GB each should be plent | |
| 17:39:05 | sean-k-mooney | you can obvioulsy slice that up and run more or less vms | |
| 17:39:32 | noonedeadpunk | there's a limit on pci deviuces... but also it's a poc more or less | |
| 17:39:59 | sean-k-mooney | so you need to allcoate 245760 2MB hugepages i think | |
| 17:40:37 | sean-k-mooney | hugepagesz=2M hugepages=245760 | |
| 17:41:03 | sean-k-mooney | with that said the server is large enough that you might consider using 1G pages just to have faster boot times | |
| 17:41:36 | sean-k-mooney | in which case the kernel command line would be `hugepagesz=1G hugepages=480` | |
| 17:41:51 | sean-k-mooney | you can allcoate these at runtim too but its recommend to do it at boot | |
| 17:41:58 | sean-k-mooney | doign it at runtime is best effort | |
| 17:42:12 | noonedeadpunk | oh, how can I allocate at runtime? | |
| 17:42:30 | noonedeadpunk | as that's part for osa I'm missing now :) | |
| 17:42:52 | sean-k-mooney | you echo a value in sys fs | |
| 17:42:52 | noonedeadpunk | and I don't want to reboot hardware on grub changes there | |
| 17:42:54 | noonedeadpunk | aha | |
| 17:42:56 | noonedeadpunk | right | |
| 17:42:58 | sean-k-mooney | let me get you an example | |
| 17:43:05 | sean-k-mooney | you can do it with systemd or udev rules too | |
| 17:43:42 | noonedeadpunk | I'm changing dev-hugepages.mount as well | |
| 17:44:16 | sean-k-mooney | echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:44:53 | sean-k-mooney | nova and libvirt have caches so you need to restart nova-compute and libvirt to make sure they see the change | |
| 17:45:05 | sean-k-mooney | so you echo the desired ammount | |
| 17:45:21 | sean-k-mooney | cat that value and you can see how many are there | |
| 17:45:38 | sean-k-mooney | wehn you write to it it will block until its done i think | |
| 17:47:47 | noonedeadpunk | ok, trying that | |
| 17:47:52 | sean-k-mooney | allocating 1G pages at runtim normally does nto work because there often isnt enouch free contigous 1G adress spaces | |
| 17:47:53 | noonedeadpunk | friday evening - perfect time :) | |
| 17:48:19 | sean-k-mooney | noonedeadpunk: well we can pick it back up on monday too. i was just leaving anyway | |
| 17:48:38 | sean-k-mooney | but ya i think nova was workign fine you just asked for more resocues then were aviable | |
| 17:48:38 | noonedeadpunk | I think I will try to go with 2MB still, as I can recall also your advice about live migrations | |
| 17:48:57 | noonedeadpunk | that qemu would operate on page size for migration | |
| 17:49:05 | noonedeadpunk | so I'm not sure I wanna do it that big | |
| 17:49:07 | sean-k-mooney | ya it jsut tends to work betterr for vms in general | |
| 17:49:26 | noonedeadpunk | Yeah, I think this what was the issue | |
| 17:49:59 | noonedeadpunk | fwiw, I also got nowhere with migrations crashing on qemu, when more then 80% of memory is used by stress... | |
| 17:50:18 | sean-k-mooney | oh anything interssting | |
| 17:50:27 | sean-k-mooney | on you said no where | |
| 17:50:29 | sean-k-mooney | ok | |
| 17:50:53 | noonedeadpunk | not really.. except that 20GB for stress out of 32GB VM is working, but 14GB stress out of 16GB VM crashing | |
| 17:51:15 | noonedeadpunk | so it's not amout of memory or intensity - it's somehow percentage of VM memory which matters | |
| 17:51:23 | noonedeadpunk | which makes no sense to me at least | |
| 17:51:36 | sean-k-mooney | ya that supper odd to me too | |
| 17:51:58 | noonedeadpunk | I also tried post copy with no difference | |
| 17:51:59 | sean-k-mooney | and it was qemu that was rashing not the guest kernel OOMing right | |
| 17:52:12 | noonedeadpunk | yeah | |
| 17:52:27 | noonedeadpunk | migration failed, reason=crashed in qemu logs | |
| 17:52:53 | sean-k-mooney | ya i havnt seen that or herd other reports | |
| 17:53:04 | sean-k-mooney | that to me say qemu bug | |
| 17:53:05 | noonedeadpunk | and tried on different generations of intels | |
| 17:53:08 | sean-k-mooney | but i dont knwo what it could be | |
| 17:55:26 | noonedeadpunk | was very easy to reproduce - get 16gb VM on hypervisor with ubuntu 22.04 or 24.04, on VM run `screen stress-ng --vm 4 --vm-bytes 14G --vm-keep`; live migrate -> profit | |
| 17:56:26 | noonedeadpunk | almost 100% reproduction rate for me | |
| 17:56:36 | noonedeadpunk | maybe 98... not sure | |
| 18:04:00 | noonedeadpunk | yeah, once I incvreazed amount of pages, instance expectedly spawning | |
| 18:06:06 | noonedeadpunk | so thanks a ton for help! | |
| 18:06:21 | noonedeadpunk | these just somehow did not add up for me | |
| 18:08:38 | gmaan | stephenfin: stephenfin gibi I replied on the v2.0 removal spec. It will break users as v2.1 is not same as v2.0 from user perspective - https://review.opendev.org/c/openstack/nova-specs/+/951949 | |
| 18:08:59 | gmaan | if somehow we can know no one use v2.0 then I will be happy to remove | |
| 18:09:50 | gmaan | and my take that time will be to bump the min microversion also but we have to be very careful about this change as it will be big impact on users | |
| 18:16:21 | sean-k-mooney | i guess installer can choose not to deploy 2.0 today we dotn in our new installer and other tools already dont deploy it too and have not for a while | |
| 19:08:04 | noonedeadpunk | In OSA v2.1 seems to be deployed like always | |
| 19:08:24 | noonedeadpunk | or well. Since Liberty: https://review.opendev.org/c/openstack/openstack-ansible/+/227839 | |
| 19:17:21 | gmaan | melwitt: about project-admin comment in manager role spec, sean-k-mooney and I replied but I agree to add some explanation in spec. While I am doing that, I wanted to check if it is ok to add that explanation in follow up and we merge current change or you would like to see that in this change itself. https://review.opendev.org/c/openstack/nova-specs/+/937650/9/specs/2025.2/approved/policy-manager-role-default.rst#29 | |
| 19:17:25 | gmaan | either is ok for me | |
| 20:17:58 | opendevreview | Merged openstack/nova master: api: Stop using wsgi.Controller.api_version to switch between API versions https://review.opendev.org/c/openstack/nova/+/936366 | |
| 20:42:23 | melwitt | gmaan: cool thanks, I'm reading through. I don't think it's necessary to follow up or change it, it was just a bit of context I was missing or didn't already know | |
| 20:43:14 | melwitt | this whole time I thought project-admin meant admin but only for a given project. if it's global, then I don't understand what's the difference between system-admin and project-admin then 😛 | |
| 20:44:40 | melwitt | but the important point is that if project-admin is global then that makes it obvious the usefulness of project-manager | |
| 21:49:37 | gmaan | melwitt: yeah, by doing back and forth on those admin, we end up keeping only one admin (legacy admin) who can do things on system level | |
| 21:51:37 | gmaan | if we could have system admin then it could make admin at different level more clear/safe to use but that confused operator instead of helpful :) | |
| 22:38:10 | opendevreview | Merged openstack/nova-specs master: Propose API policy manager role spec https://review.opendev.org/c/openstack/nova-specs/+/937650 | |
| #openstack-nova - 2025-06-07 | |||
| 01:45:26 | opendevreview | melanie witt proposed openstack/nova stable/2025.1: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952003 | |
| 01:46:41 | opendevreview | melanie witt proposed openstack/nova stable/2024.2: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952004 | |
| 01:47:25 | opendevreview | melanie witt proposed openstack/nova stable/2024.2: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952004 | |
| 01:48:47 | opendevreview | melanie witt proposed openstack/nova stable/2024.1: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952005 | |
| 02:53:05 | opendevreview | Merged openstack/nova master: api: Only check minimum API version https://review.opendev.org/c/openstack/nova/+/940325 | |
| 07:14:41 | opendevreview | Merged openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367 | |
| 16:20:53 | opendevreview | Merged openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368 | |
| #openstack-nova - 2025-06-08 | |||
| 16:38:35 | opendevreview | Merged openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369 | |
| 22:51:31 | opendevreview | Merged openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598 | |
| 23:14:09 | opendevreview | Merged openstack/nova master: doc: Add missing API samples https://review.opendev.org/c/openstack/nova/+/937012 | |
| #openstack-nova - 2025-06-09 | |||
| 11:22:50 | stephenfin | gmaan: When you're around, I think I've addressed your comments (via follow-ups) on the following: https://review.opendev.org/c/openstack/nova/+/937047/ https://review.opendev.org/c/openstack/nova/+/937048/ https://review.opendev.org/c/openstack/nova/+/937246/ | |
| 11:23:13 | stephenfin | and sean-k-mooney: you can probably re-add your +2 on https://review.opendev.org/c/openstack/nova/+/937048/ also since I have a follow-up now | |
| 11:26:36 | sean-k-mooney | ack so it was just in the test code | |
| 11:27:23 | sean-k-mooney | and only 5 tests at that | |
| 11:28:37 | sean-k-mooney | stephenfin: done | |
| 14:21:18 | stephenfin | Uggla: On https://review.opendev.org/c/openstack/nova/+/940114, I had intentionally included since-removed values in the enum ('xvpvnc') since the schemas need to work for all versions of Nova. Does that make sense? | |
| 14:22:41 | stephenfin | IMO, the api-ref should probably list those also since that's unversioned. I removed them I2f7f2379d0cd54e4d0a91008ddb44858cfc5a4cf. Perhaps I should add them back but with a warning that they won't work on modern (>= Ussuri) nova | |