Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-18
15:14:59 jamespage gibi: nope no extra specs at all
15:15:13 artom jamespage, wait, that just sounds like there's not enough RAM
15:16:11 jamespage artom: I quite agree there is not enough - but I also expected the instance not to be scheduled to a hypervisor with this memory configuration
15:16:29 artom jamespage, ah, so scheduling passes, but the instance doesn't boot because not enough RAM
15:16:57 jamespage well it does - it then sucks all of the ram up and gets OOM'ed
15:17:31 artom That's... I think that's a known issue? In the sense that Nova doesn't correctly track memory when it's a mix of hugepages and "normal" page size
15:19:37 gibi artom: yeah I think if the small pages instance doesn't request any numa related things then the NUMATopologyFilter is not applied on that instance
15:20:19 gibi but I'm sure sean-k-mooney know a lot more about this
15:20:25 gibi but most of the RH folks on PTO today
15:20:48 sean-k-mooney i just sat down to do something else
15:20:52 artom gibi, it's not even that, I think we treat all memory as available when scheduling, but when the instance gets on the host the huge pages obviously cannot be allocated to it
15:21:02 sean-k-mooney somethign about large guests?
15:21:31 gibi sh*t, sorry
15:21:46 sean-k-mooney if you dont use hw:mem_page_size=small or 4k then the numa code does not run
15:21:48 sean-k-mooney its
15:21:50 sean-k-mooney fine
15:22:00 artom Well, it's on his shoulders as well, ignoring IRC is always an option :P
15:22:06 sean-k-mooney i was just goign to check my home insurance renewal
15:22:14 gibi sean-k-mooney: it is about mixing NUMA and non NUMA guests
15:22:37 sean-k-mooney ya that is not supported today
15:22:38 gibi jamespage: above sees memory overallocation issues
15:23:07 gibi sean-k-mooney: so you say that if hw:mem_page_size=small is added then we allow mixing small and large instances?
15:23:10 sean-k-mooney you must not under any cirucmstance mix numa and non numa guest or it will break all our memory tracking
15:23:18 sean-k-mooney yes
15:23:32 sean-k-mooney if you use hw:mem_page_size=small its fine
15:23:35 sean-k-mooney brb
15:25:09 sean-k-mooney tl;dr is using hw:mem_page_small will numa affine the guest and track the memory correctly
15:25:19 sean-k-mooney if you dont it will float and cause random OOM events
15:26:03 gibi thanks!
15:26:04 sean-k-mooney placement is not enough to save us since the OOM reaper runs per numa node and also hugepage guest dont supprot oversubsricption
15:26:26 sean-k-mooney so if if you have memeroy over subscrion it wont really work right
15:27:22 sean-k-mooney we have something about this in our downstream docs but basically we say use hostaggreates to prevnt mixing numa and non numa guests
15:29:16 sean-k-mooney jamespage: if you have enuch swap by the way those other instace coudl technicaly boot but the system would likely be unstable so its better to avoid that
15:29:20 jamespage sorry - power outage just at the wrong moment
15:30:33 gibi jamespage: https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html here are the IRC logs if you are dropped
15:31:31 sean-k-mooney jamespage: the reason it can be schduled to the host is in placemnt we report total ram and then only reserver what set in host_reserver_ram or what ever that option is called
15:31:59 sean-k-mooney jamespage: we dont currently automaticlly reserve the hugepage memrory because we dont currenlty track that as a seperate pool
15:32:25 sean-k-mooney so the schduelr is only looking at total-reserved not total -reserved - hugepages
15:33:23 sean-k-mooney actully if its a recent release we dont even have a ram filter anymore so its just placment that is checking
15:33:41 jamespage sean-k-mooney: having read most of the code that was my hunch so thanks for confirming...
15:34:09 sean-k-mooney so ya that is why we say you cant mix numa and non numa today
15:34:23 sean-k-mooney if we ever get to tracking numa in placment then it will fix that
15:34:43 sean-k-mooney sicne we will have a seperate pool per page size
15:34:54 sean-k-mooney and or numa node
15:36:21 sean-k-mooney we proably could paper over this temporally with a filter similar to the nuam one or even enhance the numa one to work with non numa instance but right now we just tell people not ot do it
15:42:29 artom I guess another option would be to tell operators to include the amount of hugepages in reserved_host_ram?
15:42:34 artom It's... weird, but should work?
15:42:51 sean-k-mooney no
15:43:09 sean-k-mooney it will break schduling
15:43:10 artom Wouldn't it just prevent scheduling to a host if it doesn't have enough "normal" RAM?
15:43:44 sean-k-mooney yes but you vould not use the hugepages and normal ram
15:44:03 artom Oh, right
15:44:14 sean-k-mooney you coul only use total - reserved
15:44:49 sean-k-mooney so that is what you do if you are using the hugepage on the host
15:45:07 sean-k-mooney + the huge page reserved option
15:50:52 sean-k-mooney so we coudl set max allocation size in placement = to total -hugepages
15:51:08 sean-k-mooney that would help
15:52:11 jamespage side effect to that would be that you could only ever allocate total - hugepages on that hypervisor no?
15:52:29 sean-k-mooney no
15:53:29 jamespage oh that's the max allocation size - driving the maximum single instance footprint - I see
15:53:45 sean-k-mooney https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-inventory-detail#update-resource-provider-inventory
15:53:48 sean-k-mooney yes
15:53:51 sean-k-mooney max_unit
15:54:22 sean-k-mooney sorry was just on the phone
15:54:31 sean-k-mooney so ya total woudl be set to total memroy
15:54:37 sean-k-mooney reserved = reserved form config
15:55:09 sean-k-mooney but if we set max_unit to total-reserved-hugepages that should help
15:55:35 sean-k-mooney although it would have to be the larger of
15:55:49 sean-k-mooney total-reserved-hugepages and hugepages
15:56:30 sean-k-mooney to not limit the size of hugepage guests
15:56:51 sean-k-mooney jamespage: is it a resent release
15:57:07 sean-k-mooney if so im not sure if you can use provider.yaml to twaak the make unit
15:57:26 sean-k-mooney gibi: do you rememebr if that can tweak non custom resouce classes?
15:57:55 jamespage ussuri for this particular deploy
15:58:50 gibi Only CUSTOM_* resource classes and traits may be managed this way.
15:58:56 gibi so no
15:59:00 gibi https://docs.openstack.org/nova/latest/admin/managing-resource-providers.html
15:59:07 sean-k-mooney ya ok
15:59:13 sean-k-mooney it was afetr ussuri anyway
15:59:51 sean-k-mooney jamespage: best we likely could do is a config option that could posible be backported to set or clamp the max unit
16:00:16 sean-k-mooney or just not mix for now
16:00:17 gibi we merged in victoria
16:01:51 sean-k-mooney jamespage: sorry i cant really be more help there. you could write a custom filter to do the same thing out of tree and load that in the env since its plugable
16:02:36 sean-k-mooney for existign deployment that is the only way to make mixing numa/non numa on the same host somewhat ok but it still risks OOM events
16:02:36 jamespage sean-k-mooney: no worries - you have been more that helpful in getting my knowledge up to speed in this area :)
16:03:12 sean-k-mooney ok i need to go do some other things and my home insure is not done so ill be afk now until monday
16:03:24 sean-k-mooney enjoy your weekends folks o/
16:04:17 gibi o/
16:51:35 gibi chateaulav: sorry I run out of time today. I will try to look at the issue tomorrow
16:51:38 gibi o/
16:52:38 chateaulav thats sounds good, i think im in the right place and have been playing around with the tests for compute node.
18:33:19 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: WIP Improve remote-managed port test coverage https://review.opendev.org/c/openstack/nova/+/829974
18:34:25 opendevreview Rajat Dhasmana proposed openstack/nova master: WIP: Add support for volume backed server rebuild https://review.opendev.org/c/openstack/nova/+/820368
21:03:52 opendevreview Artom Lifshitz proposed openstack/nova master: block_device_info: Add swap to inline https://review.opendev.org/c/openstack/nova/+/826523
21:03:53 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt: Remove defunct comment https://review.opendev.org/c/openstack/nova/+/826525
21:03:53 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt: Improve creating images INFO log https://review.opendev.org/c/openstack/nova/+/826524
21:03:54 opendevreview Artom Lifshitz proposed openstack/nova master: image_meta: Add ephemeral encryption properties https://review.opendev.org/c/openstack/nova/+/760454
21:03:54 opendevreview Artom Lifshitz proposed openstack/nova master: imagebackend: default by_name image_type to config correctly https://review.opendev.org/c/openstack/nova/+/826526
21:03:55 opendevreview Artom Lifshitz proposed openstack/nova master: BlockDeviceMapping: Add is_local property https://review.opendev.org/c/openstack/nova/+/764485
21:03:55 opendevreview Artom Lifshitz proposed openstack/nova master: BlockDeviceMapping: Add encryption fields https://review.opendev.org/c/openstack/nova/+/760453

Earlier   Later