| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-18 | |||
| 13:23:02 | chateaulav | it did not like those exceptions or moving the backport for hvspec above super | |
| 13:23:52 | gibi | chateaulav: but it did not like with a different reasons. | |
| 13:24:02 | chateaulav | gotcha | |
| 13:24:08 | gibi | chateaulav: I personally would like to keep rejecting new archs like here https://review.opendev.org/c/openstack/nova/+/828369/14/nova/objects/hv_spec.py | |
| 13:24:17 | gibi | chateaulav: I know that you removed this to please grenade | |
| 13:24:27 | gibi | but I think that just hides the problem | |
| 13:25:11 | gibi | I have not time right now to proposa a patch top of yours showing how to make a unit test to show the same problem in a you local env | |
| 13:25:19 | chateaulav | ok, and thats what im starting to see and understand. getting use to zuul and where to find tthings. | |
| 13:25:21 | gibi | but I think that would be a way forward to troubleshoot | |
| 13:25:39 | chateaulav | makes sense | |
| 13:26:09 | gibi | I try to get to your problem before end of today | |
| 13:29:15 | chateaulav | appreciate it, ill be working all day on it. gonna see about that unit test | |
| 14:06:38 | opendevreview | Felix Huettner proposed openstack/nova stable/train: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829507 | |
| 14:51:46 | jamespage | o/ - is there a good reference on how scheduling should behave when hypervisors have partial hugepage memory configuration (say 150GB of 512GB) | |
| 14:51:50 | jamespage | ? | |
| 15:05:44 | gibi | jamespage: I don't think we have. if your instance needs numa topology becasue of cpu pinning or huge pages then the NUMATopologyFilter is responsible to select the proper host | |
| 15:11:07 | artom | jamespage, I don't think the "partial" matters. If the host has enough pages of the correct size to fit the instance, it should pass scheduling | |
| 15:13:09 | jamespage | gibi, artom: interestingly instances with larger page size configuration schedule fine - the problem I'm looking at happens when an instances without large pages gets scheduled to the hypervisor | |
| 15:13:52 | jamespage | deployment is using instances with large amounts of RAM - exceeding the diff between Total RAM - HugePages - Reserved by quite a bit | |
| 15:13:56 | gibi | jamespage: does instances without huge pages are they have any numa related requirements, i.e. cpu pinning? | |
| 15:14:59 | jamespage | gibi: nope no extra specs at all | |
| 15:15:13 | artom | jamespage, wait, that just sounds like there's not enough RAM | |
| 15:16:11 | jamespage | artom: I quite agree there is not enough - but I also expected the instance not to be scheduled to a hypervisor with this memory configuration | |
| 15:16:29 | artom | jamespage, ah, so scheduling passes, but the instance doesn't boot because not enough RAM | |
| 15:16:57 | jamespage | well it does - it then sucks all of the ram up and gets OOM'ed | |
| 15:17:31 | artom | That's... I think that's a known issue? In the sense that Nova doesn't correctly track memory when it's a mix of hugepages and "normal" page size | |
| 15:19:37 | gibi | artom: yeah I think if the small pages instance doesn't request any numa related things then the NUMATopologyFilter is not applied on that instance | |
| 15:20:19 | gibi | but I'm sure sean-k-mooney know a lot more about this | |
| 15:20:25 | gibi | but most of the RH folks on PTO today | |
| 15:20:48 | sean-k-mooney | i just sat down to do something else | |
| 15:20:52 | artom | gibi, it's not even that, I think we treat all memory as available when scheduling, but when the instance gets on the host the huge pages obviously cannot be allocated to it | |
| 15:21:02 | sean-k-mooney | somethign about large guests? | |
| 15:21:31 | gibi | sh*t, sorry | |
| 15:21:46 | sean-k-mooney | if you dont use hw:mem_page_size=small or 4k then the numa code does not run | |
| 15:21:48 | sean-k-mooney | its | |
| 15:21:50 | sean-k-mooney | fine | |
| 15:22:00 | artom | Well, it's on his shoulders as well, ignoring IRC is always an option :P | |
| 15:22:06 | sean-k-mooney | i was just goign to check my home insurance renewal | |
| 15:22:14 | gibi | sean-k-mooney: it is about mixing NUMA and non NUMA guests | |
| 15:22:37 | sean-k-mooney | ya that is not supported today | |
| 15:22:38 | gibi | jamespage: above sees memory overallocation issues | |
| 15:23:07 | gibi | sean-k-mooney: so you say that if hw:mem_page_size=small is added then we allow mixing small and large instances? | |
| 15:23:10 | sean-k-mooney | you must not under any cirucmstance mix numa and non numa guest or it will break all our memory tracking | |
| 15:23:18 | sean-k-mooney | yes | |
| 15:23:32 | sean-k-mooney | if you use hw:mem_page_size=small its fine | |
| 15:23:35 | sean-k-mooney | brb | |
| 15:25:09 | sean-k-mooney | tl;dr is using hw:mem_page_small will numa affine the guest and track the memory correctly | |
| 15:25:19 | sean-k-mooney | if you dont it will float and cause random OOM events | |
| 15:26:03 | gibi | thanks! | |
| 15:26:04 | sean-k-mooney | placement is not enough to save us since the OOM reaper runs per numa node and also hugepage guest dont supprot oversubsricption | |
| 15:26:26 | sean-k-mooney | so if if you have memeroy over subscrion it wont really work right | |
| 15:27:22 | sean-k-mooney | we have something about this in our downstream docs but basically we say use hostaggreates to prevnt mixing numa and non numa guests | |
| 15:29:16 | sean-k-mooney | jamespage: if you have enuch swap by the way those other instace coudl technicaly boot but the system would likely be unstable so its better to avoid that | |
| 15:29:20 | jamespage | sorry - power outage just at the wrong moment | |
| 15:30:33 | gibi | jamespage: https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html here are the IRC logs if you are dropped | |
| 15:31:31 | sean-k-mooney | jamespage: the reason it can be schduled to the host is in placemnt we report total ram and then only reserver what set in host_reserver_ram or what ever that option is called | |
| 15:31:59 | sean-k-mooney | jamespage: we dont currently automaticlly reserve the hugepage memrory because we dont currenlty track that as a seperate pool | |
| 15:32:25 | sean-k-mooney | so the schduelr is only looking at total-reserved not total -reserved - hugepages | |
| 15:33:23 | sean-k-mooney | actully if its a recent release we dont even have a ram filter anymore so its just placment that is checking | |
| 15:33:41 | jamespage | sean-k-mooney: having read most of the code that was my hunch so thanks for confirming... | |
| 15:34:09 | sean-k-mooney | so ya that is why we say you cant mix numa and non numa today | |
| 15:34:23 | sean-k-mooney | if we ever get to tracking numa in placment then it will fix that | |
| 15:34:43 | sean-k-mooney | sicne we will have a seperate pool per page size | |
| 15:34:54 | sean-k-mooney | and or numa node | |
| 15:36:21 | sean-k-mooney | we proably could paper over this temporally with a filter similar to the nuam one or even enhance the numa one to work with non numa instance but right now we just tell people not ot do it | |
| 15:42:29 | artom | I guess another option would be to tell operators to include the amount of hugepages in reserved_host_ram? | |
| 15:42:34 | artom | It's... weird, but should work? | |
| 15:42:51 | sean-k-mooney | no | |
| 15:43:09 | sean-k-mooney | it will break schduling | |
| 15:43:10 | artom | Wouldn't it just prevent scheduling to a host if it doesn't have enough "normal" RAM? | |
| 15:43:44 | sean-k-mooney | yes but you vould not use the hugepages and normal ram | |
| 15:44:03 | artom | Oh, right | |
| 15:44:14 | sean-k-mooney | you coul only use total - reserved | |
| 15:44:49 | sean-k-mooney | so that is what you do if you are using the hugepage on the host | |
| 15:45:07 | sean-k-mooney | + the huge page reserved option | |
| 15:50:52 | sean-k-mooney | so we coudl set max allocation size in placement = to total -hugepages | |
| 15:51:08 | sean-k-mooney | that would help | |
| 15:52:11 | jamespage | side effect to that would be that you could only ever allocate total - hugepages on that hypervisor no? | |
| 15:52:29 | sean-k-mooney | no | |
| 15:53:29 | jamespage | oh that's the max allocation size - driving the maximum single instance footprint - I see | |
| 15:53:45 | sean-k-mooney | https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-inventory-detail#update-resource-provider-inventory | |
| 15:53:48 | sean-k-mooney | yes | |
| 15:53:51 | sean-k-mooney | max_unit | |
| 15:54:22 | sean-k-mooney | sorry was just on the phone | |
| 15:54:31 | sean-k-mooney | so ya total woudl be set to total memroy | |
| 15:54:37 | sean-k-mooney | reserved = reserved form config | |
| 15:55:09 | sean-k-mooney | but if we set max_unit to total-reserved-hugepages that should help | |
| 15:55:35 | sean-k-mooney | although it would have to be the larger of | |
| 15:55:49 | sean-k-mooney | total-reserved-hugepages and hugepages | |
| 15:56:30 | sean-k-mooney | to not limit the size of hugepage guests | |
| 15:56:51 | sean-k-mooney | jamespage: is it a resent release | |
| 15:57:07 | sean-k-mooney | if so im not sure if you can use provider.yaml to twaak the make unit | |
| 15:57:26 | sean-k-mooney | gibi: do you rememebr if that can tweak non custom resouce classes? | |
| 15:57:55 | jamespage | ussuri for this particular deploy | |
| 15:58:50 | gibi | Only CUSTOM_* resource classes and traits may be managed this way. | |
| 15:58:56 | gibi | so no | |
| 15:59:00 | gibi | https://docs.openstack.org/nova/latest/admin/managing-resource-providers.html | |
| 15:59:07 | sean-k-mooney | ya ok | |
| 15:59:13 | sean-k-mooney | it was afetr ussuri anyway | |
| 15:59:51 | sean-k-mooney | jamespage: best we likely could do is a config option that could posible be backported to set or clamp the max unit | |