| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-18 | |||
| 07:58:53 | brinzhang | bauzas, gibi, songwenping: vGPU support in Cyborg may need a slot of PTG, do you have some suggestion? | |
| 08:00:06 | opendevreview | Felix Huettner proposed openstack/nova stable/train: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829507 | |
| 08:01:10 | brinzhang | we would like to register 2:00UTC-3:UTC at April 5, is it ok? | |
| 08:02:18 | opendevreview | Ghanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs https://review.opendev.org/c/openstack/nova/+/829626 | |
| 08:04:01 | gibi | brinzhang: I'm not sure we have a ptg etherpad yet. as far as I know RedHat folks including bauzas is PTO on today. | |
| 08:07:08 | brinzhang | gibi: ack | |
| 08:07:22 | gibi | so let's get back to this on Monday | |
| 08:07:25 | brinzhang | https://etherpad.opendev.org/p/nova-zed-ptg I saw this link, it's nothing else | |
| 08:07:52 | brinzhang | gibi: ok, we can discuss on monday ^^ | |
| 08:12:03 | gibi | chateaulav: so for example the logs from your trial with my last suggestion visible here (filtered to error only) https://zuul.opendev.org/t/openstack/build/08789d9f0b0546cb9e6fb2b6f4f0231c/log/compute1/logs/screen-n-cpu.txt?severity=4 | |
| 08:44:30 | opendevreview | Felix Huettner proposed openstack/nova stable/stein: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829859 | |
| 08:45:07 | tobias-urdin | gibi: friendly request for backport review https://review.opendev.org/c/openstack/nova/+/828407 :) | |
| 08:51:40 | opendevreview | Felix Huettner proposed openstack/nova stable/rocky: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829860 | |
| 08:55:14 | opendevreview | Felix Huettner proposed openstack/nova stable/queens: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829861 | |
| 09:15:33 | gibi | tobias-urdin: hi! I don't have +2 rights on stable branches :/ | |
| 09:17:44 | tobias-urdin | gibi: oh sorry for the noice! | |
| 09:17:55 | gibi | tobias-urdin: no worries | |
| 09:29:24 | opendevreview | Ghanshyam proposed openstack/nova master: Complete phase-1 of RBAC community-wide goal https://review.opendev.org/c/openstack/nova/+/829866 | |
| 10:51:36 | chateaulav | gibi: ok,I was trying a few things yesterday, because greens was still failing. I'll put the change back and then investigate that way. | |
| 10:51:46 | chateaulav | Grenade | |
| 10:56:29 | gibi | chateaulav: probably easier to create a unit test that calls obj_make_compatible on a populated ComputeNode object | |
| 10:56:43 | gibi | with that you can troubleshoot locally | |
| 10:59:41 | chateaulav | gibi: Yeah. Biggest thing was trying to have zuul pass which it did. I'll see about doing that, and can test to ensure the new values aren't processed | |
| 13:22:34 | chateaulav | gibi: so then with the current patchset, grenade and everything is happy. | |
| 13:23:02 | chateaulav | it did not like those exceptions or moving the backport for hvspec above super | |
| 13:23:52 | gibi | chateaulav: but it did not like with a different reasons. | |
| 13:24:02 | chateaulav | gotcha | |
| 13:24:08 | gibi | chateaulav: I personally would like to keep rejecting new archs like here https://review.opendev.org/c/openstack/nova/+/828369/14/nova/objects/hv_spec.py | |
| 13:24:17 | gibi | chateaulav: I know that you removed this to please grenade | |
| 13:24:27 | gibi | but I think that just hides the problem | |
| 13:25:11 | gibi | I have not time right now to proposa a patch top of yours showing how to make a unit test to show the same problem in a you local env | |
| 13:25:19 | chateaulav | ok, and thats what im starting to see and understand. getting use to zuul and where to find tthings. | |
| 13:25:21 | gibi | but I think that would be a way forward to troubleshoot | |
| 13:25:39 | chateaulav | makes sense | |
| 13:26:09 | gibi | I try to get to your problem before end of today | |
| 13:29:15 | chateaulav | appreciate it, ill be working all day on it. gonna see about that unit test | |
| 14:06:38 | opendevreview | Felix Huettner proposed openstack/nova stable/train: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829507 | |
| 14:51:46 | jamespage | o/ - is there a good reference on how scheduling should behave when hypervisors have partial hugepage memory configuration (say 150GB of 512GB) | |
| 14:51:50 | jamespage | ? | |
| 15:05:44 | gibi | jamespage: I don't think we have. if your instance needs numa topology becasue of cpu pinning or huge pages then the NUMATopologyFilter is responsible to select the proper host | |
| 15:11:07 | artom | jamespage, I don't think the "partial" matters. If the host has enough pages of the correct size to fit the instance, it should pass scheduling | |
| 15:13:09 | jamespage | gibi, artom: interestingly instances with larger page size configuration schedule fine - the problem I'm looking at happens when an instances without large pages gets scheduled to the hypervisor | |
| 15:13:52 | jamespage | deployment is using instances with large amounts of RAM - exceeding the diff between Total RAM - HugePages - Reserved by quite a bit | |
| 15:13:56 | gibi | jamespage: does instances without huge pages are they have any numa related requirements, i.e. cpu pinning? | |
| 15:14:59 | jamespage | gibi: nope no extra specs at all | |
| 15:15:13 | artom | jamespage, wait, that just sounds like there's not enough RAM | |
| 15:16:11 | jamespage | artom: I quite agree there is not enough - but I also expected the instance not to be scheduled to a hypervisor with this memory configuration | |
| 15:16:29 | artom | jamespage, ah, so scheduling passes, but the instance doesn't boot because not enough RAM | |
| 15:16:57 | jamespage | well it does - it then sucks all of the ram up and gets OOM'ed | |
| 15:17:31 | artom | That's... I think that's a known issue? In the sense that Nova doesn't correctly track memory when it's a mix of hugepages and "normal" page size | |
| 15:19:37 | gibi | artom: yeah I think if the small pages instance doesn't request any numa related things then the NUMATopologyFilter is not applied on that instance | |
| 15:20:19 | gibi | but I'm sure sean-k-mooney know a lot more about this | |
| 15:20:25 | gibi | but most of the RH folks on PTO today | |
| 15:20:48 | sean-k-mooney | i just sat down to do something else | |
| 15:20:52 | artom | gibi, it's not even that, I think we treat all memory as available when scheduling, but when the instance gets on the host the huge pages obviously cannot be allocated to it | |
| 15:21:02 | sean-k-mooney | somethign about large guests? | |
| 15:21:31 | gibi | sh*t, sorry | |
| 15:21:46 | sean-k-mooney | if you dont use hw:mem_page_size=small or 4k then the numa code does not run | |
| 15:21:48 | sean-k-mooney | its | |
| 15:21:50 | sean-k-mooney | fine | |
| 15:22:00 | artom | Well, it's on his shoulders as well, ignoring IRC is always an option :P | |
| 15:22:06 | sean-k-mooney | i was just goign to check my home insurance renewal | |
| 15:22:14 | gibi | sean-k-mooney: it is about mixing NUMA and non NUMA guests | |
| 15:22:37 | sean-k-mooney | ya that is not supported today | |
| 15:22:38 | gibi | jamespage: above sees memory overallocation issues | |
| 15:23:07 | gibi | sean-k-mooney: so you say that if hw:mem_page_size=small is added then we allow mixing small and large instances? | |
| 15:23:10 | sean-k-mooney | you must not under any cirucmstance mix numa and non numa guest or it will break all our memory tracking | |
| 15:23:18 | sean-k-mooney | yes | |
| 15:23:32 | sean-k-mooney | if you use hw:mem_page_size=small its fine | |
| 15:23:35 | sean-k-mooney | brb | |
| 15:25:09 | sean-k-mooney | tl;dr is using hw:mem_page_small will numa affine the guest and track the memory correctly | |
| 15:25:19 | sean-k-mooney | if you dont it will float and cause random OOM events | |
| 15:26:03 | gibi | thanks! | |
| 15:26:04 | sean-k-mooney | placement is not enough to save us since the OOM reaper runs per numa node and also hugepage guest dont supprot oversubsricption | |
| 15:26:26 | sean-k-mooney | so if if you have memeroy over subscrion it wont really work right | |
| 15:27:22 | sean-k-mooney | we have something about this in our downstream docs but basically we say use hostaggreates to prevnt mixing numa and non numa guests | |
| 15:29:16 | sean-k-mooney | jamespage: if you have enuch swap by the way those other instace coudl technicaly boot but the system would likely be unstable so its better to avoid that | |
| 15:29:20 | jamespage | sorry - power outage just at the wrong moment | |
| 15:30:33 | gibi | jamespage: https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html here are the IRC logs if you are dropped | |
| 15:31:31 | sean-k-mooney | jamespage: the reason it can be schduled to the host is in placemnt we report total ram and then only reserver what set in host_reserver_ram or what ever that option is called | |
| 15:31:59 | sean-k-mooney | jamespage: we dont currently automaticlly reserve the hugepage memrory because we dont currenlty track that as a seperate pool | |
| 15:32:25 | sean-k-mooney | so the schduelr is only looking at total-reserved not total -reserved - hugepages | |
| 15:33:23 | sean-k-mooney | actully if its a recent release we dont even have a ram filter anymore so its just placment that is checking | |
| 15:33:41 | jamespage | sean-k-mooney: having read most of the code that was my hunch so thanks for confirming... | |
| 15:34:09 | sean-k-mooney | so ya that is why we say you cant mix numa and non numa today | |
| 15:34:23 | sean-k-mooney | if we ever get to tracking numa in placment then it will fix that | |
| 15:34:43 | sean-k-mooney | sicne we will have a seperate pool per page size | |
| 15:34:54 | sean-k-mooney | and or numa node | |
| 15:36:21 | sean-k-mooney | we proably could paper over this temporally with a filter similar to the nuam one or even enhance the numa one to work with non numa instance but right now we just tell people not ot do it | |
| 15:42:29 | artom | I guess another option would be to tell operators to include the amount of hugepages in reserved_host_ram? | |
| 15:42:34 | artom | It's... weird, but should work? | |
| 15:42:51 | sean-k-mooney | no | |
| 15:43:09 | sean-k-mooney | it will break schduling | |
| 15:43:10 | artom | Wouldn't it just prevent scheduling to a host if it doesn't have enough "normal" RAM? | |
| 15:43:44 | sean-k-mooney | yes but you vould not use the hugepages and normal ram | |
| 15:44:03 | artom | Oh, right | |
| 15:44:14 | sean-k-mooney | you coul only use total - reserved | |
| 15:44:49 | sean-k-mooney | so that is what you do if you are using the hugepage on the host | |
| 15:45:07 | sean-k-mooney | + the huge page reserved option | |
| 15:50:52 | sean-k-mooney | so we coudl set max allocation size in placement = to total -hugepages | |