| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-06 | |||
| 13:22:24 | jkulik | I thought of reserved more like "No VM may cut into this, because it's provider-reserved" | |
| 13:22:43 | sean-k-mooney | yes but only if allcation ration is 1.0 | |
| 13:23:03 | sean-k-mooney | we treate reseved basically as if there is an allcoation of that size already present | |
| 13:23:44 | sean-k-mooney | but imporantly we do (total*ratio) - reserved | |
| 13:23:46 | jkulik | but before allocation_ratio is applied (at least in computing the "free" part), right? | |
| 13:23:58 | jkulik | ah. do we? | |
| 13:24:00 | sean-k-mooney | its not (total - reserved) * ratio | |
| 13:24:32 | jkulik | I always thought it would be ^ | |
| 13:24:37 | sean-k-mooney | no reseved it subtracted form total*ratio for backward compatibality | |
| 13:24:43 | sean-k-mooney | becaus that how it worked before placment | |
| 13:25:19 | jkulik | > placement/objects/allocation_candidate.py: cap = int((usage.total - usage.reserved) * allocati | |
| 13:25:19 | jkulik | on_ratio) | |
| 13:26:07 | sean-k-mooney | so https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb | |
| 13:26:22 | sean-k-mooney | is directly translated into the reseved value for memory mb | |
| 13:27:06 | jkulik | yes. it's report as-is to Placement | |
| 13:27:54 | sean-k-mooney | and the sematic of that are if you have 50G with allcaotion ration 2.0 and resved 10G we will allwo 90G of usage | |
| 13:27:55 | jkulik | https://github.com/openstack/placement/blob/master/placement/objects/allocation.py#L253 | |
| 13:28:13 | jkulik | so Placement does do (total - reserved) * allocation_ratio | |
| 13:28:26 | sean-k-mooney | that for the allcation size | |
| 13:29:18 | jkulik | that does mean, I can allocate more than total right? as long as I set `max_unit` to more than total? | |
| 13:29:54 | jkulik | but you're saying, `max_unit` is capped at `total` anyways. so that's something else, too. | |
| 13:30:17 | sean-k-mooney | so there is what placement allwos you to model | |
| 13:30:20 | sean-k-mooney | and what nova allows | |
| 13:30:25 | jkulik | hm ... ok, I think I understand why it's allowed in Placemeent now. thanks | |
| 13:30:38 | sean-k-mooney | nova does not allow a singel vm to over subscript the host resocues | |
| 13:30:43 | sean-k-mooney | that is part of our contract | |
| 13:31:12 | sean-k-mooney | that does nto mean its part fo placemetn as that might make sesne to someone. i cant think of a valid reaosn for that but it might be valid | |
| 13:31:26 | sean-k-mooney | i woudl expdect placement ot reject a invetoy whre max_ujnit exceed total | |
| 13:31:31 | sean-k-mooney | but i have not checked if ti does | |
| 13:31:45 | jkulik | Nova does allow going into reserved resources, though, when the admin requests an allocation_ratio > 1.0 | |
| 13:33:05 | sean-k-mooney | correct because again nova expect capastiy = (total * allction ratio) - reserved | |
| 13:33:05 | jkulik | Which can make sense for VCPUs, but imho does not make much sense for MEMORY_MB. I wouldn't want my hypervisor to start swapping, because an overcommitted VMs uses all of its RAM. | |
| 13:34:26 | jkulik | and since Placement takes care with the capacity check that things fit, adding a lot of logic in Nova libvirt to report different "max_unit" depending on the allocation_ratio doesn't really make sense | |
| 13:34:29 | sean-k-mooney | so we have docuemtned in nova in the past tha tyou have to multiply host_reserved_ram by the allcation ratio in the config | |
| 13:35:27 | sean-k-mooney | although its not very clear i guess if you jsut look at the config ref | |
| 13:35:35 | sean-k-mooney | so we coudl defintly improve that | |
| 13:35:36 | jkulik | I feel like that would break the allocation capacity check in Placement. | |
| 13:35:59 | sean-k-mooney | i am debating if that is a placement bug | |
| 13:36:19 | sean-k-mooney | the check looks incorect to me jsut lookign at that capstiy line but | |
| 13:36:25 | sean-k-mooney | i have not revied the code properly to say | |
| 13:37:09 | jkulik | the discussion was helpful for me nonetheless. thank you | |
| 13:38:43 | sean-k-mooney | https://github.com/openstack/nova/blob/rocky-eol/nova/scheduler/filters/ram_filter.py#L32-L65 | |
| 13:38:51 | sean-k-mooney | this is what nova used to do before placement | |
| 13:39:41 | sean-k-mooney | placement is ment ot be 1:1 compatible with that | |
| 13:40:02 | jkulik | so what*s "total_usable_ram_mb" here? | |
| 13:40:31 | sean-k-mooney | i was hoping you woudl not ask that :P | |
| 13:40:55 | sean-k-mooney | but its https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L217 | |
| 13:40:57 | jkulik | `compute.memory_mb` ... one more step | |
| 13:41:22 | sean-k-mooney | ya so that is just the total ram on the host | |
| 13:41:37 | sean-k-mooney | at least i think it is | |
| 13:42:31 | jkulik | ah, yes. We patched that downstream to properly include the reserved_host_memory_mb :D | |
| 13:42:52 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9445 | |
| 13:42:58 | jkulik | it indeed used the reserved_host_memory_mb only as "usage", not as real reduction of the available memory. | |
| 13:43:12 | sean-k-mooney | right | |
| 13:43:26 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1150-L1158 | |
| 13:43:32 | sean-k-mooney | so that is what determisn it upstream | |
| 13:43:41 | jkulik | which is inconsitent with what the vmware driver reports to Placement ... at least in our version :D | |
| 13:43:58 | sean-k-mooney | its either the file backed memory size if your using that or basically the ouput of total memory form free or in thsi cashe virsh capabliteis i belive | |
| 13:44:03 | jkulik | yeah, I read that part. that's fine. just `max_unit` seemed off to me | |
| 13:44:16 | sean-k-mooney | jkulik: well that is the other thing | |
| 13:44:25 | sean-k-mooney | this is virt driver defiened | |
| 13:44:37 | sean-k-mooney | at least to some degree | |
| 13:44:42 | jkulik | why we even looked at that was: we use hugepages from a static pool and libvirt reports not that, but the full memory of the host ... | |
| 13:44:49 | sean-k-mooney | you can use provider.yaml to overried max unit | |
| 13:45:04 | jkulik | hm! | |
| 13:45:23 | sean-k-mooney | well for hugepages you confirue the reservation diffently | |
| 13:45:49 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb | |
| 13:45:54 | sean-k-mooney | per page size per numa node | |
| 13:46:26 | sean-k-mooney | if you use hw:mem_page_size=small we also use the values form reserved_huge_pages for the system native size i.e. 4 | |
| 13:46:29 | sean-k-mooney | for 4k | |
| 13:47:13 | sean-k-mooney | so if your doing anythign related to numa you are not allowed to oversibcsribe memory | |
| 13:47:52 | jkulik | ok. then for NUMA `max_unit` would be wrong :D but nevermind | |
| 13:48:00 | sean-k-mooney | and you use `reserved_huge_pages = node:0,size:4,count=256` | |
| 13:48:47 | sean-k-mooney | well yes but if we implent numa in placment we can define the sematic of the livbirt driver to limit it | |
| 13:49:04 | sean-k-mooney | since we know that numa memory will always have an allcoaiton ration of 1.0 | |
| 13:49:18 | sean-k-mooney | the max_unit shoudl never exceed (total-reserved) in that case | |
| 13:49:36 | jkulik | yes. but that's not there, yet, right? | |
| 13:49:53 | sean-k-mooney | correct but we track numa memory diffently in nova already | |
| 13:49:59 | sean-k-mooney | and do the enfocement diffently | |
| 13:50:11 | sean-k-mooney | so placement will say this host might be valid | |
| 13:50:22 | sean-k-mooney | and hten numa toploty filter will then do the enforcement | |
| 13:51:09 | jkulik | ... and fail and then we go into retries for spawning in conductor, in case that's configured? | |
| 13:51:30 | sean-k-mooney | because numa requireds an allocation ratio of exactly 1 it drops out of the equation and max unit is just total-reserved | |
| 13:51:38 | jkulik | ah. no. it's an actual scheduler filter | |
| 13:51:52 | sean-k-mooney | right its a scheduler filter | |
| 13:52:15 | sean-k-mooney | we do the check again on the compute in case 2 diffent api request race for the last resouce on a given host | |
| 13:52:44 | sean-k-mooney | but for the most part that wont hapen in a reasonable size cloud or if you say use a weigher to spread based on ram usage | |
| 13:53:53 | jkulik | hm ... we have bad experience with spreading ... then parts of too many hosts are used, which disallows full-host-sized VMs :/ | |
| 13:54:08 | jkulik | thank you again. | |
| 13:54:23 | sean-k-mooney | :) well you can also opt to pack for that use case | |
| 13:54:28 | sean-k-mooney | but ya its a trade off | |
| 13:54:37 | sean-k-mooney | and no worries this is one of hte more complex parts of nova | |
| 14:30:33 | stephenfin | gmaan: Not necessarily for today, but what are the chances of getting the remainder of the OpenAPI series reviewed in the next few weeks? Happy to trade reviews | |
| 14:30:51 | opendevreview | Takashi Kajinami proposed openstack/placement master: Make sure [cors] allowed_origin accepts a list value https://review.opendev.org/c/openstack/placement/+/966275 | |
| 14:32:06 | stephenfin | sean-k-mooney: ditto for you: gmaan has hit the bottom 9 patches of the 28 (sorry /o\) outstanding | |
| 14:40:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: Use common utility to parse Host header https://review.opendev.org/c/openstack/nova/+/966280 | |
| 15:19:04 | opendevreview | Max proposed openstack/nova master: fix: return cross_az_attach errors directly https://review.opendev.org/c/openstack/nova/+/966284 | |
| 16:19:22 | sean-k-mooney | stephenfin: so apprenly https://review.opendev.org/c/openstack/devstack/+/966239 is the short term fix | |
| 16:19:45 | sean-k-mooney | but i also think this is likely incorrect | |
| 16:20:01 | sean-k-mooney | becasue 2025.1 shoudl still be using tempest master not a pin | |