Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-11-06
13:30:43 sean-k-mooney that is part of our contract
13:31:12 sean-k-mooney that does nto mean its part fo placemetn as that might make sesne to someone. i cant think of a valid reaosn for that but it might be valid
13:31:26 sean-k-mooney i woudl expdect placement ot reject a invetoy whre max_ujnit exceed total
13:31:31 sean-k-mooney but i have not checked if ti does
13:31:45 jkulik Nova does allow going into reserved resources, though, when the admin requests an allocation_ratio > 1.0
13:33:05 sean-k-mooney correct because again nova expect capastiy = (total * allction ratio) - reserved
13:33:05 jkulik Which can make sense for VCPUs, but imho does not make much sense for MEMORY_MB. I wouldn't want my hypervisor to start swapping, because an overcommitted VMs uses all of its RAM.
13:34:26 jkulik and since Placement takes care with the capacity check that things fit, adding a lot of logic in Nova libvirt to report different "max_unit" depending on the allocation_ratio doesn't really make sense
13:34:29 sean-k-mooney so we have docuemtned in nova in the past tha tyou have to multiply host_reserved_ram by the allcation ratio in the config
13:35:27 sean-k-mooney although its not very clear i guess if you jsut look at the config ref
13:35:35 sean-k-mooney so we coudl defintly improve that
13:35:36 jkulik I feel like that would break the allocation capacity check in Placement.
13:35:59 sean-k-mooney i am debating if that is a placement bug
13:36:19 sean-k-mooney the check looks incorect to me jsut lookign at that capstiy line but
13:36:25 sean-k-mooney i have not revied the code properly to say
13:37:09 jkulik the discussion was helpful for me nonetheless. thank you
13:38:43 sean-k-mooney https://github.com/openstack/nova/blob/rocky-eol/nova/scheduler/filters/ram_filter.py#L32-L65
13:38:51 sean-k-mooney this is what nova used to do before placement
13:39:41 sean-k-mooney placement is ment ot be 1:1 compatible with that
13:40:02 jkulik so what*s "total_usable_ram_mb" here?
13:40:31 sean-k-mooney i was hoping you woudl not ask that :P
13:40:55 sean-k-mooney but its https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L217
13:40:57 jkulik `compute.memory_mb` ... one more step
13:41:22 sean-k-mooney ya so that is just the total ram on the host
13:41:37 sean-k-mooney at least i think it is
13:42:31 jkulik ah, yes. We patched that downstream to properly include the reserved_host_memory_mb :D
13:42:52 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9445
13:42:58 jkulik it indeed used the reserved_host_memory_mb only as "usage", not as real reduction of the available memory.
13:43:12 sean-k-mooney right
13:43:26 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1150-L1158
13:43:32 sean-k-mooney so that is what determisn it upstream
13:43:41 jkulik which is inconsitent with what the vmware driver reports to Placement ... at least in our version :D
13:43:58 sean-k-mooney its either the file backed memory size if your using that or basically the ouput of total memory form free or in thsi cashe virsh capabliteis i belive
13:44:03 jkulik yeah, I read that part. that's fine. just `max_unit` seemed off to me
13:44:16 sean-k-mooney jkulik: well that is the other thing
13:44:25 sean-k-mooney this is virt driver defiened
13:44:37 sean-k-mooney at least to some degree
13:44:42 jkulik why we even looked at that was: we use hugepages from a static pool and libvirt reports not that, but the full memory of the host ...
13:44:49 sean-k-mooney you can use provider.yaml to overried max unit
13:45:04 jkulik hm!
13:45:23 sean-k-mooney well for hugepages you confirue the reservation diffently
13:45:49 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb
13:45:54 sean-k-mooney per page size per numa node
13:46:26 sean-k-mooney if you use hw:mem_page_size=small we also use the values form reserved_huge_pages for the system native size i.e. 4
13:46:29 sean-k-mooney for 4k
13:47:13 sean-k-mooney so if your doing anythign related to numa you are not allowed to oversibcsribe memory
13:47:52 jkulik ok. then for NUMA `max_unit` would be wrong :D but nevermind
13:48:00 sean-k-mooney and you use `reserved_huge_pages = node:0,size:4,count=256`
13:48:47 sean-k-mooney well yes but if we implent numa in placment we can define the sematic of the livbirt driver to limit it
13:49:04 sean-k-mooney since we know that numa memory will always have an allcoaiton ration of 1.0
13:49:18 sean-k-mooney the max_unit shoudl never exceed (total-reserved) in that case
13:49:36 jkulik yes. but that's not there, yet, right?
13:49:53 sean-k-mooney correct but we track numa memory diffently in nova already
13:49:59 sean-k-mooney and do the enfocement diffently
13:50:11 sean-k-mooney so placement will say this host might be valid
13:50:22 sean-k-mooney and hten numa toploty filter will then do the enforcement
13:51:09 jkulik ... and fail and then we go into retries for spawning in conductor, in case that's configured?
13:51:30 sean-k-mooney because numa requireds an allocation ratio of exactly 1 it drops out of the equation and max unit is just total-reserved
13:51:38 jkulik ah. no. it's an actual scheduler filter
13:51:52 sean-k-mooney right its a scheduler filter
13:52:15 sean-k-mooney we do the check again on the compute in case 2 diffent api request race for the last resouce on a given host
13:52:44 sean-k-mooney but for the most part that wont hapen in a reasonable size cloud or if you say use a weigher to spread based on ram usage
13:53:53 jkulik hm ... we have bad experience with spreading ... then parts of too many hosts are used, which disallows full-host-sized VMs :/
13:54:08 jkulik thank you again.
13:54:23 sean-k-mooney :) well you can also opt to pack for that use case
13:54:28 sean-k-mooney but ya its a trade off
13:54:37 sean-k-mooney and no worries this is one of hte more complex parts of nova
14:30:33 stephenfin gmaan: Not necessarily for today, but what are the chances of getting the remainder of the OpenAPI series reviewed in the next few weeks? Happy to trade reviews
14:30:51 opendevreview Takashi Kajinami proposed openstack/placement master: Make sure [cors] allowed_origin accepts a list value https://review.opendev.org/c/openstack/placement/+/966275
14:32:06 stephenfin sean-k-mooney: ditto for you: gmaan has hit the bottom 9 patches of the 28 (sorry /o\) outstanding
14:40:45 opendevreview Takashi Kajinami proposed openstack/nova master: Use common utility to parse Host header https://review.opendev.org/c/openstack/nova/+/966280
15:19:04 opendevreview Max proposed openstack/nova master: fix: return cross_az_attach errors directly https://review.opendev.org/c/openstack/nova/+/966284
16:19:22 sean-k-mooney stephenfin: so apprenly https://review.opendev.org/c/openstack/devstack/+/966239 is the short term fix
16:19:45 sean-k-mooney but i also think this is likely incorrect
16:20:01 sean-k-mooney becasue 2025.1 shoudl still be using tempest master not a pin
17:01:20 gmaan stephenfin: sure, not this week but I can see if I can start it again during end of next week. but it is still in my list to continue in this cycle.
17:02:08 gmaan sean-k-mooney: we should be using the master on 2025.1 but let me check chat in qa channel and see what is issue
17:02:40 stephenfin sean-k-mooney: I see there's a run-tempest-26 role in the tempest repo. I would imagine we probably want something similar if there are Python issues, but that's one for gmaan et al
17:02:48 stephenfin *new Python version issues
17:21:40 sean-k-mooney oh i ment to respoend in the ohter channle on the tempst thing lets keep that converstaion in openstack-qa my bad
17:21:52 gmaan ++
17:44:15 opendevreview Merged openstack/nova master: [CI]nova-alt-configurations tests eventlet https://review.opendev.org/c/openstack/nova/+/965922
18:50:37 opendevreview Balazs Gibizer proposed openstack/nova master: Default native threading for sch, api and metadata https://review.opendev.org/c/openstack/nova/+/965924
18:52:12 gibi sean-k-mooney: ^^ renamed the parameter as requested
18:55:41 sean-k-mooney gibi: +2 on both
19:01:07 gibi sean-k-mooney: thanks!
19:01:27 gibi dansmith: when you have time we are ready to close one item off the my eventlet list ^^ :)
19:02:35 dansmith gibi: yeah I was going to do that this morning until I saw the request..I'll circle back in a bit
19:02:41 gibi thanks
23:48:37 opendevreview Merged openstack/nova master: Move monkey_patch from init to the entrypoints https://review.opendev.org/c/openstack/nova/+/965923
#openstack-nova - 2025-11-07
05:59:29 koya Hi Uggla, bauzas, thank you very much for discussing “Instance metadata protection” at the recent OpenInfra Summit.
05:59:58 koya Based on the comments you kindly provided, I’ve been reviewing and updating (revising) the blueprint; https://blueprints.launchpad.net/nova/+spec/instance-metadata-protection. I’ll reach out to you again when it’s ready.
10:38:19 artem_vasilyev Hi, I stumbled upon the _cleanup_incomplete_migrations function and can't quite get whether this is intended logic or not. I couldn't find answers in the documentation. As I understand, the purpose of this function is to clean orphan files of already deleted instances. But when deleting instances, their migrations are also marked as soft-deleted (see https://github.com/openstack/nova/blob/master/nova/db/main/api.py#L1348). So this function
10:38:23 opendevreview Adrien Cunin proposed openstack/os-vif master: Fixed bridge name when per_port_bridge is used https://review.opendev.org/c/openstack/os-vif/+/966410
10:39:35 jkulik artem_vasilyev: fyi, your message is cut off after ". So this function"
10:40:28 artem_vasilyev thanks. So this function can't get error migrations, because it doesn't query migrations with read_deleted context. Is this the expected behavior, or am I missing something?
10:43:46 artem_vasilyev _cleanup_incomplete_migrations function is here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L11944
10:50:50 jkulik the tests mock away MigrationList.get_by_filters(), so they can't see it. from looking at the code, I'd agree with you artem_vasilyev
12:33:17 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
12:33:18 opendevreview Balazs Gibizer proposed openstack/nova master: Do not use libvirt before Host is initialized https://review.opendev.org/c/openstack/nova/+/966430

Earlier   Later