| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-06 | |||
| 13:34:29 | sean-k-mooney | so we have docuemtned in nova in the past tha tyou have to multiply host_reserved_ram by the allcation ratio in the config | |
| 13:35:27 | sean-k-mooney | although its not very clear i guess if you jsut look at the config ref | |
| 13:35:35 | sean-k-mooney | so we coudl defintly improve that | |
| 13:35:36 | jkulik | I feel like that would break the allocation capacity check in Placement. | |
| 13:35:59 | sean-k-mooney | i am debating if that is a placement bug | |
| 13:36:19 | sean-k-mooney | the check looks incorect to me jsut lookign at that capstiy line but | |
| 13:36:25 | sean-k-mooney | i have not revied the code properly to say | |
| 13:37:09 | jkulik | the discussion was helpful for me nonetheless. thank you | |
| 13:38:43 | sean-k-mooney | https://github.com/openstack/nova/blob/rocky-eol/nova/scheduler/filters/ram_filter.py#L32-L65 | |
| 13:38:51 | sean-k-mooney | this is what nova used to do before placement | |
| 13:39:41 | sean-k-mooney | placement is ment ot be 1:1 compatible with that | |
| 13:40:02 | jkulik | so what*s "total_usable_ram_mb" here? | |
| 13:40:31 | sean-k-mooney | i was hoping you woudl not ask that :P | |
| 13:40:55 | sean-k-mooney | but its https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L217 | |
| 13:40:57 | jkulik | `compute.memory_mb` ... one more step | |
| 13:41:22 | sean-k-mooney | ya so that is just the total ram on the host | |
| 13:41:37 | sean-k-mooney | at least i think it is | |
| 13:42:31 | jkulik | ah, yes. We patched that downstream to properly include the reserved_host_memory_mb :D | |
| 13:42:52 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9445 | |
| 13:42:58 | jkulik | it indeed used the reserved_host_memory_mb only as "usage", not as real reduction of the available memory. | |
| 13:43:12 | sean-k-mooney | right | |
| 13:43:26 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1150-L1158 | |
| 13:43:32 | sean-k-mooney | so that is what determisn it upstream | |
| 13:43:41 | jkulik | which is inconsitent with what the vmware driver reports to Placement ... at least in our version :D | |
| 13:43:58 | sean-k-mooney | its either the file backed memory size if your using that or basically the ouput of total memory form free or in thsi cashe virsh capabliteis i belive | |
| 13:44:03 | jkulik | yeah, I read that part. that's fine. just `max_unit` seemed off to me | |
| 13:44:16 | sean-k-mooney | jkulik: well that is the other thing | |
| 13:44:25 | sean-k-mooney | this is virt driver defiened | |
| 13:44:37 | sean-k-mooney | at least to some degree | |
| 13:44:42 | jkulik | why we even looked at that was: we use hugepages from a static pool and libvirt reports not that, but the full memory of the host ... | |
| 13:44:49 | sean-k-mooney | you can use provider.yaml to overried max unit | |
| 13:45:04 | jkulik | hm! | |
| 13:45:23 | sean-k-mooney | well for hugepages you confirue the reservation diffently | |
| 13:45:49 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb | |
| 13:45:54 | sean-k-mooney | per page size per numa node | |
| 13:46:26 | sean-k-mooney | if you use hw:mem_page_size=small we also use the values form reserved_huge_pages for the system native size i.e. 4 | |
| 13:46:29 | sean-k-mooney | for 4k | |
| 13:47:13 | sean-k-mooney | so if your doing anythign related to numa you are not allowed to oversibcsribe memory | |
| 13:47:52 | jkulik | ok. then for NUMA `max_unit` would be wrong :D but nevermind | |
| 13:48:00 | sean-k-mooney | and you use `reserved_huge_pages = node:0,size:4,count=256` | |
| 13:48:47 | sean-k-mooney | well yes but if we implent numa in placment we can define the sematic of the livbirt driver to limit it | |
| 13:49:04 | sean-k-mooney | since we know that numa memory will always have an allcoaiton ration of 1.0 | |
| 13:49:18 | sean-k-mooney | the max_unit shoudl never exceed (total-reserved) in that case | |
| 13:49:36 | jkulik | yes. but that's not there, yet, right? | |
| 13:49:53 | sean-k-mooney | correct but we track numa memory diffently in nova already | |
| 13:49:59 | sean-k-mooney | and do the enfocement diffently | |
| 13:50:11 | sean-k-mooney | so placement will say this host might be valid | |
| 13:50:22 | sean-k-mooney | and hten numa toploty filter will then do the enforcement | |
| 13:51:09 | jkulik | ... and fail and then we go into retries for spawning in conductor, in case that's configured? | |
| 13:51:30 | sean-k-mooney | because numa requireds an allocation ratio of exactly 1 it drops out of the equation and max unit is just total-reserved | |
| 13:51:38 | jkulik | ah. no. it's an actual scheduler filter | |
| 13:51:52 | sean-k-mooney | right its a scheduler filter | |
| 13:52:15 | sean-k-mooney | we do the check again on the compute in case 2 diffent api request race for the last resouce on a given host | |
| 13:52:44 | sean-k-mooney | but for the most part that wont hapen in a reasonable size cloud or if you say use a weigher to spread based on ram usage | |
| 13:53:53 | jkulik | hm ... we have bad experience with spreading ... then parts of too many hosts are used, which disallows full-host-sized VMs :/ | |
| 13:54:08 | jkulik | thank you again. | |
| 13:54:23 | sean-k-mooney | :) well you can also opt to pack for that use case | |
| 13:54:28 | sean-k-mooney | but ya its a trade off | |
| 13:54:37 | sean-k-mooney | and no worries this is one of hte more complex parts of nova | |
| 14:30:33 | stephenfin | gmaan: Not necessarily for today, but what are the chances of getting the remainder of the OpenAPI series reviewed in the next few weeks? Happy to trade reviews | |
| 14:30:51 | opendevreview | Takashi Kajinami proposed openstack/placement master: Make sure [cors] allowed_origin accepts a list value https://review.opendev.org/c/openstack/placement/+/966275 | |
| 14:32:06 | stephenfin | sean-k-mooney: ditto for you: gmaan has hit the bottom 9 patches of the 28 (sorry /o\) outstanding | |
| 14:40:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: Use common utility to parse Host header https://review.opendev.org/c/openstack/nova/+/966280 | |
| 15:19:04 | opendevreview | Max proposed openstack/nova master: fix: return cross_az_attach errors directly https://review.opendev.org/c/openstack/nova/+/966284 | |
| 16:19:22 | sean-k-mooney | stephenfin: so apprenly https://review.opendev.org/c/openstack/devstack/+/966239 is the short term fix | |
| 16:19:45 | sean-k-mooney | but i also think this is likely incorrect | |
| 16:20:01 | sean-k-mooney | becasue 2025.1 shoudl still be using tempest master not a pin | |
| 17:01:20 | gmaan | stephenfin: sure, not this week but I can see if I can start it again during end of next week. but it is still in my list to continue in this cycle. | |
| 17:02:08 | gmaan | sean-k-mooney: we should be using the master on 2025.1 but let me check chat in qa channel and see what is issue | |
| 17:02:40 | stephenfin | sean-k-mooney: I see there's a run-tempest-26 role in the tempest repo. I would imagine we probably want something similar if there are Python issues, but that's one for gmaan et al | |
| 17:02:48 | stephenfin | *new Python version issues | |
| 17:21:40 | sean-k-mooney | oh i ment to respoend in the ohter channle on the tempst thing lets keep that converstaion in openstack-qa my bad | |
| 17:21:52 | gmaan | ++ | |
| 17:44:15 | opendevreview | Merged openstack/nova master: [CI]nova-alt-configurations tests eventlet https://review.opendev.org/c/openstack/nova/+/965922 | |
| 18:50:37 | opendevreview | Balazs Gibizer proposed openstack/nova master: Default native threading for sch, api and metadata https://review.opendev.org/c/openstack/nova/+/965924 | |
| 18:52:12 | gibi | sean-k-mooney: ^^ renamed the parameter as requested | |
| 18:55:41 | sean-k-mooney | gibi: +2 on both | |
| 19:01:07 | gibi | sean-k-mooney: thanks! | |
| 19:01:27 | gibi | dansmith: when you have time we are ready to close one item off the my eventlet list ^^ :) | |
| 19:02:35 | dansmith | gibi: yeah I was going to do that this morning until I saw the request..I'll circle back in a bit | |
| 19:02:41 | gibi | thanks | |
| 23:48:37 | opendevreview | Merged openstack/nova master: Move monkey_patch from init to the entrypoints https://review.opendev.org/c/openstack/nova/+/965923 | |
| #openstack-nova - 2025-11-07 | |||
| 05:59:29 | koya | Hi Uggla, bauzas, thank you very much for discussing “Instance metadata protection” at the recent OpenInfra Summit. | |
| 05:59:58 | koya | Based on the comments you kindly provided, I’ve been reviewing and updating (revising) the blueprint; https://blueprints.launchpad.net/nova/+spec/instance-metadata-protection. I’ll reach out to you again when it’s ready. | |
| 10:38:19 | artem_vasilyev | Hi, I stumbled upon the _cleanup_incomplete_migrations function and can't quite get whether this is intended logic or not. I couldn't find answers in the documentation. As I understand, the purpose of this function is to clean orphan files of already deleted instances. But when deleting instances, their migrations are also marked as soft-deleted (see https://github.com/openstack/nova/blob/master/nova/db/main/api.py#L1348). So this function | |
| 10:38:23 | opendevreview | Adrien Cunin proposed openstack/os-vif master: Fixed bridge name when per_port_bridge is used https://review.opendev.org/c/openstack/os-vif/+/966410 | |
| 10:39:35 | jkulik | artem_vasilyev: fyi, your message is cut off after ". So this function" | |
| 10:40:28 | artem_vasilyev | thanks. So this function can't get error migrations, because it doesn't query migrations with read_deleted context. Is this the expected behavior, or am I missing something? | |
| 10:43:46 | artem_vasilyev | _cleanup_incomplete_migrations function is here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L11944 | |
| 10:50:50 | jkulik | the tests mock away MigrationList.get_by_filters(), so they can't see it. from looking at the code, I'd agree with you artem_vasilyev | |
| 12:33:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 12:33:18 | opendevreview | Balazs Gibizer proposed openstack/nova master: Do not use libvirt before Host is initialized https://review.opendev.org/c/openstack/nova/+/966430 | |
| 12:52:35 | opendevreview | Merged openstack/nova master: Default native threading for sch, api and metadata https://review.opendev.org/c/openstack/nova/+/965924 | |
| 13:34:21 | gibi | dansmith: this might be interesting to you https://bugs.launchpad.net/nova/+bug/2130881 | |
| 13:34:46 | gibi | dansmith: I'm fairly certain that we break libvirt lifecycle event handling with the patch https://review.opendev.org/c/openstack/nova/+/897076 - Sanity check that new hosts have no instances | |
| 13:35:06 | gibi | but only for the first nova-compute startup case. Later nova-compute restarts are not affected | |
| 13:36:18 | gibi | we need to reorder ComputeManager.init_host a bit to call driver.init_host before doing any libvirt driver queries | |
| 13:44:23 | gibi | I will push a fix | |
| 13:48:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Show libvirt lifecycle event issue https://review.opendev.org/c/openstack/nova/+/966436 | |
| 13:48:55 | gibi | this should show the problem clearly ^^ | |