| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-02-11 | |||
| 16:35:24 | sean-k-mooney | that provides us the cpu toplogy, numa affintity ectra | |
| 16:36:09 | sean-k-mooney | it does not tell use which cpus are reserved for exlisve use and cannot be used by libvirt to spawn a vm | |
| 16:37:11 | sp-bmilanov | I agree that this is ultimately an issue in libvirt, but I am concerned that I am not sure how long it would take this to propagate to OpenStack, and in which releases | |
| 16:38:01 | sp-bmilanov | and if there is something adequate we can do in Nova in the meantime, why not | |
| 16:38:03 | sean-k-mooney | i dont think it wold be a good idea to make nova discover this via parsing /sys or /proc | |
| 16:38:16 | bauzas | that's a good question. Usually we ask the reporter to first engage with the libvirt community to see their thoughts | |
| 16:38:21 | dansmith | I don't even understand how that _could_ be a thing | |
| 16:38:53 | sean-k-mooney | the poc patch looks at a particalar part fo /sys to get the cpu set allowed by nova's cgroup | |
| 16:38:58 | sp-bmilanov | dansmith: the use case is hyperconverged setups where there is a e.g. storage service with reserved CPUs installed on the hypervisor | |
| 16:39:05 | sean-k-mooney | but as i noted above that is nto the same as libvirts | |
| 16:39:07 | dansmith | if we need to know what libvirt has available for us and we can't ask it, I don't really see what options we have for discovering it | |
| 16:39:10 | sean-k-mooney | so it would be the wrong info | |
| 16:39:20 | dansmith | sp-bmilanov: I get the use-case totally | |
| 16:39:43 | dansmith | sean-k-mooney: does /sys's view of available CPUs change by cgroup? I had assumed not | |
| 16:40:24 | sean-k-mooney | the patch was going ot look at cgroup specific subpaths | |
| 16:40:32 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/927474/10/nova/virt/libvirt/host.py#93 | |
| 16:40:32 | dansmith | but either way, that doesn't seem like a solution because we don't know that our cgroup is the same (or configured the same) as libvirt's | |
| 16:40:39 | sean-k-mooney | /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus', | |
| 16:40:43 | dansmith | ah I see | |
| 16:41:30 | sp-bmilanov | yep, that's not ideal, but we can simplify or extend the checks as much as deemed necessary | |
| 16:41:34 | bauzas | last time we tried to look at sysfs for CPU management, it created some problems | |
| 16:42:02 | sean-k-mooney | sp-bmilanov: in general we try to aovid parsing /sys or /proc if at all posible | |
| 16:42:05 | bauzas | so I'm afraid some OSes couldn't support that | |
| 16:42:11 | sean-k-mooney | it create mantaince issues for us | |
| 16:42:23 | dansmith | a feature that works "sometimes" if a bunch of assumptions are made (i.e. libvirt and nova in the same slice) seems worse than no feature to me, in a lot of cases | |
| 16:43:03 | sean-k-mooney | if we were to do it i could maybe see doing if and only if you provide the path or slice as a config option | |
| 16:43:13 | sean-k-mooney | and only if you configure it | |
| 16:43:16 | sean-k-mooney | i.e. opt in | |
| 16:43:22 | sean-k-mooney | not on by default | |
| 16:43:24 | sp-bmilanov | sean-k-mooney that sounds good | |
| 16:43:38 | sp-bmilanov | an opt-in, if-set -like option | |
| 16:44:00 | sp-bmilanov | a bit like the CPU pinning, but more autonomous | |
| 16:44:33 | sean-k-mooney | mostly but im not sure how other feel about that | |
| 16:44:59 | bauzas | seems yet another knob to me | |
| 16:45:11 | dansmith | can we see all the slices in sysfs or only ours? | |
| 16:45:18 | bauzas | with a fragile interface and many ways to get it wrong | |
| 16:45:29 | sean-k-mooney | that depends | |
| 16:45:41 | sean-k-mooney | if your in a contianer you cant see the host one by default | |
| 16:45:41 | dansmith | if we can see all of them, then passing the name/path to the one libvirt is in so we can inspect it would be better, but idk.. seems fragile to me like bauzas says | |
| 16:46:00 | dansmith | sean-k-mooney: right, so we likely can't see libvirt's either right? | |
| 16:46:20 | sean-k-mooney | not thte way tripleo or kolla deploys nova | |
| 16:46:31 | sean-k-mooney | or our new instaler | |
| 16:46:45 | dansmith | yeah | |
| 16:46:48 | sp-bmilanov | sean-k-mooney mentioned in the change that kolla and tripleo are working around it... sean-k-mooney, where do you "not typiclaly run the nova-comptue binary with | |
| 16:46:48 | sp-bmilanov | cgroupns_mode: "host"" | |
| 16:47:11 | sean-k-mooney | right so exisitng installer only set that on libvirt | |
| 16:47:14 | sean-k-mooney | not on nova | |
| 16:47:21 | sean-k-mooney | so nova-comptue has a restricted view | |
| 16:47:37 | sp-bmilanov | ah, right.. but which installer is that? | |
| 16:47:55 | sp-bmilanov | (I might be missing Nova context) | |
| 16:48:26 | sp-bmilanov | (btw, there's the other side of the discussion, if we were to treat that currently Nova can start and schedule VMs with this misconfiguration, what would be the fix so it does not start at all (as sean-k-mooney suggested in the change comments)) | |
| 16:49:24 | sean-k-mooney | libvirt has it in kolla https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/defaults/main.yml#L11 | |
| 16:49:29 | sean-k-mooney | but not nova-compute https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/defaults/main.yml#L55 | |
| 16:49:40 | sean-k-mooney | adn its the same in tripleo and the new isntaller redhat wrote | |
| 16:50:08 | sean-k-mooney | sp-bmilanov: what installer do you use | |
| 16:50:21 | bauzas | time check : 10 mins before end of meeting | |
| 16:50:34 | sean-k-mooney | dansmith: bauzas am i right in thinking you agree this is out of scope fo nova | |
| 16:50:40 | sean-k-mooney | and in scope fo the installer | |
| 16:51:09 | sp-bmilanov | we have seen a few, but yes, kolla and no installer (opereating system packages) | |
| 16:51:20 | dansmith | I mean, that's kinda my point that this seems like us introspecting the system to dynamically configure ourselves, but in a way that can only work in certain situations | |
| 16:51:21 | bauzas | I'd tend to say yes, this sounds a tooling mechanism that could set the config correctly | |
| 16:51:51 | sean-k-mooney | i would be less agaisnt this if libvirt provided an api for this | |
| 16:51:57 | bauzas | the concern I have is that this approach doesn't sound generic at all and very targeted to a specific case | |
| 16:52:22 | sean-k-mooney | also those config options we use are ment to be virt dirver independent | |
| 16:52:37 | dansmith | right, if libvirt exposes it, that's totally different | |
| 16:53:02 | dansmith | in that case, configuring libvirt is the job of the installer, and nova can maybe even lose some config, right? or at least, not always require it to be set | |
| 16:53:05 | gibi | also it opens the question what if the cgroups config changes while nova-compute is running | |
| 16:53:18 | sean-k-mooney | fun times :) | |
| 16:53:40 | gibi | at least the current config nova has is validated at nova-compute startup | |
| 16:54:11 | sean-k-mooney | yep it wont actully catch this specific case | |
| 16:54:29 | sean-k-mooney | i.e. we validate the cpu are in the info provided by libvirt | |
| 16:54:35 | sean-k-mooney | and if they are online ectra | |
| 16:54:48 | sean-k-mooney | but we dont have visablity into cgroups at all really | |
| 16:54:53 | sean-k-mooney | at least not in this context | |
| 16:55:19 | sean-k-mooney | we have some basic check to know if its cgroups v2 vs v1 | |
| 16:56:17 | bauzas | time check : 4 mins left. | |
| 16:57:11 | gibi | this is complexity that either lives in the installer or lives in nova, as nova is big enough already I rather see this complexity added in the installer. Sure we have multiple installers so that duplicates some effort. | |
| 16:57:32 | dansmith | ++ | |
| 16:57:35 | sp-bmilanov | I was wondering if there is also a way to know this outside of reading cgroups that might be more robust | |
| 16:57:51 | sean-k-mooney | if there is im not aware of one | |
| 16:58:05 | dansmith | there is actually | |
| 16:58:11 | dansmith | ....and it's talking to libvirt :D | |
| 16:58:15 | sean-k-mooney | :) | |
| 16:58:36 | sp-bmilanov | :D | |
| 16:58:43 | bauzas | I guess we can't solve that question now | |
| 16:59:09 | bauzas | sp-bmilanov: could you at least talk to the libvirt community and see their toughts about that ? | |
| 16:59:15 | sp-bmilanov | in any case, it sounds like a good idea to initiate a discussion with libvirt devs as well | |
| 16:59:16 | bauzas | and then come back to us ? | |
| 16:59:28 | sp-bmilanov | bauzas: yep | |
| 16:59:31 | bauzas | cool | |
| 16:59:41 | bauzas | thanks then | |
| 16:59:52 | sp-bmilanov | you're welcome | |
| 17:00:04 | sp-bmilanov | thanks for the input, all | |
| 17:00:08 | bauzas | if you're OK, I think we're done for today | |
| 17:00:14 | bauzas | thanks all | |
| 17:00:17 | bauzas | #endmeeting | |
| 17:00:17 | opendevmeet | Meeting ended Tue Feb 11 17:00:17 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:00:17 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-02-11-16.01.html | |
| 17:00:17 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-02-11-16.01.txt | |
| 17:00:17 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-02-11-16.01.log.html | |