| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-02-11 | |||
| 16:19:46 | bauzas | #link https://etherpad.opendev.org/p/nova-2025.2-ptg | |
| 16:19:55 | bauzas | feel free to add your topics worth of interest into it ^ | |
| 16:21:07 | bauzas | #topic Stable Branches | |
| 16:21:16 | bauzas | elodilles_pto: oh he's not here | |
| 16:21:30 | bauzas | no worries, let's skip it for this week | |
| 16:21:37 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:21:41 | bauzas | fwiesel: anything to talk ? | |
| 16:21:51 | fwiesel | Hi, yes... some progress on the last outstanding failure. | |
| 16:22:15 | elodilles | (sorry, i'm here, just forgot to change my nick o:) but stable gates seems to be healthy, fwiw) | |
| 16:22:30 | fwiesel | I suspect I do have a possible patch for that, but now I run into a bug into Cinder. I've created a bug report for it (https://bugs.launchpad.net/cinder/+bug/2097771) and want to fix that. | |
| 16:22:35 | bauzas | elodilles: noted. | |
| 16:22:51 | bauzas | fwiesel: ack, thanks for the reporting | |
| 16:22:59 | fwiesel | If that works out, I will ask you to review my solution for nova making use of that. | |
| 16:23:05 | fwiesel | That's from my side. | |
| 16:24:35 | bauzas | thanks | |
| 16:24:54 | bauzas | #topic Open discussion | |
| 16:25:30 | bauzas | we have one topic to discuss today | |
| 16:25:38 | bauzas | sp-bmilanov -- 927474: libvirt: Fetch the available vCPUs from the respective cgroup | |
| 16:25:40 | bauzas | sp-bmilanov: around ? | |
| 16:25:44 | sp-bmilanov | hi! yep | |
| 16:25:50 | bauzas | #link https://review.opendev.org/c/openstack/nova/+/927474 | |
| 16:26:05 | bauzas | sean-k-mooney: are you here ? | |
| 16:26:13 | sean-k-mooney | yes | |
| 16:26:28 | sp-bmilanov | the topic is about how to handle the case where Nova is running in a hyperconverged setup where the are CPUs that are not available to Nova | |
| 16:26:33 | sean-k-mooney | i do not belive nova should do that but ill let sp-bmilanov explain | |
| 16:26:54 | sp-bmilanov | I was hoping to crowd-source alternatives | |
| 16:27:27 | sean-k-mooney | perhaps some context will help | |
| 16:27:49 | sp-bmilanov | the fundamental problem IMO is with libvirt -- it assumes all online CPUs == schedulable CPUs, and does not provide an API to query actually schedulable CPUs | |
| 16:27:53 | gibi | > CPUs that are not available to Nova < that feels like the existing cpuset config can handle | |
| 16:27:54 | sean-k-mooney | its possible to use the cgroups api to subdevied the host and mark some host as reserved for exclusive use fo a slice/partion | |
| 16:28:02 | sean-k-mooney | gibi: it can | |
| 16:28:20 | sean-k-mooney | sp-bmilanov would like nova to parse the cgroups toplogy and try and auto comput eit | |
| 16:28:43 | sean-k-mooney | one of my bigest issues with that is libvirt and nova-comptue might have diffent view of the cgroups | |
| 16:28:58 | sean-k-mooney | i.e. nova is deployed in a podman container and livbirt is installed on the host | |
| 16:29:12 | sp-bmilanov | sean-k-mooney: that was our initial approach, but we trying to solve the issue of Nova not using CPUs that are not availble for it | |
| 16:29:19 | gibi | what is the problem then setting the cpuset config option to only allow nova to use the CPUs the deployer wants? | |
| 16:29:33 | sean-k-mooney | sp-bmilanov: we have config options ot specify which cpu nova can use | |
| 16:29:49 | sean-k-mooney | but you explcitly do not want to use the mechanisume we provide for this | |
| 16:30:25 | sp-bmilanov | yep, we'd like to find a way in which Nova can handle this itself | |
| 16:30:49 | sp-bmilanov | else it's a runtime error on VM "creation" | |
| 16:31:13 | gibi | sp-bmilanov: why the cpuset config is not enough? | |
| 16:31:41 | sean-k-mooney | gibi: it does work but sp-bmilanov does not like the ux as they precive it as overly complex to have to configure it | |
| 16:31:43 | gibi | what is the reason you want more automatic discovery of available cpus? | |
| 16:32:43 | sean-k-mooney | nova has supported specifying which cpus can be used for guest via config since the intoduction of vcpu_pin_set in essex ish | |
| 16:33:18 | sean-k-mooney | we currently declare it to be the isntaller responcablity to configure nova for the host it is deployed on | |
| 16:33:34 | sp-bmilanov | yes, partly UX, and that we think it might be possible to do automatically | |
| 16:34:13 | sean-k-mooney | i dont think it really is, not without libvirt changes IMO | |
| 16:34:44 | sean-k-mooney | i think we would need libvirt to provide an api that told nova which cpus could be used | |
| 16:35:03 | sean-k-mooney | we currently use the capablitis api to discuver this info from libvirt | |
| 16:35:24 | sean-k-mooney | that provides us the cpu toplogy, numa affintity ectra | |
| 16:36:09 | sean-k-mooney | it does not tell use which cpus are reserved for exlisve use and cannot be used by libvirt to spawn a vm | |
| 16:37:11 | sp-bmilanov | I agree that this is ultimately an issue in libvirt, but I am concerned that I am not sure how long it would take this to propagate to OpenStack, and in which releases | |
| 16:38:01 | sp-bmilanov | and if there is something adequate we can do in Nova in the meantime, why not | |
| 16:38:03 | sean-k-mooney | i dont think it wold be a good idea to make nova discover this via parsing /sys or /proc | |
| 16:38:16 | bauzas | that's a good question. Usually we ask the reporter to first engage with the libvirt community to see their thoughts | |
| 16:38:21 | dansmith | I don't even understand how that _could_ be a thing | |
| 16:38:53 | sean-k-mooney | the poc patch looks at a particalar part fo /sys to get the cpu set allowed by nova's cgroup | |
| 16:38:58 | sp-bmilanov | dansmith: the use case is hyperconverged setups where there is a e.g. storage service with reserved CPUs installed on the hypervisor | |
| 16:39:05 | sean-k-mooney | but as i noted above that is nto the same as libvirts | |
| 16:39:07 | dansmith | if we need to know what libvirt has available for us and we can't ask it, I don't really see what options we have for discovering it | |
| 16:39:10 | sean-k-mooney | so it would be the wrong info | |
| 16:39:20 | dansmith | sp-bmilanov: I get the use-case totally | |
| 16:39:43 | dansmith | sean-k-mooney: does /sys's view of available CPUs change by cgroup? I had assumed not | |
| 16:40:24 | sean-k-mooney | the patch was going ot look at cgroup specific subpaths | |
| 16:40:32 | dansmith | but either way, that doesn't seem like a solution because we don't know that our cgroup is the same (or configured the same) as libvirt's | |
| 16:40:32 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/927474/10/nova/virt/libvirt/host.py#93 | |
| 16:40:39 | sean-k-mooney | /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus', | |
| 16:40:43 | dansmith | ah I see | |
| 16:41:30 | sp-bmilanov | yep, that's not ideal, but we can simplify or extend the checks as much as deemed necessary | |
| 16:41:34 | bauzas | last time we tried to look at sysfs for CPU management, it created some problems | |
| 16:42:02 | sean-k-mooney | sp-bmilanov: in general we try to aovid parsing /sys or /proc if at all posible | |
| 16:42:05 | bauzas | so I'm afraid some OSes couldn't support that | |
| 16:42:11 | sean-k-mooney | it create mantaince issues for us | |
| 16:42:23 | dansmith | a feature that works "sometimes" if a bunch of assumptions are made (i.e. libvirt and nova in the same slice) seems worse than no feature to me, in a lot of cases | |
| 16:43:03 | sean-k-mooney | if we were to do it i could maybe see doing if and only if you provide the path or slice as a config option | |
| 16:43:13 | sean-k-mooney | and only if you configure it | |
| 16:43:16 | sean-k-mooney | i.e. opt in | |
| 16:43:22 | sean-k-mooney | not on by default | |
| 16:43:24 | sp-bmilanov | sean-k-mooney that sounds good | |
| 16:43:38 | sp-bmilanov | an opt-in, if-set -like option | |
| 16:44:00 | sp-bmilanov | a bit like the CPU pinning, but more autonomous | |
| 16:44:33 | sean-k-mooney | mostly but im not sure how other feel about that | |
| 16:44:59 | bauzas | seems yet another knob to me | |
| 16:45:11 | dansmith | can we see all the slices in sysfs or only ours? | |
| 16:45:18 | bauzas | with a fragile interface and many ways to get it wrong | |
| 16:45:29 | sean-k-mooney | that depends | |
| 16:45:41 | dansmith | if we can see all of them, then passing the name/path to the one libvirt is in so we can inspect it would be better, but idk.. seems fragile to me like bauzas says | |
| 16:45:41 | sean-k-mooney | if your in a contianer you cant see the host one by default | |
| 16:46:00 | dansmith | sean-k-mooney: right, so we likely can't see libvirt's either right? | |
| 16:46:20 | sean-k-mooney | not thte way tripleo or kolla deploys nova | |
| 16:46:31 | sean-k-mooney | or our new instaler | |
| 16:46:45 | dansmith | yeah | |
| 16:46:48 | sp-bmilanov | cgroupns_mode: "host"" | |
| 16:46:48 | sp-bmilanov | sean-k-mooney mentioned in the change that kolla and tripleo are working around it... sean-k-mooney, where do you "not typiclaly run the nova-comptue binary with | |
| 16:47:11 | sean-k-mooney | right so exisitng installer only set that on libvirt | |
| 16:47:14 | sean-k-mooney | not on nova | |
| 16:47:21 | sean-k-mooney | so nova-comptue has a restricted view | |
| 16:47:37 | sp-bmilanov | ah, right.. but which installer is that? | |
| 16:47:55 | sp-bmilanov | (I might be missing Nova context) | |
| 16:48:26 | sp-bmilanov | (btw, there's the other side of the discussion, if we were to treat that currently Nova can start and schedule VMs with this misconfiguration, what would be the fix so it does not start at all (as sean-k-mooney suggested in the change comments)) | |