| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-02-11 | |||
| 16:18:23 | bauzas | given my on-off presence on IRC, I appreciate any way to discuss async | |
| 16:18:45 | bauzas | #topic PTG planning | |
| 16:19:14 | bauzas | well, I won't run that PTG by myself but this is a reminder that we'll have a PTG | |
| 16:19:23 | bauzas | #info Next PTG will be held on Apr 7-11 | |
| 16:19:39 | bauzas | I started to draft some etherpad for that PTG | |
| 16:19:46 | bauzas | #link https://etherpad.opendev.org/p/nova-2025.2-ptg | |
| 16:19:55 | bauzas | feel free to add your topics worth of interest into it ^ | |
| 16:21:07 | bauzas | #topic Stable Branches | |
| 16:21:16 | bauzas | elodilles_pto: oh he's not here | |
| 16:21:30 | bauzas | no worries, let's skip it for this week | |
| 16:21:37 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:21:41 | bauzas | fwiesel: anything to talk ? | |
| 16:21:51 | fwiesel | Hi, yes... some progress on the last outstanding failure. | |
| 16:22:15 | elodilles | (sorry, i'm here, just forgot to change my nick o:) but stable gates seems to be healthy, fwiw) | |
| 16:22:30 | fwiesel | I suspect I do have a possible patch for that, but now I run into a bug into Cinder. I've created a bug report for it (https://bugs.launchpad.net/cinder/+bug/2097771) and want to fix that. | |
| 16:22:35 | bauzas | elodilles: noted. | |
| 16:22:51 | bauzas | fwiesel: ack, thanks for the reporting | |
| 16:22:59 | fwiesel | If that works out, I will ask you to review my solution for nova making use of that. | |
| 16:23:05 | fwiesel | That's from my side. | |
| 16:24:35 | bauzas | thanks | |
| 16:24:54 | bauzas | #topic Open discussion | |
| 16:25:30 | bauzas | we have one topic to discuss today | |
| 16:25:38 | bauzas | sp-bmilanov -- 927474: libvirt: Fetch the available vCPUs from the respective cgroup | |
| 16:25:40 | bauzas | sp-bmilanov: around ? | |
| 16:25:44 | sp-bmilanov | hi! yep | |
| 16:25:50 | bauzas | #link https://review.opendev.org/c/openstack/nova/+/927474 | |
| 16:26:05 | bauzas | sean-k-mooney: are you here ? | |
| 16:26:13 | sean-k-mooney | yes | |
| 16:26:28 | sp-bmilanov | the topic is about how to handle the case where Nova is running in a hyperconverged setup where the are CPUs that are not available to Nova | |
| 16:26:33 | sean-k-mooney | i do not belive nova should do that but ill let sp-bmilanov explain | |
| 16:26:54 | sp-bmilanov | I was hoping to crowd-source alternatives | |
| 16:27:27 | sean-k-mooney | perhaps some context will help | |
| 16:27:49 | sp-bmilanov | the fundamental problem IMO is with libvirt -- it assumes all online CPUs == schedulable CPUs, and does not provide an API to query actually schedulable CPUs | |
| 16:27:53 | gibi | > CPUs that are not available to Nova < that feels like the existing cpuset config can handle | |
| 16:27:54 | sean-k-mooney | its possible to use the cgroups api to subdevied the host and mark some host as reserved for exclusive use fo a slice/partion | |
| 16:28:02 | sean-k-mooney | gibi: it can | |
| 16:28:20 | sean-k-mooney | sp-bmilanov would like nova to parse the cgroups toplogy and try and auto comput eit | |
| 16:28:43 | sean-k-mooney | one of my bigest issues with that is libvirt and nova-comptue might have diffent view of the cgroups | |
| 16:28:58 | sean-k-mooney | i.e. nova is deployed in a podman container and livbirt is installed on the host | |
| 16:29:12 | sp-bmilanov | sean-k-mooney: that was our initial approach, but we trying to solve the issue of Nova not using CPUs that are not availble for it | |
| 16:29:19 | gibi | what is the problem then setting the cpuset config option to only allow nova to use the CPUs the deployer wants? | |
| 16:29:33 | sean-k-mooney | sp-bmilanov: we have config options ot specify which cpu nova can use | |
| 16:29:49 | sean-k-mooney | but you explcitly do not want to use the mechanisume we provide for this | |
| 16:30:25 | sp-bmilanov | yep, we'd like to find a way in which Nova can handle this itself | |
| 16:30:49 | sp-bmilanov | else it's a runtime error on VM "creation" | |
| 16:31:13 | gibi | sp-bmilanov: why the cpuset config is not enough? | |
| 16:31:41 | sean-k-mooney | gibi: it does work but sp-bmilanov does not like the ux as they precive it as overly complex to have to configure it | |
| 16:31:43 | gibi | what is the reason you want more automatic discovery of available cpus? | |
| 16:32:43 | sean-k-mooney | nova has supported specifying which cpus can be used for guest via config since the intoduction of vcpu_pin_set in essex ish | |
| 16:33:18 | sean-k-mooney | we currently declare it to be the isntaller responcablity to configure nova for the host it is deployed on | |
| 16:33:34 | sp-bmilanov | yes, partly UX, and that we think it might be possible to do automatically | |
| 16:34:13 | sean-k-mooney | i dont think it really is, not without libvirt changes IMO | |
| 16:34:44 | sean-k-mooney | i think we would need libvirt to provide an api that told nova which cpus could be used | |
| 16:35:03 | sean-k-mooney | we currently use the capablitis api to discuver this info from libvirt | |
| 16:35:24 | sean-k-mooney | that provides us the cpu toplogy, numa affintity ectra | |
| 16:36:09 | sean-k-mooney | it does not tell use which cpus are reserved for exlisve use and cannot be used by libvirt to spawn a vm | |
| 16:37:11 | sp-bmilanov | I agree that this is ultimately an issue in libvirt, but I am concerned that I am not sure how long it would take this to propagate to OpenStack, and in which releases | |
| 16:38:01 | sp-bmilanov | and if there is something adequate we can do in Nova in the meantime, why not | |
| 16:38:03 | sean-k-mooney | i dont think it wold be a good idea to make nova discover this via parsing /sys or /proc | |
| 16:38:16 | bauzas | that's a good question. Usually we ask the reporter to first engage with the libvirt community to see their thoughts | |
| 16:38:21 | dansmith | I don't even understand how that _could_ be a thing | |
| 16:38:53 | sean-k-mooney | the poc patch looks at a particalar part fo /sys to get the cpu set allowed by nova's cgroup | |
| 16:38:58 | sp-bmilanov | dansmith: the use case is hyperconverged setups where there is a e.g. storage service with reserved CPUs installed on the hypervisor | |
| 16:39:05 | sean-k-mooney | but as i noted above that is nto the same as libvirts | |
| 16:39:07 | dansmith | if we need to know what libvirt has available for us and we can't ask it, I don't really see what options we have for discovering it | |
| 16:39:10 | sean-k-mooney | so it would be the wrong info | |
| 16:39:20 | dansmith | sp-bmilanov: I get the use-case totally | |
| 16:39:43 | dansmith | sean-k-mooney: does /sys's view of available CPUs change by cgroup? I had assumed not | |
| 16:40:24 | sean-k-mooney | the patch was going ot look at cgroup specific subpaths | |
| 16:40:32 | dansmith | but either way, that doesn't seem like a solution because we don't know that our cgroup is the same (or configured the same) as libvirt's | |
| 16:40:32 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/927474/10/nova/virt/libvirt/host.py#93 | |
| 16:40:39 | sean-k-mooney | /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus', | |
| 16:40:43 | dansmith | ah I see | |
| 16:41:30 | sp-bmilanov | yep, that's not ideal, but we can simplify or extend the checks as much as deemed necessary | |
| 16:41:34 | bauzas | last time we tried to look at sysfs for CPU management, it created some problems | |
| 16:42:02 | sean-k-mooney | sp-bmilanov: in general we try to aovid parsing /sys or /proc if at all posible | |
| 16:42:05 | bauzas | so I'm afraid some OSes couldn't support that | |
| 16:42:11 | sean-k-mooney | it create mantaince issues for us | |
| 16:42:23 | dansmith | a feature that works "sometimes" if a bunch of assumptions are made (i.e. libvirt and nova in the same slice) seems worse than no feature to me, in a lot of cases | |
| 16:43:03 | sean-k-mooney | if we were to do it i could maybe see doing if and only if you provide the path or slice as a config option | |
| 16:43:13 | sean-k-mooney | and only if you configure it | |
| 16:43:16 | sean-k-mooney | i.e. opt in | |
| 16:43:22 | sean-k-mooney | not on by default | |
| 16:43:24 | sp-bmilanov | sean-k-mooney that sounds good | |
| 16:43:38 | sp-bmilanov | an opt-in, if-set -like option | |
| 16:44:00 | sp-bmilanov | a bit like the CPU pinning, but more autonomous | |
| 16:44:33 | sean-k-mooney | mostly but im not sure how other feel about that | |
| 16:44:59 | bauzas | seems yet another knob to me | |
| 16:45:11 | dansmith | can we see all the slices in sysfs or only ours? | |
| 16:45:18 | bauzas | with a fragile interface and many ways to get it wrong | |
| 16:45:29 | sean-k-mooney | that depends | |
| 16:45:41 | dansmith | if we can see all of them, then passing the name/path to the one libvirt is in so we can inspect it would be better, but idk.. seems fragile to me like bauzas says | |
| 16:45:41 | sean-k-mooney | if your in a contianer you cant see the host one by default | |
| 16:46:00 | dansmith | sean-k-mooney: right, so we likely can't see libvirt's either right? | |
| 16:46:20 | sean-k-mooney | not thte way tripleo or kolla deploys nova | |
| 16:46:31 | sean-k-mooney | or our new instaler | |
| 16:46:45 | dansmith | yeah | |
| 16:46:48 | sp-bmilanov | cgroupns_mode: "host"" | |
| 16:46:48 | sp-bmilanov | sean-k-mooney mentioned in the change that kolla and tripleo are working around it... sean-k-mooney, where do you "not typiclaly run the nova-comptue binary with | |
| 16:47:11 | sean-k-mooney | right so exisitng installer only set that on libvirt | |