Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-02-11
16:21:16 bauzas elodilles_pto: oh he's not here
16:21:30 bauzas no worries, let's skip it for this week
16:21:37 bauzas #topic vmwareapi 3rd-party CI efforts Highlights
16:21:41 bauzas fwiesel: anything to talk ?
16:21:51 fwiesel Hi, yes... some progress on the last outstanding failure.
16:22:15 elodilles (sorry, i'm here, just forgot to change my nick o:) but stable gates seems to be healthy, fwiw)
16:22:30 fwiesel I suspect I do have a possible patch for that, but now I run into a bug into Cinder. I've created a bug report for it (https://bugs.launchpad.net/cinder/+bug/2097771) and want to fix that.
16:22:35 bauzas elodilles: noted.
16:22:51 bauzas fwiesel: ack, thanks for the reporting
16:22:59 fwiesel If that works out, I will ask you to review my solution for nova making use of that.
16:23:05 fwiesel That's from my side.
16:24:35 bauzas thanks
16:24:54 bauzas #topic Open discussion
16:25:30 bauzas we have one topic to discuss today
16:25:38 bauzas sp-bmilanov -- 927474: libvirt: Fetch the available vCPUs from the respective cgroup
16:25:40 bauzas sp-bmilanov: around ?
16:25:44 sp-bmilanov hi! yep
16:25:50 bauzas #link https://review.opendev.org/c/openstack/nova/+/927474
16:26:05 bauzas sean-k-mooney: are you here ?
16:26:13 sean-k-mooney yes
16:26:28 sp-bmilanov the topic is about how to handle the case where Nova is running in a hyperconverged setup where the are CPUs that are not available to Nova
16:26:33 sean-k-mooney i do not belive nova should do that but ill let sp-bmilanov explain
16:26:54 sp-bmilanov I was hoping to crowd-source alternatives
16:27:27 sean-k-mooney perhaps some context will help
16:27:49 sp-bmilanov the fundamental problem IMO is with libvirt -- it assumes all online CPUs == schedulable CPUs, and does not provide an API to query actually schedulable CPUs
16:27:53 gibi > CPUs that are not available to Nova < that feels like the existing cpuset config can handle
16:27:54 sean-k-mooney its possible to use the cgroups api to subdevied the host and mark some host as reserved for exclusive use fo a slice/partion
16:28:02 sean-k-mooney gibi: it can
16:28:20 sean-k-mooney sp-bmilanov would like nova to parse the cgroups toplogy and try and auto comput eit
16:28:43 sean-k-mooney one of my bigest issues with that is libvirt and nova-comptue might have diffent view of the cgroups
16:28:58 sean-k-mooney i.e. nova is deployed in a podman container and livbirt is installed on the host
16:29:12 sp-bmilanov sean-k-mooney: that was our initial approach, but we trying to solve the issue of Nova not using CPUs that are not availble for it
16:29:19 gibi what is the problem then setting the cpuset config option to only allow nova to use the CPUs the deployer wants?
16:29:33 sean-k-mooney sp-bmilanov: we have config options ot specify which cpu nova can use
16:29:49 sean-k-mooney but you explcitly do not want to use the mechanisume we provide for this
16:30:25 sp-bmilanov yep, we'd like to find a way in which Nova can handle this itself
16:30:49 sp-bmilanov else it's a runtime error on VM "creation"
16:31:13 gibi sp-bmilanov: why the cpuset config is not enough?
16:31:41 sean-k-mooney gibi: it does work but sp-bmilanov does not like the ux as they precive it as overly complex to have to configure it
16:31:43 gibi what is the reason you want more automatic discovery of available cpus?
16:32:43 sean-k-mooney nova has supported specifying which cpus can be used for guest via config since the intoduction of vcpu_pin_set in essex ish
16:33:18 sean-k-mooney we currently declare it to be the isntaller responcablity to configure nova for the host it is deployed on
16:33:34 sp-bmilanov yes, partly UX, and that we think it might be possible to do automatically
16:34:13 sean-k-mooney i dont think it really is, not without libvirt changes IMO
16:34:44 sean-k-mooney i think we would need libvirt to provide an api that told nova which cpus could be used
16:35:03 sean-k-mooney we currently use the capablitis api to discuver this info from libvirt
16:35:24 sean-k-mooney that provides us the cpu toplogy, numa affintity ectra
16:36:09 sean-k-mooney it does not tell use which cpus are reserved for exlisve use and cannot be used by libvirt to spawn a vm
16:37:11 sp-bmilanov I agree that this is ultimately an issue in libvirt, but I am concerned that I am not sure how long it would take this to propagate to OpenStack, and in which releases
16:38:01 sp-bmilanov and if there is something adequate we can do in Nova in the meantime, why not
16:38:03 sean-k-mooney i dont think it wold be a good idea to make nova discover this via parsing /sys or /proc
16:38:16 bauzas that's a good question. Usually we ask the reporter to first engage with the libvirt community to see their thoughts
16:38:21 dansmith I don't even understand how that _could_ be a thing
16:38:53 sean-k-mooney the poc patch looks at a particalar part fo /sys to get the cpu set allowed by nova's cgroup
16:38:58 sp-bmilanov dansmith: the use case is hyperconverged setups where there is a e.g. storage service with reserved CPUs installed on the hypervisor
16:39:05 sean-k-mooney but as i noted above that is nto the same as libvirts
16:39:07 dansmith if we need to know what libvirt has available for us and we can't ask it, I don't really see what options we have for discovering it
16:39:10 sean-k-mooney so it would be the wrong info
16:39:20 dansmith sp-bmilanov: I get the use-case totally
16:39:43 dansmith sean-k-mooney: does /sys's view of available CPUs change by cgroup? I had assumed not
16:40:24 sean-k-mooney the patch was going ot look at cgroup specific subpaths
16:40:32 dansmith but either way, that doesn't seem like a solution because we don't know that our cgroup is the same (or configured the same) as libvirt's
16:40:32 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/927474/10/nova/virt/libvirt/host.py#93
16:40:39 sean-k-mooney /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus',
16:40:43 dansmith ah I see
16:41:30 sp-bmilanov yep, that's not ideal, but we can simplify or extend the checks as much as deemed necessary
16:41:34 bauzas last time we tried to look at sysfs for CPU management, it created some problems
16:42:02 sean-k-mooney sp-bmilanov: in general we try to aovid parsing /sys or /proc if at all posible
16:42:05 bauzas so I'm afraid some OSes couldn't support that
16:42:11 sean-k-mooney it create mantaince issues for us
16:42:23 dansmith a feature that works "sometimes" if a bunch of assumptions are made (i.e. libvirt and nova in the same slice) seems worse than no feature to me, in a lot of cases
16:43:03 sean-k-mooney if we were to do it i could maybe see doing if and only if you provide the path or slice as a config option
16:43:13 sean-k-mooney and only if you configure it
16:43:16 sean-k-mooney i.e. opt in
16:43:22 sean-k-mooney not on by default
16:43:24 sp-bmilanov sean-k-mooney that sounds good
16:43:38 sp-bmilanov an opt-in, if-set -like option
16:44:00 sp-bmilanov a bit like the CPU pinning, but more autonomous
16:44:33 sean-k-mooney mostly but im not sure how other feel about that
16:44:59 bauzas seems yet another knob to me
16:45:11 dansmith can we see all the slices in sysfs or only ours?
16:45:18 bauzas with a fragile interface and many ways to get it wrong
16:45:29 sean-k-mooney that depends
16:45:41 dansmith if we can see all of them, then passing the name/path to the one libvirt is in so we can inspect it would be better, but idk.. seems fragile to me like bauzas says
16:45:41 sean-k-mooney if your in a contianer you cant see the host one by default
16:46:00 dansmith sean-k-mooney: right, so we likely can't see libvirt's either right?
16:46:20 sean-k-mooney not thte way tripleo or kolla deploys nova
16:46:31 sean-k-mooney or our new instaler
16:46:45 dansmith yeah
16:46:48 sp-bmilanov cgroupns_mode: "host""
16:46:48 sp-bmilanov sean-k-mooney mentioned in the change that kolla and tripleo are working around it... sean-k-mooney, where do you "not typiclaly run the nova-comptue binary with
16:47:11 sean-k-mooney right so exisitng installer only set that on libvirt
16:47:14 sean-k-mooney not on nova
16:47:21 sean-k-mooney so nova-comptue has a restricted view
16:47:37 sp-bmilanov ah, right.. but which installer is that?
16:47:55 sp-bmilanov (I might be missing Nova context)
16:48:26 sp-bmilanov (btw, there's the other side of the discussion, if we were to treat that currently Nova can start and schedule VMs with this misconfiguration, what would be the fix so it does not start at all (as sean-k-mooney suggested in the change comments))
16:49:24 sean-k-mooney libvirt has it in kolla https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/defaults/main.yml#L11
16:49:29 sean-k-mooney but not nova-compute https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/defaults/main.yml#L55
16:49:40 sean-k-mooney adn its the same in tripleo and the new isntaller redhat wrote

Earlier   Later