| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-02-11 | |||
| 16:08:33 | bauzas | when looking at https://review.opendev.org/c/openstack/nova/+/940642, even by rechecking, I got the same exceptions on nova-next and nova-multi-cell jobs | |
| 16:08:37 | bauzas | for the same test | |
| 16:09:06 | bauzas | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ba0/940642/5/check/nova-next/ba009c2/testr_results.html | |
| 16:09:12 | bauzas | https://6ae015e4c924496dfda3-74c26bb67ca36871e200d07f3210a3f0.ssl.cf1.rackcdn.com/940642/5/check/nova-multi-cell/c2a2f68/testr_results.html | |
| 16:09:19 | bauzas | have people seen the same failure ? | |
| 16:09:43 | bauzas | last time I looked, I saw nova-next failing once over 5 times | |
| 16:10:00 | bauzas | this would create some problems later in the week | |
| 16:10:39 | bauzas | so I wonder whether we should skip that specific test | |
| 16:11:26 | Uggla | @bauzas, the one I pushed yesterday pass the tests | |
| 16:12:01 | bauzas | hmmm | |
| 16:12:11 | bauzas | okay, will wee | |
| 16:12:13 | bauzas | see* | |
| 16:12:41 | Uggla | maybe you were not unlucky. | |
| 16:13:25 | bauzas | oh, nvm found the root case, PEBKAC | |
| 16:13:55 | bauzas | https://6ae015e4c924496dfda3-74c26bb67ca36871e200d07f3210a3f0.ssl.cf1.rackcdn.com/940642/5/check/nova-multi-cell/c2a2f68/controller/logs/screen-n-sch.txt | |
| 16:14:02 | bauzas | this is because of my patch | |
| 16:14:09 | bauzas | so we can move on | |
| 16:14:29 | bauzas | any other failure to discuss ? | |
| 16:15:05 | bauzas | #topic Release Planning | |
| 16:15:30 | bauzas | #link https://releases.openstack.org/epoxy/schedule.html | |
| 16:15:35 | bauzas | #info Nova deadlines are set in the above schedule | |
| 16:15:41 | bauzas | #info 2 weeks before Feature Freeze | |
| 16:16:01 | bauzas | so maybe we should go straight to the next topic | |
| 16:16:07 | bauzas | #topic Review priorities | |
| 16:16:14 | bauzas | #link https://etherpad.opendev.org/p/nova-2025.1-status | |
| 16:16:39 | bauzas | I'll be honest, I was swamped by other priorities but I want to review a long list of changes this week | |
| 16:16:45 | bauzas | that becomes super urgent | |
| 16:17:35 | dviroel | fyi, I added at line 131 links to the blueprint and gerrit topic wrt scheduler-hints-in-server-details | |
| 16:17:45 | dviroel | please take a look when you folks have some time :) | |
| 16:18:02 | bauzas | noted, appreciated | |
| 16:18:23 | bauzas | given my on-off presence on IRC, I appreciate any way to discuss async | |
| 16:18:45 | bauzas | #topic PTG planning | |
| 16:19:14 | bauzas | well, I won't run that PTG by myself but this is a reminder that we'll have a PTG | |
| 16:19:23 | bauzas | #info Next PTG will be held on Apr 7-11 | |
| 16:19:39 | bauzas | I started to draft some etherpad for that PTG | |
| 16:19:46 | bauzas | #link https://etherpad.opendev.org/p/nova-2025.2-ptg | |
| 16:19:55 | bauzas | feel free to add your topics worth of interest into it ^ | |
| 16:21:07 | bauzas | #topic Stable Branches | |
| 16:21:16 | bauzas | elodilles_pto: oh he's not here | |
| 16:21:30 | bauzas | no worries, let's skip it for this week | |
| 16:21:37 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:21:41 | bauzas | fwiesel: anything to talk ? | |
| 16:21:51 | fwiesel | Hi, yes... some progress on the last outstanding failure. | |
| 16:22:15 | elodilles | (sorry, i'm here, just forgot to change my nick o:) but stable gates seems to be healthy, fwiw) | |
| 16:22:30 | fwiesel | I suspect I do have a possible patch for that, but now I run into a bug into Cinder. I've created a bug report for it (https://bugs.launchpad.net/cinder/+bug/2097771) and want to fix that. | |
| 16:22:35 | bauzas | elodilles: noted. | |
| 16:22:51 | bauzas | fwiesel: ack, thanks for the reporting | |
| 16:22:59 | fwiesel | If that works out, I will ask you to review my solution for nova making use of that. | |
| 16:23:05 | fwiesel | That's from my side. | |
| 16:24:35 | bauzas | thanks | |
| 16:24:54 | bauzas | #topic Open discussion | |
| 16:25:30 | bauzas | we have one topic to discuss today | |
| 16:25:38 | bauzas | sp-bmilanov -- 927474: libvirt: Fetch the available vCPUs from the respective cgroup | |
| 16:25:40 | bauzas | sp-bmilanov: around ? | |
| 16:25:44 | sp-bmilanov | hi! yep | |
| 16:25:50 | bauzas | #link https://review.opendev.org/c/openstack/nova/+/927474 | |
| 16:26:05 | bauzas | sean-k-mooney: are you here ? | |
| 16:26:13 | sean-k-mooney | yes | |
| 16:26:28 | sp-bmilanov | the topic is about how to handle the case where Nova is running in a hyperconverged setup where the are CPUs that are not available to Nova | |
| 16:26:33 | sean-k-mooney | i do not belive nova should do that but ill let sp-bmilanov explain | |
| 16:26:54 | sp-bmilanov | I was hoping to crowd-source alternatives | |
| 16:27:27 | sean-k-mooney | perhaps some context will help | |
| 16:27:49 | sp-bmilanov | the fundamental problem IMO is with libvirt -- it assumes all online CPUs == schedulable CPUs, and does not provide an API to query actually schedulable CPUs | |
| 16:27:53 | gibi | > CPUs that are not available to Nova < that feels like the existing cpuset config can handle | |
| 16:27:54 | sean-k-mooney | its possible to use the cgroups api to subdevied the host and mark some host as reserved for exclusive use fo a slice/partion | |
| 16:28:02 | sean-k-mooney | gibi: it can | |
| 16:28:20 | sean-k-mooney | sp-bmilanov would like nova to parse the cgroups toplogy and try and auto comput eit | |
| 16:28:43 | sean-k-mooney | one of my bigest issues with that is libvirt and nova-comptue might have diffent view of the cgroups | |
| 16:28:58 | sean-k-mooney | i.e. nova is deployed in a podman container and livbirt is installed on the host | |
| 16:29:12 | sp-bmilanov | sean-k-mooney: that was our initial approach, but we trying to solve the issue of Nova not using CPUs that are not availble for it | |
| 16:29:19 | gibi | what is the problem then setting the cpuset config option to only allow nova to use the CPUs the deployer wants? | |
| 16:29:33 | sean-k-mooney | sp-bmilanov: we have config options ot specify which cpu nova can use | |
| 16:29:49 | sean-k-mooney | but you explcitly do not want to use the mechanisume we provide for this | |
| 16:30:25 | sp-bmilanov | yep, we'd like to find a way in which Nova can handle this itself | |
| 16:30:49 | sp-bmilanov | else it's a runtime error on VM "creation" | |
| 16:31:13 | gibi | sp-bmilanov: why the cpuset config is not enough? | |
| 16:31:41 | sean-k-mooney | gibi: it does work but sp-bmilanov does not like the ux as they precive it as overly complex to have to configure it | |
| 16:31:43 | gibi | what is the reason you want more automatic discovery of available cpus? | |
| 16:32:43 | sean-k-mooney | nova has supported specifying which cpus can be used for guest via config since the intoduction of vcpu_pin_set in essex ish | |
| 16:33:18 | sean-k-mooney | we currently declare it to be the isntaller responcablity to configure nova for the host it is deployed on | |
| 16:33:34 | sp-bmilanov | yes, partly UX, and that we think it might be possible to do automatically | |
| 16:34:13 | sean-k-mooney | i dont think it really is, not without libvirt changes IMO | |
| 16:34:44 | sean-k-mooney | i think we would need libvirt to provide an api that told nova which cpus could be used | |
| 16:35:03 | sean-k-mooney | we currently use the capablitis api to discuver this info from libvirt | |
| 16:35:24 | sean-k-mooney | that provides us the cpu toplogy, numa affintity ectra | |
| 16:36:09 | sean-k-mooney | it does not tell use which cpus are reserved for exlisve use and cannot be used by libvirt to spawn a vm | |
| 16:37:11 | sp-bmilanov | I agree that this is ultimately an issue in libvirt, but I am concerned that I am not sure how long it would take this to propagate to OpenStack, and in which releases | |
| 16:38:01 | sp-bmilanov | and if there is something adequate we can do in Nova in the meantime, why not | |
| 16:38:03 | sean-k-mooney | i dont think it wold be a good idea to make nova discover this via parsing /sys or /proc | |
| 16:38:16 | bauzas | that's a good question. Usually we ask the reporter to first engage with the libvirt community to see their thoughts | |
| 16:38:21 | dansmith | I don't even understand how that _could_ be a thing | |
| 16:38:53 | sean-k-mooney | the poc patch looks at a particalar part fo /sys to get the cpu set allowed by nova's cgroup | |
| 16:38:58 | sp-bmilanov | dansmith: the use case is hyperconverged setups where there is a e.g. storage service with reserved CPUs installed on the hypervisor | |
| 16:39:05 | sean-k-mooney | but as i noted above that is nto the same as libvirts | |
| 16:39:07 | dansmith | if we need to know what libvirt has available for us and we can't ask it, I don't really see what options we have for discovering it | |
| 16:39:10 | sean-k-mooney | so it would be the wrong info | |
| 16:39:20 | dansmith | sp-bmilanov: I get the use-case totally | |
| 16:39:43 | dansmith | sean-k-mooney: does /sys's view of available CPUs change by cgroup? I had assumed not | |
| 16:40:24 | sean-k-mooney | the patch was going ot look at cgroup specific subpaths | |
| 16:40:32 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/927474/10/nova/virt/libvirt/host.py#93 | |