| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-21 | |||
| 08:23:05 | bauzas | DOOOOOH | |
| 08:24:01 | gibi | bauzas: https://review.opendev.org/c/opendev/irc-meetings/+/810165 | |
| 08:25:16 | bauzas | gibi: thanks, was looking on it...after finding the Simpson video :) | |
| 08:26:54 | gibi | bauzas: on the today meeting we should also talk about the release liaison role | |
| 08:27:07 | bauzas | gibi: yeah, I was about to ask for it | |
| 08:27:11 | gibi | cool | |
| 08:27:13 | bauzas | also, we have other liaisons fwiw | |
| 08:28:07 | gibi | for me only the release one was visible | |
| 08:28:34 | bauzas | technically we have other roles in our team :) https://wiki.openstack.org/wiki/Nova#People | |
| 08:28:55 | bauzas | but meh | |
| 08:29:10 | bauzas | it's not like we have 50 engineers wondering how to help our community :) | |
| 08:31:27 | lyarwood | \o mornig | |
| 08:35:18 | elodilles | gibi: yes, i'll be there on the meeting :) | |
| 08:38:06 | gibi | elodilles: do you already see a common solution forming from stable perspective? | |
| 08:42:25 | gibi | lyarwood: o/ | |
| 08:45:19 | elodilles | o/ | |
| 08:46:41 | elodilles | gibi: btw, I agree with your option 2 but needs to be checked with infra I think | |
| 08:48:27 | gibi | elodilles: yeah, I can accept that option 2 needs more effort probaly to make it right so if we need a short term solution then I'm fine with either 1 or 3. 1 has the limit that we need to cut RC this week. 3 has the effect that we will never turn the job back again | |
| 08:50:06 | gibi | hm feels like the gate is clogged | |
| 08:50:08 | gibi | :/ | |
| 09:14:46 | bauzas | gibi: elodilles: fwiw, I'm in favor of bumping the dep now, which is option 1 and revisit the issue at the PTG | |
| 09:15:04 | bauzas | short-term, we need to addree the fire | |
| 09:15:07 | bauzas | address* | |
| 09:15:23 | bauzas | long-term, we could draft a plan on how to avoid fire next time | |
| 09:28:44 | gibi | bauzas: I was cheated a bit with 1). on stable/xena this is viable as we can have RC2. But on older stable branches we only have the option of 2) or 3) (or going against the stable req policy) | |
| 09:37:57 | bauzas | gibi: what policy issue do you see ? we can bump a .y release which can need to upgrade a dependency, right? | |
| 09:41:27 | alexe9191 | good day everyone :) after restarting the scheduler after an OpenStack upgrades all of the schedulers are running 100% cpu and are taking a lot of time building the cache. Scheduling is not working of course as the schedulers are busy creating this cache. | |
| 09:41:58 | alexe9191 | I am wondering if adding more schedulers would solve this problem ? current resources allocated to each scheduler is about 8G of memory and 8 CPUS | |
| 09:42:26 | alexe9191 | but I do not think that the # of cpu matters as it seems that the python process is single threaded. I also did not find any "workers" configuration for the scheduler in the newton version. | |
| 09:44:07 | gibi | bauzas: on stable we should not bump major version of a dependency I think | |
| 09:44:23 | bauzas | for a .z release, yes | |
| 09:44:53 | gibi | can we do that for a .y. release? | |
| 09:45:15 | bauzas | lemme try to look at the Openstack semver rules | |
| 09:45:24 | bauzas | that's not really a "stable policy" AFAIK | |
| 09:45:37 | bauzas | at least, I can't find anything in https://docs.openstack.org/project-team-guide/stable-branches.html | |
| 09:46:02 | bauzas | there it is https://docs.openstack.org/pbr/latest/user/semver.html#semantic-versioning-specification-semver | |
| 09:48:52 | bauzas | alas, can't find anything specific | |
| 09:49:24 | bauzas | we won't technically change placement API if we bump the minimum | |
| 09:49:54 | bauzas | gibi: I guess we should raise this question to the wider community, maybe | |
| 09:50:11 | bauzas | and see whether they freak out about upgrading our deps in a .y release | |
| 09:50:52 | alexe9191 | anyone on the scheduling issue :) ? | |
| 09:54:02 | opendevreview | Merged openstack/placement master: Bump min decorator to 4.0.0 https://review.opendev.org/c/openstack/placement/+/810001 | |
| 09:54:17 | bauzas | alexe9191: we superseded the idea to have multiple workers by telling we should rather use Placement | |
| 09:54:48 | bauzas | alexe9191: for the CPU usage, try to get a GMR | |
| 09:55:17 | bauzas | alexe9191: https://docs.openstack.org/nova/latest/reference/gmr.html | |
| 09:55:35 | bauzas | so you should see the greenlets and greenthreads concurrently running | |
| 09:56:17 | alexe9191 | I am still on the way to upgrade to rocky and to use the placement fully. I can not drop the sceduler in newton as far as I know | |
| 09:56:36 | bauzas | nova-scheduler will continue to exist | |
| 09:56:55 | bauzas | it's just that scheduler will call out placement for getting a list of candidates before running | |
| 09:57:15 | bauzas | which reduces the amount of complexity by adding a single-lock mechanism | |
| 09:58:11 | alexe9191 | We will start using the placement UI as recommended. However we need to upgrade first. We are coming from Kilo and need to transit first at newton before we continue the migrations. | |
| 09:59:11 | alexe9191 | The scheduler is running at 100% cpu for about an hour now and updating host stats/aggregates etc as I can see in the log files. And as I understand, this needs to be done first before the scheduler starts scheduling anything by either using placement or using the filters. | |
| 09:59:32 | alexe9191 | So my question is, would adding more workers speed up the process of building that cache? or it's just a matter of waiting | |
| 10:02:16 | gibi | bauzas: good idea to start a discussion around this in the wider community | |
| 10:14:46 | bauzas | alexe9191: honestly, updating the cache shouldn't take one hour | |
| 10:14:50 | bauzas | you have another issue | |
| 10:15:09 | alexe9191 | What could that be? | |
| 10:15:32 | bauzas | I dunno, get the GMR | |
| 10:15:40 | alexe9191 | I see a ton of those messages `Update host state with aggregates` | |
| 10:15:54 | bauzas | how many aggregates do you have ? | |
| 10:15:57 | alexe9191 | We also have about 900 hosts and more than 10K vms | |
| 10:16:04 | alexe9191 | one moment | |
| 10:16:25 | alexe9191 | 9 | |
| 10:16:57 | alexe9191 | I also see a lot of those "Update host state with service dict" | |
| 10:22:53 | bauzas | 9 can't be an issue | |
| 10:23:26 | bauzas | again, please do a GMR and see what threads run | |
| 10:23:40 | alexe9191 | I am honestly not sure what you mean when you say do a GMR? :) | |
| 10:24:24 | alexe9191 | ah `kill -USR2 8675 ` | |
| 10:24:32 | alexe9191 | that wouldn't terminate the process right ? | |
| 10:37:35 | alexe9191 | Nova did not generate any report | |
| 10:42:09 | sean-k-mooney | alexe9191: it will more or less kille the process | |
| 10:42:18 | sean-k-mooney | the report is stored in the nova log | |
| 10:42:24 | sean-k-mooney | where ever you redirect that too | |
| 10:42:33 | sean-k-mooney | to fix the process you will need to restart it | |
| 10:43:58 | alexe9191 | I did send the USR2 signal but the logs where empty | |
| 10:44:05 | alexe9191 | well not empty I did not see any GMR logs | |
| 10:44:15 | sean-k-mooney | what process did you send it to | |
| 10:44:19 | alexe9191 | nova-scheduler | |
| 10:44:23 | alexe9191 | and also nova-api as a test | |
| 10:44:43 | alexe9191 | I am actually wondering if having debug enabled is causing an overhead? | |
| 10:44:43 | sean-k-mooney | nova-api wont work if its running under appache or uwsgi | |
| 10:44:47 | sean-k-mooney | but the shcheduler should | |
| 10:44:56 | sean-k-mooney | debug logging | |
| 10:44:59 | alexe9191 | it's not it's running under dumb-init. All of the processes are running in containers. | |
| 10:45:05 | sean-k-mooney | am if you have slow disks then it could | |
| 10:45:17 | sean-k-mooney | oh this is kolla | |
| 10:45:25 | alexe9191 | Yeah more or less, kolla containers | |
| 10:45:30 | sean-k-mooney | so dumb-init is likely not passing the signal | |
| 10:45:48 | alexe9191 | hmmmm | |
| 10:45:53 | sean-k-mooney | that said if you used the pid of the python process not dumb-init it should work | |
| 10:46:04 | alexe9191 | Let me try it out again | |
| 10:50:39 | alexe9191 | Nothing | |
| 10:50:48 | alexe9191 | I do however see a lot of those in the logs "Lock "host_instance" acquired by "nova.scheduler.host_manager.sync_instance_info"" | |
| 10:51:16 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Add regression test for bug #1908075 https://review.opendev.org/c/openstack/nova/+/810191 | |
| 11:13:54 | alexe9191 | and now after about 90 minutes it's done | |
| 11:30:37 | gibi | bauzas: could you please triage this vgpu bug https://bugs.launchpad.net/nova/+bug/1943933 ? | |
| 11:33:10 | sean-k-mooney | i think the way our config is ment to work is we list the adress of the parent | |
| 11:33:37 | sean-k-mooney | and then report the quantity of the mdev type that can be created for that device | |
| 11:34:05 | gibi | ohh so the reported wants to partially report the possible mdevs from a physical device | |
| 11:34:20 | sean-k-mooney | i think so | |