Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-21
08:20:44 bauzas gibi: I was thinking about it
08:20:53 bauzas I can do it :)
08:20:55 gibi elodilles: will you be available on todays meeting to talk about the stable req bump problem?
08:21:04 gibi bauzas: then I officially give you the baton :)
08:21:14 bauzas huuuuuuuuuuh
08:23:00 bauzas https://www.youtube.com/watch?v=43RID9cIEAE
08:23:05 bauzas DOOOOOH
08:24:01 gibi bauzas: https://review.opendev.org/c/opendev/irc-meetings/+/810165
08:25:16 bauzas gibi: thanks, was looking on it...after finding the Simpson video :)
08:26:54 gibi bauzas: on the today meeting we should also talk about the release liaison role
08:27:07 bauzas gibi: yeah, I was about to ask for it
08:27:11 gibi cool
08:27:13 bauzas also, we have other liaisons fwiw
08:28:07 gibi for me only the release one was visible
08:28:34 bauzas technically we have other roles in our team :) https://wiki.openstack.org/wiki/Nova#People
08:28:55 bauzas but meh
08:29:10 bauzas it's not like we have 50 engineers wondering how to help our community :)
08:31:27 lyarwood \o mornig
08:35:18 elodilles gibi: yes, i'll be there on the meeting :)
08:38:06 gibi elodilles: do you already see a common solution forming from stable perspective?
08:42:25 gibi lyarwood: o/
08:45:19 elodilles o/
08:46:41 elodilles gibi: btw, I agree with your option 2 but needs to be checked with infra I think
08:48:27 gibi elodilles: yeah, I can accept that option 2 needs more effort probaly to make it right so if we need a short term solution then I'm fine with either 1 or 3. 1 has the limit that we need to cut RC this week. 3 has the effect that we will never turn the job back again
08:50:06 gibi hm feels like the gate is clogged
08:50:08 gibi :/
09:14:46 bauzas gibi: elodilles: fwiw, I'm in favor of bumping the dep now, which is option 1 and revisit the issue at the PTG
09:15:04 bauzas short-term, we need to addree the fire
09:15:07 bauzas address*
09:15:23 bauzas long-term, we could draft a plan on how to avoid fire next time
09:28:44 gibi bauzas: I was cheated a bit with 1). on stable/xena this is viable as we can have RC2. But on older stable branches we only have the option of 2) or 3) (or going against the stable req policy)
09:37:57 bauzas gibi: what policy issue do you see ? we can bump a .y release which can need to upgrade a dependency, right?
09:41:27 alexe9191 good day everyone :) after restarting the scheduler after an OpenStack upgrades all of the schedulers are running 100% cpu and are taking a lot of time building the cache. Scheduling is not working of course as the schedulers are busy creating this cache.
09:41:58 alexe9191 I am wondering if adding more schedulers would solve this problem ? current resources allocated to each scheduler is about 8G of memory and 8 CPUS
09:42:26 alexe9191 but I do not think that the # of cpu matters as it seems that the python process is single threaded. I also did not find any "workers" configuration for the scheduler in the newton version.
09:44:07 gibi bauzas: on stable we should not bump major version of a dependency I think
09:44:23 bauzas for a .z release, yes
09:44:53 gibi can we do that for a .y. release?
09:45:15 bauzas lemme try to look at the Openstack semver rules
09:45:24 bauzas that's not really a "stable policy" AFAIK
09:45:37 bauzas at least, I can't find anything in https://docs.openstack.org/project-team-guide/stable-branches.html
09:46:02 bauzas there it is https://docs.openstack.org/pbr/latest/user/semver.html#semantic-versioning-specification-semver
09:48:52 bauzas alas, can't find anything specific
09:49:24 bauzas we won't technically change placement API if we bump the minimum
09:49:54 bauzas gibi: I guess we should raise this question to the wider community, maybe
09:50:11 bauzas and see whether they freak out about upgrading our deps in a .y release
09:50:52 alexe9191 anyone on the scheduling issue :) ?
09:54:02 opendevreview Merged openstack/placement master: Bump min decorator to 4.0.0 https://review.opendev.org/c/openstack/placement/+/810001
09:54:17 bauzas alexe9191: we superseded the idea to have multiple workers by telling we should rather use Placement
09:54:48 bauzas alexe9191: for the CPU usage, try to get a GMR
09:55:17 bauzas alexe9191: https://docs.openstack.org/nova/latest/reference/gmr.html
09:55:35 bauzas so you should see the greenlets and greenthreads concurrently running
09:56:17 alexe9191 I am still on the way to upgrade to rocky and to use the placement fully. I can not drop the sceduler in newton as far as I know
09:56:36 bauzas nova-scheduler will continue to exist
09:56:55 bauzas it's just that scheduler will call out placement for getting a list of candidates before running
09:57:15 bauzas which reduces the amount of complexity by adding a single-lock mechanism
09:58:11 alexe9191 We will start using the placement UI as recommended. However we need to upgrade first. We are coming from Kilo and need to transit first at newton before we continue the migrations.
09:59:11 alexe9191 The scheduler is running at 100% cpu for about an hour now and updating host stats/aggregates etc as I can see in the log files. And as I understand, this needs to be done first before the scheduler starts scheduling anything by either using placement or using the filters.
09:59:32 alexe9191 So my question is, would adding more workers speed up the process of building that cache? or it's just a matter of waiting
10:02:16 gibi bauzas: good idea to start a discussion around this in the wider community
10:14:46 bauzas alexe9191: honestly, updating the cache shouldn't take one hour
10:14:50 bauzas you have another issue
10:15:09 alexe9191 What could that be?
10:15:32 bauzas I dunno, get the GMR
10:15:40 alexe9191 I see a ton of those messages `Update host state with aggregates`
10:15:54 bauzas how many aggregates do you have ?
10:15:57 alexe9191 We also have about 900 hosts and more than 10K vms
10:16:04 alexe9191 one moment
10:16:25 alexe9191 9
10:16:57 alexe9191 I also see a lot of those "Update host state with service dict"
10:22:53 bauzas 9 can't be an issue
10:23:26 bauzas again, please do a GMR and see what threads run
10:23:40 alexe9191 I am honestly not sure what you mean when you say do a GMR? :)
10:24:24 alexe9191 ah `kill -USR2 8675 `
10:24:32 alexe9191 that wouldn't terminate the process right ?
10:37:35 alexe9191 Nova did not generate any report
10:42:09 sean-k-mooney alexe9191: it will more or less kille the process
10:42:18 sean-k-mooney the report is stored in the nova log
10:42:24 sean-k-mooney where ever you redirect that too
10:42:33 sean-k-mooney to fix the process you will need to restart it
10:43:58 alexe9191 I did send the USR2 signal but the logs where empty
10:44:05 alexe9191 well not empty I did not see any GMR logs
10:44:15 sean-k-mooney what process did you send it to
10:44:19 alexe9191 nova-scheduler
10:44:23 alexe9191 and also nova-api as a test
10:44:43 alexe9191 I am actually wondering if having debug enabled is causing an overhead?
10:44:43 sean-k-mooney nova-api wont work if its running under appache or uwsgi
10:44:47 sean-k-mooney but the shcheduler should
10:44:56 sean-k-mooney debug logging
10:44:59 alexe9191 it's not it's running under dumb-init. All of the processes are running in containers.
10:45:05 sean-k-mooney am if you have slow disks then it could
10:45:17 sean-k-mooney oh this is kolla
10:45:25 alexe9191 Yeah more or less, kolla containers
10:45:30 sean-k-mooney so dumb-init is likely not passing the signal
10:45:48 alexe9191 hmmmm
10:45:53 sean-k-mooney that said if you used the pid of the python process not dumb-init it should work
10:46:04 alexe9191 Let me try it out again
10:50:39 alexe9191 Nothing
10:50:48 alexe9191 I do however see a lot of those in the logs "Lock "host_instance" acquired by "nova.scheduler.host_manager.sync_instance_info""
10:51:16 opendevreview Vlad Gusev proposed openstack/nova stable/stein: Add regression test for bug #1908075 https://review.opendev.org/c/openstack/nova/+/810191

Earlier   Later