Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-21
07:16:30 opendevreview Balazs Gibizer proposed openstack/nova stable/xena: [stable-only]Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/nova/+/809760
07:16:36 opendevreview Balazs Gibizer proposed openstack/nova stable/xena: [stable-only]Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/nova/+/809760
07:17:00 gibi zigo, artom, clarkb: I've updated the stable/xena setup patches
07:17:51 gibi I think in general the backport check helps enforceing that we only merge patches in a good branch order
07:18:07 gibi new stable branch setup seems to be an edge case
07:19:26 bauzas good morning folks
07:20:41 bauzas gibi: hmmmm
07:21:18 gibi bauzas: good morning
07:21:28 bauzas gibi: modifying the commit msg would mean that the CI job would accept it ?
07:21:53 bauzas haha
07:21:55 bauzas "Stable branch requires either cherry-pick -x headers or [stable-only] tag!"
07:22:03 gibi bauzas: yes, either the patch needs to have a cherry-picked from line where the hash points to a merged change or the patch needs to state that it is stable only
07:22:09 bauzas saw in https://zuul.opendev.org/t/openstack/build/97c7712edfe34c5386c0cc2fc37ef466
07:22:20 bauzas gibi: kk, +2 then
07:24:02 bauzas gibi: looks like we need to do this also for all of the changes https://review.opendev.org/q/owner:infra-root%2540openstack.org+(project:openstack/nova+OR+project:openstack/placement)+is:open
07:28:51 bauzas ah no
07:29:01 bauzas just rechecks because of timeouts in placement
07:34:53 gibi I think we dont have the backport check in placement
07:35:31 gibi the placement lower constraint timeout needs https://review.opendev.org/c/openstack/placement/+/810001 to merge first
07:48:17 bauzas gibi: this is correct, hence my "ah no" :)
07:48:32 bauzas (placement doesn't verify the backports)
07:49:12 bauzas gibi: and shit, you're right
07:49:18 bauzas we need a second core
07:51:17 bauzas lyarwood: I know your placement knowledge is the same than old greek, but we'd appreciate a +W for a dependency bump https://review.opendev.org/c/openstack/placement/+/810001 in order to fix the gate
07:56:32 gibi bauzas: also the stable team needs to make a short term decision what to do with the lower constraint bump on stable/xean
07:56:53 bauzas we have an open change about it
07:56:54 gibi it is not really allowed, but we can push an RC2 to be in the xena release
07:57:11 bauzas https://review.opendev.org/c/openstack/placement/+/787863
07:57:28 gibi yeah that also possible
07:57:32 gibi turning off the job
07:57:57 bauzas honestly, I have no opinion and this could be discussed at the PTG
07:58:14 bauzas but we need to consider the bump, agreed
07:58:25 gibi so 1) bump lower constarint and release RC2 so we are not breaking stable policy abour reqs 2) try to pin virtualenv during tox installation to avoid unpinned setuptools on stable (this is the long term solution) 3) turn off the lower job
07:59:27 gibi and the problem is not placement specific but hitting a lot of projects
07:59:31 gibi nova is not impacted
07:59:44 gibi so a common solution would be good
08:20:18 bauzas gibi: let's discuss this during our meeting tonight
08:20:26 gibi sure
08:20:35 gibi bauzas: btw would you like to take over charing of the meeting?
08:20:44 bauzas gibi: I was thinking about it
08:20:53 bauzas I can do it :)
08:20:55 gibi elodilles: will you be available on todays meeting to talk about the stable req bump problem?
08:21:04 gibi bauzas: then I officially give you the baton :)
08:21:14 bauzas huuuuuuuuuuh
08:23:00 bauzas https://www.youtube.com/watch?v=43RID9cIEAE
08:23:05 bauzas DOOOOOH
08:24:01 gibi bauzas: https://review.opendev.org/c/opendev/irc-meetings/+/810165
08:25:16 bauzas gibi: thanks, was looking on it...after finding the Simpson video :)
08:26:54 gibi bauzas: on the today meeting we should also talk about the release liaison role
08:27:07 bauzas gibi: yeah, I was about to ask for it
08:27:11 gibi cool
08:27:13 bauzas also, we have other liaisons fwiw
08:28:07 gibi for me only the release one was visible
08:28:34 bauzas technically we have other roles in our team :) https://wiki.openstack.org/wiki/Nova#People
08:28:55 bauzas but meh
08:29:10 bauzas it's not like we have 50 engineers wondering how to help our community :)
08:31:27 lyarwood \o mornig
08:35:18 elodilles gibi: yes, i'll be there on the meeting :)
08:38:06 gibi elodilles: do you already see a common solution forming from stable perspective?
08:42:25 gibi lyarwood: o/
08:45:19 elodilles o/
08:46:41 elodilles gibi: btw, I agree with your option 2 but needs to be checked with infra I think
08:48:27 gibi elodilles: yeah, I can accept that option 2 needs more effort probaly to make it right so if we need a short term solution then I'm fine with either 1 or 3. 1 has the limit that we need to cut RC this week. 3 has the effect that we will never turn the job back again
08:50:06 gibi hm feels like the gate is clogged
08:50:08 gibi :/
09:14:46 bauzas gibi: elodilles: fwiw, I'm in favor of bumping the dep now, which is option 1 and revisit the issue at the PTG
09:15:04 bauzas short-term, we need to addree the fire
09:15:07 bauzas address*
09:15:23 bauzas long-term, we could draft a plan on how to avoid fire next time
09:28:44 gibi bauzas: I was cheated a bit with 1). on stable/xena this is viable as we can have RC2. But on older stable branches we only have the option of 2) or 3) (or going against the stable req policy)
09:37:57 bauzas gibi: what policy issue do you see ? we can bump a .y release which can need to upgrade a dependency, right?
09:41:27 alexe9191 good day everyone :) after restarting the scheduler after an OpenStack upgrades all of the schedulers are running 100% cpu and are taking a lot of time building the cache. Scheduling is not working of course as the schedulers are busy creating this cache.
09:41:58 alexe9191 I am wondering if adding more schedulers would solve this problem ? current resources allocated to each scheduler is about 8G of memory and 8 CPUS
09:42:26 alexe9191 but I do not think that the # of cpu matters as it seems that the python process is single threaded. I also did not find any "workers" configuration for the scheduler in the newton version.
09:44:07 gibi bauzas: on stable we should not bump major version of a dependency I think
09:44:23 bauzas for a .z release, yes
09:44:53 gibi can we do that for a .y. release?
09:45:15 bauzas lemme try to look at the Openstack semver rules
09:45:24 bauzas that's not really a "stable policy" AFAIK
09:45:37 bauzas at least, I can't find anything in https://docs.openstack.org/project-team-guide/stable-branches.html
09:46:02 bauzas there it is https://docs.openstack.org/pbr/latest/user/semver.html#semantic-versioning-specification-semver
09:48:52 bauzas alas, can't find anything specific
09:49:24 bauzas we won't technically change placement API if we bump the minimum
09:49:54 bauzas gibi: I guess we should raise this question to the wider community, maybe
09:50:11 bauzas and see whether they freak out about upgrading our deps in a .y release
09:50:52 alexe9191 anyone on the scheduling issue :) ?
09:54:02 opendevreview Merged openstack/placement master: Bump min decorator to 4.0.0 https://review.opendev.org/c/openstack/placement/+/810001
09:54:17 bauzas alexe9191: we superseded the idea to have multiple workers by telling we should rather use Placement
09:54:48 bauzas alexe9191: for the CPU usage, try to get a GMR
09:55:17 bauzas alexe9191: https://docs.openstack.org/nova/latest/reference/gmr.html
09:55:35 bauzas so you should see the greenlets and greenthreads concurrently running
09:56:17 alexe9191 I am still on the way to upgrade to rocky and to use the placement fully. I can not drop the sceduler in newton as far as I know
09:56:36 bauzas nova-scheduler will continue to exist
09:56:55 bauzas it's just that scheduler will call out placement for getting a list of candidates before running
09:57:15 bauzas which reduces the amount of complexity by adding a single-lock mechanism
09:58:11 alexe9191 We will start using the placement UI as recommended. However we need to upgrade first. We are coming from Kilo and need to transit first at newton before we continue the migrations.
09:59:11 alexe9191 The scheduler is running at 100% cpu for about an hour now and updating host stats/aggregates etc as I can see in the log files. And as I understand, this needs to be done first before the scheduler starts scheduling anything by either using placement or using the filters.
09:59:32 alexe9191 So my question is, would adding more workers speed up the process of building that cache? or it's just a matter of waiting
10:02:16 gibi bauzas: good idea to start a discussion around this in the wider community
10:14:46 bauzas alexe9191: honestly, updating the cache shouldn't take one hour

Earlier   Later