| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-21 | |||
| 07:21:18 | gibi | bauzas: good morning | |
| 07:21:28 | bauzas | gibi: modifying the commit msg would mean that the CI job would accept it ? | |
| 07:21:53 | bauzas | haha | |
| 07:21:55 | bauzas | "Stable branch requires either cherry-pick -x headers or [stable-only] tag!" | |
| 07:22:03 | gibi | bauzas: yes, either the patch needs to have a cherry-picked from line where the hash points to a merged change or the patch needs to state that it is stable only | |
| 07:22:09 | bauzas | saw in https://zuul.opendev.org/t/openstack/build/97c7712edfe34c5386c0cc2fc37ef466 | |
| 07:22:20 | bauzas | gibi: kk, +2 then | |
| 07:24:02 | bauzas | gibi: looks like we need to do this also for all of the changes https://review.opendev.org/q/owner:infra-root%2540openstack.org+(project:openstack/nova+OR+project:openstack/placement)+is:open | |
| 07:28:51 | bauzas | ah no | |
| 07:29:01 | bauzas | just rechecks because of timeouts in placement | |
| 07:34:53 | gibi | I think we dont have the backport check in placement | |
| 07:35:31 | gibi | the placement lower constraint timeout needs https://review.opendev.org/c/openstack/placement/+/810001 to merge first | |
| 07:48:17 | bauzas | gibi: this is correct, hence my "ah no" :) | |
| 07:48:32 | bauzas | (placement doesn't verify the backports) | |
| 07:49:12 | bauzas | gibi: and shit, you're right | |
| 07:49:18 | bauzas | we need a second core | |
| 07:51:17 | bauzas | lyarwood: I know your placement knowledge is the same than old greek, but we'd appreciate a +W for a dependency bump https://review.opendev.org/c/openstack/placement/+/810001 in order to fix the gate | |
| 07:56:32 | gibi | bauzas: also the stable team needs to make a short term decision what to do with the lower constraint bump on stable/xean | |
| 07:56:53 | bauzas | we have an open change about it | |
| 07:56:54 | gibi | it is not really allowed, but we can push an RC2 to be in the xena release | |
| 07:57:11 | bauzas | https://review.opendev.org/c/openstack/placement/+/787863 | |
| 07:57:28 | gibi | yeah that also possible | |
| 07:57:32 | gibi | turning off the job | |
| 07:57:57 | bauzas | honestly, I have no opinion and this could be discussed at the PTG | |
| 07:58:14 | bauzas | but we need to consider the bump, agreed | |
| 07:58:25 | gibi | so 1) bump lower constarint and release RC2 so we are not breaking stable policy abour reqs 2) try to pin virtualenv during tox installation to avoid unpinned setuptools on stable (this is the long term solution) 3) turn off the lower job | |
| 07:59:27 | gibi | and the problem is not placement specific but hitting a lot of projects | |
| 07:59:31 | gibi | nova is not impacted | |
| 07:59:44 | gibi | so a common solution would be good | |
| 08:20:18 | bauzas | gibi: let's discuss this during our meeting tonight | |
| 08:20:26 | gibi | sure | |
| 08:20:35 | gibi | bauzas: btw would you like to take over charing of the meeting? | |
| 08:20:44 | bauzas | gibi: I was thinking about it | |
| 08:20:53 | bauzas | I can do it :) | |
| 08:20:55 | gibi | elodilles: will you be available on todays meeting to talk about the stable req bump problem? | |
| 08:21:04 | gibi | bauzas: then I officially give you the baton :) | |
| 08:21:14 | bauzas | huuuuuuuuuuh | |
| 08:23:00 | bauzas | https://www.youtube.com/watch?v=43RID9cIEAE | |
| 08:23:05 | bauzas | DOOOOOH | |
| 08:24:01 | gibi | bauzas: https://review.opendev.org/c/opendev/irc-meetings/+/810165 | |
| 08:25:16 | bauzas | gibi: thanks, was looking on it...after finding the Simpson video :) | |
| 08:26:54 | gibi | bauzas: on the today meeting we should also talk about the release liaison role | |
| 08:27:07 | bauzas | gibi: yeah, I was about to ask for it | |
| 08:27:11 | gibi | cool | |
| 08:27:13 | bauzas | also, we have other liaisons fwiw | |
| 08:28:07 | gibi | for me only the release one was visible | |
| 08:28:34 | bauzas | technically we have other roles in our team :) https://wiki.openstack.org/wiki/Nova#People | |
| 08:28:55 | bauzas | but meh | |
| 08:29:10 | bauzas | it's not like we have 50 engineers wondering how to help our community :) | |
| 08:31:27 | lyarwood | \o mornig | |
| 08:35:18 | elodilles | gibi: yes, i'll be there on the meeting :) | |
| 08:38:06 | gibi | elodilles: do you already see a common solution forming from stable perspective? | |
| 08:42:25 | gibi | lyarwood: o/ | |
| 08:45:19 | elodilles | o/ | |
| 08:46:41 | elodilles | gibi: btw, I agree with your option 2 but needs to be checked with infra I think | |
| 08:48:27 | gibi | elodilles: yeah, I can accept that option 2 needs more effort probaly to make it right so if we need a short term solution then I'm fine with either 1 or 3. 1 has the limit that we need to cut RC this week. 3 has the effect that we will never turn the job back again | |
| 08:50:06 | gibi | hm feels like the gate is clogged | |
| 08:50:08 | gibi | :/ | |
| 09:14:46 | bauzas | gibi: elodilles: fwiw, I'm in favor of bumping the dep now, which is option 1 and revisit the issue at the PTG | |
| 09:15:04 | bauzas | short-term, we need to addree the fire | |
| 09:15:07 | bauzas | address* | |
| 09:15:23 | bauzas | long-term, we could draft a plan on how to avoid fire next time | |
| 09:28:44 | gibi | bauzas: I was cheated a bit with 1). on stable/xena this is viable as we can have RC2. But on older stable branches we only have the option of 2) or 3) (or going against the stable req policy) | |
| 09:37:57 | bauzas | gibi: what policy issue do you see ? we can bump a .y release which can need to upgrade a dependency, right? | |
| 09:41:27 | alexe9191 | good day everyone :) after restarting the scheduler after an OpenStack upgrades all of the schedulers are running 100% cpu and are taking a lot of time building the cache. Scheduling is not working of course as the schedulers are busy creating this cache. | |
| 09:41:58 | alexe9191 | I am wondering if adding more schedulers would solve this problem ? current resources allocated to each scheduler is about 8G of memory and 8 CPUS | |
| 09:42:26 | alexe9191 | but I do not think that the # of cpu matters as it seems that the python process is single threaded. I also did not find any "workers" configuration for the scheduler in the newton version. | |
| 09:44:07 | gibi | bauzas: on stable we should not bump major version of a dependency I think | |
| 09:44:23 | bauzas | for a .z release, yes | |
| 09:44:53 | gibi | can we do that for a .y. release? | |
| 09:45:15 | bauzas | lemme try to look at the Openstack semver rules | |
| 09:45:24 | bauzas | that's not really a "stable policy" AFAIK | |
| 09:45:37 | bauzas | at least, I can't find anything in https://docs.openstack.org/project-team-guide/stable-branches.html | |
| 09:46:02 | bauzas | there it is https://docs.openstack.org/pbr/latest/user/semver.html#semantic-versioning-specification-semver | |
| 09:48:52 | bauzas | alas, can't find anything specific | |
| 09:49:24 | bauzas | we won't technically change placement API if we bump the minimum | |
| 09:49:54 | bauzas | gibi: I guess we should raise this question to the wider community, maybe | |
| 09:50:11 | bauzas | and see whether they freak out about upgrading our deps in a .y release | |
| 09:50:52 | alexe9191 | anyone on the scheduling issue :) ? | |
| 09:54:02 | opendevreview | Merged openstack/placement master: Bump min decorator to 4.0.0 https://review.opendev.org/c/openstack/placement/+/810001 | |
| 09:54:17 | bauzas | alexe9191: we superseded the idea to have multiple workers by telling we should rather use Placement | |
| 09:54:48 | bauzas | alexe9191: for the CPU usage, try to get a GMR | |
| 09:55:17 | bauzas | alexe9191: https://docs.openstack.org/nova/latest/reference/gmr.html | |
| 09:55:35 | bauzas | so you should see the greenlets and greenthreads concurrently running | |
| 09:56:17 | alexe9191 | I am still on the way to upgrade to rocky and to use the placement fully. I can not drop the sceduler in newton as far as I know | |
| 09:56:36 | bauzas | nova-scheduler will continue to exist | |
| 09:56:55 | bauzas | it's just that scheduler will call out placement for getting a list of candidates before running | |
| 09:57:15 | bauzas | which reduces the amount of complexity by adding a single-lock mechanism | |
| 09:58:11 | alexe9191 | We will start using the placement UI as recommended. However we need to upgrade first. We are coming from Kilo and need to transit first at newton before we continue the migrations. | |
| 09:59:11 | alexe9191 | The scheduler is running at 100% cpu for about an hour now and updating host stats/aggregates etc as I can see in the log files. And as I understand, this needs to be done first before the scheduler starts scheduling anything by either using placement or using the filters. | |
| 09:59:32 | alexe9191 | So my question is, would adding more workers speed up the process of building that cache? or it's just a matter of waiting | |
| 10:02:16 | gibi | bauzas: good idea to start a discussion around this in the wider community | |
| 10:14:46 | bauzas | alexe9191: honestly, updating the cache shouldn't take one hour | |
| 10:14:50 | bauzas | you have another issue | |
| 10:15:09 | alexe9191 | What could that be? | |
| 10:15:32 | bauzas | I dunno, get the GMR | |
| 10:15:40 | alexe9191 | I see a ton of those messages `Update host state with aggregates` | |
| 10:15:54 | bauzas | how many aggregates do you have ? | |
| 10:15:57 | alexe9191 | We also have about 900 hosts and more than 10K vms | |
| 10:16:04 | alexe9191 | one moment | |