| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 08:51:44 | bauzas | well, I still don't get it | |
| 08:52:09 | bauzas | that's surely me, but if the operator said "okay, let's go with PCPU", that's because he opted-in, right? | |
| 08:52:35 | stephenfin | they've opted in by setting '[compute] cpu_dedicated_set' on some compute nodes, yes | |
| 08:53:08 | bauzas | but we said that it wouldn't provide PCPU inventories until all computes are opted in, right? | |
| 08:53:46 | alex_xu | bauzas: no, the upgrade plan is changed | |
| 08:53:47 | bauzas | actually, we send new inventories after opting in with the conf opt, you're right | |
| 08:53:59 | bauzas | I remember +2ing this | |
| 08:54:44 | bauzas | alex_xu: stephenfin: honesly, I feel it's hard to manage both at the same time, and I voiced it a couple of times in the spec review | |
| 08:55:13 | alex_xu | bauzas: here is the thing manage same time https://review.opendev.org/#/c/671801/46/nova/scheduler/manager.py@174 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 08:57:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 08:57:24 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 08:57:27 | stephenfin | bauzas: I'm not sure what else there is to do though. We have to do this, because no one has an alternative | |
| 08:57:42 | stephenfin | and we need to get to a future with PCPU. We can't just keep kicking the can down the road :( | |
| 08:57:55 | bauzas | I don't disagree with you | |
| 08:58:06 | bauzas | I'm just trying to make sure we don't shoot ourselves in the feet | |
| 08:58:34 | bauzas | that whole upgrade plan gave me brainaches when I was reviewing the spec | |
| 08:58:41 | stephenfin | oh, I've been working plenty hard to make sure we don't that :) | |
| 08:58:57 | stephenfin | changing the world (as far as placement is concerned) is hard | |
| 08:59:42 | stephenfin | as is getting things through the gate right now, it seems :D | |
| 08:59:52 | stephenfin | so many false negatives... | |
| 09:00:32 | efried | o/ nova | |
| 09:00:37 | bauzas | ... | |
| 09:00:42 | efried | The gate appears to be well and truly f'ed. | |
| 09:00:52 | stephenfin | Yes. Yes it does. | |
| 09:01:06 | efried | There was a configparser thing, which may be fixed? | |
| 09:01:18 | efried | But I haven't seen any successful runs even since that happened | |
| 09:02:15 | stephenfin | I'm not sure. Takashi-san (ignorance time: should you use the first name or second?) has been furiously rechecking some Mox -> mock and its failed everytime with a different error | |
| 09:02:37 | bauzas | lemme look at logstash | |
| 09:02:47 | bauzas | any bug tracking this ? | |
| 09:02:56 | bauzas | any exception snippet I could use ? | |
| 09:03:31 | stephenfin | bauzas: Not really. It's a simple dependency issue https://zuul.opendev.org/t/openstack/build/f4c728cbb77e48b7a149ea952d2ca2ec/log/job-output.txt#862 | |
| 09:03:45 | bauzas | that's enough for looking at occurrences | |
| 09:04:32 | stephenfin | it was hitting anything using Python 2.7. Here's the functional test from the same change (the other one was the py27 unit test) https://zuul.opendev.org/t/openstack/build/f862fb763ee54e96b0a844eaf6f00b57/log/job-output.txt#944 | |
| 09:04:44 | stephenfin | I guess they dropped support for Python 2 too \o/ | |
| 09:08:40 | luyao | stephenfin: I see you maybe busy for your own patch, very appreciate if you can look the vpmems again when you get time. NUMA issue is fixed and CI for vpmems works well, the tests passed on the last two patches. | |
| 09:09:01 | stephenfin | luyao: Yup, don't worry, I'll get to it before the end of my day | |
| 09:09:15 | stephenfin | Should be a straightforward +2 at this point, I'd imagine | |
| 09:09:25 | luyao | stephenfin: many thanks! :D | |
| 09:09:29 | bauzas | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3A%20Could%20not%20find%20a%20version%20that%20satisfies%20the%20requirement%20configparser%5C%22 | |
| 09:09:36 | bauzas | looks like it's solved ^ | |
| 09:09:41 | efried | stephenfin: meanwhile, what's the status of cpu-resources? Ready for a re-look? | |
| 09:10:16 | stephenfin | efried: one patch, yes, but alex_xu has some concerns around the other patch | |
| 09:10:34 | efried | bauzas: are you savvy on the quota issue or do we need to wait for Dan? | |
| 09:10:49 | alex_xu | efried: yes, the only https://review.opendev.org/#/c/671801/, I +2 all other patches | |
| 09:10:59 | stephenfin | efried: In short, the way we're doing the retry is to only retry if placement doesn't provide any matches | |
| 09:11:19 | stephenfin | However, placement could return some matches and the NUMATopologyFilter (or other filters) could reject those | |
| 09:11:49 | efried | ugh | |
| 09:11:50 | stephenfin | So alex_xu is suggesting that we should both the request to placement and the request to the filter scheduler inside the try-try again logic | |
| 09:11:51 | bauzas | efried: well, I'm not a quota specialist, but looks like melwitt's concerns were addressed | |
| 09:11:57 | bauzas | efried: I can take a look on it | |
| 09:12:19 | alex_xu | efried: may help you understand the problem https://etherpad.openstack.org/p/pcpu_fallback | |
| 09:12:33 | efried | stephenfin: or what we could do is just do both GET /a_c queries, cat the vcpu ones after the pcpu ones, and run the filter once. | |
| 09:12:55 | bauzas | we are technically sharding the capacity, right? | |
| 09:13:09 | stephenfin | efried: I thought of that but doesn't the filter scheduler shuffle the allocation candidates? | |
| 09:13:20 | efried | placement does the shuffle | |
| 09:13:24 | stephenfin | ohhhh | |
| 09:13:29 | efried | I don't know if the filter scheduler also does | |
| 09:14:30 | efried | stephenfin: this scenario only applies if PCPUs were requested, right? | |
| 09:14:43 | efried | Meaning if you have the granny switch flipped *and* you're on a flavor that's requesting dedicated? | |
| 09:15:55 | efried | Doing a double GET /a_c would be expensive, so it would be nice if we were only doing it in a corner case. | |
| 09:18:57 | gibi | efried: if there is more than one candidate for a single host then the scheduler will only look for the first candidate (per host) | |
| 09:19:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 09:20:21 | gibi | efried: that will complicate my work a bit ^^ | |
| 09:21:03 | efried | gibi: just a rebase, shouldn't affect anything | |
| 09:21:48 | efried | gibi: only one candidate per host -- yeah, that kills it (<== stephenfin) | |
| 09:23:06 | gibi | efried: I have ongoing work in that series locally, so I have to restach top of the rebase | |
| 09:23:11 | gibi | efried: I will manage | |
| 09:23:14 | efried | oh shit, I'm sorry gibi | |
| 09:23:25 | luyao | sean-k-mooney: Hi Sean, are you around. | |
| 09:23:29 | gibi | efried: no worries, I think I can do some interactive rebase magic | |
| 09:23:40 | efried | gibi: I didn't rebase anything except the first patch | |
| 09:23:57 | efried | so you may just be able to proceed as you were | |
| 09:24:02 | gibi | efried: ack. I will rebase my work on top of that | |
| 09:24:24 | efried | I was just trying to do what little I could to get the gate moving | |
| 09:24:34 | gibi | efried: sure, I understand your motives | |
| 09:24:39 | efried | that patch had already failed py37 on the f'in innodb thing | |
| 09:24:41 | stephenfin | efried, gibi: That shouldn't matter, should it? | |
| 09:25:04 | stephenfin | I mean, if a host is reporting PCPU then by definition we shouldn't try using it for VCPU | |
| 09:25:07 | efried | stephenfin: It makes it a very imperfect solution, but I guess it still narrows the window | |
| 09:25:11 | efried | oh | |
| 09:25:13 | gibi | efried: no worries. I guess I'm overreacted | |
| 09:25:30 | efried | that's a good point stephenfin... I think | |
| 09:25:48 | stephenfin | the crucial thing is does the filter scheduler shuffle hosts? | |
| 09:25:57 | stephenfin | *shuffle allocation candidates | |
| 09:25:58 | gibi | stephenfin: I see. so either there will be more than once candidate but then we are OK with the first. Or there only be candidate with VCPU and then we fallback to that | |
| 09:26:17 | gibi | stephenfin: there is some processing involved let me find a link | |
| 09:26:34 | efried | gibi: actually, I'm not sure if you had hoped to merge more in that series than is already +Wed, but it might be best to wait until after FF to do *anything* else. Esp since gate resources are going to be really scarce for the forseeable future. | |
| 09:27:29 | gibi | efried: Matt and I has hopes for the series. He said he could be able to +2 the rest today. But I accept the bad news if the gate is sad | |
| 09:27:55 | stephenfin | gibi: I think so. Either we'll return a candidate for host from the PCPU query (host has 'cpu_dedicated_set' configured)... | |
| 09:28:02 | stephenfin | or we'll return a candidate for a host from the VCPU query (host has 'vcpu_pin_set' or 'cpu_shared_set' configured or nothing)... | |
| 09:28:22 | gibi | stephenfin: https://github.com/openstack/nova/blob/5fa49cd0b8b6015aa61b4312b2ce1ae780c42c64/nova/scheduler/manager.py#L176 | |
| 09:28:34 | stephenfin | or we'll return the same candidate for both (host has 'cpu_dedicated_set' *and* 'cpu_shared_set') but ignore the one from the VCPU request | |
| 09:28:44 | gibi | stephenfin: sound OK to me | |
| 09:28:44 | stephenfin | because the PCPU query was done first | |