| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 08:18:13 | gibi | lyarwood: yepp, today is still crazy FF day | |
| 08:19:44 | lyarwood | ack thanks I'll hold off in that case | |
| 08:24:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make SRIOV computes non symmetric in func test https://review.opendev.org/681667 | |
| 08:34:11 | stephenfin | alex_xu: Looking | |
| 08:35:30 | stephenfin | alex_xu: That looks correct. I started work on the fix for that yesterday but haven't finished. It only happens if you have two compute nodes though, right? | |
| 08:36:35 | alex_xu | stephenfin: it can be happened for 100 nodes, if all 100 nodes can get from placement allocation candidtes, but failed in later filtering | |
| 08:37:04 | stephenfin | both cases? | |
| 08:37:15 | alex_xu | stephenfin: yes, see the line 25 | |
| 08:37:24 | alex_xu | stephenfin: I think the root cause is that | |
| 08:38:29 | stephenfin | but it'll only happen if we attempt to use the same host | |
| 08:38:32 | alex_xu | I don't think I figure out the root cause yesterday, it isn't just about resize | |
| 08:38:46 | alex_xu | what about case1 | |
| 08:39:41 | alex_xu | stephenfin: I think anycase about placement can return candidates, but nova scheduler filter refuse the request. | |
| 08:39:54 | alex_xu | then we won't fallback | |
| 08:41:34 | stephenfin | true | |
| 08:41:43 | stephenfin | hmm | |
| 08:42:06 | stephenfin | I'm still tempted to think it's not a huge issue. You'd need an awful lot to go against you | |
| 08:42:09 | alex_xu | stephenfin: but need to think about that whether we can have such worse case, like 100 nodes failed at filter not the placement | |
| 08:42:44 | stephenfin | Namely that you're in the middle of an upgrade and you've got a lot of failures in the compute node | |
| 08:42:59 | stephenfin | I'd be tempted to just add a workaround option | |
| 08:42:59 | alex_xu | I guess it will bad in the every begining of upgrade, like you only have 5 nodes or 10 nodes in T | |
| 08:43:21 | stephenfin | That, when enabled, will trigger "request VCPU by default" | |
| 08:43:45 | stephenfin | so if someone ran into this in the middle of an upgrade, they'd set the config option and we wouldn't request PCPU until they've a few more compute nodes upgraded | |
| 08:43:50 | stephenfin | It might be overkill thoughj | |
| 08:43:52 | alex_xu | ok, the operator can switch that when he confidence he has enough PCPU node | |
| 08:43:52 | stephenfin | *though | |
| 08:43:57 | stephenfin | yeah | |
| 08:44:25 | alex_xu | stephenfin: I guess someone may say we add more job for operator | |
| 08:44:48 | alex_xu | stephenfin: how deficult if we fallback the whole placment+filtering | |
| 08:44:57 | stephenfin | I'm looking now | |
| 08:46:28 | alex_xu | I'm thinking we don't want to fallback all the time, how to do that check condition when filter failed | |
| 08:48:56 | stephenfin | alex_xu: Okay, I think it should be possible. Leave it with me :) | |
| 08:49:17 | alex_xu | stephenfin: so cool :) | |
| 08:50:19 | bauzas | alex_xu: stephenfin: I looked at your convo, can you please summarize me why the filter would different from placement ? | |
| 08:50:36 | alex_xu | bauzas: here is two cases I found https://etherpad.openstack.org/p/pcpu_fallback | |
| 08:50:38 | alex_xu | maybe more | |
| 08:51:02 | stephenfin | bauzas: We try to get PCPU from placement, and if that returns nothing we try for VCPU | |
| 08:51:11 | stephenfin | It's to minimize the upgrade impact | |
| 08:51:41 | stephenfin | However, if the first request does return some stuff, but those things fail the filters, we don't do the fallback for VCPU | |
| 08:51:44 | bauzas | well, I still don't get it | |
| 08:52:09 | bauzas | that's surely me, but if the operator said "okay, let's go with PCPU", that's because he opted-in, right? | |
| 08:52:35 | stephenfin | they've opted in by setting '[compute] cpu_dedicated_set' on some compute nodes, yes | |
| 08:53:08 | bauzas | but we said that it wouldn't provide PCPU inventories until all computes are opted in, right? | |
| 08:53:46 | alex_xu | bauzas: no, the upgrade plan is changed | |
| 08:53:47 | bauzas | actually, we send new inventories after opting in with the conf opt, you're right | |
| 08:53:59 | bauzas | I remember +2ing this | |
| 08:54:44 | bauzas | alex_xu: stephenfin: honesly, I feel it's hard to manage both at the same time, and I voiced it a couple of times in the spec review | |
| 08:55:13 | alex_xu | bauzas: here is the thing manage same time https://review.opendev.org/#/c/671801/46/nova/scheduler/manager.py@174 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 08:57:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 08:57:24 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 08:57:27 | stephenfin | bauzas: I'm not sure what else there is to do though. We have to do this, because no one has an alternative | |
| 08:57:42 | stephenfin | and we need to get to a future with PCPU. We can't just keep kicking the can down the road :( | |
| 08:57:55 | bauzas | I don't disagree with you | |
| 08:58:06 | bauzas | I'm just trying to make sure we don't shoot ourselves in the feet | |
| 08:58:34 | bauzas | that whole upgrade plan gave me brainaches when I was reviewing the spec | |
| 08:58:41 | stephenfin | oh, I've been working plenty hard to make sure we don't that :) | |
| 08:58:57 | stephenfin | changing the world (as far as placement is concerned) is hard | |
| 08:59:42 | stephenfin | as is getting things through the gate right now, it seems :D | |
| 08:59:52 | stephenfin | so many false negatives... | |
| 09:00:32 | efried | o/ nova | |
| 09:00:37 | bauzas | ... | |
| 09:00:42 | efried | The gate appears to be well and truly f'ed. | |
| 09:00:52 | stephenfin | Yes. Yes it does. | |
| 09:01:06 | efried | There was a configparser thing, which may be fixed? | |
| 09:01:18 | efried | But I haven't seen any successful runs even since that happened | |
| 09:02:15 | stephenfin | I'm not sure. Takashi-san (ignorance time: should you use the first name or second?) has been furiously rechecking some Mox -> mock and its failed everytime with a different error | |
| 09:02:37 | bauzas | lemme look at logstash | |
| 09:02:47 | bauzas | any bug tracking this ? | |
| 09:02:56 | bauzas | any exception snippet I could use ? | |
| 09:03:31 | stephenfin | bauzas: Not really. It's a simple dependency issue https://zuul.opendev.org/t/openstack/build/f4c728cbb77e48b7a149ea952d2ca2ec/log/job-output.txt#862 | |
| 09:03:45 | bauzas | that's enough for looking at occurrences | |
| 09:04:32 | stephenfin | it was hitting anything using Python 2.7. Here's the functional test from the same change (the other one was the py27 unit test) https://zuul.opendev.org/t/openstack/build/f862fb763ee54e96b0a844eaf6f00b57/log/job-output.txt#944 | |
| 09:04:44 | stephenfin | I guess they dropped support for Python 2 too \o/ | |
| 09:08:40 | luyao | stephenfin: I see you maybe busy for your own patch, very appreciate if you can look the vpmems again when you get time. NUMA issue is fixed and CI for vpmems works well, the tests passed on the last two patches. | |
| 09:09:01 | stephenfin | luyao: Yup, don't worry, I'll get to it before the end of my day | |
| 09:09:15 | stephenfin | Should be a straightforward +2 at this point, I'd imagine | |
| 09:09:25 | luyao | stephenfin: many thanks! :D | |
| 09:09:29 | bauzas | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3A%20Could%20not%20find%20a%20version%20that%20satisfies%20the%20requirement%20configparser%5C%22 | |
| 09:09:36 | bauzas | looks like it's solved ^ | |
| 09:09:41 | efried | stephenfin: meanwhile, what's the status of cpu-resources? Ready for a re-look? | |
| 09:10:16 | stephenfin | efried: one patch, yes, but alex_xu has some concerns around the other patch | |
| 09:10:34 | efried | bauzas: are you savvy on the quota issue or do we need to wait for Dan? | |
| 09:10:49 | alex_xu | efried: yes, the only https://review.opendev.org/#/c/671801/, I +2 all other patches | |
| 09:10:59 | stephenfin | efried: In short, the way we're doing the retry is to only retry if placement doesn't provide any matches | |
| 09:11:19 | stephenfin | However, placement could return some matches and the NUMATopologyFilter (or other filters) could reject those | |
| 09:11:49 | efried | ugh | |
| 09:11:50 | stephenfin | So alex_xu is suggesting that we should both the request to placement and the request to the filter scheduler inside the try-try again logic | |
| 09:11:51 | bauzas | efried: well, I'm not a quota specialist, but looks like melwitt's concerns were addressed | |
| 09:11:57 | bauzas | efried: I can take a look on it | |
| 09:12:19 | alex_xu | efried: may help you understand the problem https://etherpad.openstack.org/p/pcpu_fallback | |
| 09:12:33 | efried | stephenfin: or what we could do is just do both GET /a_c queries, cat the vcpu ones after the pcpu ones, and run the filter once. | |
| 09:12:55 | bauzas | we are technically sharding the capacity, right? | |
| 09:13:09 | stephenfin | efried: I thought of that but doesn't the filter scheduler shuffle the allocation candidates? | |
| 09:13:20 | efried | placement does the shuffle | |