| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 08:42:44 | stephenfin | Namely that you're in the middle of an upgrade and you've got a lot of failures in the compute node | |
| 08:42:59 | alex_xu | I guess it will bad in the every begining of upgrade, like you only have 5 nodes or 10 nodes in T | |
| 08:42:59 | stephenfin | I'd be tempted to just add a workaround option | |
| 08:43:21 | stephenfin | That, when enabled, will trigger "request VCPU by default" | |
| 08:43:45 | stephenfin | so if someone ran into this in the middle of an upgrade, they'd set the config option and we wouldn't request PCPU until they've a few more compute nodes upgraded | |
| 08:43:50 | stephenfin | It might be overkill thoughj | |
| 08:43:52 | stephenfin | *though | |
| 08:43:52 | alex_xu | ok, the operator can switch that when he confidence he has enough PCPU node | |
| 08:43:57 | stephenfin | yeah | |
| 08:44:25 | alex_xu | stephenfin: I guess someone may say we add more job for operator | |
| 08:44:48 | alex_xu | stephenfin: how deficult if we fallback the whole placment+filtering | |
| 08:44:57 | stephenfin | I'm looking now | |
| 08:46:28 | alex_xu | I'm thinking we don't want to fallback all the time, how to do that check condition when filter failed | |
| 08:48:56 | stephenfin | alex_xu: Okay, I think it should be possible. Leave it with me :) | |
| 08:49:17 | alex_xu | stephenfin: so cool :) | |
| 08:50:19 | bauzas | alex_xu: stephenfin: I looked at your convo, can you please summarize me why the filter would different from placement ? | |
| 08:50:36 | alex_xu | bauzas: here is two cases I found https://etherpad.openstack.org/p/pcpu_fallback | |
| 08:50:38 | alex_xu | maybe more | |
| 08:51:02 | stephenfin | bauzas: We try to get PCPU from placement, and if that returns nothing we try for VCPU | |
| 08:51:11 | stephenfin | It's to minimize the upgrade impact | |
| 08:51:41 | stephenfin | However, if the first request does return some stuff, but those things fail the filters, we don't do the fallback for VCPU | |
| 08:51:44 | bauzas | well, I still don't get it | |
| 08:52:09 | bauzas | that's surely me, but if the operator said "okay, let's go with PCPU", that's because he opted-in, right? | |
| 08:52:35 | stephenfin | they've opted in by setting '[compute] cpu_dedicated_set' on some compute nodes, yes | |
| 08:53:08 | bauzas | but we said that it wouldn't provide PCPU inventories until all computes are opted in, right? | |
| 08:53:46 | alex_xu | bauzas: no, the upgrade plan is changed | |
| 08:53:47 | bauzas | actually, we send new inventories after opting in with the conf opt, you're right | |
| 08:53:59 | bauzas | I remember +2ing this | |
| 08:54:44 | bauzas | alex_xu: stephenfin: honesly, I feel it's hard to manage both at the same time, and I voiced it a couple of times in the spec review | |
| 08:55:13 | alex_xu | bauzas: here is the thing manage same time https://review.opendev.org/#/c/671801/46/nova/scheduler/manager.py@174 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 08:57:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 08:57:24 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 08:57:27 | stephenfin | bauzas: I'm not sure what else there is to do though. We have to do this, because no one has an alternative | |
| 08:57:42 | stephenfin | and we need to get to a future with PCPU. We can't just keep kicking the can down the road :( | |
| 08:57:55 | bauzas | I don't disagree with you | |
| 08:58:06 | bauzas | I'm just trying to make sure we don't shoot ourselves in the feet | |
| 08:58:34 | bauzas | that whole upgrade plan gave me brainaches when I was reviewing the spec | |
| 08:58:41 | stephenfin | oh, I've been working plenty hard to make sure we don't that :) | |
| 08:58:57 | stephenfin | changing the world (as far as placement is concerned) is hard | |
| 08:59:42 | stephenfin | as is getting things through the gate right now, it seems :D | |
| 08:59:52 | stephenfin | so many false negatives... | |
| 09:00:32 | efried | o/ nova | |
| 09:00:37 | bauzas | ... | |
| 09:00:42 | efried | The gate appears to be well and truly f'ed. | |
| 09:00:52 | stephenfin | Yes. Yes it does. | |
| 09:01:06 | efried | There was a configparser thing, which may be fixed? | |
| 09:01:18 | efried | But I haven't seen any successful runs even since that happened | |
| 09:02:15 | stephenfin | I'm not sure. Takashi-san (ignorance time: should you use the first name or second?) has been furiously rechecking some Mox -> mock and its failed everytime with a different error | |
| 09:02:37 | bauzas | lemme look at logstash | |
| 09:02:47 | bauzas | any bug tracking this ? | |
| 09:02:56 | bauzas | any exception snippet I could use ? | |
| 09:03:31 | stephenfin | bauzas: Not really. It's a simple dependency issue https://zuul.opendev.org/t/openstack/build/f4c728cbb77e48b7a149ea952d2ca2ec/log/job-output.txt#862 | |
| 09:03:45 | bauzas | that's enough for looking at occurrences | |
| 09:04:32 | stephenfin | it was hitting anything using Python 2.7. Here's the functional test from the same change (the other one was the py27 unit test) https://zuul.opendev.org/t/openstack/build/f862fb763ee54e96b0a844eaf6f00b57/log/job-output.txt#944 | |
| 09:04:44 | stephenfin | I guess they dropped support for Python 2 too \o/ | |
| 09:08:40 | luyao | stephenfin: I see you maybe busy for your own patch, very appreciate if you can look the vpmems again when you get time. NUMA issue is fixed and CI for vpmems works well, the tests passed on the last two patches. | |
| 09:09:01 | stephenfin | luyao: Yup, don't worry, I'll get to it before the end of my day | |
| 09:09:15 | stephenfin | Should be a straightforward +2 at this point, I'd imagine | |
| 09:09:25 | luyao | stephenfin: many thanks! :D | |
| 09:09:29 | bauzas | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3A%20Could%20not%20find%20a%20version%20that%20satisfies%20the%20requirement%20configparser%5C%22 | |
| 09:09:36 | bauzas | looks like it's solved ^ | |
| 09:09:41 | efried | stephenfin: meanwhile, what's the status of cpu-resources? Ready for a re-look? | |
| 09:10:16 | stephenfin | efried: one patch, yes, but alex_xu has some concerns around the other patch | |
| 09:10:34 | efried | bauzas: are you savvy on the quota issue or do we need to wait for Dan? | |
| 09:10:49 | alex_xu | efried: yes, the only https://review.opendev.org/#/c/671801/, I +2 all other patches | |
| 09:10:59 | stephenfin | efried: In short, the way we're doing the retry is to only retry if placement doesn't provide any matches | |
| 09:11:19 | stephenfin | However, placement could return some matches and the NUMATopologyFilter (or other filters) could reject those | |
| 09:11:49 | efried | ugh | |
| 09:11:50 | stephenfin | So alex_xu is suggesting that we should both the request to placement and the request to the filter scheduler inside the try-try again logic | |
| 09:11:51 | bauzas | efried: well, I'm not a quota specialist, but looks like melwitt's concerns were addressed | |
| 09:11:57 | bauzas | efried: I can take a look on it | |
| 09:12:19 | alex_xu | efried: may help you understand the problem https://etherpad.openstack.org/p/pcpu_fallback | |
| 09:12:33 | efried | stephenfin: or what we could do is just do both GET /a_c queries, cat the vcpu ones after the pcpu ones, and run the filter once. | |
| 09:12:55 | bauzas | we are technically sharding the capacity, right? | |
| 09:13:09 | stephenfin | efried: I thought of that but doesn't the filter scheduler shuffle the allocation candidates? | |
| 09:13:20 | efried | placement does the shuffle | |
| 09:13:24 | stephenfin | ohhhh | |
| 09:13:29 | efried | I don't know if the filter scheduler also does | |
| 09:14:30 | efried | stephenfin: this scenario only applies if PCPUs were requested, right? | |
| 09:14:43 | efried | Meaning if you have the granny switch flipped *and* you're on a flavor that's requesting dedicated? | |
| 09:15:55 | efried | Doing a double GET /a_c would be expensive, so it would be nice if we were only doing it in a corner case. | |
| 09:18:57 | gibi | efried: if there is more than one candidate for a single host then the scheduler will only look for the first candidate (per host) | |
| 09:19:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 09:20:21 | gibi | efried: that will complicate my work a bit ^^ | |
| 09:21:03 | efried | gibi: just a rebase, shouldn't affect anything | |
| 09:21:48 | efried | gibi: only one candidate per host -- yeah, that kills it (<== stephenfin) | |
| 09:23:06 | gibi | efried: I have ongoing work in that series locally, so I have to restach top of the rebase | |
| 09:23:11 | gibi | efried: I will manage | |
| 09:23:14 | efried | oh shit, I'm sorry gibi | |
| 09:23:25 | luyao | sean-k-mooney: Hi Sean, are you around. | |
| 09:23:29 | gibi | efried: no worries, I think I can do some interactive rebase magic | |
| 09:23:40 | efried | gibi: I didn't rebase anything except the first patch | |
| 09:23:57 | efried | so you may just be able to proceed as you were | |
| 09:24:02 | gibi | efried: ack. I will rebase my work on top of that | |