| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 06:28:15 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 06:28:15 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 06:28:16 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 06:41:37 | alex_xu | stephenfin: sean-k-mooney the fallback case I understand, https://etherpad.openstack.org/p/pcpu_fallback | |
| 06:45:27 | gibi | good morning nova | |
| 06:46:02 | alex_xu | gibi: good morning | |
| 07:05:51 | openstackgerrit | hutianhao27 proposed openstack/nova master: Fix bug directory left after cold migration https://review.opendev.org/681652 | |
| 07:13:17 | bauzas | good morning nova | |
| 07:15:02 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 07:15:02 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 07:15:03 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 07:15:03 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 07:15:04 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 07:15:04 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 07:15:05 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 07:15:05 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 07:30:24 | openstackgerrit | Brin Zhang proposed openstack/nova master: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 07:50:01 | gibi | do we have a solution for the gate failure: ERROR: Could not find a version that satisfies the requirement configparser===4.0.1 | |
| 07:50:04 | gibi | ? | |
| 07:50:24 | alex_xu | gibi: I saw this https://review.opendev.org/#/c/681630/ | |
| 07:51:24 | alex_xu | so it suppose to be work now? | |
| 07:52:53 | gibi | alex_xu: thanks, then I go and recheclk | |
| 07:52:55 | gibi | recheck | |
| 08:17:44 | lyarwood | Have we cut M3 yet? I've been holding off for a while on posting bugfixes to avoid clogging up the gate. | |
| 08:18:01 | lyarwood | wait, the deadline is today, ignore me. | |
| 08:18:13 | gibi | lyarwood: yepp, today is still crazy FF day | |
| 08:19:44 | lyarwood | ack thanks I'll hold off in that case | |
| 08:24:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make SRIOV computes non symmetric in func test https://review.opendev.org/681667 | |
| 08:34:11 | stephenfin | alex_xu: Looking | |
| 08:35:30 | stephenfin | alex_xu: That looks correct. I started work on the fix for that yesterday but haven't finished. It only happens if you have two compute nodes though, right? | |
| 08:36:35 | alex_xu | stephenfin: it can be happened for 100 nodes, if all 100 nodes can get from placement allocation candidtes, but failed in later filtering | |
| 08:37:04 | stephenfin | both cases? | |
| 08:37:15 | alex_xu | stephenfin: yes, see the line 25 | |
| 08:37:24 | alex_xu | stephenfin: I think the root cause is that | |
| 08:38:29 | stephenfin | but it'll only happen if we attempt to use the same host | |
| 08:38:32 | alex_xu | I don't think I figure out the root cause yesterday, it isn't just about resize | |
| 08:38:46 | alex_xu | what about case1 | |
| 08:39:41 | alex_xu | stephenfin: I think anycase about placement can return candidates, but nova scheduler filter refuse the request. | |
| 08:39:54 | alex_xu | then we won't fallback | |
| 08:41:34 | stephenfin | true | |
| 08:41:43 | stephenfin | hmm | |
| 08:42:06 | stephenfin | I'm still tempted to think it's not a huge issue. You'd need an awful lot to go against you | |
| 08:42:09 | alex_xu | stephenfin: but need to think about that whether we can have such worse case, like 100 nodes failed at filter not the placement | |
| 08:42:44 | stephenfin | Namely that you're in the middle of an upgrade and you've got a lot of failures in the compute node | |
| 08:42:59 | alex_xu | I guess it will bad in the every begining of upgrade, like you only have 5 nodes or 10 nodes in T | |
| 08:42:59 | stephenfin | I'd be tempted to just add a workaround option | |
| 08:43:21 | stephenfin | That, when enabled, will trigger "request VCPU by default" | |
| 08:43:45 | stephenfin | so if someone ran into this in the middle of an upgrade, they'd set the config option and we wouldn't request PCPU until they've a few more compute nodes upgraded | |
| 08:43:50 | stephenfin | It might be overkill thoughj | |
| 08:43:52 | stephenfin | *though | |
| 08:43:52 | alex_xu | ok, the operator can switch that when he confidence he has enough PCPU node | |
| 08:43:57 | stephenfin | yeah | |
| 08:44:25 | alex_xu | stephenfin: I guess someone may say we add more job for operator | |
| 08:44:48 | alex_xu | stephenfin: how deficult if we fallback the whole placment+filtering | |
| 08:44:57 | stephenfin | I'm looking now | |
| 08:46:28 | alex_xu | I'm thinking we don't want to fallback all the time, how to do that check condition when filter failed | |
| 08:48:56 | stephenfin | alex_xu: Okay, I think it should be possible. Leave it with me :) | |
| 08:49:17 | alex_xu | stephenfin: so cool :) | |
| 08:50:19 | bauzas | alex_xu: stephenfin: I looked at your convo, can you please summarize me why the filter would different from placement ? | |
| 08:50:36 | alex_xu | bauzas: here is two cases I found https://etherpad.openstack.org/p/pcpu_fallback | |
| 08:50:38 | alex_xu | maybe more | |
| 08:51:02 | stephenfin | bauzas: We try to get PCPU from placement, and if that returns nothing we try for VCPU | |
| 08:51:11 | stephenfin | It's to minimize the upgrade impact | |
| 08:51:41 | stephenfin | However, if the first request does return some stuff, but those things fail the filters, we don't do the fallback for VCPU | |
| 08:51:44 | bauzas | well, I still don't get it | |
| 08:52:09 | bauzas | that's surely me, but if the operator said "okay, let's go with PCPU", that's because he opted-in, right? | |
| 08:52:35 | stephenfin | they've opted in by setting '[compute] cpu_dedicated_set' on some compute nodes, yes | |
| 08:53:08 | bauzas | but we said that it wouldn't provide PCPU inventories until all computes are opted in, right? | |
| 08:53:46 | alex_xu | bauzas: no, the upgrade plan is changed | |
| 08:53:47 | bauzas | actually, we send new inventories after opting in with the conf opt, you're right | |
| 08:53:59 | bauzas | I remember +2ing this | |
| 08:54:44 | bauzas | alex_xu: stephenfin: honesly, I feel it's hard to manage both at the same time, and I voiced it a couple of times in the spec review | |
| 08:55:13 | alex_xu | bauzas: here is the thing manage same time https://review.opendev.org/#/c/671801/46/nova/scheduler/manager.py@174 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 08:57:20 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 08:57:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 08:57:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 08:57:23 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 08:57:24 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 08:57:27 | stephenfin | bauzas: I'm not sure what else there is to do though. We have to do this, because no one has an alternative | |
| 08:57:42 | stephenfin | and we need to get to a future with PCPU. We can't just keep kicking the can down the road :( | |
| 08:57:55 | bauzas | I don't disagree with you | |
| 08:58:06 | bauzas | I'm just trying to make sure we don't shoot ourselves in the feet | |
| 08:58:34 | bauzas | that whole upgrade plan gave me brainaches when I was reviewing the spec | |
| 08:58:41 | stephenfin | oh, I've been working plenty hard to make sure we don't that :) | |
| 08:58:57 | stephenfin | changing the world (as far as placement is concerned) is hard | |
| 08:59:42 | stephenfin | as is getting things through the gate right now, it seems :D | |
| 08:59:52 | stephenfin | so many false negatives... | |
| 09:00:32 | efried | o/ nova | |
| 09:00:37 | bauzas | ... | |
| 09:00:42 | efried | The gate appears to be well and truly f'ed. | |
| 09:00:52 | stephenfin | Yes. Yes it does. | |
| 09:01:06 | efried | There was a configparser thing, which may be fixed? | |
| 09:01:18 | efried | But I haven't seen any successful runs even since that happened | |
| 09:02:15 | stephenfin | I'm not sure. Takashi-san (ignorance time: should you use the first name or second?) has been furiously rechecking some Mox -> mock and its failed everytime with a different error | |
| 09:02:37 | bauzas | lemme look at logstash | |
| 09:02:47 | bauzas | any bug tracking this ? | |
| 09:02:56 | bauzas | any exception snippet I could use ? | |