| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 08:59:52 | stephenfin | so many false negatives... | |
| 09:00:32 | efried | o/ nova | |
| 09:00:37 | bauzas | ... | |
| 09:00:42 | efried | The gate appears to be well and truly f'ed. | |
| 09:00:52 | stephenfin | Yes. Yes it does. | |
| 09:01:06 | efried | There was a configparser thing, which may be fixed? | |
| 09:01:18 | efried | But I haven't seen any successful runs even since that happened | |
| 09:02:15 | stephenfin | I'm not sure. Takashi-san (ignorance time: should you use the first name or second?) has been furiously rechecking some Mox -> mock and its failed everytime with a different error | |
| 09:02:37 | bauzas | lemme look at logstash | |
| 09:02:47 | bauzas | any bug tracking this ? | |
| 09:02:56 | bauzas | any exception snippet I could use ? | |
| 09:03:31 | stephenfin | bauzas: Not really. It's a simple dependency issue https://zuul.opendev.org/t/openstack/build/f4c728cbb77e48b7a149ea952d2ca2ec/log/job-output.txt#862 | |
| 09:03:45 | bauzas | that's enough for looking at occurrences | |
| 09:04:32 | stephenfin | it was hitting anything using Python 2.7. Here's the functional test from the same change (the other one was the py27 unit test) https://zuul.opendev.org/t/openstack/build/f862fb763ee54e96b0a844eaf6f00b57/log/job-output.txt#944 | |
| 09:04:44 | stephenfin | I guess they dropped support for Python 2 too \o/ | |
| 09:08:40 | luyao | stephenfin: I see you maybe busy for your own patch, very appreciate if you can look the vpmems again when you get time. NUMA issue is fixed and CI for vpmems works well, the tests passed on the last two patches. | |
| 09:09:01 | stephenfin | luyao: Yup, don't worry, I'll get to it before the end of my day | |
| 09:09:15 | stephenfin | Should be a straightforward +2 at this point, I'd imagine | |
| 09:09:25 | luyao | stephenfin: many thanks! :D | |
| 09:09:29 | bauzas | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3A%20Could%20not%20find%20a%20version%20that%20satisfies%20the%20requirement%20configparser%5C%22 | |
| 09:09:36 | bauzas | looks like it's solved ^ | |
| 09:09:41 | efried | stephenfin: meanwhile, what's the status of cpu-resources? Ready for a re-look? | |
| 09:10:16 | stephenfin | efried: one patch, yes, but alex_xu has some concerns around the other patch | |
| 09:10:34 | efried | bauzas: are you savvy on the quota issue or do we need to wait for Dan? | |
| 09:10:49 | alex_xu | efried: yes, the only https://review.opendev.org/#/c/671801/, I +2 all other patches | |
| 09:10:59 | stephenfin | efried: In short, the way we're doing the retry is to only retry if placement doesn't provide any matches | |
| 09:11:19 | stephenfin | However, placement could return some matches and the NUMATopologyFilter (or other filters) could reject those | |
| 09:11:49 | efried | ugh | |
| 09:11:50 | stephenfin | So alex_xu is suggesting that we should both the request to placement and the request to the filter scheduler inside the try-try again logic | |
| 09:11:51 | bauzas | efried: well, I'm not a quota specialist, but looks like melwitt's concerns were addressed | |
| 09:11:57 | bauzas | efried: I can take a look on it | |
| 09:12:19 | alex_xu | efried: may help you understand the problem https://etherpad.openstack.org/p/pcpu_fallback | |
| 09:12:33 | efried | stephenfin: or what we could do is just do both GET /a_c queries, cat the vcpu ones after the pcpu ones, and run the filter once. | |
| 09:12:55 | bauzas | we are technically sharding the capacity, right? | |
| 09:13:09 | stephenfin | efried: I thought of that but doesn't the filter scheduler shuffle the allocation candidates? | |
| 09:13:20 | efried | placement does the shuffle | |
| 09:13:24 | stephenfin | ohhhh | |
| 09:13:29 | efried | I don't know if the filter scheduler also does | |
| 09:14:30 | efried | stephenfin: this scenario only applies if PCPUs were requested, right? | |
| 09:14:43 | efried | Meaning if you have the granny switch flipped *and* you're on a flavor that's requesting dedicated? | |
| 09:15:55 | efried | Doing a double GET /a_c would be expensive, so it would be nice if we were only doing it in a corner case. | |
| 09:18:57 | gibi | efried: if there is more than one candidate for a single host then the scheduler will only look for the first candidate (per host) | |
| 09:19:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 09:20:21 | gibi | efried: that will complicate my work a bit ^^ | |
| 09:21:03 | efried | gibi: just a rebase, shouldn't affect anything | |
| 09:21:48 | efried | gibi: only one candidate per host -- yeah, that kills it (<== stephenfin) | |
| 09:23:06 | gibi | efried: I have ongoing work in that series locally, so I have to restach top of the rebase | |
| 09:23:11 | gibi | efried: I will manage | |
| 09:23:14 | efried | oh shit, I'm sorry gibi | |
| 09:23:25 | luyao | sean-k-mooney: Hi Sean, are you around. | |
| 09:23:29 | gibi | efried: no worries, I think I can do some interactive rebase magic | |
| 09:23:40 | efried | gibi: I didn't rebase anything except the first patch | |
| 09:23:57 | efried | so you may just be able to proceed as you were | |
| 09:24:02 | gibi | efried: ack. I will rebase my work on top of that | |
| 09:24:24 | efried | I was just trying to do what little I could to get the gate moving | |
| 09:24:34 | gibi | efried: sure, I understand your motives | |
| 09:24:39 | efried | that patch had already failed py37 on the f'in innodb thing | |
| 09:24:41 | stephenfin | efried, gibi: That shouldn't matter, should it? | |
| 09:25:04 | stephenfin | I mean, if a host is reporting PCPU then by definition we shouldn't try using it for VCPU | |
| 09:25:07 | efried | stephenfin: It makes it a very imperfect solution, but I guess it still narrows the window | |
| 09:25:11 | efried | oh | |
| 09:25:13 | gibi | efried: no worries. I guess I'm overreacted | |
| 09:25:30 | efried | that's a good point stephenfin... I think | |
| 09:25:48 | stephenfin | the crucial thing is does the filter scheduler shuffle hosts? | |
| 09:25:57 | stephenfin | *shuffle allocation candidates | |
| 09:25:58 | gibi | stephenfin: I see. so either there will be more than once candidate but then we are OK with the first. Or there only be candidate with VCPU and then we fallback to that | |
| 09:26:17 | gibi | stephenfin: there is some processing involved let me find a link | |
| 09:26:34 | efried | gibi: actually, I'm not sure if you had hoped to merge more in that series than is already +Wed, but it might be best to wait until after FF to do *anything* else. Esp since gate resources are going to be really scarce for the forseeable future. | |
| 09:27:29 | gibi | efried: Matt and I has hopes for the series. He said he could be able to +2 the rest today. But I accept the bad news if the gate is sad | |
| 09:27:55 | stephenfin | gibi: I think so. Either we'll return a candidate for host from the PCPU query (host has 'cpu_dedicated_set' configured)... | |
| 09:28:02 | stephenfin | or we'll return a candidate for a host from the VCPU query (host has 'vcpu_pin_set' or 'cpu_shared_set' configured or nothing)... | |
| 09:28:22 | gibi | stephenfin: https://github.com/openstack/nova/blob/5fa49cd0b8b6015aa61b4312b2ce1ae780c42c64/nova/scheduler/manager.py#L176 | |
| 09:28:34 | stephenfin | or we'll return the same candidate for both (host has 'cpu_dedicated_set' *and* 'cpu_shared_set') but ignore the one from the VCPU request | |
| 09:28:44 | gibi | stephenfin: sound OK to me | |
| 09:28:44 | stephenfin | because the PCPU query was done first | |
| 09:28:49 | stephenfin | schweet | |
| 09:28:53 | gibi | stephenfin: but look at the link. We bould dicts :/ | |
| 09:29:38 | bauzas | stephenfin: gibi: efried: alex_xu: seriously, I'm still continuing to consider we overthink the problem | |
| 09:29:56 | gibi | stephenfin: hm, that dict is not a problem the allocation requests still in a list | |
| 09:30:06 | efried | gibi: but only per host | |
| 09:30:09 | bauzas | I mean, once the operator opts in, he litterally shards his cloud in twice | |
| 09:30:28 | gibi | efried: but the fallback is needed per host, isn't it? | |
| 09:30:36 | bauzas | if we say the opt-in is per compute, that means the sharding can be long | |
| 09:30:53 | efried | gibi: actually, maybe so. | |
| 09:31:18 | bauzas | that's why the more I see our convos, the best i think it would be to just consider a global option that would dictate all computes sending PCPU inventories | |
| 09:31:24 | gibi | stephenfin: we take the first ar per host https://github.com/openstack/nova/blob/5fa49cd0b8b6015aa61b4312b2ce1ae780c42c64/nova/scheduler/filter_scheduler.py#L238 | |
| 09:31:27 | efried | gibi, stephenfin: as long as it's okay if we try has-only-VCPU candidates before we try a has-PCPU candidate | |
| 09:31:52 | stephenfin | efried: It's not an issue. As noted, the NUMATopologyFilter or virt driver will fail that request | |
| 09:31:56 | efried | ...which should still be fine, because the ntf will remove... yeah | |
| 09:32:38 | efried | so again, stephenfin, under what circumstances are we worried about this? | |
| 09:32:39 | stephenfin | bauzas: We sort of have that - the compute nodes won't send PCPU inventory until 'cpu_dedicated_set' is set | |
| 09:32:58 | bauzas | stephenfin: I know, my opinion is just "doc it, dude" | |
| 09:33:01 | efried | If it's only in the "sharded" case as bauzas says, then I think it may be acceptable to take the double GET /a_c hit. | |
| 09:33:37 | stephenfin | efried: The current solution (where we only try the second 'GET /a_c' if the first doesn't return anything) or the modified one we're suggesting here (where we always make two requests and concat them) ? | |
| 09:33:50 | efried | but if it can happen in a fully-pre or fully-post cloud, we should think it through more carefully. | |
| 09:34:06 | cdent | in train two a_c hits is pretty cheap | |
| 09:34:06 | bauzas | adding more complexity into a abstract API that would add special-cases in order to fix a very temporary situation is IMHO like using a big hammer | |
| 09:34:07 | efried | I'm talking about doing double GET /a_c preemptively. | |
| 09:34:28 | efried | cdent: even for CERN? | |
| 09:34:35 | cdent | yeah | |