| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-27 | |||
| 13:42:10 | mriedem | mgagne: i've been meaning to follow up with you about the caching scheduler removal... | |
| 13:43:50 | gibi | mriedem: does this ignoring behavior needs to be guarded with a new api microversion? I hope not as maintaining the old behavior is not easily possible for nested | |
| 13:44:49 | mriedem | gibi: no i don't think so, | |
| 13:44:55 | mriedem | we're already ignoring force for some filters like i said | |
| 13:45:07 | mriedem | a microversion would be more of a signaling mechanism, | |
| 13:45:31 | mriedem | plus things like nova CLI that do version discovery and pass the latest microversion by default would simply opt into the new microversion where force is never passed | |
| 13:46:15 | mriedem | iow, force is legacy from when we just had claims in the compute and overcomitting was "simpler" | |
| 13:46:21 | gibi | mriedem: so there would be a microversion to remove the force flag from the API but ignoring the force flag would be retroactive in every version | |
| 13:46:44 | bauzas | mriedem: I'm in and out due to some customer issue, but I'll reply to you later on | |
| 13:46:54 | mriedem | well, i don't know if we should blatantly ignore it if we can help it | |
| 13:47:21 | mriedem | but i'm not sure how we determine if we should ignore it or not for nested RPs | |
| 13:47:28 | gibi | mriedem: we cannot easily help it in case of nested, and sooner or later (after NUMA) every instance will be nested | |
| 13:47:39 | mriedem | b/c like you said, the instance might have allocations on nested RPs on the source host but maybe not the forced dest? | |
| 13:47:52 | mriedem | well, every instance with numa allocations | |
| 13:47:57 | mriedem | not all instances are those kinds though | |
| 13:48:26 | gibi | mriedem: if VCPU resources are moved to NUMA RP then the simple instance will become nested | |
| 13:53:19 | mriedem | are we talking about doing that? | |
| 13:53:31 | mriedem | i guess for PCPU/VCPU modeling? | |
| 13:53:46 | mriedem | if so, shiiiiit that reshape is going to be big in a public cloud | |
| 13:53:52 | gibi | mriedem: honestly I don't know what will be the numa model in plaacement | |
| 13:55:27 | gibi | mriedem: https://review.openstack.org/#/c/555081/18/specs/stein/approved/cpu-resources.rst@261 | |
| 13:55:42 | gibi | mriedem: in this spec VCPU is under NUMA | |
| 14:00:39 | mriedem | gibi: ok left a comment / question in the upgrade impact section, | |
| 14:00:51 | mriedem | we also said at the ptg that that spec wouldn't be a priority for stein | |
| 14:00:58 | gibi | mriedem: correct | |
| 14:01:09 | mriedem | just getting vgpu reshaping done is going to be a big hurdle | |
| 14:01:48 | gibi | mriedem: yeah so vgpu and bandwidth using instance will be the first nested instances. So during this time it make sense to keep a separate code path for them | |
| 14:02:14 | gibi | mriedem: I try to implement what you described. checking the nestedness in conductor and ignoring the force flag for nested | |
| 14:24:36 | openstackgerrit | Christoph Manns proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748 | |
| 14:24:57 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | |
| 14:34:14 | openstackgerrit | Mark Goddard proposed openstack/nova master: Don't emit warning when ironic properties are zero https://review.openstack.org/605754 | |
| 14:36:56 | mnaser | filtering happens after placement, correct? | |
| 14:37:03 | mnaser | is there no warning message that says "i couldn't find anything?" | |
| 14:37:26 | mnaser | i guess i can just rely on conductor's "Setting instance to ERROR state." | |
| 14:37:39 | bauzas | mriedem: I'm still working on the vgpu reshape patch, and yes, it's a big hurdle :( | |
| 14:38:22 | bauzas | gibi: FWIW, I need to split https://review.openstack.org/#/c/552924/ in two, one targeted for Stein with no NUMA affinity | |
| 14:38:26 | openstackgerrit | Mark Goddard proposed openstack/nova master: Don't emit warning when ironic properties are zero https://review.openstack.org/605754 | |
| 14:38:48 | bauzas | mnaser: you're right, filters are called after we found an allocation candidate | |
| 14:38:56 | gibi | bauzas: ack | |
| 14:38:57 | openstackgerrit | Merged openstack/nova master: consumer gen: more tests for delete allocation cases https://review.openstack.org/591811 | |
| 14:39:21 | mnaser | thanks bauzas, im relying on this logstash query to monitor those: tags:nova AND message:"Setting instance to ERROR state." AND message:NoValidHost_Remote | |
| 14:39:29 | bauzas | mnaser: but you should at least still have the filtering logs | |
| 14:40:02 | mnaser | yeah the filter logs are there but if 0 computes match at the end, but i'm curious if there's any warning if no allocation candidates that come in the first place | |
| 14:40:05 | bauzas | hah, yeah, but we provide an INFO log saying "heh, 0 hosts found" | |
| 14:40:30 | mnaser | so i guess if its gets 0 allocation candidates, it'll just go through filters and end up with 0 anyways | |
| 14:40:32 | bauzas | I also think we tell in the logs whether we found no candidates after placement | |
| 14:40:39 | mnaser | let me verify | |
| 14:40:43 | bauzas | mnaser: sec, checking the gate | |
| 14:40:46 | bauzas | ok | |
| 14:40:58 | mriedem | i don't think we do, | |
| 14:41:20 | mriedem | we just pass an empty list to ComputeNodeList.get_by_uuids() which returns an empty list and passes that down to the filter scheduler driver which then raises NoValidHost | |
| 14:41:54 | mriedem | oh i guess we log something at debug, | |
| 14:41:58 | mriedem | but that won't be indexed by logstash | |
| 14:42:14 | mnaser | yeah we dont do that, too much data | |
| 14:42:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L150 | |
| 14:42:32 | bauzas | mnaser: yup, we do http://logs.openstack.org/72/585672/7/check/tempest-full-py3/f1d0b34/controller/logs/screen-n-sch.txt.gz#_Sep_26_19_02_54_285546 | |
| 14:42:59 | mriedem | mnaser: do you have a failure log to check? | |
| 14:43:03 | mriedem | placement should be logging some stuff now too | |
| 14:43:16 | mnaser | mriedem: i might if it hasnt rotated out | |
| 14:43:24 | bauzas | mriedem: see, we put an info log on how many hosts we got from placement ^ | |
| 14:43:31 | mriedem | as to which "filters" in placement resulted in 0 allocation candidates | |
| 14:43:43 | mriedem | Sep 26 19:02:54.285546 ubuntu-xenial-rax-ord-0002338068 nova-scheduler[18720]: DEBUG nova.filters [None req-8a6074ae-e62f-4ac7-a525-4c411c130c39 tempest-AutoAllocateNetworkTest-2112321998 tempest-AutoAllocateNetworkTest-2112321998] Starting with 1 host(s) {{(pid=20232) get_filtered_objects /opt/stack/nova/nova/filters.py:70}} | |
| 14:43:45 | bauzas | oh shit, that's DEBUG | |
| 14:43:47 | mnaser | yeah | |
| 14:43:47 | mnaser | mriedem: that is a LOG.debug() so a normal deployment wont see it | |
| 14:43:50 | mriedem | right | |
| 14:43:51 | bauzas | my bad | |
| 14:43:54 | mriedem | GAWD! | |
| 14:44:13 | mnaser | i think it's useful to get that warning because a lot of times when placement isnt returning anything | |
| 14:44:15 | bauzas | we info out when we have the filtering results | |
| 14:44:16 | mnaser | i would get really confused | |
| 14:44:19 | mriedem | logging https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L150 at INFO might be ok | |
| 14:44:26 | mnaser | yeah but we don't even pass things down to filter | |
| 14:44:29 | bauzas | WTF | |
| 14:44:30 | mnaser | if we get 0 allocation candidates | |
| 14:44:39 | mnaser | if i understand what mriedem linked above | |
| 14:44:42 | bauzas | that's correct | |
| 14:44:50 | bauzas | we just said "meh, that's bad" | |
| 14:45:11 | bauzas | my point was, if we end up with 0 hosts from filtering, some INFO log is done | |
| 14:45:30 | mnaser | yeah that scenario is taken care of i agree | |
| 14:45:30 | bauzas | so, having the pre-filtering result to be INFO seems consistent and valid to me | |
| 14:45:44 | bauzas | lemme dig the code | |
| 14:45:47 | mnaser | i'd even go as far as say that's a warning | |
| 14:45:56 | mnaser | https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L150 -- just switch that to warning ? | |
| 14:46:11 | bauzas | but I'm pretty sure we say it's INFO (and no ERROR or warning, because a capacity problem isn't a scheduling problem) | |
| 14:46:29 | mnaser | "change debug level for more info" | |
| 14:46:31 | bauzas | mnaser: I'd advocate for INFO | |
| 14:46:36 | bauzas | no WARN | |
| 14:46:48 | mnaser | it would be consistent with the other stuff | |
| 14:46:52 | bauzas | lemme find the existing log we raise post-filtering | |
| 14:46:53 | mriedem | bauzas: unrelated, but is it just me or do we persist RequestSpec.requested_destination? | |
| 14:46:57 | mriedem | and probably shouldn't... | |
| 14:47:03 | mnaser | bauzas: its info, i have an entry here | |
| 14:47:23 | mnaser | bauzas: 2018-09-27 12:37:00.467 394218 INFO nova.filters [<snip>] Filter ComputeFilter returned 0 hosts | |
| 14:47:23 | bauzas | mriedem: wait, wait wait | |
| 14:47:31 | mriedem | mnaser: i'd say info | |
| 14:47:39 | bauzas | mriedem: probably yet another PEBKAC then | |
| 14:47:41 | mriedem | it's not a warning if someone is trying to resize to a flavor that won't fit anywhere | |
| 14:47:48 | bauzas | (for the persisted field) | |
| 14:47:53 | bauzas | mriedem: zactly | |