| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-27 | |||
| 13:22:25 | mriedem | in the case of nested RP allocations + force, we'd just go through the scheduler with the requested host but kind of ignore the force flag | |
| 13:22:50 | mriedem | i.e. you can live migrate today by specifying a specific host w/o forcing it and we'll validate that requested host with the scheduler | |
| 13:22:54 | mriedem | it sounds like you want to do the same | |
| 13:22:58 | mriedem | s/want/need/ | |
| 13:23:25 | gibi | mriedem: that seems to be the way forward. So now I go and dig that code path | |
| 13:23:30 | gibi | mriedem: thanks | |
| 13:24:07 | mriedem | in the case of an unforced live migration to a specific host, we just set the RequestSpec.requested_destination to the requested host/node and send that to the scheduler | |
| 13:24:24 | mriedem | it sounds like you'd just need some logic up-front to determine, is this an instance that has allocations on nested RPs | |
| 13:24:28 | mriedem | and if so, ignore the force flag | |
| 13:25:12 | gibi | mriedem: yeah, forced live migration will be less forced as it can return NoValidHost after this change | |
| 13:25:13 | mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/compute/api.py#L4374 | |
| 13:25:44 | mriedem | gibi: and i think that's ok - forced live migration could always fail even if you bypassed the scheduler b/c the conductor task still does some prechecks on the forced host | |
| 13:25:45 | gibi | mriedem: which means if every instance will be on nested RP then force livemigration will be equal to non forced livemigration with provided host | |
| 13:26:01 | mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L160 | |
| 13:26:13 | mriedem | like, self._check_host_is_up(self.destination) is the ComputeFilter | |
| 13:26:20 | mriedem | self._check_destination_has_enough_memory() is the RamFilter | |
| 13:26:27 | mriedem | etc | |
| 13:26:56 | mriedem | i'd really like to just get rid of that _check_requested_destination method | |
| 13:27:11 | gibi | mriedem: OK, this seems to be a way forward to eventually get rid of the force live migration altogether | |
| 13:27:28 | jaypipes | mriedem: "recheck slow node" <-- worst 90s band name EVAH. | |
| 13:27:46 | mriedem | jaypipes: it's definitely not as good as butthole surfers | |
| 13:27:52 | jaypipes | indeed. | |
| 13:28:22 | jaypipes | true fact: my high school band in the early 90s was called "Slow Children at Play". | |
| 13:28:46 | mriedem | highly offensive | |
| 13:28:50 | jaypipes | embarrassing, I know. | |
| 13:29:43 | mriedem | https://www.youtube.com/watch?v=SkjJLQUhxks | |
| 13:29:49 | mriedem | now i know what needs to be playing this morning | |
| 13:32:44 | mriedem | bauzas: depending on where you check if you should ignore the force flag it could get messy, | |
| 13:32:45 | mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/compute/api.py#L4359 | |
| 13:32:52 | mriedem | because the api determines what gets passed to conductor | |
| 13:32:56 | mriedem | it's all very tightly coupled | |
| 13:33:21 | mriedem | i.e. if force is True, the requested host parameter isn't passed to conductor, but the request spec is set with the requested_destination | |
| 13:33:51 | mriedem | but if force is True, the request spec is untouched and the requested host is passed to conductor | |
| 13:33:55 | mriedem | and conductor has to know what that means | |
| 13:34:10 | gibi | mriedem: I see | |
| 13:34:21 | mriedem | i assume the logic to determine if the instance has allocations against nested RPs should probably live in conductor | |
| 13:34:25 | mriedem | so we don't block the API response | |
| 13:34:34 | mriedem | although having said that.... | |
| 13:34:52 | mriedem | before some microversion, live migration is aysync rpc call from api-conductor-scheduler until we pick a host and cast to it | |
| 13:35:15 | mriedem | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id31 | |
| 13:35:34 | mriedem | meaning it's pretty easy to timeout the API while picking hosts before 2.34 | |
| 13:35:56 | mriedem | cold migrate is the same way - it's all synchronous until we cast to the chosen compute | |
| 13:36:01 | mriedem | :( | |
| 13:36:42 | mriedem | anywho, i'd recommend doing your nested RP calculations in conductor rather than api | |
| 13:37:04 | gibi | mriedem: ack | |
| 13:37:21 | mriedem | maybe you don't need to look at placement? maybe you can just glean if it's got nested rps by looking at the flavor? | |
| 13:37:52 | mriedem | i guess you care about allocations for ports | |
| 13:38:16 | gibi | mriedem: even if flavor has granular groups I don't know if the host the instance is on are already reshaped to nested or not | |
| 13:38:26 | mriedem | right yeah | |
| 13:39:17 | gibi | mriedem: can we simply ignore the force flag for every instance not just for the nested ones? I mean the end user will not know if his instance is nested or not | |
| 13:40:22 | mriedem | it's not really about the owner of the instance (the user), force is for the admin | |
| 13:40:44 | gibi | mriedem: true, I missed that | |
| 13:41:01 | mriedem | if we're really going down a path of blatantly ignoring the force parameter, we should probably consider just deprecating it in the api | |
| 13:41:19 | mriedem | like i said, we kind of already ignore it today for ram/disk/vcpu | |
| 13:41:32 | mriedem | plus some other sanity checks that conductor does | |
| 13:41:53 | mriedem | caching scheduler is the only thing that would still truly force today since it doesn't create allocations | |
| 13:41:59 | mriedem | but we can maybe remove the caching scheduler now | |
| 13:42:10 | mriedem | mgagne: i've been meaning to follow up with you about the caching scheduler removal... | |
| 13:43:50 | gibi | mriedem: does this ignoring behavior needs to be guarded with a new api microversion? I hope not as maintaining the old behavior is not easily possible for nested | |
| 13:44:49 | mriedem | gibi: no i don't think so, | |
| 13:44:55 | mriedem | we're already ignoring force for some filters like i said | |
| 13:45:07 | mriedem | a microversion would be more of a signaling mechanism, | |
| 13:45:31 | mriedem | plus things like nova CLI that do version discovery and pass the latest microversion by default would simply opt into the new microversion where force is never passed | |
| 13:46:15 | mriedem | iow, force is legacy from when we just had claims in the compute and overcomitting was "simpler" | |
| 13:46:21 | gibi | mriedem: so there would be a microversion to remove the force flag from the API but ignoring the force flag would be retroactive in every version | |
| 13:46:44 | bauzas | mriedem: I'm in and out due to some customer issue, but I'll reply to you later on | |
| 13:46:54 | mriedem | well, i don't know if we should blatantly ignore it if we can help it | |
| 13:47:21 | mriedem | but i'm not sure how we determine if we should ignore it or not for nested RPs | |
| 13:47:28 | gibi | mriedem: we cannot easily help it in case of nested, and sooner or later (after NUMA) every instance will be nested | |
| 13:47:39 | mriedem | b/c like you said, the instance might have allocations on nested RPs on the source host but maybe not the forced dest? | |
| 13:47:52 | mriedem | well, every instance with numa allocations | |
| 13:47:57 | mriedem | not all instances are those kinds though | |
| 13:48:26 | gibi | mriedem: if VCPU resources are moved to NUMA RP then the simple instance will become nested | |
| 13:53:19 | mriedem | are we talking about doing that? | |
| 13:53:31 | mriedem | i guess for PCPU/VCPU modeling? | |
| 13:53:46 | mriedem | if so, shiiiiit that reshape is going to be big in a public cloud | |
| 13:53:52 | gibi | mriedem: honestly I don't know what will be the numa model in plaacement | |
| 13:55:27 | gibi | mriedem: https://review.openstack.org/#/c/555081/18/specs/stein/approved/cpu-resources.rst@261 | |
| 13:55:42 | gibi | mriedem: in this spec VCPU is under NUMA | |
| 14:00:39 | mriedem | gibi: ok left a comment / question in the upgrade impact section, | |
| 14:00:51 | mriedem | we also said at the ptg that that spec wouldn't be a priority for stein | |
| 14:00:58 | gibi | mriedem: correct | |
| 14:01:09 | mriedem | just getting vgpu reshaping done is going to be a big hurdle | |
| 14:01:48 | gibi | mriedem: yeah so vgpu and bandwidth using instance will be the first nested instances. So during this time it make sense to keep a separate code path for them | |
| 14:02:14 | gibi | mriedem: I try to implement what you described. checking the nestedness in conductor and ignoring the force flag for nested | |
| 14:24:36 | openstackgerrit | Christoph Manns proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748 | |
| 14:24:57 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | |
| 14:34:14 | openstackgerrit | Mark Goddard proposed openstack/nova master: Don't emit warning when ironic properties are zero https://review.openstack.org/605754 | |
| 14:36:56 | mnaser | filtering happens after placement, correct? | |
| 14:37:03 | mnaser | is there no warning message that says "i couldn't find anything?" | |
| 14:37:26 | mnaser | i guess i can just rely on conductor's "Setting instance to ERROR state." | |
| 14:37:39 | bauzas | mriedem: I'm still working on the vgpu reshape patch, and yes, it's a big hurdle :( | |
| 14:38:22 | bauzas | gibi: FWIW, I need to split https://review.openstack.org/#/c/552924/ in two, one targeted for Stein with no NUMA affinity | |
| 14:38:26 | openstackgerrit | Mark Goddard proposed openstack/nova master: Don't emit warning when ironic properties are zero https://review.openstack.org/605754 | |
| 14:38:48 | bauzas | mnaser: you're right, filters are called after we found an allocation candidate | |
| 14:38:56 | gibi | bauzas: ack | |
| 14:38:57 | openstackgerrit | Merged openstack/nova master: consumer gen: more tests for delete allocation cases https://review.openstack.org/591811 | |
| 14:39:21 | mnaser | thanks bauzas, im relying on this logstash query to monitor those: tags:nova AND message:"Setting instance to ERROR state." AND message:NoValidHost_Remote | |
| 14:39:29 | bauzas | mnaser: but you should at least still have the filtering logs | |
| 14:40:02 | mnaser | yeah the filter logs are there but if 0 computes match at the end, but i'm curious if there's any warning if no allocation candidates that come in the first place | |