| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-27 | |||
| 08:41:25 | stephenfin | kashyap: I assume you're going to follow that up with a patch to bump the current minimums? | |
| 08:41:39 | kashyap | stephenfin: Yeah, indeed. | |
| 08:41:56 | kashyap | stephenfin: Want to ACK the above, already got one from gibi | |
| 08:42:22 | stephenfin | I can. Just reading through the notes on the Wiki first | |
| 08:42:25 | kashyap | Sure | |
| 08:42:59 | kashyap | stephenfin: In short: Oracle Linux already has the relevant versions we bumped to and SLES will have it (Colleen confirmed on the review). | |
| 08:43:06 | kashyap | And the rest of the distributions already have it. | |
| 08:43:26 | kashyap | That's the thread on the list: lists.openstack.org/pipermail/openstack-operators/2018-September/015929.html | |
| 08:43:57 | kashyap | Clicakable: http://lists.openstack.org/pipermail/openstack-operators/2018-September/015929.html | |
| 08:53:38 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Live migration of instances https://review.openstack.org/270116 | |
| 09:17:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628 | |
| 09:34:16 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ib6f95c22ffd3ea235b60db4da32094d49c2efa2a https://review.openstack.org/604743 | |
| 10:00:36 | openstackgerrit | Chen proposed openstack/nova master: remove commented-out code https://review.openstack.org/605635 | |
| 10:18:03 | openstackgerrit | Brin Zhang proposed openstack/nova master: Specifies the storage backend to boot instance https://review.openstack.org/579360 | |
| 10:35:30 | openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | |
| 10:35:34 | openstackgerrit | tianhui proposed openstack/nova master: Update doc: launch-instance-from-volume https://review.openstack.org/605640 | |
| 11:15:48 | openstackgerrit | Merged openstack/nova master: Pick next minimum libvirt / QEMU versions for "T" release https://review.openstack.org/605060 | |
| 11:27:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Followup for Iba230201803ef3d33bccaaf83eb10453eea43f20 https://review.openstack.org/605653 | |
| 11:57:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Iba230201803ef3d33bccaaf83eb10453eea43f20 https://review.openstack.org/605653 | |
| 11:57:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ie991d4b53e9bb5e7ec26da99219178ab7695abf6 https://review.openstack.org/605658 | |
| 12:01:13 | gibi | jaypipes: I've left answer to your question in https://review.openstack.org/#/c/591811/ | |
| 12:13:12 | mdbooth | Hmm, I just discovered that initialisation by defaultdict isn't threadsafe | |
| 12:13:58 | cdent | mdbooth: that statement has an mdbooth number of 7 | |
| 12:14:32 | mdbooth | It means that if you've got: foo = defaultdict(threading.Lock) | |
| 12:14:47 | mdbooth | and 2 threads concurrently do: foo['thing'] | |
| 12:14:57 | mdbooth | They might get different threading.Lock objects | |
| 12:16:54 | cdent | there's some interesting but potentially old discussion at https://stackoverflow.com/questions/17682484/is-collections-defaultdict-thread-safe | |
| 12:17:18 | mdbooth | cdent: Yep, that's where I read it :) | |
| 12:43:06 | openstackgerrit | huanhongda proposed openstack/nova master: Allow to attach/detach port when vm_state is soft-delete https://review.openstack.org/605602 | |
| 12:48:20 | openstackgerrit | Elod Illes proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078 | |
| 12:50:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Null out instance.availability_zone on shelve offload https://review.openstack.org/599087 | |
| 12:50:55 | mriedem | johnthetubaguy: alex_xu: ^ was just a rebase due to a new notification test that needed a sample update | |
| 13:11:08 | gibi | mriedem: is the skip_filters flag of the scheduler you mention in https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L103-L108 a reality or just a dream? | |
| 13:12:28 | gibi | mriedem: I'm trying to solve the nested force migration issues in https://review.openstack.org/#/c/604084/2/nova/tests/functional/test_servers.py@4897 and I think that skip_filters flag is the solution | |
| 13:17:30 | mriedem | gibi: i'm not working on it if that's what you mean | |
| 13:18:09 | mriedem | but i think the idea is still ok - pass a flag to the scheduler to not run filtered hosts, just do the claim on the forced host | |
| 13:18:32 | mriedem | treats the scheduler more like a library | |
| 13:18:42 | mriedem | the point of that todo was more about DRYing up the code | |
| 13:18:52 | gibi | mriedem: would scheduler still call GET allocation_candidates? | |
| 13:19:25 | mriedem | if it needs those to make the claim, then i guess? but that kind of goes against what forcing a host is all about, | |
| 13:19:38 | mriedem | which is, claims be damned this is where i want the thing to go | |
| 13:19:46 | mriedem | but having said that, | |
| 13:19:58 | mriedem | we already broke that contract with https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L103-L108 | |
| 13:20:09 | mriedem | because we have to keep allocations straight in placement | |
| 13:20:17 | mriedem | or that's what we said to justify that change at the time | |
| 13:20:40 | mriedem | honestly force with bypassing the scheduler was always a terrible idea | |
| 13:20:47 | gibi | mriedem: when we forcing a host we cannot blindly copy the source allocation to the dest host in case of nested | |
| 13:21:08 | gibi | mriedem: as we only know the dest host rp_uuid but not the nested rps | |
| 13:21:35 | mriedem | so we need a new set of allocation candidates | |
| 13:21:37 | gibi | mriedem: one way to solve that is to call GET a_c but limit the search for the given dest host | |
| 13:21:51 | mriedem | that seems reasonable | |
| 13:21:53 | mriedem | essentially, | |
| 13:22:25 | mriedem | in the case of nested RP allocations + force, we'd just go through the scheduler with the requested host but kind of ignore the force flag | |
| 13:22:50 | mriedem | i.e. you can live migrate today by specifying a specific host w/o forcing it and we'll validate that requested host with the scheduler | |
| 13:22:54 | mriedem | it sounds like you want to do the same | |
| 13:22:58 | mriedem | s/want/need/ | |
| 13:23:25 | gibi | mriedem: that seems to be the way forward. So now I go and dig that code path | |
| 13:23:30 | gibi | mriedem: thanks | |
| 13:24:07 | mriedem | in the case of an unforced live migration to a specific host, we just set the RequestSpec.requested_destination to the requested host/node and send that to the scheduler | |
| 13:24:24 | mriedem | it sounds like you'd just need some logic up-front to determine, is this an instance that has allocations on nested RPs | |
| 13:24:28 | mriedem | and if so, ignore the force flag | |
| 13:25:12 | gibi | mriedem: yeah, forced live migration will be less forced as it can return NoValidHost after this change | |
| 13:25:13 | mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/compute/api.py#L4374 | |
| 13:25:44 | mriedem | gibi: and i think that's ok - forced live migration could always fail even if you bypassed the scheduler b/c the conductor task still does some prechecks on the forced host | |
| 13:25:45 | gibi | mriedem: which means if every instance will be on nested RP then force livemigration will be equal to non forced livemigration with provided host | |
| 13:26:01 | mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L160 | |
| 13:26:13 | mriedem | like, self._check_host_is_up(self.destination) is the ComputeFilter | |
| 13:26:20 | mriedem | self._check_destination_has_enough_memory() is the RamFilter | |
| 13:26:27 | mriedem | etc | |
| 13:26:56 | mriedem | i'd really like to just get rid of that _check_requested_destination method | |
| 13:27:11 | gibi | mriedem: OK, this seems to be a way forward to eventually get rid of the force live migration altogether | |
| 13:27:28 | jaypipes | mriedem: "recheck slow node" <-- worst 90s band name EVAH. | |
| 13:27:46 | mriedem | jaypipes: it's definitely not as good as butthole surfers | |
| 13:27:52 | jaypipes | indeed. | |
| 13:28:22 | jaypipes | true fact: my high school band in the early 90s was called "Slow Children at Play". | |
| 13:28:46 | mriedem | highly offensive | |
| 13:28:50 | jaypipes | embarrassing, I know. | |
| 13:29:43 | mriedem | https://www.youtube.com/watch?v=SkjJLQUhxks | |
| 13:29:49 | mriedem | now i know what needs to be playing this morning | |
| 13:32:44 | mriedem | bauzas: depending on where you check if you should ignore the force flag it could get messy, | |
| 13:32:45 | mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/compute/api.py#L4359 | |
| 13:32:52 | mriedem | because the api determines what gets passed to conductor | |
| 13:32:56 | mriedem | it's all very tightly coupled | |
| 13:33:21 | mriedem | i.e. if force is True, the requested host parameter isn't passed to conductor, but the request spec is set with the requested_destination | |
| 13:33:51 | mriedem | but if force is True, the request spec is untouched and the requested host is passed to conductor | |
| 13:33:55 | mriedem | and conductor has to know what that means | |
| 13:34:10 | gibi | mriedem: I see | |
| 13:34:21 | mriedem | i assume the logic to determine if the instance has allocations against nested RPs should probably live in conductor | |
| 13:34:25 | mriedem | so we don't block the API response | |
| 13:34:34 | mriedem | although having said that.... | |
| 13:34:52 | mriedem | before some microversion, live migration is aysync rpc call from api-conductor-scheduler until we pick a host and cast to it | |
| 13:35:15 | mriedem | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id31 | |
| 13:35:34 | mriedem | meaning it's pretty easy to timeout the API while picking hosts before 2.34 | |
| 13:35:56 | mriedem | cold migrate is the same way - it's all synchronous until we cast to the chosen compute | |
| 13:36:01 | mriedem | :( | |
| 13:36:42 | mriedem | anywho, i'd recommend doing your nested RP calculations in conductor rather than api | |
| 13:37:04 | gibi | mriedem: ack | |
| 13:37:21 | mriedem | maybe you don't need to look at placement? maybe you can just glean if it's got nested rps by looking at the flavor? | |
| 13:37:52 | mriedem | i guess you care about allocations for ports | |
| 13:38:16 | gibi | mriedem: even if flavor has granular groups I don't know if the host the instance is on are already reshaped to nested or not | |