| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-23 | |||
| 16:36:36 | leakypipes | dansmith: there will be a point (already reached?) where this stuff is just too expensive to try and calculate for each scheduling request and makes the interface between scheduler and placement overly cumbersome. | |
| 16:37:04 | dansmith | leakypipes: I don't think this has anything to do with what scheduler asks of placement | |
| 16:37:17 | dansmith | leakypipes: what is described in the spec is a simple change to the current hard affinity filter | |
| 16:37:31 | dansmith | and what I was proposing would be similar but with some different math | |
| 16:37:56 | dansmith | the biggest change is "oh now we're going to have extra specs on instance groups" | |
| 16:38:08 | dansmith | which I think is a very big step we should not take lightly | |
| 16:38:11 | dansmith | (or maybe at all) | |
| 16:38:50 | dansmith | especially my point about changing that value on an existing group, because people will expect that if they change that from 3 to 2, existing instances get moved around | |
| 16:38:54 | dansmith | and that ain't hap'nan | |
| 16:48:02 | leakypipes | dansmith, jroll: if you want some mind-bending reading from k8s on this subject: https://github.com/kubernetes/kubernetes/pull/18265/files | |
| 16:48:17 | leakypipes | dansmith, jroll: note that k8s affinity stuff is still in beta... | |
| 16:48:50 | jroll | oh my | |
| 16:49:29 | mriedem | dansmith: i think the 'extra spec' term in his spec is a mistake, or just a WIP thing, | |
| 16:49:39 | mriedem | i don't think we should add a random bag of extra specs to instance groups either | |
| 16:50:04 | mriedem | dansmith: i also said updating the max-per-host value on an existing group isn't going to happen | |
| 16:50:15 | mriedem | for the same reasons we don't allow changing policy on an existing group, or changing membership | |
| 16:51:00 | leakypipes | what about an awkward bag of flaccid specs? | |
| 16:51:27 | mriedem | if it's as simple as putting an attribute on a group in a hard affinity policy that says, instead of allowing no more than 1 vm from this group per host, you can allow up to 3 (or whatever), then that seems to fit the bill | |
| 16:51:35 | mriedem | leakypipes: they have pills for that | |
| 16:51:41 | leakypipes | lol | |
| 16:51:49 | mriedem | let me get you frank thomas' number | |
| 16:52:00 | mriedem | actually, jimmy johnson sells them too and he already lives in FL | |
| 16:52:32 | cfriesen | dansmith: we originally had metadata in instance groups, but it got pulled out due to not really having any users | |
| 16:52:32 | mriedem | sorry i got distracted, was hastily throwing together a PBC... | |
| 16:52:54 | mriedem | cfriesen: yeah that was linked into the spec and was something i didn't even know existed | |
| 16:53:04 | mriedem | metadata in general makes our lives terrible | |
| 16:53:12 | mriedem | like aggregate meta, and flavor extra specs | |
| 16:53:23 | mriedem | i realize it's use though | |
| 16:54:06 | cfriesen | dansmith: it'd be possible to prohibit changing the value on an existing group to something that would result in the current spread being invalid....alternately you could just allow that and document that it'll only affect the *next* scheduling decision. | |
| 16:54:39 | mriedem | cfriesen: to determine if the new requested value would invalidate things would mean running through the scheduler all over again | |
| 16:54:44 | mriedem | and you could still get it wrong | |
| 16:54:53 | cfriesen | mriedem: there has been that recurring spec for allowing instances to be added/subtracted from a group | |
| 16:54:54 | mriedem | which is why we have the late affinity check on the compute hosts | |
| 16:55:12 | mriedem | cfriesen: i remember powervc pushing it back in kilo but that's been abandoned for a long time | |
| 16:55:39 | cfriesen | arguably we've got races all over with instance group affinity, so what's one more. :) | |
| 16:55:41 | dansmith | cfriesen: yeah then the group is in violation of policy with no way of getting it out, unless you do your own shuffling | |
| 16:56:07 | dansmith | mriedem: and yeah I'd rather a real attribute, but I also think we're just going to end up with unlimited attributes for other things like this | |
| 16:56:15 | dansmith | mriedem: so I dunno.. it's a can of worms | |
| 16:57:53 | mriedem | idk, if this is the first time somtehing like this has come up in the last what 5 years? | |
| 16:58:23 | cfriesen | for what it's worth, internally we added a "best-effort" flag to the "hard" affinity/antiaffinity policies to allow us to migrate instances off a compute node for maintenance | |
| 16:58:23 | mriedem | it would get annoying if, over time, we had a bunch of attributes on groups that only applied to specific policies | |
| 16:58:37 | mriedem | cfriesen: so soft affinity? | |
| 16:58:56 | cfriesen | no, you can turn the best-effort flag on and off dynamically | |
| 16:59:17 | cfriesen | so you'd normally run with it strict, but if you need to take down a compute node you can set best-effort, move everything, then turn it back off | |
| 16:59:41 | cfriesen | otherwise if you've got hard-affinity you can't migrate any of them | |
| 17:00:04 | mriedem | which is why people want a force flag on evacuate, live migrate (and now cold migrate) | |
| 17:00:06 | leakypipes | mriedem: and the late affinity check is (the only?) remaining upcall from a cell to API, no? | |
| 17:00:15 | mriedem | leakypipes: hells no | |
| 17:00:32 | leakypipes | mriedem: it's not an upcall? | |
| 17:00:35 | mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | |
| 17:00:38 | mriedem | it is an upcall | |
| 17:00:42 | mriedem | but it's not the only one | |
| 17:00:44 | leakypipes | oh, not the only reminaing.. | |
| 17:00:46 | mriedem | we've got aggregates too | |
| 17:01:04 | mriedem | so aggregates and affinity are the remaining upcall issues | |
| 17:01:21 | mriedem | but right now there are 2 each | |
| 17:01:24 | mriedem | so 4 upcall issues | |
| 17:01:33 | leakypipes | mriedem: then you'll LOVE my aggregate affinity spec! :) now with moar AFFINITY and moar AGGREGATES! | |
| 17:01:48 | mriedem | leakypipes: i already said 'upcall upcall upcall' on that spec several times :) | |
| 17:01:54 | leakypipes | I know :) | |
| 17:02:20 | mriedem | i think i also hedged with something like, 'but we already do this in a few other places so people already have to rely on it, so maybe another log on the fire doesn't kill us' | |
| 17:02:55 | cfriesen | On a totally different topic...has anyone ever heard of nova allocating duplicate network interfaces? (So the user boots while asking for 2 network interfaces, and nova allocates two ports on each network.) | |
| 17:03:08 | mriedem | yes | |
| 17:03:10 | mriedem | that's old news | |
| 17:03:25 | mriedem | tempest has a test for it also i think | |
| 17:03:31 | mriedem | been around since juno? | |
| 17:05:28 | cfriesen | I mean nova is allocating twice as many as were asked for. | |
| 17:05:47 | mriedem | double your pleasure | |
| 17:05:49 | mriedem | idk, going to lunch | |
| 17:14:43 | efried | Greetings from JFK airport | |
| 17:27:30 | mnaser | ok | |
| 17:27:33 | mnaser | im convinced grenade is broken | |
| 17:27:43 | mnaser | for stable/pike | |
| 17:28:05 | efried | Isn't that mnaser guy known for being johnny-on-the-spot for grenade fixes? | |
| 17:28:16 | mnaser | only when i have to :( | |
| 17:28:29 | mnaser | Host 'ubuntu-xenial-rax-dfw-0002683360' is not mapped to any cell | |
| 17:28:36 | mnaser | we keep getting this in multinode | |
| 17:29:49 | efried | It would seem odd that cell discovery isn't being run. Like, nothing would ever work. | |
| 17:30:00 | efried | And that's pretty much the only thing I know about cells. | |
| 17:30:40 | mnaser | efried: indeed seems to be the case. looks like there was a change to add 'CELLSV2_SETUP=singleconductor' in there, so not sure if that might have affected it | |
| 17:31:13 | efried | mnaser: You're going to have something of a hard time finding a core today, but I'll +1 your fix :) | |
| 17:31:31 | mnaser | efried: looks like there's only 3 cores for grenade too.. | |
| 17:32:49 | efried | Looks like qa-release is included by inheritance. | |
| 17:33:02 | efried | So seven | |
| 17:35:12 | mnaser | ok looks like this runs => nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@10.209.130.218:5672/ | |
| 17:35:39 | mnaser | but discover_hosts is never called | |
| 17:36:22 | efried | And simple_cell_setup doesn't run discovery itself? | |
| 17:36:52 | efried | I remember having to fix this around pike timeframe. | |
| 17:37:02 | mnaser | efried: going through the code it looks like it does call _map_cell_and_hosts() | |
| 17:47:51 | openstack | Launchpad bug 1708039 in devstack "gate-grenade-dsvm-neutron-multinode-ubuntu-xenial fails with "No host-to-cell mapping found for selected host"" [Medium,Fix released] - Assigned to Sean Dague (sdague) | |
| 17:47:51 | mnaser | sigh https://bugs.launchpad.net/grenade/+bug/1708039 looks like it was 'supposed' to be fixed | |
| 17:49:09 | mnaser | it looks like it regressed | |
| 17:49:12 | mnaser | and its all stable/pike hits | |
| 17:54:40 | mnaser | "Didn't find service registered by hostname after 60 seconds" .. found it | |
| 17:54:49 | mnaser | it's actually listed but the bash for some reason doesnt find it | |
| 17:55:51 | efried | leakypipes: You around today? | |
| 17:55:58 | efried | Hoho, it's Friday | |
| 17:56:14 | andreaf | mriedem hey I'm setting up a zuul-v3 multinode job, and everything works fine apart from nova that gives me "Host is not mapped to any cell" http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/ara/result/525a60bd-ac22-4fc4-9db7-e61fce8ac1f5/ | |
| 17:56:45 | andreaf | mriedem: I compared configs in localrc and nova and I don't see anything obvious - do you have any idea about what this could be? | |
| 17:56:52 | leakypipes | fried_rice_jfk: ues | |