Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-23
16:35:46 jroll :P
16:36:36 leakypipes dansmith: there will be a point (already reached?) where this stuff is just too expensive to try and calculate for each scheduling request and makes the interface between scheduler and placement overly cumbersome.
16:37:04 dansmith leakypipes: I don't think this has anything to do with what scheduler asks of placement
16:37:17 dansmith leakypipes: what is described in the spec is a simple change to the current hard affinity filter
16:37:31 dansmith and what I was proposing would be similar but with some different math
16:37:56 dansmith the biggest change is "oh now we're going to have extra specs on instance groups"
16:38:08 dansmith which I think is a very big step we should not take lightly
16:38:11 dansmith (or maybe at all)
16:38:50 dansmith especially my point about changing that value on an existing group, because people will expect that if they change that from 3 to 2, existing instances get moved around
16:38:54 dansmith and that ain't hap'nan
16:48:02 leakypipes dansmith, jroll: if you want some mind-bending reading from k8s on this subject: https://github.com/kubernetes/kubernetes/pull/18265/files
16:48:17 leakypipes dansmith, jroll: note that k8s affinity stuff is still in beta...
16:48:50 jroll oh my
16:49:29 mriedem dansmith: i think the 'extra spec' term in his spec is a mistake, or just a WIP thing,
16:49:39 mriedem i don't think we should add a random bag of extra specs to instance groups either
16:50:04 mriedem dansmith: i also said updating the max-per-host value on an existing group isn't going to happen
16:50:15 mriedem for the same reasons we don't allow changing policy on an existing group, or changing membership
16:51:00 leakypipes what about an awkward bag of flaccid specs?
16:51:27 mriedem if it's as simple as putting an attribute on a group in a hard affinity policy that says, instead of allowing no more than 1 vm from this group per host, you can allow up to 3 (or whatever), then that seems to fit the bill
16:51:35 mriedem leakypipes: they have pills for that
16:51:41 leakypipes lol
16:51:49 mriedem let me get you frank thomas' number
16:52:00 mriedem actually, jimmy johnson sells them too and he already lives in FL
16:52:32 mriedem sorry i got distracted, was hastily throwing together a PBC...
16:52:32 cfriesen dansmith: we originally had metadata in instance groups, but it got pulled out due to not really having any users
16:52:54 mriedem cfriesen: yeah that was linked into the spec and was something i didn't even know existed
16:53:04 mriedem metadata in general makes our lives terrible
16:53:12 mriedem like aggregate meta, and flavor extra specs
16:53:23 mriedem i realize it's use though
16:54:06 cfriesen dansmith: it'd be possible to prohibit changing the value on an existing group to something that would result in the current spread being invalid....alternately you could just allow that and document that it'll only affect the *next* scheduling decision.
16:54:39 mriedem cfriesen: to determine if the new requested value would invalidate things would mean running through the scheduler all over again
16:54:44 mriedem and you could still get it wrong
16:54:53 cfriesen mriedem: there has been that recurring spec for allowing instances to be added/subtracted from a group
16:54:54 mriedem which is why we have the late affinity check on the compute hosts
16:55:12 mriedem cfriesen: i remember powervc pushing it back in kilo but that's been abandoned for a long time
16:55:39 cfriesen arguably we've got races all over with instance group affinity, so what's one more. :)
16:55:41 dansmith cfriesen: yeah then the group is in violation of policy with no way of getting it out, unless you do your own shuffling
16:56:07 dansmith mriedem: and yeah I'd rather a real attribute, but I also think we're just going to end up with unlimited attributes for other things like this
16:56:15 dansmith mriedem: so I dunno.. it's a can of worms
16:57:53 mriedem idk, if this is the first time somtehing like this has come up in the last what 5 years?
16:58:23 mriedem it would get annoying if, over time, we had a bunch of attributes on groups that only applied to specific policies
16:58:23 cfriesen for what it's worth, internally we added a "best-effort" flag to the "hard" affinity/antiaffinity policies to allow us to migrate instances off a compute node for maintenance
16:58:37 mriedem cfriesen: so soft affinity?
16:58:56 cfriesen no, you can turn the best-effort flag on and off dynamically
16:59:17 cfriesen so you'd normally run with it strict, but if you need to take down a compute node you can set best-effort, move everything, then turn it back off
16:59:41 cfriesen otherwise if you've got hard-affinity you can't migrate any of them
17:00:04 mriedem which is why people want a force flag on evacuate, live migrate (and now cold migrate)
17:00:06 leakypipes mriedem: and the late affinity check is (the only?) remaining upcall from a cell to API, no?
17:00:15 mriedem leakypipes: hells no
17:00:32 leakypipes mriedem: it's not an upcall?
17:00:35 mriedem https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls
17:00:38 mriedem it is an upcall
17:00:42 mriedem but it's not the only one
17:00:44 leakypipes oh, not the only reminaing..
17:00:46 mriedem we've got aggregates too
17:01:04 mriedem so aggregates and affinity are the remaining upcall issues
17:01:21 mriedem but right now there are 2 each
17:01:24 mriedem so 4 upcall issues
17:01:33 leakypipes mriedem: then you'll LOVE my aggregate affinity spec! :) now with moar AFFINITY and moar AGGREGATES!
17:01:48 mriedem leakypipes: i already said 'upcall upcall upcall' on that spec several times :)
17:01:54 leakypipes I know :)
17:02:20 mriedem i think i also hedged with something like, 'but we already do this in a few other places so people already have to rely on it, so maybe another log on the fire doesn't kill us'
17:02:55 cfriesen On a totally different topic...has anyone ever heard of nova allocating duplicate network interfaces? (So the user boots while asking for 2 network interfaces, and nova allocates two ports on each network.)
17:03:08 mriedem yes
17:03:10 mriedem that's old news
17:03:25 mriedem tempest has a test for it also i think
17:03:31 mriedem been around since juno?
17:05:28 cfriesen I mean nova is allocating twice as many as were asked for.
17:05:47 mriedem double your pleasure
17:05:49 mriedem idk, going to lunch
17:14:43 efried Greetings from JFK airport
17:27:30 mnaser ok
17:27:33 mnaser im convinced grenade is broken
17:27:43 mnaser for stable/pike
17:28:05 efried Isn't that mnaser guy known for being johnny-on-the-spot for grenade fixes?
17:28:16 mnaser only when i have to :(
17:28:29 mnaser Host 'ubuntu-xenial-rax-dfw-0002683360' is not mapped to any cell
17:28:36 mnaser we keep getting this in multinode
17:29:49 efried It would seem odd that cell discovery isn't being run. Like, nothing would ever work.
17:30:00 efried And that's pretty much the only thing I know about cells.
17:30:40 mnaser efried: indeed seems to be the case. looks like there was a change to add 'CELLSV2_SETUP=singleconductor' in there, so not sure if that might have affected it
17:31:13 efried mnaser: You're going to have something of a hard time finding a core today, but I'll +1 your fix :)
17:31:31 mnaser efried: looks like there's only 3 cores for grenade too..
17:32:49 efried Looks like qa-release is included by inheritance.
17:33:02 efried So seven
17:35:12 mnaser ok looks like this runs => nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@10.209.130.218:5672/
17:35:39 mnaser but discover_hosts is never called
17:36:22 efried And simple_cell_setup doesn't run discovery itself?
17:36:52 efried I remember having to fix this around pike timeframe.
17:37:02 mnaser efried: going through the code it looks like it does call _map_cell_and_hosts()
17:47:51 mnaser sigh https://bugs.launchpad.net/grenade/+bug/1708039 looks like it was 'supposed' to be fixed
17:47:51 openstack Launchpad bug 1708039 in devstack "gate-grenade-dsvm-neutron-multinode-ubuntu-xenial fails with "No host-to-cell mapping found for selected host"" [Medium,Fix released] - Assigned to Sean Dague (sdague)
17:49:09 mnaser it looks like it regressed
17:49:12 mnaser and its all stable/pike hits
17:54:40 mnaser "Didn't find service registered by hostname after 60 seconds" .. found it
17:54:49 mnaser it's actually listed but the bash for some reason doesnt find it
17:55:51 efried leakypipes: You around today?
17:55:58 efried Hoho, it's Friday
17:56:14 andreaf mriedem hey I'm setting up a zuul-v3 multinode job, and everything works fine apart from nova that gives me "Host is not mapped to any cell" http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/ara/result/525a60bd-ac22-4fc4-9db7-e61fce8ac1f5/
17:56:45 andreaf mriedem: I compared configs in localrc and nova and I don't see anything obvious - do you have any idea about what this could be?

Earlier   Later