Posted	Nick	Remark
#openstack-nova - 2017-04-19
14:13:41	mriedem	i liked the idea of posting the allocations from the scheduler where if we needed to retry we already have the list of filtered hosts, but we know that's a toss up perf wise
14:14:01	mriedem	if we have to rpc call into the compute to get overhead, i think we then have to move everything into conductor
14:14:08	dansmith	mriedem: well, and scheduler calling to compute is worse than conductor
14:14:09	dansmith	much worse
14:14:10	dansmith	yeah
14:15:06	dansmith	so, another way to look at this:
14:15:24	dansmith	we're talking about having resource overrides in the flavor
14:15:53	dansmith	we could also just say that the drivers can't calculate different values for the resources, and have to build the vm from the things they're given,
14:16:00	bauzas	sorry, was afk
14:16:08	dansmith	which in the case of hyperv means you get $mem fewer MBs of disk,
14:16:13	dansmith	and in the case of libvirt, you get one fewer vcpu
14:16:34	dansmith	and if you don't want that as an operator, then you override the actual values for those with room for the slack, leaving the display values as what people will really get
14:16:56	dansmith	for people that really use the same flavor for two hypervisor types (if that really happens) then things get ugly
14:17:18	dansmith	but I kinda feel like the cloudy approach here is to not be as snowflake-y on the virt side
14:17:26	bauzas	could someone tell which problem you folks are discussing ? #1 reschedules in a cellsv2 world or #2 overhead values for instance given by virt drivers ?
14:17:31	mriedem	that also pushes a lot of complexity on the operator
14:17:37	dansmith	it does, and that sucks
14:17:39	mriedem	bauzas: #2
14:18:15	dansmith	mriedem: so, we should probably go the extra-call-to-compute route for now and if that ends up being too expensive, then punt later
14:18:31	dansmith	we already do that for things like getting predicted-but-never-correct bdm device names right?
14:18:37	mriedem	yeah, we do that from the api
14:18:42	bauzas	so, my take on that is that overheads, since are only virt-related are things which wouldn't be exposed to users
14:18:43	mriedem	when attaching a volume
14:18:54	dansmith	so here's another potential optimization:
14:18:57	bauzas	users ask for memory sizes they know
14:19:10	bauzas	they don't know how much the hypervisor has
14:19:14	bauzas	so
14:19:25	dansmith	we could do that call once, and record the value we got for the flavor id, hypervisor type and version, and only do that once per boot if we have to try several hosts in a single cycle
14:19:53	mriedem	oh btw, this is nice in the py35 tests: "Fatal Python error: Cannot recover from stack overflow."
14:19:54	dansmith	mriedem: like, if we have to try four libvirt hosts in a single boot call, we only have to call to one of the computes to get the overhead
14:19:56	mriedem	sdague: ^
14:20:00	bauzas	when the scheduler is choosing a destination, he has to pick it based on the user request, but related to the total amount of memory is left
14:20:21	mriedem	dansmith: yeah that's not a bad idea
14:20:27	mriedem	cache the results in conductor
14:20:36	mriedem	take the rpc call hit once
14:20:38	dansmith	we need more information than just the hostname coming back from scheduler though, else we have to hit the db ourselves to look up the host info again
14:20:43	dansmith	yeah
14:21:02	dansmith	either once per call or once per conductor worker
14:21:06	dansmith	or memcache or something
14:21:15	mriedem	i assumed once per worker
14:21:22	dansmith	well, that could get big,
14:21:33	mriedem	sure, if we use the cache utils stuff,
14:21:33	dansmith	as there are tons of workers, and potentially lots of flavors and stuff
14:21:35	mriedem	it's configurable
14:21:41	dansmith	yeah, that's the way to go I think
14:21:43	mriedem	default is per worker in memory cache
14:21:48	mriedem	if you want memcache, configure it
14:21:50	dansmith	yeah
14:22:02	bauzas	wait
14:22:30	bauzas	why placement couldn't compare the user flavor vs. the size of what's left, ie. inventory - instances ?
14:22:30	mriedem	gcb: i'm seeing "Fatal Python error: Cannot recover from stack overflow." in the py35 job for nova unit tests randomly
14:22:35	mriedem	gcb: http://logs.openstack.org/97/456397/3/check/gate-nova-python35/05322ec/console.html#_2017-04-19_12_45_10_724578
14:22:47	bauzas	instances being "instance.flavor + instance.overhead" ?
14:22:51	mriedem	bauzas: placement doesn't have the overhead info
14:23:05	bauzas	mriedem: placement knows inventory I know
14:23:16	mriedem	bauzas: we're saying we need to get the overhead from the compute before we post the allocations
14:23:20	mriedem	to take into account the overhead
14:23:33	mriedem	then placement tells us if that works or not for the inventory on the provider
14:23:44	mriedem	if POST /allocations/instance fails, then we retry through the scheduler
14:24:06	mriedem	dan was just saying that we can cache the rpc call results in conductor - the results of the call to the compute to get the overhead
14:24:23	mriedem	per flavorid/hypervisortype/version
14:24:25	bauzas	mriedem: that would assume the overhead is not calculated per instance
14:24:32	mriedem	huh?
14:24:36	gcb	mriedem: I saw the email , trying to figure out the root cause and fix it
14:24:39	mriedem	the overhead is calculated per instance
14:24:46	mriedem	gcb: this is a different error :)
14:25:01	bauzas	mriedem: yup, so we would query the compute every time a boot request comes in to know the overhead, right?
14:25:06	mriedem	bauzas: yes
14:25:09	mriedem	and cache the results
14:25:25	bauzas	mriedem: cache what ? given we don't know which compute
14:25:34	gcb	mriedem, wow, let me check the details
14:25:39	mriedem	bauzas: we'd do this in conductor after we get a host from the scheduler
14:25:41	bauzas	we would have to cache overheads per flavor for all computes
14:25:47	mriedem	bauzas: right
14:25:47	dansmith	no
14:25:48	bauzas	mriedem: oh, after ?
14:25:57	bauzas	erm, not sure I like that
14:26:07	dansmith	per hypervisor type/version
14:26:10	bauzas	I'd tend to prefer considering that the flavor is what the user requests
14:26:24	mriedem	bauzas: that's not how it works today
14:26:36	bauzas	and placement should verify against the inventory which would be decremented by the sum of overheads
14:26:52	mriedem	i request 8GB of RAM for my vm, but xen says i really need 8GB + 256MB or something
14:27:02	bauzas	at the moment, placement matches against the inventory minus the sum of allocations
14:27:04	lyarwood	mriedem: ack sure looking now
14:27:26	mriedem	bauzas: "and placement should verify against the inventory which would be decremented by the sum of overheads" - yes, but we have to get the overheads to placement
14:27:32	bauzas	I'm proposing to match against the inventory minus the allocations+overhead
14:27:33	mriedem	we do that via the allocations
14:27:42	mriedem	bauzas: yes, we all agree on that
14:27:51	mriedem	that's not new
14:27:54	dansmith	the fact that we have larger allocations than the flavor is not ideal, but at the moment we have no choice
14:28:15	bauzas	but you would not just compare with the flavor for the instance, but flavor+cached_overhead ?
14:28:25	bauzas	well, huge thing
14:28:40	mriedem	the allocations are made up of the flavor today
14:28:42	bauzas	dansmith: mriedem: okay, I think why you'd like to cache that
14:29:46	mriedem	should i go back and update the spec with a summary?
14:30:16	mriedem	bauzas: unfortunately for you, this means redoing a large part of the spec where you changed it from posting allocations in conductor to scheduler, now back to conductor :(
14:31:07	cdent	mriedem: what does that mean with regard to the reasons it moved back to the scheduler in the first place?
14:31:53	cdent	conductor ->(reasons)->sched->(new reasons)->conductor
14:32:53	bauzas	mriedem: this is fine https://www.nytimes.com/2016/08/06/arts/this-is-fine-meme-dog-fire.html
14:33:13	mriedem	cdent: we moved it to scheduler because that's where the filtered host list is, so retries there might be theoretically faster,

Earlier   Later