Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-23
15:46:32 sean-k-mooney i guess this is one of those things that an operator should not do. i would feel more comfortable with always going to the schduler but i guess that would be a behavior change so it would need at least a microversion bump.
15:50:19 sean-k-mooney stephenfin: fyi, when cpus and memory are modeled in placement as the numa topoloy can be altered via the image metadata you will have to recalulate it on rebuild and check if the placement allocation still fufil the request even when we use the same image since we do not use a cached copy of the image metadata.
15:53:43 stephenfin sean-k-mooney: Sounds fair. Best tell jaypipes to get a move on then ;)
15:53:50 s10 So if we want to start 1000 new instances with 6 nova-scheduler process, without this step (if we commented it out), all of them end up being started tens of minutes faster.
15:53:50 s10 Our tests show, that this step (loading list of instances to the host_state) takes ~10 seconds for deployments with thousands of instances and hundreds of hosts.
15:53:50 s10 Hi. Is there any possibility to somehow optimise host_state.update(), which runs on every instance scheduling? Specifically this: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L181
15:55:18 sean-k-mooney stephenfin: well personally i think simply always going to the schduler on a rebuild would be a better change since the numatopolgy filter would assert the correct behavior
15:55:25 cdent s10: I'd recommend posting to the openstack-dev list about that. It's a topic that I've seen come up from multiple peope lately, and they're not all here rightnow.
15:56:45 mriedem s10: yeah it's a known issue, i have a semi-related patch, sec
15:57:11 mriedem s10: https://review.openstack.org/#/c/569247/
15:57:16 s10 I see https://bugs.launchpad.net/nova/+bug/1737465
15:57:17 openstack Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:57:19 s10 We use Pike
15:57:28 mriedem s10: yes same related bug
15:57:28 sean-k-mooney mriedem: s10 is this not required to make sure we are not using statle hostsate info?
15:58:05 mriedem sean-k-mooney: yes it is
15:58:08 mriedem for the affinity filters
15:58:49 mriedem s10: for starters, i've been hoping someone with a large enough deployment can tell me if this makes an improvement https://review.openstack.org/#/c/569247/ because CERN was reporting the same issue
15:59:01 mriedem s10: if you can test that out we could move it forward
15:59:02 sean-k-mooney mriedem: so commenting it out while faster would be incorrect unless we updated the afinit filters to get the updated info but that would break multicreate
15:59:12 mriedem there are more ways to optimize these calls
15:59:53 s10 Yes, we can't really comment them out, because we rely on the anti-affinity filter for the octavia load-balancers.
16:00:26 mriedem the information is also somewhat cached and the computes rpc cast to the scheduler whenever an instance is created/deleted/moved assuming you have that setup
16:00:32 s10 And we use host_state.instances in our custom RealRamFilter...
16:00:40 mriedem that's the track_instance_changes config option,
16:01:19 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes
16:01:57 mriedem if you're doing split MQ though then the comptues can't reach the scheduler
16:01:59 mriedem and that doesn't help
16:02:32 mriedem we also know that we can optimize here https://review.openstack.org/#/c/571928/2/nova/scheduler/host_manager.py with some db join magic
16:02:37 sean-k-mooney mriedem: looking at https://review.openstack.org/#/c/569247 the optimisationis storing a list of uuids per host rather then the full instance objects correct? i can see how that would work for the in tree affintiy filters and the num instance weigher
16:03:01 mriedem sean-k-mooney: well it's also the db query
16:03:19 mriedem rather than query the full instance objects and such per host, just get the uuids
16:04:06 mriedem s10: if you can test out https://review.openstack.org/#/c/569247/ and report back that would be very helpful in moving this forward
16:04:38 sean-k-mooney mriedem: sure that makes sense.
16:04:51 cdent mriedem, s10: whatever you work out here, it would still be great to see some kind of summary to the mailing list
16:05:08 s10 mriedem: Ok, I will test it
16:05:31 s10 mriedem: could this change be cherry-picked without rework to Pike?
16:06:52 sean-k-mooney s10 it looks pretty trivial in terms of code change. this would not qualify for backport however as it is not fixing a bug. it is a performance enhancement
16:08:45 mriedem s10: probably, there was another change merged in the series before it that you'll need https://review.openstack.org/#/q/topic:bug/1737465+(status:open+OR+status:merged)
16:08:55 mriedem https://review.openstack.org/#/q/Iccefbfdfa578515a004ef6ac718bac1a49d5c5fd
16:09:00 mriedem you'll need that in pike first
16:09:18 mriedem and then https://review.openstack.org/#/c/569247/
16:15:15 s10 Our out-of-tree filter rely on host_state.instances.values(), we need values of the instance.memory_mb. Will we be hit with a lazy-load penalty in this case, with https://review.openstack.org/#/c/570083/?
16:16:53 dansmith yup
16:18:40 sean-k-mooney s10: what does your real ram filter actully check vs what placement will do?
16:19:12 sean-k-mooney s10: are you trying to disable oversubscription?
16:20:19 s10 sean-k-mooney: actual usage of the ram on compute nodes. Every QEMU process produces some memory overhead (up to 500-1000mb in some cases, depends on disk usage and rbd cached, I believe). Placement doesn't know anything about it and can schedule instance to the host, where real size of free ram is less than reserved_memory_mb.
16:20:51 mriedem yeah we don't track overhead in placement, that's a known thing
16:21:27 mriedem and we've said for now, buffer that with reserved_host_memory_mb
16:21:40 sean-k-mooney s10: how are you getting the real host memory usage?
16:23:02 mriedem s10: also fyi http://lists.openstack.org/pipermail/openstack-dev/2018-May/130624.html
16:23:04 s10 We patched nova/scheduler/host_manager.py, added self.memory_mb_used = compute.memory_mb_used
16:23:21 mriedem s10: then why would you need host_state.instances.values()?
16:23:54 s10 Because without it we can't compute reserved_memory_mb, it's in config...
16:24:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
16:24:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
16:24:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931
16:24:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Cache is_bfv check in ResourceTracker https://review.openstack.org/584962
16:24:26 s10 Maybe we should pass reserved_memory_mb same way as we pass memory_mb_used...
16:25:01 sean-k-mooney s10: it would make sense to me to just embed the reserved memory in the host state object
16:25:24 mriedem to be clear, CONF.reserved_host_memory_mb is used to reserve capacity for MEMORY_MB in the compute node resource provider's inventory in placement
16:25:35 mriedem which is used to filter hosts before they even get to your filter
16:25:58 mriedem if you need to reserve space on the host for qemu/rbd, you should increase reserved_host_memory_mb per host
16:26:32 s10 But it is still calculated per compute, based on the config in nova-compute, right?
16:26:51 mriedem the config is read in compute yes
16:27:17 mriedem compute node memory_mb_used is later adjusted in the resource tracker based on the instance flavor
16:27:20 mriedem per instance on that host
16:27:43 sean-k-mooney mriedem: ture but s10's filter would catch the case where they did not reserve enough and the free ram is actully less then the reserved
16:27:52 mriedem but that information is also available via placement and is used to calculate the allocation candidates that the scheduler gets before we even hit the filters
16:28:28 mriedem sean-k-mooney: you might need to be conservative then in the reserved memory_mb
16:28:31 sean-k-mooney mriedem: placement only has allocations based on the flavor request. not the actul memory used by the instance
16:28:32 mriedem if this is a persistent problem
16:29:24 mriedem Kevin_Zheng: fyi https://review.openstack.org/#/c/584962/
16:29:32 s10 We used reserved_host_memory_mb for that purpose, but it doesn't work well for us, some instances produces memory overhead over time, and in this case we might end up reserving 50% of host memory
16:29:55 sean-k-mooney mriedem: im just wondering could we symplify this by comparing free_ram_mb > reseved in the fileter if we added reserved to the hostsate object
16:30:21 mriedem s10: hmm, ok, it would be helpful to write something up about this problem in the openstack-dev mailing list, tag the subject line with [nova] and describe the issue and what your scheduler filter does to resolve it
16:30:36 mriedem s10: upstream changes only affect out of tree filters while those filters are out of tree,
16:30:47 mriedem if your filter were in tree, we would have considered this when making that change
16:31:18 sean-k-mooney s10: well one of the main issue with reserved_host_memory_mb is that it does not take account of numa topology of the host so your instance can be killed by oom if you are not useing hugepages and a numa node is exausted but non local memory is free
16:31:26 mriedem sean-k-mooney: maybe, but solutions should be discussed in the ML
16:31:41 s10 Thank you. I will test performance with fixes for the host_state, and ask our developers to change our filter not to use host_state.
16:31:51 sean-k-mooney mriedem: sure :) it was just a taught
16:32:28 mriedem mostly just because this is a bad week with FF on thursday
16:32:35 mriedem don't really have time to design solutions for things that can go into the ML
16:35:00 sean-k-mooney s10: is there a bug open for your usecase? its 2 late to adress in rocky but we could likely adrss in early stien if we captured the request.
16:38:03 s10 sean-k-mooney: I didn't see bugs, related to real memory overhead problem. Only remember, that there was a thread in openstack-operators (?)
16:40:47 s10 https://bugs.launchpad.net/nova/+bug/1683858
16:40:48 openstack Launchpad bug 1683858 in OpenStack Compute (nova) "Allocation records do not contain overhead information" [Medium,Won't fix]
16:41:22 sean-k-mooney s10: based on the wording i assume that was suggesting this in placement hence the wont fix.
16:41:59 sean-k-mooney s10: i dont think placement is the right place to adress this but an in tree filter would be potentially reasonable.
16:44:57 mriedem stephenfin: we've got a runway slot opening so i'm going to throw the numa vswitch series back in ther
16:44:58 mriedem *there
16:44:59 mriedem just FYI
16:45:16 stephenfin Sweet. Should have that move thing done in a short bit
16:49:58 sean-k-mooney s10: i wont promise anything but i have added this usecase to my whiteboard of random TODOs. i might see if i can come up with somthing for stien if there is still interest.
16:50:56 AJaeger nova cores, could you review https://review.openstack.org/#/c/583023/ to move the job in-tree, please? that'S a followup to work for os-vif. stephenfin and sean-k-mooney, want to look at it again, please?
16:52:50 stephenfin AJaeger: ack
16:53:21 stephenfin lyarwood: I didn't get to https://review.openstack.org/#/c/562072/ yet :( It's first thing I'm doing tomorrow morning though, I promise
16:53:38 mriedem dansmith should probably look at those
16:53:45 mriedem given evacness

Earlier   Later