Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-23
15:55:18 sean-k-mooney stephenfin: well personally i think simply always going to the schduler on a rebuild would be a better change since the numatopolgy filter would assert the correct behavior
15:55:25 cdent s10: I'd recommend posting to the openstack-dev list about that. It's a topic that I've seen come up from multiple peope lately, and they're not all here rightnow.
15:56:45 mriedem s10: yeah it's a known issue, i have a semi-related patch, sec
15:57:11 mriedem s10: https://review.openstack.org/#/c/569247/
15:57:16 s10 I see https://bugs.launchpad.net/nova/+bug/1737465
15:57:17 openstack Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:57:19 s10 We use Pike
15:57:28 sean-k-mooney mriedem: s10 is this not required to make sure we are not using statle hostsate info?
15:57:28 mriedem s10: yes same related bug
15:58:05 mriedem sean-k-mooney: yes it is
15:58:08 mriedem for the affinity filters
15:58:49 mriedem s10: for starters, i've been hoping someone with a large enough deployment can tell me if this makes an improvement https://review.openstack.org/#/c/569247/ because CERN was reporting the same issue
15:59:01 mriedem s10: if you can test that out we could move it forward
15:59:02 sean-k-mooney mriedem: so commenting it out while faster would be incorrect unless we updated the afinit filters to get the updated info but that would break multicreate
15:59:12 mriedem there are more ways to optimize these calls
15:59:53 s10 Yes, we can't really comment them out, because we rely on the anti-affinity filter for the octavia load-balancers.
16:00:26 mriedem the information is also somewhat cached and the computes rpc cast to the scheduler whenever an instance is created/deleted/moved assuming you have that setup
16:00:32 s10 And we use host_state.instances in our custom RealRamFilter...
16:00:40 mriedem that's the track_instance_changes config option,
16:01:19 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes
16:01:57 mriedem if you're doing split MQ though then the comptues can't reach the scheduler
16:01:59 mriedem and that doesn't help
16:02:32 mriedem we also know that we can optimize here https://review.openstack.org/#/c/571928/2/nova/scheduler/host_manager.py with some db join magic
16:02:37 sean-k-mooney mriedem: looking at https://review.openstack.org/#/c/569247 the optimisationis storing a list of uuids per host rather then the full instance objects correct? i can see how that would work for the in tree affintiy filters and the num instance weigher
16:03:01 mriedem sean-k-mooney: well it's also the db query
16:03:19 mriedem rather than query the full instance objects and such per host, just get the uuids
16:04:06 mriedem s10: if you can test out https://review.openstack.org/#/c/569247/ and report back that would be very helpful in moving this forward
16:04:38 sean-k-mooney mriedem: sure that makes sense.
16:04:51 cdent mriedem, s10: whatever you work out here, it would still be great to see some kind of summary to the mailing list
16:05:08 s10 mriedem: Ok, I will test it
16:05:31 s10 mriedem: could this change be cherry-picked without rework to Pike?
16:06:52 sean-k-mooney s10 it looks pretty trivial in terms of code change. this would not qualify for backport however as it is not fixing a bug. it is a performance enhancement
16:08:45 mriedem s10: probably, there was another change merged in the series before it that you'll need https://review.openstack.org/#/q/topic:bug/1737465+(status:open+OR+status:merged)
16:08:55 mriedem https://review.openstack.org/#/q/Iccefbfdfa578515a004ef6ac718bac1a49d5c5fd
16:09:00 mriedem you'll need that in pike first
16:09:18 mriedem and then https://review.openstack.org/#/c/569247/
16:15:15 s10 Our out-of-tree filter rely on host_state.instances.values(), we need values of the instance.memory_mb. Will we be hit with a lazy-load penalty in this case, with https://review.openstack.org/#/c/570083/?
16:16:53 dansmith yup
16:18:40 sean-k-mooney s10: what does your real ram filter actully check vs what placement will do?
16:19:12 sean-k-mooney s10: are you trying to disable oversubscription?
16:20:19 s10 sean-k-mooney: actual usage of the ram on compute nodes. Every QEMU process produces some memory overhead (up to 500-1000mb in some cases, depends on disk usage and rbd cached, I believe). Placement doesn't know anything about it and can schedule instance to the host, where real size of free ram is less than reserved_memory_mb.
16:20:51 mriedem yeah we don't track overhead in placement, that's a known thing
16:21:27 mriedem and we've said for now, buffer that with reserved_host_memory_mb
16:21:40 sean-k-mooney s10: how are you getting the real host memory usage?
16:23:02 mriedem s10: also fyi http://lists.openstack.org/pipermail/openstack-dev/2018-May/130624.html
16:23:04 s10 We patched nova/scheduler/host_manager.py, added self.memory_mb_used = compute.memory_mb_used
16:23:21 mriedem s10: then why would you need host_state.instances.values()?
16:23:54 s10 Because without it we can't compute reserved_memory_mb, it's in config...
16:24:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
16:24:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
16:24:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931
16:24:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Cache is_bfv check in ResourceTracker https://review.openstack.org/584962
16:24:26 s10 Maybe we should pass reserved_memory_mb same way as we pass memory_mb_used...
16:25:01 sean-k-mooney s10: it would make sense to me to just embed the reserved memory in the host state object
16:25:24 mriedem to be clear, CONF.reserved_host_memory_mb is used to reserve capacity for MEMORY_MB in the compute node resource provider's inventory in placement
16:25:35 mriedem which is used to filter hosts before they even get to your filter
16:25:58 mriedem if you need to reserve space on the host for qemu/rbd, you should increase reserved_host_memory_mb per host
16:26:32 s10 But it is still calculated per compute, based on the config in nova-compute, right?
16:26:51 mriedem the config is read in compute yes
16:27:17 mriedem compute node memory_mb_used is later adjusted in the resource tracker based on the instance flavor
16:27:20 mriedem per instance on that host
16:27:43 sean-k-mooney mriedem: ture but s10's filter would catch the case where they did not reserve enough and the free ram is actully less then the reserved
16:27:52 mriedem but that information is also available via placement and is used to calculate the allocation candidates that the scheduler gets before we even hit the filters
16:28:28 mriedem sean-k-mooney: you might need to be conservative then in the reserved memory_mb
16:28:31 sean-k-mooney mriedem: placement only has allocations based on the flavor request. not the actul memory used by the instance
16:28:32 mriedem if this is a persistent problem
16:29:24 mriedem Kevin_Zheng: fyi https://review.openstack.org/#/c/584962/
16:29:32 s10 We used reserved_host_memory_mb for that purpose, but it doesn't work well for us, some instances produces memory overhead over time, and in this case we might end up reserving 50% of host memory
16:29:55 sean-k-mooney mriedem: im just wondering could we symplify this by comparing free_ram_mb > reseved in the fileter if we added reserved to the hostsate object
16:30:21 mriedem s10: hmm, ok, it would be helpful to write something up about this problem in the openstack-dev mailing list, tag the subject line with [nova] and describe the issue and what your scheduler filter does to resolve it
16:30:36 mriedem s10: upstream changes only affect out of tree filters while those filters are out of tree,
16:30:47 mriedem if your filter were in tree, we would have considered this when making that change
16:31:18 sean-k-mooney s10: well one of the main issue with reserved_host_memory_mb is that it does not take account of numa topology of the host so your instance can be killed by oom if you are not useing hugepages and a numa node is exausted but non local memory is free
16:31:26 mriedem sean-k-mooney: maybe, but solutions should be discussed in the ML
16:31:41 s10 Thank you. I will test performance with fixes for the host_state, and ask our developers to change our filter not to use host_state.
16:31:51 sean-k-mooney mriedem: sure :) it was just a taught
16:32:28 mriedem mostly just because this is a bad week with FF on thursday
16:32:35 mriedem don't really have time to design solutions for things that can go into the ML
16:35:00 sean-k-mooney s10: is there a bug open for your usecase? its 2 late to adress in rocky but we could likely adrss in early stien if we captured the request.
16:38:03 s10 sean-k-mooney: I didn't see bugs, related to real memory overhead problem. Only remember, that there was a thread in openstack-operators (?)
16:40:47 s10 https://bugs.launchpad.net/nova/+bug/1683858
16:40:48 openstack Launchpad bug 1683858 in OpenStack Compute (nova) "Allocation records do not contain overhead information" [Medium,Won't fix]
16:41:22 sean-k-mooney s10: based on the wording i assume that was suggesting this in placement hence the wont fix.
16:41:59 sean-k-mooney s10: i dont think placement is the right place to adress this but an in tree filter would be potentially reasonable.
16:44:57 mriedem stephenfin: we've got a runway slot opening so i'm going to throw the numa vswitch series back in ther
16:44:58 mriedem *there
16:44:59 mriedem just FYI
16:45:16 stephenfin Sweet. Should have that move thing done in a short bit
16:49:58 sean-k-mooney s10: i wont promise anything but i have added this usecase to my whiteboard of random TODOs. i might see if i can come up with somthing for stien if there is still interest.
16:50:56 AJaeger nova cores, could you review https://review.openstack.org/#/c/583023/ to move the job in-tree, please? that'S a followup to work for os-vif. stephenfin and sean-k-mooney, want to look at it again, please?
16:52:50 stephenfin AJaeger: ack
16:53:21 stephenfin lyarwood: I didn't get to https://review.openstack.org/#/c/562072/ yet :( It's first thing I'm doing tomorrow morning though, I promise
16:53:38 mriedem dansmith should probably look at those
16:53:45 mriedem given evacness
16:53:59 stephenfin mriedem: Suits me down to the ground
16:54:57 sean-k-mooney AJaeger: will nova beable to import the job from os-vif? i just kicked off an experimental job to make sure it all works
16:57:45 openstackgerrit Merged openstack/nova master: perform reshaper operations in single transaction https://review.openstack.org/582383
16:57:53 openstackgerrit Merged openstack/nova master: Refactor _heal_instances_in_cell https://review.openstack.org/577896
16:58:34 AJaeger sean-k-mooney: that'S how zuul was designed ;)
16:58:42 AJaeger sean-k-mooney: but yeah, I just have done that - thanks

Earlier   Later