Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-23
15:28:41 mriedem no request spec
15:28:41 mriedem ?
15:28:49 sean-k-mooney mriedem: even if the same image is passed we should proably reschdule as the image metadata could have changed
15:28:50 mriedem no new image?
15:29:15 mriedem stephenfin: if the image doesn't change, we don't call the scheduler
15:30:04 mriedem sean-k-mooney: yeah, maybe, which would be new logic in the api to detect that since we persist the original image properties on the instance system_metadata i think
15:30:44 stephenfin mriedem: Right, so I can't recalculate NUMA affinity for any network changes. That's OK, yeah?
15:30:50 dansmith mriedem: right, and I think we said that we shouldn't refuse a rebuild because an image has changed metadata since the user then has very little recourse
15:31:29 mriedem dansmith: i don't remember talking about image meta changing on the original image, but ok
15:31:34 dansmith I do
15:31:42 sean-k-mooney mriedem: hum well if we are using the cached copy it should safe but if we pull them down from glance then we might violate one of the image requests or not be able to supprot one of them
15:31:46 mriedem stephenfin: yes because we assume the network affinity on the same host for the same image hasn't changed
15:32:35 stephenfin mriedem: Sweet
15:32:43 stephenfin https://imgflip.com/i/2ejmw4
15:34:15 sean-k-mooney dansmith: mriedem do we also use the embeded flavor on rebuild? changes to image meta on rebuild is logically the same as changes to flavour extra specs so we should proably have the same behavior
15:34:26 dansmith we do
15:35:06 dansmith I'm not actually sure we do use the embedded _image_ properties for rebuild, but I'm saying I don't think that's a problem
15:35:16 mriedem https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3172
15:35:17 sean-k-mooney i think we dont
15:35:20 dansmith because changing image metadata such that people can't continue to be on a host they're currently on is pretty bad
15:35:25 dansmith and
15:35:44 dansmith running instances aren't going to move, so not allowing a rebuild is very surprising to the user who knows nothing of such changes
15:35:56 sean-k-mooney dansmith: it might be unitentionaly also image metadata and images can be set by enduesrs
15:36:37 dansmith yes, but they don't see the scheduling policy
15:39:11 sean-k-mooney dansmith: so ya i just checked locally we do use the updated image meta on rebuild
15:46:32 sean-k-mooney i guess this is one of those things that an operator should not do. i would feel more comfortable with always going to the schduler but i guess that would be a behavior change so it would need at least a microversion bump.
15:50:19 sean-k-mooney stephenfin: fyi, when cpus and memory are modeled in placement as the numa topoloy can be altered via the image metadata you will have to recalulate it on rebuild and check if the placement allocation still fufil the request even when we use the same image since we do not use a cached copy of the image metadata.
15:53:43 stephenfin sean-k-mooney: Sounds fair. Best tell jaypipes to get a move on then ;)
15:53:50 s10 Hi. Is there any possibility to somehow optimise host_state.update(), which runs on every instance scheduling? Specifically this: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L181
15:53:50 s10 Our tests show, that this step (loading list of instances to the host_state) takes ~10 seconds for deployments with thousands of instances and hundreds of hosts.
15:53:50 s10 So if we want to start 1000 new instances with 6 nova-scheduler process, without this step (if we commented it out), all of them end up being started tens of minutes faster.
15:55:18 sean-k-mooney stephenfin: well personally i think simply always going to the schduler on a rebuild would be a better change since the numatopolgy filter would assert the correct behavior
15:55:25 cdent s10: I'd recommend posting to the openstack-dev list about that. It's a topic that I've seen come up from multiple peope lately, and they're not all here rightnow.
15:56:45 mriedem s10: yeah it's a known issue, i have a semi-related patch, sec
15:57:11 mriedem s10: https://review.openstack.org/#/c/569247/
15:57:16 s10 I see https://bugs.launchpad.net/nova/+bug/1737465
15:57:17 openstack Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:57:19 s10 We use Pike
15:57:28 sean-k-mooney mriedem: s10 is this not required to make sure we are not using statle hostsate info?
15:57:28 mriedem s10: yes same related bug
15:58:05 mriedem sean-k-mooney: yes it is
15:58:08 mriedem for the affinity filters
15:58:49 mriedem s10: for starters, i've been hoping someone with a large enough deployment can tell me if this makes an improvement https://review.openstack.org/#/c/569247/ because CERN was reporting the same issue
15:59:01 mriedem s10: if you can test that out we could move it forward
15:59:02 sean-k-mooney mriedem: so commenting it out while faster would be incorrect unless we updated the afinit filters to get the updated info but that would break multicreate
15:59:12 mriedem there are more ways to optimize these calls
15:59:53 s10 Yes, we can't really comment them out, because we rely on the anti-affinity filter for the octavia load-balancers.
16:00:26 mriedem the information is also somewhat cached and the computes rpc cast to the scheduler whenever an instance is created/deleted/moved assuming you have that setup
16:00:32 s10 And we use host_state.instances in our custom RealRamFilter...
16:00:40 mriedem that's the track_instance_changes config option,
16:01:19 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes
16:01:57 mriedem if you're doing split MQ though then the comptues can't reach the scheduler
16:01:59 mriedem and that doesn't help
16:02:32 mriedem we also know that we can optimize here https://review.openstack.org/#/c/571928/2/nova/scheduler/host_manager.py with some db join magic
16:02:37 sean-k-mooney mriedem: looking at https://review.openstack.org/#/c/569247 the optimisationis storing a list of uuids per host rather then the full instance objects correct? i can see how that would work for the in tree affintiy filters and the num instance weigher
16:03:01 mriedem sean-k-mooney: well it's also the db query
16:03:19 mriedem rather than query the full instance objects and such per host, just get the uuids
16:04:06 mriedem s10: if you can test out https://review.openstack.org/#/c/569247/ and report back that would be very helpful in moving this forward
16:04:38 sean-k-mooney mriedem: sure that makes sense.
16:04:51 cdent mriedem, s10: whatever you work out here, it would still be great to see some kind of summary to the mailing list
16:05:08 s10 mriedem: Ok, I will test it
16:05:31 s10 mriedem: could this change be cherry-picked without rework to Pike?
16:06:52 sean-k-mooney s10 it looks pretty trivial in terms of code change. this would not qualify for backport however as it is not fixing a bug. it is a performance enhancement
16:08:45 mriedem s10: probably, there was another change merged in the series before it that you'll need https://review.openstack.org/#/q/topic:bug/1737465+(status:open+OR+status:merged)
16:08:55 mriedem https://review.openstack.org/#/q/Iccefbfdfa578515a004ef6ac718bac1a49d5c5fd
16:09:00 mriedem you'll need that in pike first
16:09:18 mriedem and then https://review.openstack.org/#/c/569247/
16:15:15 s10 Our out-of-tree filter rely on host_state.instances.values(), we need values of the instance.memory_mb. Will we be hit with a lazy-load penalty in this case, with https://review.openstack.org/#/c/570083/?
16:16:53 dansmith yup
16:18:40 sean-k-mooney s10: what does your real ram filter actully check vs what placement will do?
16:19:12 sean-k-mooney s10: are you trying to disable oversubscription?
16:20:19 s10 sean-k-mooney: actual usage of the ram on compute nodes. Every QEMU process produces some memory overhead (up to 500-1000mb in some cases, depends on disk usage and rbd cached, I believe). Placement doesn't know anything about it and can schedule instance to the host, where real size of free ram is less than reserved_memory_mb.
16:20:51 mriedem yeah we don't track overhead in placement, that's a known thing
16:21:27 mriedem and we've said for now, buffer that with reserved_host_memory_mb
16:21:40 sean-k-mooney s10: how are you getting the real host memory usage?
16:23:02 mriedem s10: also fyi http://lists.openstack.org/pipermail/openstack-dev/2018-May/130624.html
16:23:04 s10 We patched nova/scheduler/host_manager.py, added self.memory_mb_used = compute.memory_mb_used
16:23:21 mriedem s10: then why would you need host_state.instances.values()?
16:23:54 s10 Because without it we can't compute reserved_memory_mb, it's in config...
16:24:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
16:24:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
16:24:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931
16:24:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Cache is_bfv check in ResourceTracker https://review.openstack.org/584962
16:24:26 s10 Maybe we should pass reserved_memory_mb same way as we pass memory_mb_used...
16:25:01 sean-k-mooney s10: it would make sense to me to just embed the reserved memory in the host state object
16:25:24 mriedem to be clear, CONF.reserved_host_memory_mb is used to reserve capacity for MEMORY_MB in the compute node resource provider's inventory in placement
16:25:35 mriedem which is used to filter hosts before they even get to your filter
16:25:58 mriedem if you need to reserve space on the host for qemu/rbd, you should increase reserved_host_memory_mb per host
16:26:32 s10 But it is still calculated per compute, based on the config in nova-compute, right?
16:26:51 mriedem the config is read in compute yes
16:27:17 mriedem compute node memory_mb_used is later adjusted in the resource tracker based on the instance flavor
16:27:20 mriedem per instance on that host
16:27:43 sean-k-mooney mriedem: ture but s10's filter would catch the case where they did not reserve enough and the free ram is actully less then the reserved
16:27:52 mriedem but that information is also available via placement and is used to calculate the allocation candidates that the scheduler gets before we even hit the filters
16:28:28 mriedem sean-k-mooney: you might need to be conservative then in the reserved memory_mb
16:28:31 sean-k-mooney mriedem: placement only has allocations based on the flavor request. not the actul memory used by the instance
16:28:32 mriedem if this is a persistent problem
16:29:24 mriedem Kevin_Zheng: fyi https://review.openstack.org/#/c/584962/
16:29:32 s10 We used reserved_host_memory_mb for that purpose, but it doesn't work well for us, some instances produces memory overhead over time, and in this case we might end up reserving 50% of host memory
16:29:55 sean-k-mooney mriedem: im just wondering could we symplify this by comparing free_ram_mb > reseved in the fileter if we added reserved to the hostsate object

Earlier   Later