Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-23
15:21:27 mriedem if the rebuild + new image doesn't fit on the current host
15:21:41 mriedem instance goes to error state and you have to fix it
15:22:19 mriedem bauzas: ping https://review.openstack.org/#/c/583715/
15:23:10 mriedem bauzas: also ping https://review.openstack.org/#/c/581813/ - that's the nova-status check for request spec migrations which means if we get that in rocky, we can drop the reqspec migrations and compat code in stein
15:23:15 mriedem which is something i desperatly want to do
15:23:21 sean-k-mooney mriedem: right i was assuming we would hit the retry fileter and try another host but i guess not. in my mind i was assuming we did the host affinity for rebuil via a weigher rather then filtering to just the current host
15:23:21 mriedem that and spellcheck
15:23:43 mriedem sean-k-mooney: we pass a hint to the scheduler telling it to filter on the existing instance host
15:24:42 sean-k-mooney mriedem: cool ill look into how that work more closely in the future. good to know
15:26:06 mriedem stephenfin: so i assume you might have 2 patches: 1 to add the physnet and tunneled plumbing to the nova.network.model.Network object and populate it during _build_network_info_model, and then another change to call some utils method in the various conductor spots to populate ReuqestSpec.network_metadata using instance.info_cache.network_info
15:26:34 stephenfin mriedem: That's exactly what I have, albeit rolled into one patch
15:26:38 mriedem stephenfin: btw, this change is doing the same call spots in conductor that you'll need https://review.openstack.org/#/c/583715/
15:26:57 stephenfin mriedem: However, are you sure about the "scheduler getting called" bit. Comments/logic in the code suggest otherwise mriedem: Are you sure. Your comments here say something else https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L901-L916
15:27:20 mriedem https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L920
15:27:25 stephenfin By rebuild I'm referring to the 'openstack server rebuild $NAME" command
15:27:56 mriedem note the "*unless* a new image # was specified"
15:28:09 stephenfin Riiiight, missed that
15:28:10 mriedem the rebuild_instance method in conductor manager is totally coupled to how the api code works
15:28:31 stephenfin So, what do I do in the "no new instance" case
15:28:35 stephenfin Happily carry on?
15:28:41 mriedem no request spec
15:28:41 mriedem ?
15:28:49 sean-k-mooney mriedem: even if the same image is passed we should proably reschdule as the image metadata could have changed
15:28:50 mriedem no new image?
15:29:15 mriedem stephenfin: if the image doesn't change, we don't call the scheduler
15:30:04 mriedem sean-k-mooney: yeah, maybe, which would be new logic in the api to detect that since we persist the original image properties on the instance system_metadata i think
15:30:44 stephenfin mriedem: Right, so I can't recalculate NUMA affinity for any network changes. That's OK, yeah?
15:30:50 dansmith mriedem: right, and I think we said that we shouldn't refuse a rebuild because an image has changed metadata since the user then has very little recourse
15:31:29 mriedem dansmith: i don't remember talking about image meta changing on the original image, but ok
15:31:34 dansmith I do
15:31:42 sean-k-mooney mriedem: hum well if we are using the cached copy it should safe but if we pull them down from glance then we might violate one of the image requests or not be able to supprot one of them
15:31:46 mriedem stephenfin: yes because we assume the network affinity on the same host for the same image hasn't changed
15:32:35 stephenfin mriedem: Sweet
15:32:43 stephenfin https://imgflip.com/i/2ejmw4
15:34:15 sean-k-mooney dansmith: mriedem do we also use the embeded flavor on rebuild? changes to image meta on rebuild is logically the same as changes to flavour extra specs so we should proably have the same behavior
15:34:26 dansmith we do
15:35:06 dansmith I'm not actually sure we do use the embedded _image_ properties for rebuild, but I'm saying I don't think that's a problem
15:35:16 mriedem https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3172
15:35:17 sean-k-mooney i think we dont
15:35:20 dansmith because changing image metadata such that people can't continue to be on a host they're currently on is pretty bad
15:35:25 dansmith and
15:35:44 dansmith running instances aren't going to move, so not allowing a rebuild is very surprising to the user who knows nothing of such changes
15:35:56 sean-k-mooney dansmith: it might be unitentionaly also image metadata and images can be set by enduesrs
15:36:37 dansmith yes, but they don't see the scheduling policy
15:39:11 sean-k-mooney dansmith: so ya i just checked locally we do use the updated image meta on rebuild
15:46:32 sean-k-mooney i guess this is one of those things that an operator should not do. i would feel more comfortable with always going to the schduler but i guess that would be a behavior change so it would need at least a microversion bump.
15:50:19 sean-k-mooney stephenfin: fyi, when cpus and memory are modeled in placement as the numa topoloy can be altered via the image metadata you will have to recalulate it on rebuild and check if the placement allocation still fufil the request even when we use the same image since we do not use a cached copy of the image metadata.
15:53:43 stephenfin sean-k-mooney: Sounds fair. Best tell jaypipes to get a move on then ;)
15:53:50 s10 Hi. Is there any possibility to somehow optimise host_state.update(), which runs on every instance scheduling? Specifically this: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L181
15:53:50 s10 Our tests show, that this step (loading list of instances to the host_state) takes ~10 seconds for deployments with thousands of instances and hundreds of hosts.
15:53:50 s10 So if we want to start 1000 new instances with 6 nova-scheduler process, without this step (if we commented it out), all of them end up being started tens of minutes faster.
15:55:18 sean-k-mooney stephenfin: well personally i think simply always going to the schduler on a rebuild would be a better change since the numatopolgy filter would assert the correct behavior
15:55:25 cdent s10: I'd recommend posting to the openstack-dev list about that. It's a topic that I've seen come up from multiple peope lately, and they're not all here rightnow.
15:56:45 mriedem s10: yeah it's a known issue, i have a semi-related patch, sec
15:57:11 mriedem s10: https://review.openstack.org/#/c/569247/
15:57:16 s10 I see https://bugs.launchpad.net/nova/+bug/1737465
15:57:17 openstack Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:57:19 s10 We use Pike
15:57:28 sean-k-mooney mriedem: s10 is this not required to make sure we are not using statle hostsate info?
15:57:28 mriedem s10: yes same related bug
15:58:05 mriedem sean-k-mooney: yes it is
15:58:08 mriedem for the affinity filters
15:58:49 mriedem s10: for starters, i've been hoping someone with a large enough deployment can tell me if this makes an improvement https://review.openstack.org/#/c/569247/ because CERN was reporting the same issue
15:59:01 mriedem s10: if you can test that out we could move it forward
15:59:02 sean-k-mooney mriedem: so commenting it out while faster would be incorrect unless we updated the afinit filters to get the updated info but that would break multicreate
15:59:12 mriedem there are more ways to optimize these calls
15:59:53 s10 Yes, we can't really comment them out, because we rely on the anti-affinity filter for the octavia load-balancers.
16:00:26 mriedem the information is also somewhat cached and the computes rpc cast to the scheduler whenever an instance is created/deleted/moved assuming you have that setup
16:00:32 s10 And we use host_state.instances in our custom RealRamFilter...
16:00:40 mriedem that's the track_instance_changes config option,
16:01:19 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes
16:01:57 mriedem if you're doing split MQ though then the comptues can't reach the scheduler
16:01:59 mriedem and that doesn't help
16:02:32 mriedem we also know that we can optimize here https://review.openstack.org/#/c/571928/2/nova/scheduler/host_manager.py with some db join magic
16:02:37 sean-k-mooney mriedem: looking at https://review.openstack.org/#/c/569247 the optimisationis storing a list of uuids per host rather then the full instance objects correct? i can see how that would work for the in tree affintiy filters and the num instance weigher
16:03:01 mriedem sean-k-mooney: well it's also the db query
16:03:19 mriedem rather than query the full instance objects and such per host, just get the uuids
16:04:06 mriedem s10: if you can test out https://review.openstack.org/#/c/569247/ and report back that would be very helpful in moving this forward
16:04:38 sean-k-mooney mriedem: sure that makes sense.
16:04:51 cdent mriedem, s10: whatever you work out here, it would still be great to see some kind of summary to the mailing list
16:05:08 s10 mriedem: Ok, I will test it
16:05:31 s10 mriedem: could this change be cherry-picked without rework to Pike?
16:06:52 sean-k-mooney s10 it looks pretty trivial in terms of code change. this would not qualify for backport however as it is not fixing a bug. it is a performance enhancement
16:08:45 mriedem s10: probably, there was another change merged in the series before it that you'll need https://review.openstack.org/#/q/topic:bug/1737465+(status:open+OR+status:merged)
16:08:55 mriedem https://review.openstack.org/#/q/Iccefbfdfa578515a004ef6ac718bac1a49d5c5fd
16:09:00 mriedem you'll need that in pike first
16:09:18 mriedem and then https://review.openstack.org/#/c/569247/
16:15:15 s10 Our out-of-tree filter rely on host_state.instances.values(), we need values of the instance.memory_mb. Will we be hit with a lazy-load penalty in this case, with https://review.openstack.org/#/c/570083/?
16:16:53 dansmith yup
16:18:40 sean-k-mooney s10: what does your real ram filter actully check vs what placement will do?
16:19:12 sean-k-mooney s10: are you trying to disable oversubscription?
16:20:19 s10 sean-k-mooney: actual usage of the ram on compute nodes. Every QEMU process produces some memory overhead (up to 500-1000mb in some cases, depends on disk usage and rbd cached, I believe). Placement doesn't know anything about it and can schedule instance to the host, where real size of free ram is less than reserved_memory_mb.
16:20:51 mriedem yeah we don't track overhead in placement, that's a known thing
16:21:27 mriedem and we've said for now, buffer that with reserved_host_memory_mb
16:21:40 sean-k-mooney s10: how are you getting the real host memory usage?
16:23:02 mriedem s10: also fyi http://lists.openstack.org/pipermail/openstack-dev/2018-May/130624.html
16:23:04 s10 We patched nova/scheduler/host_manager.py, added self.memory_mb_used = compute.memory_mb_used
16:23:21 mriedem s10: then why would you need host_state.instances.values()?
16:23:54 s10 Because without it we can't compute reserved_memory_mb, it's in config...
16:24:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715

Earlier   Later