Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-23
15:16:14 mriedem stephenfin: depends, what does the RUN_ON_REBUILD flag say for NUMATopologyFilter?
15:16:21 mriedem sean-k-mooney: not in all cases, and not for all filters
15:16:32 mriedem stephenfin: and to be clear, you're talking about rebuild, not evacuate, yes?
15:16:46 mriedem RUN_ON_REBUILD = True
15:16:46 mriedem """Filter on requested NUMA topology."""
15:16:46 mriedem stephenfin: class NUMATopologyFilter(filters.BaseHostFilter):
15:17:03 stephenfin mriedem: It's True and yes
15:17:18 mriedem stephenfin: so then you need to put the stuff in the request spec if we're going through the scheduler during rebuild b/c the image changes
15:17:29 mriedem because image_props.get('hw_cpu_policy')] and image_props.get('hw_cpu_thread_policy')]
15:17:34 mriedem if the image changes, the numa stuff can change
15:17:51 mriedem stephenfin: rebuild isn't a move, but we have to validate the new image
15:18:17 mriedem *validate the new image still works on the current host
15:19:12 sean-k-mooney mriedem: a rebuilt instance could land on a different host then the instace was on previously though right. its effectivly a new boot with the same port/volumes as the original instance
15:19:34 mriedem sean-k-mooney: no
15:19:44 mriedem sean-k-mooney: are you talking about evacuate?
15:20:16 sean-k-mooney no i taught if we rebuilt and the previous host was nolonger valid we would schdule to another host that was
15:20:23 sean-k-mooney i gues we just fail to rebuild
15:21:19 mriedem you get a novalidhost
15:21:27 mriedem if the rebuild + new image doesn't fit on the current host
15:21:41 mriedem instance goes to error state and you have to fix it
15:22:19 mriedem bauzas: ping https://review.openstack.org/#/c/583715/
15:23:10 mriedem bauzas: also ping https://review.openstack.org/#/c/581813/ - that's the nova-status check for request spec migrations which means if we get that in rocky, we can drop the reqspec migrations and compat code in stein
15:23:15 mriedem which is something i desperatly want to do
15:23:21 mriedem that and spellcheck
15:23:21 sean-k-mooney mriedem: right i was assuming we would hit the retry fileter and try another host but i guess not. in my mind i was assuming we did the host affinity for rebuil via a weigher rather then filtering to just the current host
15:23:43 mriedem sean-k-mooney: we pass a hint to the scheduler telling it to filter on the existing instance host
15:24:42 sean-k-mooney mriedem: cool ill look into how that work more closely in the future. good to know
15:26:06 mriedem stephenfin: so i assume you might have 2 patches: 1 to add the physnet and tunneled plumbing to the nova.network.model.Network object and populate it during _build_network_info_model, and then another change to call some utils method in the various conductor spots to populate ReuqestSpec.network_metadata using instance.info_cache.network_info
15:26:34 stephenfin mriedem: That's exactly what I have, albeit rolled into one patch
15:26:38 mriedem stephenfin: btw, this change is doing the same call spots in conductor that you'll need https://review.openstack.org/#/c/583715/
15:26:57 stephenfin mriedem: However, are you sure about the "scheduler getting called" bit. Comments/logic in the code suggest otherwise mriedem: Are you sure. Your comments here say something else https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L901-L916
15:27:20 mriedem https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L920
15:27:25 stephenfin By rebuild I'm referring to the 'openstack server rebuild $NAME" command
15:27:56 mriedem note the "*unless* a new image # was specified"
15:28:09 stephenfin Riiiight, missed that
15:28:10 mriedem the rebuild_instance method in conductor manager is totally coupled to how the api code works
15:28:31 stephenfin So, what do I do in the "no new instance" case
15:28:35 stephenfin Happily carry on?
15:28:41 mriedem ?
15:28:41 mriedem no request spec
15:28:49 sean-k-mooney mriedem: even if the same image is passed we should proably reschdule as the image metadata could have changed
15:28:50 mriedem no new image?
15:29:15 mriedem stephenfin: if the image doesn't change, we don't call the scheduler
15:30:04 mriedem sean-k-mooney: yeah, maybe, which would be new logic in the api to detect that since we persist the original image properties on the instance system_metadata i think
15:30:44 stephenfin mriedem: Right, so I can't recalculate NUMA affinity for any network changes. That's OK, yeah?
15:30:50 dansmith mriedem: right, and I think we said that we shouldn't refuse a rebuild because an image has changed metadata since the user then has very little recourse
15:31:29 mriedem dansmith: i don't remember talking about image meta changing on the original image, but ok
15:31:34 dansmith I do
15:31:42 sean-k-mooney mriedem: hum well if we are using the cached copy it should safe but if we pull them down from glance then we might violate one of the image requests or not be able to supprot one of them
15:31:46 mriedem stephenfin: yes because we assume the network affinity on the same host for the same image hasn't changed
15:32:35 stephenfin mriedem: Sweet
15:32:43 stephenfin https://imgflip.com/i/2ejmw4
15:34:15 sean-k-mooney dansmith: mriedem do we also use the embeded flavor on rebuild? changes to image meta on rebuild is logically the same as changes to flavour extra specs so we should proably have the same behavior
15:34:26 dansmith we do
15:35:06 dansmith I'm not actually sure we do use the embedded _image_ properties for rebuild, but I'm saying I don't think that's a problem
15:35:16 mriedem https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3172
15:35:17 sean-k-mooney i think we dont
15:35:20 dansmith because changing image metadata such that people can't continue to be on a host they're currently on is pretty bad
15:35:25 dansmith and
15:35:44 dansmith running instances aren't going to move, so not allowing a rebuild is very surprising to the user who knows nothing of such changes
15:35:56 sean-k-mooney dansmith: it might be unitentionaly also image metadata and images can be set by enduesrs
15:36:37 dansmith yes, but they don't see the scheduling policy
15:39:11 sean-k-mooney dansmith: so ya i just checked locally we do use the updated image meta on rebuild
15:46:32 sean-k-mooney i guess this is one of those things that an operator should not do. i would feel more comfortable with always going to the schduler but i guess that would be a behavior change so it would need at least a microversion bump.
15:50:19 sean-k-mooney stephenfin: fyi, when cpus and memory are modeled in placement as the numa topoloy can be altered via the image metadata you will have to recalulate it on rebuild and check if the placement allocation still fufil the request even when we use the same image since we do not use a cached copy of the image metadata.
15:53:43 stephenfin sean-k-mooney: Sounds fair. Best tell jaypipes to get a move on then ;)
15:53:50 s10 So if we want to start 1000 new instances with 6 nova-scheduler process, without this step (if we commented it out), all of them end up being started tens of minutes faster.
15:53:50 s10 Our tests show, that this step (loading list of instances to the host_state) takes ~10 seconds for deployments with thousands of instances and hundreds of hosts.
15:53:50 s10 Hi. Is there any possibility to somehow optimise host_state.update(), which runs on every instance scheduling? Specifically this: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L181
15:55:18 sean-k-mooney stephenfin: well personally i think simply always going to the schduler on a rebuild would be a better change since the numatopolgy filter would assert the correct behavior
15:55:25 cdent s10: I'd recommend posting to the openstack-dev list about that. It's a topic that I've seen come up from multiple peope lately, and they're not all here rightnow.
15:56:45 mriedem s10: yeah it's a known issue, i have a semi-related patch, sec
15:57:11 mriedem s10: https://review.openstack.org/#/c/569247/
15:57:16 s10 I see https://bugs.launchpad.net/nova/+bug/1737465
15:57:17 openstack Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:57:19 s10 We use Pike
15:57:28 mriedem s10: yes same related bug
15:57:28 sean-k-mooney mriedem: s10 is this not required to make sure we are not using statle hostsate info?
15:58:05 mriedem sean-k-mooney: yes it is
15:58:08 mriedem for the affinity filters
15:58:49 mriedem s10: for starters, i've been hoping someone with a large enough deployment can tell me if this makes an improvement https://review.openstack.org/#/c/569247/ because CERN was reporting the same issue
15:59:01 mriedem s10: if you can test that out we could move it forward
15:59:02 sean-k-mooney mriedem: so commenting it out while faster would be incorrect unless we updated the afinit filters to get the updated info but that would break multicreate
15:59:12 mriedem there are more ways to optimize these calls
15:59:53 s10 Yes, we can't really comment them out, because we rely on the anti-affinity filter for the octavia load-balancers.
16:00:26 mriedem the information is also somewhat cached and the computes rpc cast to the scheduler whenever an instance is created/deleted/moved assuming you have that setup
16:00:32 s10 And we use host_state.instances in our custom RealRamFilter...
16:00:40 mriedem that's the track_instance_changes config option,
16:01:19 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes
16:01:57 mriedem if you're doing split MQ though then the comptues can't reach the scheduler
16:01:59 mriedem and that doesn't help
16:02:32 mriedem we also know that we can optimize here https://review.openstack.org/#/c/571928/2/nova/scheduler/host_manager.py with some db join magic
16:02:37 sean-k-mooney mriedem: looking at https://review.openstack.org/#/c/569247 the optimisationis storing a list of uuids per host rather then the full instance objects correct? i can see how that would work for the in tree affintiy filters and the num instance weigher
16:03:01 mriedem sean-k-mooney: well it's also the db query
16:03:19 mriedem rather than query the full instance objects and such per host, just get the uuids
16:04:06 mriedem s10: if you can test out https://review.openstack.org/#/c/569247/ and report back that would be very helpful in moving this forward
16:04:38 sean-k-mooney mriedem: sure that makes sense.
16:04:51 cdent mriedem, s10: whatever you work out here, it would still be great to see some kind of summary to the mailing list
16:05:08 s10 mriedem: Ok, I will test it
16:05:31 s10 mriedem: could this change be cherry-picked without rework to Pike?

Earlier   Later