Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
16:18:07 sean-k-mooney wasnt that the suggeted default in the spec
16:18:08 bauzas sean-k-mooney: correct
16:18:09 stephenfin It's dumb but running tox cripples Bluejeans and any day that I've loads of meetings (like today) means I can't kick off anything locally that's going to run in the background
16:18:20 bauzas sean-k-mooney: nope, the default was 'nothing changes'
16:18:39 dansmith bauzas: yes I think that'd be ideal
16:18:41 sean-k-mooney oh well prefered was just going to be impleneted by a weigher right
16:19:02 bauzas dansmith: okay, I'll -W the spec and work on a new PS
16:19:07 dansmith cool
16:19:09 sean-k-mooney we will prefer host that can provide numa affinty but not filter out any
16:19:17 sean-k-mooney like the pci weigher
16:19:20 mnaser mriedem: SELECT instances.uuid, COUNT(virtual_interfaces.uuid) FROM instances LEFT JOIN virtual_interfaces ON virtual_interfaces.instance_uuid = instances.uuid WHERE instances.deleted=0 GROUP BY instances.uuid;
16:19:26 mnaser shows 1/2 but no 0's
16:19:38 bauzas sean-k-mooney: yup, just a way to see whether we can have *some affinity*
16:19:51 bauzas if no affinity possible, fine
16:19:55 mnaser so I guess I can cherry-pick that code and run it once..
16:20:07 sean-k-mooney yep
16:20:16 sean-k-mooney that is why i assumed it would be the default
16:20:32 sean-k-mooney becaue its best effort
16:20:32 mriedem mnaser: cool, let me know how it goes
16:20:36 bauzas sean-k-mooney: the concern wasn't really the default
16:20:39 bauzas value*
16:20:56 bauzas sean-k-mooney: the concern is more whether we want to introduce more knobs, and 'required' needed those
16:21:24 openstackgerrit Mohammed Naser proposed openstack/nova stable/rocky: Force refresh instance info_cache during heal https://review.openstack.org/653040
16:21:27 bauzas anyway, I'll just write a new revision, and people could chime on it
16:22:16 stephenfin bauzas, sean-k-mooney: RE: the cpu-resources discussion above, it's not nova-compute that makes the call to placement
16:22:46 bauzas stephenfin: which call are we talking about ?
16:23:04 stephenfin the claim
16:23:44 bauzas yup, I don't disagree
16:23:52 bauzas but then I don't get your point
16:24:05 mnaser at least it applies cleanly
16:24:24 bauzas my upgrade concerns are about when and how we should transform inventories
16:24:36 bauzas (and move allocations accordingly)
16:25:08 stephenfin bauzas: We'd either have to request a certain amount of PCPU resources, which couldn't be fulfilled since we're not reporting any inventory (because the config values haven't been set)
16:25:35 stephenfin Or we'd have to keep requesting VCPU resources for everything, which borks the whole idea
16:25:44 bauzas I tend for the latter
16:26:22 bauzas if people start asking PCPU resources, they necessarly have to be fulfilled by hosts ready to accept them, I don't disagree
16:27:05 bauzas but then, Train is necessarly a mitigation release
16:27:10 bauzas because of computes
16:27:21 bauzas we did had the same problem with rolling upgrades
16:27:42 bauzas you can't really make use of a feature unless all computes are up to date in general, in particular when it comes to resources usage
16:29:30 stephenfin bauzas: hmm, that removes our ability to transform 'hw:cpu_policy=dedicated' under the hood though
16:29:32 bauzas that doesn't mean operators can't use PCPU requests with Train
16:29:47 stephenfin and if a CPU is part of the PCPU pool, it can't be part of the VCPU poool
16:29:53 bauzas but they absolutely need to converge all their inventories *before* they use the PCPU request
16:31:15 stephenfin without the transform in place, any existing instances can't be migrated (they wouldn't be using PCPUs but rather the legacy 'hw:cpu_policy=dedicated' extra spec) plus flavours and images would not be requesting the correct stuff
16:32:01 stephenfin The point is I think we need that rewriting in place, and if we need that then we need some initial PCPU inventories in place as soon as the upgrade is in place, otherwise the ability to do move existing instances or create new ones is gone :(
16:32:22 stephenfin This could really do with a call, I think
16:34:47 sean-k-mooney probably
16:36:48 bauzas stephenfin: I have to leave in a few, but I'll summarize my thoughts
16:37:19 bauzas we could just leave existing options as they are and leave inventories be VCPU
16:37:23 bauzas but
16:38:15 bauzas once operator sets config options (and then we can discuss on this specific trigger), then we do a reshape for this host and split VCPU inventory into VCPU and PCPU inventories
16:38:23 bauzas and we accordingly move allocations
16:38:51 bauzas with the slight detail that we verify resources *before* doing the reshape so we can raise an exception
16:38:59 bauzas at compute startup
16:39:55 bauzas for requests, we somehow need to make sure that we transform requests by using PCPU based on a specific point in time
16:40:00 bauzas either by providing a flag
16:40:11 bauzas or by verifying the PCPU inventories
16:40:42 bauzas stephenfin: thoughts on that ?
16:40:58 bauzas others: too
16:43:52 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Re-propose stable device rescue for Train https://review.openstack.org/651151
16:45:40 stephenfin bauzas: If that happens though, then everything has to be done at once.
16:46:03 bauzas stephenfin: that happens what ?
16:46:15 bauzas asking for PCPU ?
16:46:36 stephenfin if we wait until some point in time to start reporting PCPU
16:47:37 stephenfin so if we say the operator setting this configuration option is that point in time, then we must also ensure the operator also twists the knob that says "transform all my legacy extra specs/image meta to PCPU requests"
16:48:10 efried We said we weren't going to allow reshapes except on upgrade boundaries. When "we" said that, I was a dissenting vote. There are a number of specs we're discussing in the current cycle where that is going to need to be re-evaluated.
16:48:28 stephenfin (that knob has to exist because there is no a point in time where we can have no PCPU resources so therefore we can't always transform)
16:49:25 efried We must either allow reshapes "any time" (conceivably with a compute service restart), or stick to the above guns and require a host to be cleared out before configuration tweaks are done. The latter means we're not reshaping, just shuffling inventory around in a regular update_provider_tree code path, because we don't have allocations to dork with.
16:50:10 bauzas efried: we said earlier that the latter is highly terrible for ops
16:50:23 bauzas earlier being last week
16:50:39 bauzas efried: so that's why I'm considering a reshape at compute startup
16:50:49 bauzas based on config flags modification
16:50:58 dansmith efried: what you mean is make reshapes obligatory and not contingent on a config value or something right?
16:50:59 efried I'm in favor of that.
16:51:08 bauzas and the request knob be ops-driven, I like this
16:51:29 dansmith efried: because I think the two are not mutually exclusive, if the shape of the reshape would be defined by something in config (i.e. how many cpus are dedicated vs. shared)
16:51:43 efried dansmith: I mean allow reshapes to happen when they need to happen, rather than restricting them to upgrade boundaries. That's what bauzas is talking about as well.
16:52:05 efried Yes, dansmith and bauzas we still need to fail the reshape if it entails moving allocations in an impossible way.
16:52:05 dansmith efried: right I know, but I think there's subtlety here
16:52:19 bauzas efried: to be fair, VGPU reshapes are done on compute startup already, not upgrade: )
16:52:40 bauzas of course, it will in theory run once, after upgrading
16:52:49 dansmith efried: we have to maintain config compatibility, but if if we don't have information in the N-1 config to do the reshape, then we have to be able to punt the reshape (triggered by an upgrade) until after the config is updated
16:52:53 efried like if you suddenly specify your PCPU pinset to be empty, but have instances running with dedicated CPUs, that's a fail.
16:53:05 dansmith and, I agree that if you have to change how many pcpus are dedicated, we have to reshape again
16:53:16 efried I think we're on the same page
16:53:20 bauzas efried: that's why I proposed to check the allocations and inventories *before* providing the reshape
16:53:51 bauzas if the operator changes the config, but placement says "sorry but you can't", then the compute will fail to restart
16:54:22 dansmith efried: I think the thing I don't want, which I expressed as "only at upgrade time" is something like we reshape every time we restart compute because we decide we can arrange things better, or some state in the db has changed, but reshape due to a config/structural change makes sense
16:54:29 bauzas if the operator changes the config, and placement resources are okay, then the driver returns a ReshapeNeeded
16:54:35 efried dansmith: wfm
16:54:39 bauzas and then the new inventories and allocations
16:54:57 bauzas okay, so dansmith, efried and I are on the same page
16:55:25 bauzas there is one last concern from stephenfin about the config knob for the PCPU request
16:55:44 efried So e.g. in the PCPU spec, we're inferring the counts and pinsets of VCPUs vs PCPUs based on existing conf options.
16:55:49 bauzas but I think it's okay to make the request transformation to be "config-driven"
16:56:15 efried So the operator needs to change to the new config in such a way that it *exactly* matches what we inferred, right?
16:56:27 bauzas so, the operator would basically tell when he's okay to count PCPUs (ie. probably after the whole nodes config change)
16:56:59 efried Otherwise we don't just need a reshape (move allocations) - we would also possibly need to re-pin guests to different physical processors and such.
16:57:06 bauzas efried: no, I'm saying that existing config will report VCPUs anyway

Earlier   Later