Posted	Nick	Remark
#openstack-nova - 2020-04-06
11:15:50	sean-k-mooney	well regardign fancy new keys. i would rather just define a custom: name spaces that should be used for user defined extra_specs and declare all other namespaced keys as owned by nova
11:16:16	sean-k-mooney	and if a new intree filter wants to add keys it gets its own namespace
11:16:44	sean-k-mooney	if our of tree filter need to define keys the do it via custom:
11:17:11	sean-k-mooney	ideally with a prefix e.g. custom:myfilter_mykey
11:30:14	sean-k-mooney	ierdem: i assume you have oversubsrtion enabled on your cloud?
11:30:25	sean-k-mooney	ierdem: is the host over subsibed
11:31:53	sean-k-mooney	ierdem: it should like you either have someing in the vm that is broken/maliusly consumeing cpu, the vm workload need more vcpus or the host is over subsribed
11:32:54	sean-k-mooney	ierdem: simply seeing high cpu usage in the guest is not an indication that something is wrong form a nova point of view
11:33:36	ierdem	hmm, how can i check if the host is oversubscribed?
11:34:32	ierdem	by the way, before resizing vCPU count, it was working fine
11:35:00	kplant	a resize can move the instance to another hypervisor
11:36:41	kplant	if you just do a 'nova hypervisor-show <uuid>'
11:36:49	kplant	you can check vcpus vs vcpus_used
11:36:58	kplant	if vcpus_used > vcpus; you're over subscribed
11:37:38	ierdem	ok, thanks i will try and return to you
11:38:12	kplant	also vcpus_used == vcpus is a bad idea, unless nova knows about vcpus you're reserving
11:38:53	sean-k-mooney	yes by default resize to same host is disabled so it normally does a move and will only stay on the same host if the weigher consider it to be the best host
11:39:13	sean-k-mooney	if you enable same host resize in the config
11:40:28	sean-k-mooney	kplant: if you use vcpu_pin_set or (cpu_shared_set and cpu_dedicated_set) then yes vcpu_used == vcpu shared is fine
11:40:59	kplant	yeah, if not ideal :-)
11:41:18	sean-k-mooney	ideally you should sue the *_sets for doing host reservation instead fo the reserved_host_cpus
11:41:25	sean-k-mooney	*use
11:42:18	sean-k-mooney	if you use the set you can choose which cpus to reserve and then you can use systemd to run the host process only on thos cores
11:42:34	kplant	you can go further and use the bootloader
11:42:56	sean-k-mooney	using isolcpus? is so you should really avoid that
11:43:07	kplant	my only complaint with *_sets is you can't have open ended ranges
11:43:13	kplant	like "3-"
11:43:24	kplant	makes it easier to deal with hosts with, not vastly, different configs
11:43:40	sean-k-mooney	you can do negations
11:43:51	sean-k-mooney	"^0-2"
11:43:59	kplant	ooo
11:44:10	sean-k-mooney	i think negated ranges works
11:44:18	sean-k-mooney	you can negate indivcual cores
11:45:01	sean-k-mooney	we prably could add open ended ranges too you could always file a bug/blueprint
11:45:27	kplant	i think negated ranges provide the same result
11:45:38	kplant	nice
11:46:03	sean-k-mooney	ill triple check the code to make sure that works
11:46:16	sean-k-mooney	we support it for cpu_realtime_mask
11:46:27	sean-k-mooney	but i think its supported in teh *_sets too
11:47:59	sean-k-mooney	kplant: this is the code and it inclde the negation example for 1 core
11:48:20	openstackgerrit	Balazs Gibizer proposed openstack/nova-specs master: template: consider openstack client besides novaclient https://review.opendev.org/717722
11:48:46	sean-k-mooney	kplant: yep negated ranges should work https://github.com/openstack/nova/blob/31aa4a6d7f0b8c301b093ad176ee2b44b5d3cec8/nova/virt/hardware.py#L134-L140
11:50:05	sean-k-mooney	sorry miss read that
11:50:53	sean-k-mooney	the negation only works with 1 cpu
11:51:20	kplant	looking at 113 -> 116
11:51:26	kplant	i think you might have been right originally
11:51:36	kplant	it tests for ^str == '^'
11:51:39	kplant	then proceeds as normal
11:51:47	kplant	the next try/catch is for a range
11:52:38	kplant	oof, that's ignoring what you highlighted though
11:53:32	sean-k-mooney	https://github.com/openstack/nova/blob/824bc358c25564ed6603d8f94587abd8902fe5af/nova/tests/unit/virt/test_hardware.py#L132-L133
11:53:38	openstackgerrit	Balazs Gibizer proposed openstack/nova master: doc: require openstack client change for every new API microversion https://review.opendev.org/717727
11:53:41	kplant	that's still not a show stopper
11:53:41	sean-k-mooney	unit tests are awsome :)
11:53:48	sean-k-mooney	although docs would also help
11:54:01	kplant	as long as the _set string is _only_ negations, every other vcpu should implicitly be available for scheduling, no?
11:54:52	sean-k-mooney	that is the behavior i would want as an enduser
11:56:26	sean-k-mooney	but no unfortunetly that is not what the behvior is today
11:56:30	kplant	i probably just over thought it and gave myself a reason to be lazy and use reserved_host_cpus instead of cpu_dedicated_set
11:56:41	kplant	oh, really?
11:57:00	openstackgerrit	Merged openstack/nova master: Add info about affinity requests to the troubleshooting doc https://review.opendev.org/715092
11:57:01	sean-k-mooney	it starts with an empty set cpuset_ids = set()
11:57:08	openstackgerrit	Merged openstack/nova master: Stabilize functional tests https://review.opendev.org/717070
11:57:09	kplant	eek
11:57:15	openstackgerrit	Merged openstack/nova master: Introduce scope_types in security groups policy https://review.opendev.org/716786
11:57:16	sean-k-mooney	and after we loop over every thing cpuset_ids -= cpuset_reject_ids
11:57:39	sean-k-mooney	it would not be hard to add that behavior.
11:58:02	kplant	i would expect assuming all cpus to actually be faster code
11:58:13	kplant	that's more likely in line with what the result will be
11:58:37	sean-k-mooney	kplant: that code does not know how many cpus you have
11:59:44	kplant	fair, it would come from sql
12:00:45	sean-k-mooney	not quite where this is used is on the comptue node before we store the info in the db but anyway if you do have a propoasl for improving this feel free to write it up
12:02:09	kplant	is this the better behavior? the current behavior forces the user to be explicit
12:02:39	kplant	i guess i can always submit it and get input that way
12:06:40	sean-k-mooney	kplant: if you dont set the config options all cores are assumed to be usable
12:06:59	sean-k-mooney	kplant: so the idea was of you opt in you should say what you want
12:08:35	kplant	i think that mindset still applies, just change the behavior from cpu_*_set to a merge behavior instead of replace
12:08:43	kplant	i think that's reasonable
12:08:47	sean-k-mooney	kplant: that said i have wanted to remove the reserved_host_cpus options since we first added vcpu_pin_set so makeing it nicer to use the reserved_host_cpus will help with that goal
12:08:49	kplant	just want to make sure before i waste time with a bp
12:10:26	kplant	waste other people's time*
12:10:33	sean-k-mooney	kplant: well we would need to keep backwards compatbliy so we could not make it merge by defaul but we could change the behavior so that if you only specify negation then we woud assume all cpus were valid and apply the negation
12:11:23	kplant	very fair point
12:11:37	kplant	that would make cpu_dedicated_set = "4" == all cpus
12:12:26	sean-k-mooney	kplant: ill file a bug
12:12:58	kplant	appreciate that
12:16:05	sean-k-mooney	kplant: the only issue really is that now that we have two ranges cpu_share_set and cpu_dedicated_set
12:16:18	sean-k-mooney	it become less uesful but its still useful
12:17:13	kplant	i guess the winner between the two would be the more explicit option?
12:17:21	kplant	share_set: "1-5"
12:17:25	kplant	dedicate_set: "3"
12:17:34	sean-k-mooney	no you get an error if you do that
12:17:50	sean-k-mooney	and i dont think we want that much magic in the config option parsing
12:18:33	kplant	that works
12:19:14	kplant	so i guess here's a difficult question
12:19:27	kplant	if you do mix shared and dedicated on the same host, and leave N cpus unspecified
12:19:34	kplant	are they shared? are they dedicated?
12:19:49	kplant	with the current implementation they're neither
12:21:57	sean-k-mooney	https://bugs.launchpad.net/nova/+bug/1871096
12:21:58	openstack	Launchpad bug 1871096 in OpenStack Compute (nova) "when only a negation is specified for cpu_*_sets we should assume all cpus are vaild and subtract the negated cpus" [Wishlist,Triaged]

Earlier   Later