Posted	Nick	Remark
#openstack-nova - 2019-11-20
16:02:02	efried	dansmith: I can live with that; same as the requirement for the keymgr to exist.
16:02:04	dansmith	efried: because the user doesn't know any of this, other than that "every openstack cloud they use seems to be different and incompatible"
16:02:20	fgonzales3246	hello all, I'm configuring SSL connections into my MySQL database for the components and when I activate this into the dabase and in each connection my controller starts to slow down and the requests fail due to timeout.
16:02:23	dansmith	efried: ack, I'd much rather do that, or at least do that in addition to the other
16:02:52	efried	dansmith: IMO it's better to block operations you know are broken than to simply state they're not supported and let them sh*t all over themselves if tried.
16:02:59	fgonzales3246	I tried to isolate one component like, cinder, but it occurs the same... seems that the requests doesn't go to the compute node
16:03:06	efried	but I think the counterargument has been that we want to be able to enable the operation without needing a new microversion.
16:03:32	fgonzales3246	but the services are all up and running, do you know what could I be missing? Something related to nova-conductor or something else?
16:03:38	dansmith	efried: of course, I'm saying having "the plan" be a thousand places where seemingly fundamental operations can fail with the appropriate combination of obscure backend configs sucks from the user's perspective
16:03:40	efried	anyway, in this case we're not talking API changes, and we're talking blocking at boot rather than (well, in addition to) shelve, so I'm good.
16:03:51	efried	dansmith: totally agree.
16:03:56	efried	block it if you know it's broken.
16:04:09	dansmith	...but try not to rely on that in design
16:04:27	efried	now I just hope I can detect a swift
16:04:46	efried	oh,
16:05:03	efried	cause I wanted to add a check for "key manager configured" to the "do we enable vTPM in the first place" logic
16:05:11	efried	but couldn't find a reliable way to inspect CONF for that
16:05:20	dansmith	you need the catalog, no?
16:05:24	efried	but yeah,
16:05:36	efried	I can at least look in the catalog to make sure there's a keymgr endpoint.
16:05:40	efried	um
16:05:59	efried	except I'm doing this on init_host and don't have a context?
16:06:05	dansmith	no,
16:06:10	dansmith	you do this at boot time for the instance
16:06:17	dansmith	non-tpm instances don't care about this
16:06:37	dansmith	what you want to avoid is agreeing to create an instance for the user which in six weeks will be unsnapshottable
16:06:44	efried	Sorry, I'm talking about something slightly different: the part where the libvirt driver decides whether to set the "can I vTPM?" flag.
16:06:53	dansmith	I'm not talking about that
16:06:55	efried	...causing it to expose the traits
16:06:57	efried	I know, I am.
16:07:02	efried	I understand and agree with what you're talking about.
16:07:16	dansmith	that too may be good, but expecting keystone is up and reachable when compute starts is not okay I think
16:07:33	efried	orly? ... okay.
16:07:34	dansmith	computes on the other side of a network partition shouldn't just fail to start I think
16:07:42	efried	fair enough
16:07:54	dansmith	they'll currently wait for conductor if it's not up yet, so...
16:08:25	efried	yeah, I don't want to be the guy to introduce a network ping into init_host.
16:08:45	dansmith	we have that :)
16:08:49	dansmith	I added it
16:09:01	efried	from compute driver or from compute manager?
16:09:04	dansmith	...but it's just a ping() rpc method compute makes to conductor to see if it's alive before it goes :)
16:09:08	dansmith	manager
16:09:22	efried	yeah, that's in manager. I'm talking about libvirt compute driver's init_host, which seems... dirtier.
16:09:37	dansmith	yes, I'm just pointing out that I called it ping :)
16:10:42	efried	okay, so lacking a jroll for live discussion, I'll mod the spec as if this is the plan, and ping for reviews.
16:10:45	efried	Thanks dansmith
16:11:14	dansmith	sounds good
16:13:27	mnaser	sean-k-mooney: so my new map is 3C per NUMA node (6 threads total), ~32GB per NUMA node, with quite interesting latency numbers
16:13:37	mnaser	https://www.irccloud.com/pastebin/91D5sGuS/
16:13:53	openstackgerrit	Stephen Finucane proposed openstack/nova master: zuul: Remove unnecessary 'tox_install_siblings' https://review.opendev.org/695235
16:16:19	efried	mriedem: the barbican-gating-on-py3 thing from yesterday seems to be busted https://review.opendev.org/#/c/695052/ -- would you mind having a quick look at some point?
16:17:51	mnaser	w
16:22:13	stephenfin	efried: didn't AJaeger have a fix for that?
16:22:26	sean-k-mooney	mnaser: that kindof look like what i would expect
16:22:30	stephenfin	efried: https://review.opendev.org/#/c/689461/
16:22:32	efried	stephenfin: yes, gmann said something in -barbican
16:22:40	sean-k-mooney	as the numa distance increase teh latnecy does too
16:22:50	stephenfin	ah, maybe not the same thing
16:22:52	stephenfin	nvm me
16:23:04	mnaser	sean-k-mooney: i also bumped the "4-link xGMI max speed" setting to 18Gbps .. but the machine is now crashing (it was set to 10.667 earlier)
16:23:09	efried	stephenfin: yeah, he pointed to https://review.opendev.org/#/c/689458/ (same thing but in train)
16:23:14	mnaser	so ill reset it back to that, and hopefully the latency stays at these numbers
16:23:34	gmann	stephenfin: yeah, stable/train backport will fix the baribican. but still asking the reason to infra on ML for learning.
16:23:51	stephenfin	sweet
16:24:10	sean-k-mooney	but it lends more evidence to the idea we should look at cache affinity and numa distance when selecting vm numa node mappings
16:24:39	mnaser	sean-k-mooney: yep, giving an instance two NUMA nodes that are '32' away vs '10' ..
16:24:54	sean-k-mooney	mnaser: can you dump the numa distance info
16:24:56	mnaser	or rather 11/12
16:25:15	sean-k-mooney	i would be gould to confimr the latency corralate with the numa distance reported
16:25:24	mnaser	https://www.irccloud.com/pastebin/GwxGM24b/
16:25:47	mnaser	they're not that consistent but yeah
16:26:14	sean-k-mooney	ya it does if we look noes 8-15 have the same numa distacne of 32
16:26:32	mnaser	which are all socket #2
16:26:36	sean-k-mooney	yep
16:27:18	sean-k-mooney	we see 2-7 are all reported at numa distance of 12 and have almost the same latency
16:27:24	gmann	we need some review from mriedem and tonyb here to get these backport in (pinged dave-mccowan barbican stable core also) - we need to get these merged as soon as we can - https://review.opendev.org/#/q/I24a46d0d7476203feccb1250d4ce3ad94b2e0ecd
16:27:38	mnaser	sean-k-mooney: struggling what the 11/12 are there for
16:29:00	sean-k-mooney	ya its not fully clear but we could use this info to optimise slightly
16:29:25	sean-k-mooney	i suspect that is related to how the dies are grouped in pairs
16:29:35	sean-k-mooney	distance of 11 is to the other ccx in the pari
16:29:49	sean-k-mooney	12 is the other ccx on the socket no in the pair
16:29:56	sean-k-mooney	and 32 is cross socket in this case
16:32:09	mnaser	sean-k-mooney: ok that makes sense then, i will try to reset the infinity fabric speed down to see if the machine stops crashing :\
16:34:28	Sundar	sean-k-mooney: Thanks a lot for reviewing the Cyborg spec despite your busy schedule. All the best.
16:40:15	mriedem	gmann: the only one passing is stable/train
16:42:30	efried	mriedem: it sounds like if that one merges it might fix master (I still don't understand how though)
16:42:59	gmann	mriedem: yeah some unrelated failure. i did recheck on stein. after train cherry pick merged we can check if problem is sovled. otherwise need to get all in
16:43:30	gmann	it all depends on how zuul pick the job definition due to those branche variant.
16:44:16	gmann	we can try one by one. last option to avoid barbican gate is to make grenade job as py2 explicitly.
16:44:25	gmann	*gate break
16:47:01	mriedem	efried: because the job definitions are branch specific and grenade deploys the old side using stable/train and the new side using master
16:47:17	mriedem	so sometimes to fix a thing on master for grenade you need to do something in master-1
16:47:48	mnaser	sean-k-mooney: ok making good progress, according to numastat it ended up scheduling to numa cell0 and cell1 which means this is the most "optimized" setup
16:48:22	mnaser	topology is 8 sockets, i wonder if it would be better to model the topology as "2 sockets, 4 cores, 1 threads" or "2 sockets, 2 cores, 2 threads" too
16:56:19	efried	dansmith: are there any operations other than shelve for which the swift blob would be appropriate?
16:57:15	dansmith	well, what are you going to do about snapshot?
16:57:23	dansmith	snapshot gets a fresh tpm?
16:57:34	dansmith	that doesn't work well for people that use snapshot as a backup mechanism
16:57:42	dansmith	speaking of that, the backup api :)
16:58:48	mnaser	burn it with fire

Earlier   Later