Posted	Nick	Remark
#openstack-nova - 2019-11-04
15:25:47	donnyd	Oh, well I know I avoid that issue by using local storage on FN. Not sure what ovh has on the backend of their instances
15:26:10	efried	not asking for action on your part donnyd, just advice. If a patch has 10 zuul jobs an any one of them lands on ovh and chokes on I/O and times out, the whole patch has to be retried, which is a royal PITA. Been happening a really lot over the last week or two.
15:26:43	efried	Yeah, I guess they would need to look into where the actual bottleneck is.
15:26:53	efried	not sure if there's any way to tell from here via grafana...
15:26:55	donnyd	The reality is we should try and label jobs by what they are bound by
15:27:09	efried	heh, we were just talking about your email from Sept.
15:27:25	donnyd	If there is a cpu bound job, scheduling on FN would be less optimal
15:27:44	efried	I couldn't see anywhere infra had responded to that idea, but I thought I remembered them shutting it down hard for some reason.
15:27:50	cdent	efried, mriedem : what I meant by less cpu, was less cpu for all nodes, not per node
15:27:51	donnyd	But IO bound jobs will go like stink on FN
15:27:53	cdent	so that we can run fewr jobs
15:28:13	cdent	because we run too many at once
15:28:22	donnyd	I shut down FN when I put in my gen set
15:28:24	cdent	because oversubscribers are liars
15:28:48	efried	cdent: okay, that's what I was suggesting too.
15:28:53	donnyd	I oversub FN by a small margin, but not much
15:29:07	donnyd	Usually 100% of the memory is utilized
15:29:08	efried	so like a white lie?
15:29:25	cdent	donnyd in my experience what you're doing seems to be working much better than some of the other providers, so kudos to you
15:29:31	donnyd	But CPU can be oversub like 1.125
15:29:50	donnyd	cdent: I need faster CPUs for sure
15:30:01	donnyd	Mine are so slow compared to others
15:30:12	mriedem	looking dstat when this messaging timeout happened https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4 around the time of the timeout cpu usage is low, io is basically 0 and load is spiked
15:30:31	donnyd	The biggest difference for FN is local NVME storage for all instances
15:30:43	cdent	question: isn't this something infra will already have a plan and solution for?
15:30:52	cdent	this is what cloud mgt/provisioning is all about, presumably?
15:31:11	mriedem	i wonder if these providers live migrate the ci vms around frequently?
15:31:30	donnyd	If there was a way to benchmark providers and then prefer them for jobs that are bound by something I think we could optimize the CI
15:32:19	donnyd	Butttt.... if that provider breaks or goes away the issue becomes jobs failing because they were dependent on that super fast thing provider_x does
15:32:28	mriedem	cdent: infra is just tenants so i'm not sure how much control they have
15:32:48	mriedem	besides may requesting minimums in a flavor or something
15:32:49	mriedem	?
15:32:51	cdent	mriedem: but presumably they can tweak their consumption?
15:32:53	mriedem	*maybe
15:32:54	donnyd	mriedem: well a simple benchmark could expose what jobs run optimal
15:34:19	donnyd	That was my idea and it got quickly struck down because of the issue listed above efried
15:35:42	donnyd	I do all kinds of custom things and so does mnaser at vexxhost to support the CI... not sure how much usage it gets though. I think limestone is on board with it as well
15:35:45	efried	ah, thanks donnyd, I remember now
15:35:51	donnyd	NP
15:36:01	donnyd	Hopefully that helps
15:36:17	efried	So really $provider needs to be able to run $job. In this case ovh needs to go figure out what the bottleneck is and fix it.
15:36:31	donnyd	Bingo
15:36:44	efried	even if it's by reducing the number of nodes they provide
15:36:49	donnyd	But they probably don't even know its an issue till someone complains
15:36:59	efried	not sure where else to complain
15:37:08	donnyd	probably infra
15:37:10	efried	openstack-discuss@ has a thread
15:37:44	donnyd	You can always schedule the job using maybe the numa label made for sean-k-mooney
15:38:00	efried	I guess we wait until after the summit then, infra folks are going to be pretty sparse for the next week I imagine.
15:38:05	donnyd	That would ensure the job runs on FN vexxhost or limestone if I am not mistaken
15:38:12	efried	donnyd: yeah, but you have to put that in the job def itself, right?
15:38:26	efried	You can't do it one off to make a particular patch merge
15:38:29	donnyd	Yea, you have to assign it a label
15:38:31	donnyd	No
15:38:44	efried	that's our problem here, getting approved patches through the gate.
15:38:44	donnyd	Sry, that is correct efried
15:39:16	donnyd	Well patch the job with that label... then it will run anyways wont it
15:39:20	efried	heh, we should just change all our devstack-based job defs to run on the fast providers...
15:39:33	donnyd	Well I wouldn't call FN fast
15:39:36	efried	wait until somebody notices
15:39:41	efried	okay, s/fast/reliable/
15:39:43	donnyd	Just better at IO bound jobs
15:40:06	efried	Yeah, I would rather my patch take 2h and succeed than 1.5h and fail.
15:40:17	donnyd	I'm sure ironic and tripleo hate FN because my CPUs are old
15:40:41	donnyd	And they are mostly CPU bound
15:40:54	donnyd	Give it a swing with the label
15:41:05	donnyd	And if it works, you have a bandaid
15:42:21	openstackgerrit	Eric Fried proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
16:14:29	efried	donnyd: I wasn't serious about the label. That would be pretty publicly greedy of us.
16:19:19	dansmith	mriedem: question for you in here while I look at the test: https://review.opendev.org/#/c/635646/48/nova/conductor/tasks/cross_cell_migrate.py
16:19:50	dansmith	holy hell, everything in the check/gate are running tests and only minutes old
16:19:58	dansmith	we should send the whole community to china more often
16:24:15	donnyd	Well not really efried
16:24:45	donnyd	If a job has requirements to run and a specific label meets the requirements then I don't see the issue
16:25:02	donnyd	But it's your call... just pointing out the options
16:26:44	mriedem	dansmith: thanks, replied
16:27:18	mriedem	dansmith: is my sarcasm detector picking this up correctly? https://review.opendev.org/#/c/635080/48/nova/tests/unit/compute/test_compute_mgr.py@10545
16:27:32	dansmith	mriedem: yes.
16:28:02	mriedem	heh, verbose commentage is how i keep from feeling lonely
16:34:55	dansmith	mriedem: oops, my "...yes I see" was supposed to go into the fault_clone complaint nit
16:35:13	dansmith	went back to note that and picked the wrong comment to edit
16:35:48	dansmith	mriedem: one more question
16:40:30	openstackgerrit	Merged openstack/nova master: Use long_rpc_timeout in conductor migrate_server RPC API call https://review.opendev.org/692550
16:41:03	mriedem	replied
16:43:04	mriedem	agree about the mapping == add a unit test wrinkle for that?
16:43:23	mriedem	will do - and the variable name nits
16:45:58	dansmith	cool
16:48:27	efried	dansmith: I just put a bunch more words in the 'flavor groups' spec. Would you still be -1 if the answer to the backward compat question were "you can ignore flavor groups and keep doing extra specs forever"? IMO there's still too much complexity (most of which still isn't addressed) to make it worthwhile.
16:49:08	efried	whoah, that timeout patch merged already?? Didn't we approve it like an hour ago??
16:49:13	dansmith	efried: you mean would I still be effectively -2 I assume
16:49:21	efried	yeah
16:50:22	efried	pretty sure we're in agreement on it, I'm just wondering how aggressively I/we should say "despite feeding you paths to address all the holes, don't bother doing that, because this is going to die anyway"
16:50:22	dansmith	efried: and without reading, you mean keeping extra_specs for the hard flavor case and only allowing the groups for compose-ability? I still think it's practically useless without some way to define which things can be composed, and agree it's far too much change to too many fundamental things to be worth it or viable
16:50:41	efried	yeah, that's more or less what I said.
16:50:44	efried	in the review
16:51:11	dansmith	I think I'm still effectively -2 even with keeping extra_specs for those reasons, yeah
16:51:46	dansmith	I think that it's legit to -2 something like this on "this isn't the direction we want to go" and if you and I both -2 it for that reason, I think that is a fairly tight case
16:51:54	dansmith	it's not like we haven't considered it or reviewed it in detail
16:52:23	dansmith	certainly don't want to just flip him the bird, but you wanted to be more upfront with people (as do I) so... that'd be pretty upfront
16:52:29	efried	yeah, right now we're both -1.9 I think, but I don't want the author to take our -1s as a message that he should go try to fill it in.

Earlier   Later