| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-04 | |||
| 15:30:01 | donnyd | Mine are so slow compared to others | |
| 15:30:12 | mriedem | looking dstat when this messaging timeout happened https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4 around the time of the timeout cpu usage is low, io is basically 0 and load is spiked | |
| 15:30:31 | donnyd | The biggest difference for FN is local NVME storage for all instances | |
| 15:30:43 | cdent | question: isn't this something infra will already have a plan and solution for? | |
| 15:30:52 | cdent | this is what cloud mgt/provisioning is all about, presumably? | |
| 15:31:11 | mriedem | i wonder if these providers live migrate the ci vms around frequently? | |
| 15:31:30 | donnyd | If there was a way to benchmark providers and then prefer them for jobs that are bound by something I think we could optimize the CI | |
| 15:32:19 | donnyd | Butttt.... if that provider breaks or goes away the issue becomes jobs failing because they were dependent on that super fast thing provider_x does | |
| 15:32:28 | mriedem | cdent: infra is just tenants so i'm not sure how much control they have | |
| 15:32:48 | mriedem | besides may requesting minimums in a flavor or something | |
| 15:32:49 | mriedem | ? | |
| 15:32:51 | cdent | mriedem: but presumably they can tweak their consumption? | |
| 15:32:53 | mriedem | *maybe | |
| 15:32:54 | donnyd | mriedem: well a simple benchmark could expose what jobs run optimal | |
| 15:34:19 | donnyd | That was my idea and it got quickly struck down because of the issue listed above efried | |
| 15:35:42 | donnyd | I do all kinds of custom things and so does mnaser at vexxhost to support the CI... not sure how much usage it gets though. I think limestone is on board with it as well | |
| 15:35:45 | efried | ah, thanks donnyd, I remember now | |
| 15:35:51 | donnyd | NP | |
| 15:36:01 | donnyd | Hopefully that helps | |
| 15:36:17 | efried | So really $provider needs to be able to run $job. In this case ovh needs to go figure out what the bottleneck is and fix it. | |
| 15:36:31 | donnyd | Bingo | |
| 15:36:44 | efried | even if it's by reducing the number of nodes they provide | |
| 15:36:49 | donnyd | But they probably don't even know its an issue till someone complains | |
| 15:36:59 | efried | not sure where else to complain | |
| 15:37:08 | donnyd | probably infra | |
| 15:37:10 | efried | openstack-discuss@ has a thread | |
| 15:37:44 | donnyd | You can always schedule the job using maybe the numa label made for sean-k-mooney | |
| 15:38:00 | efried | I guess we wait until after the summit then, infra folks are going to be pretty sparse for the next week I imagine. | |
| 15:38:05 | donnyd | That would ensure the job runs on FN vexxhost or limestone if I am not mistaken | |
| 15:38:12 | efried | donnyd: yeah, but you have to put that in the job def itself, right? | |
| 15:38:26 | efried | You can't do it one off to make a particular patch merge | |
| 15:38:29 | donnyd | Yea, you have to assign it a label | |
| 15:38:31 | donnyd | No | |
| 15:38:44 | donnyd | Sry, that is correct efried | |
| 15:38:44 | efried | that's our problem here, getting approved patches through the gate. | |
| 15:39:16 | donnyd | Well patch the job with that label... then it will run anyways wont it | |
| 15:39:20 | efried | heh, we should just change all our devstack-based job defs to run on the fast providers... | |
| 15:39:33 | donnyd | Well I wouldn't call FN fast | |
| 15:39:36 | efried | wait until somebody notices | |
| 15:39:41 | efried | okay, s/fast/reliable/ | |
| 15:39:43 | donnyd | Just better at IO bound jobs | |
| 15:40:06 | efried | Yeah, I would rather my patch take 2h and succeed than 1.5h and fail. | |
| 15:40:17 | donnyd | I'm sure ironic and tripleo hate FN because my CPUs are old | |
| 15:40:41 | donnyd | And they are mostly CPU bound | |
| 15:40:54 | donnyd | Give it a swing with the label | |
| 15:41:05 | donnyd | And if it works, you have a bandaid | |
| 15:42:21 | openstackgerrit | Eric Fried proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | |
| 16:14:29 | efried | donnyd: I wasn't serious about the label. That would be pretty publicly greedy of us. | |
| 16:19:19 | dansmith | mriedem: question for you in here while I look at the test: https://review.opendev.org/#/c/635646/48/nova/conductor/tasks/cross_cell_migrate.py | |
| 16:19:50 | dansmith | holy hell, everything in the check/gate are running tests and only minutes old | |
| 16:19:58 | dansmith | we should send the whole community to china more often | |
| 16:24:15 | donnyd | Well not really efried | |
| 16:24:45 | donnyd | If a job has requirements to run and a specific label meets the requirements then I don't see the issue | |
| 16:25:02 | donnyd | But it's your call... just pointing out the options | |
| 16:26:44 | mriedem | dansmith: thanks, replied | |
| 16:27:18 | mriedem | dansmith: is my sarcasm detector picking this up correctly? https://review.opendev.org/#/c/635080/48/nova/tests/unit/compute/test_compute_mgr.py@10545 | |
| 16:27:32 | dansmith | mriedem: yes. | |
| 16:28:02 | mriedem | heh, verbose commentage is how i keep from feeling lonely | |
| 16:34:55 | dansmith | mriedem: oops, my "...yes I see" was supposed to go into the fault_clone complaint nit | |
| 16:35:13 | dansmith | went back to note that and picked the wrong comment to edit | |
| 16:35:48 | dansmith | mriedem: one more question | |
| 16:40:30 | openstackgerrit | Merged openstack/nova master: Use long_rpc_timeout in conductor migrate_server RPC API call https://review.opendev.org/692550 | |
| 16:41:03 | mriedem | replied | |
| 16:43:04 | mriedem | agree about the mapping == add a unit test wrinkle for that? | |
| 16:43:23 | mriedem | will do - and the variable name nits | |
| 16:45:58 | dansmith | cool | |
| 16:48:27 | efried | dansmith: I just put a bunch more words in the 'flavor groups' spec. Would you still be -1 if the answer to the backward compat question were "you can ignore flavor groups and keep doing extra specs forever"? IMO there's still too much complexity (most of which still isn't addressed) to make it worthwhile. | |
| 16:49:08 | efried | whoah, that timeout patch merged already?? Didn't we approve it like an hour ago?? | |
| 16:49:13 | dansmith | efried: you mean would I still be effectively -2 I assume | |
| 16:49:21 | efried | yeah | |
| 16:50:22 | dansmith | efried: and without reading, you mean keeping extra_specs for the hard flavor case and only allowing the groups for compose-ability? I still think it's practically useless without some way to define which things can be composed, and agree it's far too much change to too many fundamental things to be worth it or viable | |
| 16:50:22 | efried | pretty sure we're in agreement on it, I'm just wondering how aggressively I/we should say "despite feeding you paths to address all the holes, don't bother doing that, because this is going to die anyway" | |
| 16:50:41 | efried | yeah, that's more or less what I said. | |
| 16:50:44 | efried | in the review | |
| 16:51:11 | dansmith | I think I'm still effectively -2 even with keeping extra_specs for those reasons, yeah | |
| 16:51:46 | dansmith | I think that it's legit to -2 something like this on "this isn't the direction we want to go" and if you and I both -2 it for that reason, I think that is a fairly tight case | |
| 16:51:54 | dansmith | it's not like we haven't considered it or reviewed it in detail | |
| 16:52:23 | dansmith | certainly don't want to just flip him the bird, but you wanted to be more upfront with people (as do I) so... that'd be pretty upfront | |
| 16:52:29 | efried | yeah, right now we're both -1.9 I think, but I don't want the author to take our -1s as a message that he should go try to fill it in. | |
| 16:52:38 | dansmith | yeah, fair point | |
| 16:52:54 | dansmith | I tried to call out my effectively -2ness for that reason, but maybe -2 would be better | |
| 16:53:11 | efried | so if you wouldn't mind reading my latest comments (when you get a chance) and if you agree with what I'm saying, go ahead and -2 and I'll follow suit. | |
| 16:53:21 | dansmith | sure | |
| 16:53:22 | efried | thx | |
| 16:53:46 | mriedem | i have run into "I don't want the author to take our -1s as a message that he should go try to fill it in." on a couple of specs for ussuri even though i've said a few times, "i don't think we need to do this, the existing alternatives or sufficient or this doesn't fit with the project IMO" but the specs continue to be updated | |
| 16:54:07 | mriedem | *are sufficient | |
| 16:54:29 | mriedem | i probably haven't -2ed because they are mostly trivial things to do, just plumbing, | |
| 16:54:40 | mriedem | not anywhere near the complexity of this flavor groups tihng | |
| 16:54:42 | mriedem | *thing | |
| 16:56:10 | cdent | Explicit words might be the way to go. | |
| 16:57:27 | dansmith | cdent: my words were explicit, but I hedged on the vote | |
| 16:57:44 | dansmith | because people sometimes accuse me of closing the door to further reviews with a -2 or even a -1 | |
| 16:58:02 | cdent | well closing the door is what's wanted here, yeah? | |
| 16:58:22 | dansmith | well, | |
| 16:58:38 | dansmith | yes, but I want to do that in concert with other reviewers, like efried and I just did | |
| 16:59:20 | dansmith | so which do I need to kill? there is public, private, and shared | |
| 16:59:25 | dansmith | oops, wrong window | |
| 16:59:42 | cdent | all three /me is all for early kills | |
| 16:59:45 | cdent | ! | |
| 17:02:48 | mriedem | https://review.opendev.org/#/c/682302/ and https://review.opendev.org/#/c/580336/ are the ones i'm struggling with fwiw | |