| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-10 | |||
| 19:37:56 | dansmith | mriedem: tbh I feel like I have a bit of review fatigue on this quota stuff | |
| 19:38:20 | dansmith | not sure I've seen an actual review from you on this one | |
| 19:38:29 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: claim resources in scheduler https://review.openstack.org/476632 | |
| 19:38:30 | mriedem | you haven't | |
| 19:38:33 | jaypipes | mriedem: fixed :( ^ | |
| 19:38:42 | dansmith | I guess we're not likely to see one from sdague in the near future either, eh? | |
| 19:39:12 | mriedem | probably not. i plan on reviewing that quotas one. | |
| 19:39:26 | jaypipes | mriedem: I will put a patch up renaming delete_allocation_for_instance() to delete_allocations_for_consumer() to remedy my need for grammatical correctness. | |
| 19:43:34 | sdague | dansmith: yeh, not this week | |
| 19:44:00 | sdague | I can get my head back into it next week, but I don't know which dates we're racing against now | |
| 19:44:19 | dansmith | yeah, need it before next week I think | |
| 19:52:26 | mriedem | sdague: feature freeze is july 27 | |
| 19:52:31 | mriedem | so we're less than 3 weeks from FF | |
| 19:53:09 | mriedem | jaypipes: a couple of comments in https://review.openstack.org/#/c/476632/ | |
| 19:53:25 | mriedem | jaypipes: dansmith: in a nutshell on that one, | |
| 19:53:47 | mriedem | i think it means we potentially have problems for bursts of instances in the same request with that change until we start doing retries in the scheduler | |
| 19:54:08 | mriedem | and/or we start randomizing which allocation request match we pick for a given compute node | |
| 19:55:39 | jaypipes | mriedem: do you want me to do retries in this patch? | |
| 19:55:49 | jaypipes | mriedem: I was working on that in a separate followup. | |
| 19:56:04 | jaypipes | mriedem: but if you're concerned about this, I suppose I could put it all in one patch. | |
| 19:56:11 | jaypipes | mriedem: was trying to keep the LOCC low. | |
| 19:56:16 | mriedem | a single change makes it harder to review for sure | |
| 19:56:17 | mriedem | yeah | |
| 19:56:42 | mriedem | i guess we assume no one is doing CD | |
| 19:57:12 | mriedem | jaypipes: how about line up the retries change on top, | |
| 19:57:13 | jaypipes | mriedem: answered you. | |
| 19:57:16 | mriedem | and then we'll get them in order | |
| 19:57:22 | jaypipes | mriedem: sure thing. | |
| 19:57:22 | mriedem | so there isn't like a week between changes | |
| 19:57:30 | jaypipes | mriedem: gimme about an hour. | |
| 20:00:31 | cdent | is this canon: “i guess we assume no one is doing CD” ? If so, life just got a lot easier. | |
| 20:00:49 | mriedem | no, it's not | |
| 20:01:19 | mriedem | dims is supposed to make huawei public cloud doing CD in the next couple of weeks :) | |
| 20:01:25 | cdent | do you take bribes? | |
| 20:02:54 | mriedem | https://docs.openstack.org/nova/latest/project_scope.html#upgrade-expectations | |
| 20:03:01 | mriedem | "In addition, we also support users upgrading from the master branch, technically, between any two commits within the same release cycle." | |
| 20:03:39 | cdent | oh yeah, I know, I was just thinking maybe the world had collapsed | |
| 20:30:03 | jaypipes | mriedem: so... problem. | |
| 20:30:17 | jaypipes | mriedem: I need to do the alternatives stuff before the retries. | |
| 20:30:29 | jaypipes | mriedem: because the retries need to loop over the alternatives. | |
| 20:30:34 | mriedem | right, b/c we need the alternatives to do retries | |
| 20:30:35 | mriedem | yup | |
| 20:30:49 | jaypipes | mriedem: I don't necessarily need to *return* (over RPC) the alternative yet though. | |
| 20:34:20 | mriedem | correctamundo | |
| 20:34:35 | mriedem | so it's like 3 changes on top of this, maybe 2 | |
| 20:34:40 | mriedem | 1. determine alternatives, | |
| 20:34:45 | mriedem | 2. retry on alternatives in scheduler | |
| 20:34:52 | mriedem | 3. pass alternatives to conductor | |
| 20:35:00 | mriedem | and then 4,5,6,7,8 do things with those in the cell | |
| 20:35:05 | jaypipes | yeah | |
| 20:38:50 | jaypipes | mriedem, dansmith: did we ever decide how many alternatives we wanted to return? | |
| 20:39:02 | jaypipes | mriedem, dansmith: limit to 10 per instance UUID maybe? | |
| 20:39:08 | dansmith | jaypipes: CONF.num_retries | |
| 20:39:13 | dansmith | times num_instances | |
| 20:39:13 | jaypipes | ah | |
| 20:39:14 | jaypipes | gotcha. | |
| 20:41:27 | edleafe | jaypipes: this was the consensus a couple of months ago: https://review.openstack.org/#/c/465767/ | |
| 20:41:51 | edleafe | default to CONF.num_retries | |
| 20:42:17 | edleafe | also, won't the order need to be different? | |
| 20:43:06 | edleafe | IOW, claim fails on "best" host, so you try claiming next one. Once you succeed, you select N alternates from the next N hosts in the weighed list that are in the same cell as the successful claimed host | |
| 20:43:18 | jaypipes | edleafe: the consensus was to create a new CONF option build_max_attempts? | |
| 20:44:42 | dansmith | that sounds like a terrible idea | |
| 20:44:45 | dansmith | num_retries and build_max_attempts is highly confusing | |
| 20:45:08 | dansmith | we have the former, I see no reason for a billionth conf var | |
| 20:45:54 | edleafe | jaypipes: sorry, it was supposed to be 'max_attempts' | |
| 20:47:47 | mriedem | plus host_subset_size | |
| 20:48:05 | openstackgerrit | Merged openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/476631 | |
| 20:48:13 | mriedem | we're also going to have to build in the filtering for the alternatives by cell | |
| 20:48:35 | mriedem | i'm guessing we're going to want to cache that as well | |
| 20:48:39 | edleafe | mriedem: host_subset_size only comes into play in the scheduler. The conductor wouldn't use it | |
| 20:48:42 | mriedem | so we don't lookup hostmappings every time we schedule an instance | |
| 20:48:55 | mriedem | edleafe: i was just commenting on the number of existing options | |
| 20:49:01 | dansmith | mriedem: we discussed that yeah, bauzas had a plan for just filtering the alternatives with matching cells | |
| 20:49:17 | dansmith | it was done in his set I think | |
| 20:49:36 | edleafe | it was also done here: https://review.openstack.org/#/c/465768/8/nova/scheduler/filter_scheduler.py | |
| 20:49:37 | mriedem | something else nagging me a bit here, | |
| 20:49:47 | edleafe | (that was before moving to scheduler manager) | |
| 20:50:03 | jaypipes | fuck the instance group shit. | |
| 20:50:10 | mriedem | say it's a create request with 100 servers | |
| 20:50:21 | mriedem | we get our allocation candidates and make claims on them | |
| 20:50:35 | smcginnis | jaypipes: How do you really feel? | |
| 20:50:47 | mriedem | then we're going to send some alternatives down, but couldn't those alternatives be full by the time we're doing with the claims? | |
| 20:50:49 | jaypipes | smcginnis: sorry, that wasn't meant for public consumption. | |
| 20:50:53 | smcginnis | :) | |
| 20:50:53 | mriedem | as the alternatives aren't really a fresh set? | |
| 20:51:18 | mriedem | s/doing/done/ | |
| 20:51:19 | edleafe | mriedem: there could be dupes in the alternatives for each of the 100, too | |
| 20:51:20 | jaypipes | mriedem: yes, you're absolutely correct. this is the problem with num_instances > 1 :( | |
| 20:51:34 | edleafe | mriedem: just like the selected host could be the same for some of the 100 | |
| 20:51:55 | dansmith | jaypipes: fwiw, I was just (minutes ago) helping a real nova user debug a problem with --num_instances=100 :/ | |
| 20:52:01 | mriedem | edleafe: well filtering dupes from the list of alternatives seems trivial, just do a set based on rp uuid | |
| 20:52:14 | edleafe | mriedem: why? | |
| 20:52:23 | mriedem | i get the feeling that huawei's users are also lovers of multiple create | |
| 20:52:30 | edleafe | with the scheduler claiming, the only reason a build will fail will be a host issue | |
| 20:52:32 | mriedem | b/c i've seen bug reports from them about it | |
| 20:52:32 | jaypipes | it's an NFV thing... | |
| 20:52:51 | edleafe | so most will succeed, and we throw away the alternatives | |
| 20:52:55 | mriedem | edleafe: i assume you and i are talking about different things then | |
| 20:52:57 | dansmith | while googling for something, I also found a bug filed against another platform with "nova has --num_instances and I use it a lot, can't use your platform without it...." | |
| 20:53:09 | mriedem | ha, that's a first | |