| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-13 | |||
| 21:47:17 | melwitt | oh, hm | |
| 21:47:45 | mriedem | https://gist.github.com/mriedem/cece80e5ab375c2fad6d9d599851c2dc | |
| 21:48:19 | mriedem | i think i know why, | |
| 21:48:27 | mriedem | in master the context is targeted when you get the instance | |
| 21:48:31 | mriedem | in ocata it's not, | |
| 21:48:46 | mriedem | so in this change when it's looking up the flavor from the instance, the context isn't targeted to cell0 | |
| 21:49:29 | mriedem | but we want the quotas in the main cell db, | |
| 21:49:49 | mriedem | so i'm going to have to do the same helper method that i had on master for getting the flavor from a targeted context before creating the reservation | |
| 21:49:49 | mriedem | ugh | |
| 21:50:45 | openstackgerrit | Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842 | |
| 21:58:26 | dansmith | mriedem: how do we not trip over that in tempest? | |
| 21:58:34 | mriedem | dansmith: it's for this patch, | |
| 21:58:38 | mriedem | as a result of this patch i mean | |
| 21:58:46 | mriedem | i've got it sorted out, it's just kind of a mess for the backport, | |
| 21:58:52 | mriedem | since it's a mix of the 2 changes on master | |
| 21:59:02 | mriedem | you'll see shortly | |
| 21:59:03 | dansmith | oh, I see I thought you meant the regression test doesn't run on ocata before the change | |
| 22:00:18 | openstackgerrit | melanie witt proposed openstack/nova master: Count resources to check quota for cells https://review.openstack.org/416521 | |
| 22:00:18 | openstackgerrit | melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | |
| 22:00:19 | openstackgerrit | melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | |
| 22:01:26 | mriedem | i should have probably backported this first to see how it would work | |
| 22:04:32 | mriedem | looks like we have another ocata regression https://review.openstack.org/#/c/444106/ | |
| 22:10:02 | dansmith | hmm, I thought we did that _on_ retry | |
| 22:11:59 | dansmith | populate_filter_properties() creates the retry of the selected host | |
| 22:12:02 | dansmith | which we are calling already | |
| 22:12:18 | dansmith | oh | |
| 22:12:30 | dansmith | only if populate_retry has done the initial work first I guess | |
| 22:12:31 | dansmith | huh | |
| 22:14:26 | mriedem | gd how many postgres packages do i really need to install just to run pep8 | |
| 22:15:14 | mriedem | sudo apt-get install -y (*pg*|*pq*) | |
| 22:15:21 | mriedem | install -y *everything* | |
| 22:16:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add SameCellFilter https://review.openstack.org/445035 | |
| 22:20:04 | mriedem | we should have a functional regression test for that retry fix, | |
| 22:21:01 | mriedem | should be relatively simple, run 2 nova-compute services, make the scheduler pick the first one and make that one fail to trigger a retry, and then make sure it's actually retried to the 2nd host | |
| 22:21:54 | dansmith | is that enough? it might do that by itself by chance | |
| 22:22:53 | openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in aggregates.py https://review.openstack.org/445230 | |
| 22:23:18 | openstackgerrit | Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842 | |
| 22:23:26 | openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208 | |
| 22:23:47 | mriedem | dansmith: how? | |
| 22:23:53 | mriedem | w/o digging into details i guess i don't know | |
| 22:24:11 | mriedem | i'd also like to be able to force the scheduler to pick a host in the test that we've setup to fail | |
| 22:24:28 | mriedem | i'm not entirely sure how to do that in functional tests w/o stubbing something out | |
| 22:24:29 | dansmith | if you asked the scheduler to pick what would have been its second choice and it retries, it'd choose the other one without this fix | |
| 22:24:35 | dansmith | right exactly | |
| 22:24:45 | mriedem | yeah we have to make the test pick the hosts in the order we need for the recreate | |
| 22:24:49 | mriedem | that's the tricky part, | |
| 22:24:54 | mriedem | but doesn't seem too hard | |
| 22:25:15 | dansmith | don't we shuffle the results? | |
| 22:25:16 | mriedem | i'd think a weigher could be used | |
| 22:25:32 | dansmith | yeah maybe | |
| 22:25:44 | mriedem | that's scheduler black magic to me though | |
| 22:26:08 | dansmith | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L124-L124 | |
| 22:26:24 | mriedem | i don't want to make a big deal out of it, but this is a complicated piece of code and we clearly don't have it covered sufficiently in tests | |
| 22:26:32 | dansmith | you'd have to set the subset size to 1 | |
| 22:26:48 | melwitt | it defaults to 1, so no shuffle by default | |
| 22:26:57 | dansmith | okay | |
| 22:27:12 | mriedem | are either of you interested in writing that functional test? :) | |
| 22:27:13 | dansmith | that seems like an unfortunate default for regular people | |
| 22:27:30 | mriedem | it's only random if the weights are equal isn't it? | |
| 22:28:06 | dansmith | wait, | |
| 22:28:14 | dansmith | oh, I see | |
| 22:28:42 | dansmith | huh, I never knew this picked a single host by default | |
| 22:28:44 | melwitt | mriedem: I can give it a go. I've written a weird retry test before https://github.com/openstack/nova/blob/master/nova/tests/functional/test_compute_mgr.py#L40 | |
| 22:28:57 | melwitt | I agree with you that we need coverage on this stuff | |
| 22:30:44 | melwitt | I think maybe the subset thing is only a problem if you get parallel requests often (tm) that all pick the same host and retry. we did so we increased it from 1 | |
| 22:31:19 | dansmith | sure, but that's a common complaint | |
| 22:31:32 | dansmith | that and running multiple schedulers choosing the same host | |
| 22:32:00 | melwitt | yeah. I guess no one thought of changing the default | |
| 22:32:13 | dansmith | I just can't think of a good reason to have the default be 1 | |
| 22:32:34 | dansmith | if it's larger it doesn't hurt low-traffic clouds, but being =1 does hurt high traffic ones by default | |
| 22:32:40 | mriedem | isn't this exactly what johnthetubaguy was saying this morning, why people don't use the filter scheduler with packing? | |
| 22:32:46 | dansmith | mriedem: yes | |
| 22:32:57 | mriedem | because we don't randomize anything | |
| 22:33:00 | dansmith | mriedem: but remember I said "but we randomize the result" | |
| 22:33:04 | melwitt | mriedem today mentioned a good point in a review somewhere I think (?) that it might be since people want to stack instances rather than spread them usually | |
| 22:33:08 | dansmith | mriedem: and he said "right, but at a certain level it doesn't matter" | |
| 22:33:15 | melwitt | well, I typed that way too slowly | |
| 22:34:19 | mriedem | melwitt: yes because then you can't place large instances | |
| 22:34:29 | dansmith | mriedem: for RAX-level, I can imagine it's not enough, but for medium sizes I would expect this works fine | |
| 22:34:49 | melwitt | yeah. well with subset = 1, that means no randomization. are you saying there's a different randomize somewhere? | |
| 22:34:49 | dansmith | mriedem: but we're not doing anyone favors by defaulting that to 1, IMHO | |
| 22:35:20 | dansmith | melwitt: no, I'm saying even with that on, it's not enough to satisfy rax's need for parallelism | |
| 22:35:33 | melwitt | oh | |
| 22:36:48 | dansmith | if you set it to infinity it defeats all your weighting right? so there's probably some inflection point where it starts hurting you | |
| 22:37:52 | melwitt | yeah, I'm not sure how that randomize plays with the weighing part | |
| 22:38:25 | dansmith | I think we get back an ordered list from the weighers, | |
| 22:38:36 | melwitt | unless it just shuffles the weighed order of N subset | |
| 22:38:38 | dansmith | and that subset size randomizes the N best-weighed hosts | |
| 22:38:52 | melwitt | yeah | |
| 22:38:59 | dansmith | so if you set it to N=$n_computes, you get no weighing | |
| 22:39:03 | melwitt | right | |
| 22:39:11 | dansmith | which I hear is bad and stuff | |
| 22:39:15 | melwitt | hah | |
| 22:44:11 | melwitt | so I guess a good-ish reason to default to 1 is because the actual good default will vary so much from deployment to deployment | |
| 22:44:59 | dansmith | I dunno, if people have less than say five candidate machines they're probably running a pretty damned small cloud | |
| 22:45:17 | dansmith | shooting for the middle and tweaking up or down depending on your size seems reasonable, | |
| 22:45:32 | dansmith | but yeah, someone will probably be confused with a two-node test cloud if it's !=1 | |
| 22:45:33 | melwitt | yeah, so I was thinking if you have 10 compute nodes and your subset defaulted to 2, then you could run into problems trying to schedule large instances? | |
| 22:45:34 | dansmith | just unfortunate | |
| 22:45:50 | dansmith | no the weighed hosts are all candidates | |