| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-13 | |||
| 21:04:31 | dansmith | satyar: this is: https://review.openstack.org/#/c/443395/ | |
| 21:04:45 | mriedem | well, | |
| 21:04:54 | mriedem | on master it's that and https://review.openstack.org/#/c/443403/ | |
| 21:05:07 | mriedem | on ocata it should just be https://review.openstack.org/#/c/443395/ and i'm working on the backport but hitting some issues | |
| 21:06:02 | satyar | ohh ok will be testing it Thanks :) | |
| 21:06:25 | satyar | thanks dansmith and mriedem | |
| 21:08:57 | mriedem | satyar: i've marked the other bug as a duplicate | |
| 21:09:09 | satyar | to? | |
| 21:09:42 | satyar | ok got it | |
| 21:09:45 | openstack | Launchpad bug 1670627 in OpenStack Compute (nova) ocata "quota is always in-use after delete the ERROR instances " [Critical,Confirmed] | |
| 21:09:45 | mriedem | https://bugs.launchpad.net/nova/+bug/1670627 | |
| 21:10:26 | openstackgerrit | Dan Smith proposed openstack/nova master: Add SameCellFilter https://review.openstack.org/445035 | |
| 21:11:32 | satyar | ideally it should be opposite that bug should be duplicate of the first one ;) but its fine | |
| 21:11:59 | satyar | aslong as we fix the issue :) | |
| 21:12:04 | mriedem | satyar: yeah, i guess triage failed us, it went under the one it did because someone showed up in the channel last wednesday and brought it up | |
| 21:12:17 | mriedem | right place right time | |
| 21:12:21 | satyar | :P | |
| 21:23:23 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova-specs master: Live Migration of Rescued Instances https://review.openstack.org/347161 | |
| 21:28:43 | openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208 | |
| 21:33:06 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: Reset instance status on invalid host https://review.openstack.org/401009 | |
| 21:35:25 | mriedem | wtf, in ocata, when the instance is in cell0 and i go to delete it, i get a 404, but when i do a GET i get a response | |
| 21:43:27 | melwitt | mriedem: are you using novaclient to delete it or the REST API directly? because I feel like I have seen that before when I had bugs in my own code because of how novaclient does a GET before doing the delete. I can't remember the details off the top of my head | |
| 21:46:34 | mriedem | melwitt: rest api, this is that functional regression test for the quota patch | |
| 21:46:54 | mriedem | https://review.openstack.org/#/c/443395/ but on ocata | |
| 21:47:15 | mriedem | the functional regression test fails waiting for the server to be gone after the delete request, and it's because the delete request fails with a 404 when looking up the instance.flavor | |
| 21:47:17 | melwitt | oh, hm | |
| 21:47:45 | mriedem | https://gist.github.com/mriedem/cece80e5ab375c2fad6d9d599851c2dc | |
| 21:48:19 | mriedem | i think i know why, | |
| 21:48:27 | mriedem | in master the context is targeted when you get the instance | |
| 21:48:31 | mriedem | in ocata it's not, | |
| 21:48:46 | mriedem | so in this change when it's looking up the flavor from the instance, the context isn't targeted to cell0 | |
| 21:49:29 | mriedem | but we want the quotas in the main cell db, | |
| 21:49:49 | mriedem | ugh | |
| 21:49:49 | mriedem | so i'm going to have to do the same helper method that i had on master for getting the flavor from a targeted context before creating the reservation | |
| 21:50:45 | openstackgerrit | Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842 | |
| 21:58:26 | dansmith | mriedem: how do we not trip over that in tempest? | |
| 21:58:34 | mriedem | dansmith: it's for this patch, | |
| 21:58:38 | mriedem | as a result of this patch i mean | |
| 21:58:46 | mriedem | i've got it sorted out, it's just kind of a mess for the backport, | |
| 21:58:52 | mriedem | since it's a mix of the 2 changes on master | |
| 21:59:02 | mriedem | you'll see shortly | |
| 21:59:03 | dansmith | oh, I see I thought you meant the regression test doesn't run on ocata before the change | |
| 22:00:18 | openstackgerrit | melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | |
| 22:00:18 | openstackgerrit | melanie witt proposed openstack/nova master: Count resources to check quota for cells https://review.openstack.org/416521 | |
| 22:00:19 | openstackgerrit | melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | |
| 22:01:26 | mriedem | i should have probably backported this first to see how it would work | |
| 22:04:32 | mriedem | looks like we have another ocata regression https://review.openstack.org/#/c/444106/ | |
| 22:10:02 | dansmith | hmm, I thought we did that _on_ retry | |
| 22:11:59 | dansmith | populate_filter_properties() creates the retry of the selected host | |
| 22:12:02 | dansmith | which we are calling already | |
| 22:12:18 | dansmith | oh | |
| 22:12:30 | dansmith | only if populate_retry has done the initial work first I guess | |
| 22:12:31 | dansmith | huh | |
| 22:14:26 | mriedem | gd how many postgres packages do i really need to install just to run pep8 | |
| 22:15:14 | mriedem | sudo apt-get install -y (*pg*|*pq*) | |
| 22:15:21 | mriedem | install -y *everything* | |
| 22:16:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add SameCellFilter https://review.openstack.org/445035 | |
| 22:20:04 | mriedem | we should have a functional regression test for that retry fix, | |
| 22:21:01 | mriedem | should be relatively simple, run 2 nova-compute services, make the scheduler pick the first one and make that one fail to trigger a retry, and then make sure it's actually retried to the 2nd host | |
| 22:21:54 | dansmith | is that enough? it might do that by itself by chance | |
| 22:22:53 | openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in aggregates.py https://review.openstack.org/445230 | |
| 22:23:18 | openstackgerrit | Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842 | |
| 22:23:26 | openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208 | |
| 22:23:47 | mriedem | dansmith: how? | |
| 22:23:53 | mriedem | w/o digging into details i guess i don't know | |
| 22:24:11 | mriedem | i'd also like to be able to force the scheduler to pick a host in the test that we've setup to fail | |
| 22:24:28 | mriedem | i'm not entirely sure how to do that in functional tests w/o stubbing something out | |
| 22:24:29 | dansmith | if you asked the scheduler to pick what would have been its second choice and it retries, it'd choose the other one without this fix | |
| 22:24:35 | dansmith | right exactly | |
| 22:24:45 | mriedem | yeah we have to make the test pick the hosts in the order we need for the recreate | |
| 22:24:49 | mriedem | that's the tricky part, | |
| 22:24:54 | mriedem | but doesn't seem too hard | |
| 22:25:15 | dansmith | don't we shuffle the results? | |
| 22:25:16 | mriedem | i'd think a weigher could be used | |
| 22:25:32 | dansmith | yeah maybe | |
| 22:25:44 | mriedem | that's scheduler black magic to me though | |
| 22:26:08 | dansmith | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L124-L124 | |
| 22:26:24 | mriedem | i don't want to make a big deal out of it, but this is a complicated piece of code and we clearly don't have it covered sufficiently in tests | |
| 22:26:32 | dansmith | you'd have to set the subset size to 1 | |
| 22:26:48 | melwitt | it defaults to 1, so no shuffle by default | |
| 22:26:57 | dansmith | okay | |
| 22:27:12 | mriedem | are either of you interested in writing that functional test? :) | |
| 22:27:13 | dansmith | that seems like an unfortunate default for regular people | |
| 22:27:30 | mriedem | it's only random if the weights are equal isn't it? | |
| 22:28:06 | dansmith | wait, | |
| 22:28:14 | dansmith | oh, I see | |
| 22:28:42 | dansmith | huh, I never knew this picked a single host by default | |
| 22:28:44 | melwitt | mriedem: I can give it a go. I've written a weird retry test before https://github.com/openstack/nova/blob/master/nova/tests/functional/test_compute_mgr.py#L40 | |
| 22:28:57 | melwitt | I agree with you that we need coverage on this stuff | |
| 22:30:44 | melwitt | I think maybe the subset thing is only a problem if you get parallel requests often (tm) that all pick the same host and retry. we did so we increased it from 1 | |
| 22:31:19 | dansmith | sure, but that's a common complaint | |
| 22:31:32 | dansmith | that and running multiple schedulers choosing the same host | |
| 22:32:00 | melwitt | yeah. I guess no one thought of changing the default | |
| 22:32:13 | dansmith | I just can't think of a good reason to have the default be 1 | |
| 22:32:34 | dansmith | if it's larger it doesn't hurt low-traffic clouds, but being =1 does hurt high traffic ones by default | |
| 22:32:40 | mriedem | isn't this exactly what johnthetubaguy was saying this morning, why people don't use the filter scheduler with packing? | |
| 22:32:46 | dansmith | mriedem: yes | |
| 22:32:57 | mriedem | because we don't randomize anything | |
| 22:33:00 | dansmith | mriedem: but remember I said "but we randomize the result" | |
| 22:33:04 | melwitt | mriedem today mentioned a good point in a review somewhere I think (?) that it might be since people want to stack instances rather than spread them usually | |