Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-15
20:58:55 mriedem so, i'm going to just start with a code comment in the request spec object
20:59:04 mriedem so when i look at this in a year, i won't have to do all of the detective work again
20:59:59 bauzas fine by me
21:04:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a note about force_hosts only ever having a single value https://review.openstack.org/446175
21:04:42 mriedem there you go ^
21:04:49 openstackgerrit Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393
21:06:09 mriedem ok, well, back to figuring out how i can mock the scheduler to send build requests to certain hosts in order
21:07:28 bauzas melwitt: https://review.openstack.org/#/c/446175/1 could be worth of interest
21:07:51 bauzas mriedem: what's your concern ?
21:08:06 mriedem bauzas: i'm writing a functional test to recreate https://review.openstack.org/#/c/444106/
21:08:07 bauzas mriedem: you want the scheduler to return you a specific destination ?
21:08:21 mriedem i have 2 hosts, i want the scheduler to send them in order
21:10:34 mriedem i think i can just use the disk weigher and fake it out
21:13:50 mriedem oh now i see a bug in the docs
21:15:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix the help for the disk_weight_multiplier option https://review.openstack.org/446177
21:16:31 bauzas mriedem: the scheduler is only returning one host, so 2 calls one after the other ?
21:17:21 mriedem bauzas: i need the test to make the scheduler put the server on the hosts in order, where the first host will fail during driver.spawn which would normally trigger a retry to the 2nd host
21:17:30 mriedem i've got 2 hosts in the test
21:17:45 mriedem i want to trick the scheduler into picking the hosts in order
21:17:52 mriedem so i really just need it to pick the bad one first
21:18:06 mriedem i guess maybe it doesn't matter which order as long as i can keep track of the first host that's called
21:18:18 mriedem i'll just do that
21:19:05 bauzas I see
21:21:32 bauzas mriedem: tbh, just asks the scheduler for one host, kill the spawn and then the reschedule will put that host in the retry list
21:22:02 bauzas in case retry doesn't work, then you'll get the first node
21:22:07 mriedem the scheduler will try to send it back to the same host?
21:22:40 bauzas yup, because the claim will be reversed
21:23:33 bauzas well, there is some random
21:23:52 mriedem ok, so i'm going to try and test this in the non-crazy way
21:24:02 mriedem which is if i have 2 hosts and the first fails, i expect it to go to the 2nd host
21:24:14 mriedem see _add_retry_host in the scheduler utils
21:24:19 mriedem that's why i thought it worked this way
21:24:41 bauzas you can trick the scheduler
21:24:43 mriedem ah RetryFilter
21:24:59 bauzas yeah
21:25:22 bauzas that's why we had NoValidHosts for a spawn issue
21:25:32 bauzas now, it's a separate exception
21:25:57 bauzas but yeah, if we retry and we don't have enough computes, then the RetryFilter will return 0 hosts
21:26:51 bauzas as I said, you can trick what the scheduler returns by avoiding random choice if host_subset_size opt is having a 1 value
21:27:48 bauzas which is the default
21:28:04 bauzas so, that's why the scheduler always returns the same compute
21:29:33 bauzas mriedem: FWIW, I'm tempted to add a section in https://etherpad.openstack.org/p/BOS-Nova-brainstorming saying 'add your name if you want to go to the Summit but still waits for your management" :p
21:30:06 mriedem bauzas: that kind of passive aggressiveness is reserved for midwesterners in the US
21:30:24 mriedem i'm also setting host_subset_size=2
21:30:29 mriedem i'm setting all the things
21:30:39 bauzas if so, you'd random between two hosts
21:31:15 mriedem but the RetryFilter, which is enabled by default, should filter out the first failed host
21:31:28 mriedem anyway, i'll just get this going and see
21:33:39 bauzas mriedem: sure but if we don't have the retry dict, then you could miss the problem
21:34:00 bauzas what you want to cover
21:47:20 edleafe jaypipes: want me to fix the pep8 issue on https://review.openstack.org/441544 ?
21:47:45 jaypipes edleafe: lol, pls do. :)
21:47:52 edleafe k
21:48:14 jaypipes edleafe: serves me right for not tox -epep8'ing.
21:49:39 openstackgerrit Ed Leafe proposed openstack/nova master: virt: implement get_inventory() for Ironic https://review.openstack.org/441544
21:49:47 edleafe jaypipes: fixed ^^
21:50:24 jaypipes cheers, I wallaby'd it.
22:02:33 openstackgerrit Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669
22:08:07 jaypipes mriedem, dansmith, melwitt: one of you want some review bait? :) https://review.openstack.org/#/c/444497/
22:14:55 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in admin_password.py https://review.openstack.org/443957
22:15:14 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208
22:15:25 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in aggregates.py https://review.openstack.org/445230
22:15:44 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in assisted_volume_snapshots.py https://review.openstack.org/445238
22:32:38 mriedem jaypipes: no more for me today
22:32:50 mriedem melwitt: here is that functional regression test for the retry bug ^
22:32:50 openstack bug 1671648 in OpenStack Compute (nova) "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Shunli Zhou (shunliz)
22:32:50 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add a functional regression/recreate test for bug 1671648 https://review.openstack.org/446209
22:35:40 melwitt mriedem: cool, will check it out
22:45:18 melwitt jaypipes: +W
23:12:17 mriedem1 aha, the chance scheduler doesn't set filter limits
23:12:22 mriedem1 b/c it doesn't do filtering
23:12:58 mriedem and the scheduler filter uses RequestSpec.from_primitives which defaults 'limits' to {}
23:13:06 melwitt mriedem: ah, right. need to self.flags for filterscheduler
23:13:28 mriedem i don't want to use the filter scheduler actually,
23:13:31 dansmith melwitt: were you going to split up the third quota patch at all?
23:13:36 mriedem i started with that and the default enabled filters kicked out my hosts
23:13:50 dansmith I've just been going back and forth between a real file and a test file for a while,
23:14:08 dansmith and I think it should be easy to cleave off some large related chunks purely by tagging files that go together
23:14:16 dansmith is there any reason to keep this monolithic?
23:14:41 melwitt no, other than I didn't do much thinking about how to logically split it into smaller chunks
23:14:58 melwitt I had a hell of a time just getting it to work, so it stayed monolithic
23:15:16 dansmith lemme see if I can split off some chunks that look obvious to me
23:16:06 melwitt thanks. suggestions on how to split it would be appreciated
23:17:09 melwitt mriedem: I thought I had seen an example of a func test that used filterscheduler but I'm not finding anything so far
23:17:55 melwitt oh, I was grepping for the wrong thing "FilterScheduler". it looks like this to configure self.flags(driver='filter_scheduler', group='scheduler')
23:20:50 melwitt and I just read back and saw you don't want the filter scheduler. nevermind
23:23:13 mriedem right, filter scheduler is the default
23:23:31 mriedem but it has a bunch of filters enabled by default, so i'd have to wittle that list of filters down to what works for this test, which might just be the RetryFilter,
23:23:39 mriedem i'm just trying to get the base things working
23:24:59 melwitt mriedem: well, you can self.flags the enabled filters too. so maybe just put RetryFilter in there by itself
23:25:22 mriedem yeah i thought about that,
23:25:23 melwitt if the bug is about something that RetryFilter needs to populate, I mean
23:25:34 melwitt if it's unrelated, then you don't need filter scheduler
23:25:36 mriedem RetryFilter doesn't populate the thing,
23:25:40 mriedem it checks the populated value
23:26:11 mriedem anyway, the functional test is now failing and going back and forth between hosts so i'm doing something else wrong here
23:26:16 mriedem with the fix i mean
23:26:19 mriedem so it's retrying, but failing
23:27:12 mriedem even though i'm using that flag to signal when we should fail and when we shouldn't
23:27:16 melwitt so the RetryFilter is what skips hosts that have already been attempted. so it seems like that would be your problem, right?

Earlier   Later