Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-15
21:08:07 bauzas mriedem: you want the scheduler to return you a specific destination ?
21:08:21 mriedem i have 2 hosts, i want the scheduler to send them in order
21:10:34 mriedem i think i can just use the disk weigher and fake it out
21:13:50 mriedem oh now i see a bug in the docs
21:15:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix the help for the disk_weight_multiplier option https://review.openstack.org/446177
21:16:31 bauzas mriedem: the scheduler is only returning one host, so 2 calls one after the other ?
21:17:21 mriedem bauzas: i need the test to make the scheduler put the server on the hosts in order, where the first host will fail during driver.spawn which would normally trigger a retry to the 2nd host
21:17:30 mriedem i've got 2 hosts in the test
21:17:45 mriedem i want to trick the scheduler into picking the hosts in order
21:17:52 mriedem so i really just need it to pick the bad one first
21:18:06 mriedem i guess maybe it doesn't matter which order as long as i can keep track of the first host that's called
21:18:18 mriedem i'll just do that
21:19:05 bauzas I see
21:21:32 bauzas mriedem: tbh, just asks the scheduler for one host, kill the spawn and then the reschedule will put that host in the retry list
21:22:02 bauzas in case retry doesn't work, then you'll get the first node
21:22:07 mriedem the scheduler will try to send it back to the same host?
21:22:40 bauzas yup, because the claim will be reversed
21:23:33 bauzas well, there is some random
21:23:52 mriedem ok, so i'm going to try and test this in the non-crazy way
21:24:02 mriedem which is if i have 2 hosts and the first fails, i expect it to go to the 2nd host
21:24:14 mriedem see _add_retry_host in the scheduler utils
21:24:19 mriedem that's why i thought it worked this way
21:24:41 bauzas you can trick the scheduler
21:24:43 mriedem ah RetryFilter
21:24:59 bauzas yeah
21:25:22 bauzas that's why we had NoValidHosts for a spawn issue
21:25:32 bauzas now, it's a separate exception
21:25:57 bauzas but yeah, if we retry and we don't have enough computes, then the RetryFilter will return 0 hosts
21:26:51 bauzas as I said, you can trick what the scheduler returns by avoiding random choice if host_subset_size opt is having a 1 value
21:27:48 bauzas which is the default
21:28:04 bauzas so, that's why the scheduler always returns the same compute
21:29:33 bauzas mriedem: FWIW, I'm tempted to add a section in https://etherpad.openstack.org/p/BOS-Nova-brainstorming saying 'add your name if you want to go to the Summit but still waits for your management" :p
21:30:06 mriedem bauzas: that kind of passive aggressiveness is reserved for midwesterners in the US
21:30:24 mriedem i'm also setting host_subset_size=2
21:30:29 mriedem i'm setting all the things
21:30:39 bauzas if so, you'd random between two hosts
21:31:15 mriedem but the RetryFilter, which is enabled by default, should filter out the first failed host
21:31:28 mriedem anyway, i'll just get this going and see
21:33:39 bauzas mriedem: sure but if we don't have the retry dict, then you could miss the problem
21:34:00 bauzas what you want to cover
21:47:20 edleafe jaypipes: want me to fix the pep8 issue on https://review.openstack.org/441544 ?
21:47:45 jaypipes edleafe: lol, pls do. :)
21:47:52 edleafe k
21:48:14 jaypipes edleafe: serves me right for not tox -epep8'ing.
21:49:39 openstackgerrit Ed Leafe proposed openstack/nova master: virt: implement get_inventory() for Ironic https://review.openstack.org/441544
21:49:47 edleafe jaypipes: fixed ^^
21:50:24 jaypipes cheers, I wallaby'd it.
22:02:33 openstackgerrit Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669
22:08:07 jaypipes mriedem, dansmith, melwitt: one of you want some review bait? :) https://review.openstack.org/#/c/444497/
22:14:55 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in admin_password.py https://review.openstack.org/443957
22:15:14 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208
22:15:25 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in aggregates.py https://review.openstack.org/445230
22:15:44 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in assisted_volume_snapshots.py https://review.openstack.org/445238
22:32:38 mriedem jaypipes: no more for me today
22:32:50 mriedem melwitt: here is that functional regression test for the retry bug ^
22:32:50 openstack bug 1671648 in OpenStack Compute (nova) "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Shunli Zhou (shunliz)
22:32:50 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add a functional regression/recreate test for bug 1671648 https://review.openstack.org/446209
22:35:40 melwitt mriedem: cool, will check it out
22:45:18 melwitt jaypipes: +W
23:12:17 mriedem1 aha, the chance scheduler doesn't set filter limits
23:12:22 mriedem1 b/c it doesn't do filtering
23:12:58 mriedem and the scheduler filter uses RequestSpec.from_primitives which defaults 'limits' to {}
23:13:06 melwitt mriedem: ah, right. need to self.flags for filterscheduler
23:13:28 mriedem i don't want to use the filter scheduler actually,
23:13:31 dansmith melwitt: were you going to split up the third quota patch at all?
23:13:36 mriedem i started with that and the default enabled filters kicked out my hosts
23:13:50 dansmith I've just been going back and forth between a real file and a test file for a while,
23:14:08 dansmith and I think it should be easy to cleave off some large related chunks purely by tagging files that go together
23:14:16 dansmith is there any reason to keep this monolithic?
23:14:41 melwitt no, other than I didn't do much thinking about how to logically split it into smaller chunks
23:14:58 melwitt I had a hell of a time just getting it to work, so it stayed monolithic
23:15:16 dansmith lemme see if I can split off some chunks that look obvious to me
23:16:06 melwitt thanks. suggestions on how to split it would be appreciated
23:17:09 melwitt mriedem: I thought I had seen an example of a func test that used filterscheduler but I'm not finding anything so far
23:17:55 melwitt oh, I was grepping for the wrong thing "FilterScheduler". it looks like this to configure self.flags(driver='filter_scheduler', group='scheduler')
23:20:50 melwitt and I just read back and saw you don't want the filter scheduler. nevermind
23:23:13 mriedem right, filter scheduler is the default
23:23:31 mriedem but it has a bunch of filters enabled by default, so i'd have to wittle that list of filters down to what works for this test, which might just be the RetryFilter,
23:23:39 mriedem i'm just trying to get the base things working
23:24:59 melwitt mriedem: well, you can self.flags the enabled filters too. so maybe just put RetryFilter in there by itself
23:25:22 mriedem yeah i thought about that,
23:25:23 melwitt if the bug is about something that RetryFilter needs to populate, I mean
23:25:34 melwitt if it's unrelated, then you don't need filter scheduler
23:25:36 mriedem RetryFilter doesn't populate the thing,
23:25:40 mriedem it checks the populated value
23:26:11 mriedem anyway, the functional test is now failing and going back and forth between hosts so i'm doing something else wrong here
23:26:16 mriedem with the fix i mean
23:26:19 mriedem so it's retrying, but failing
23:27:12 mriedem even though i'm using that flag to signal when we should fail and when we shouldn't
23:27:16 melwitt so the RetryFilter is what skips hosts that have already been attempted. so it seems like that would be your problem, right?
23:27:24 melwitt (I'm reading in the filter)
23:28:04 melwitt but that implies it would go on forever otherwise? that doesn't really make sense
23:33:15 mriedem i can try self.flags(enabled_filters=['RetryFilter'], group='filter_scheduler') and see what happens
23:36:49 mriedem yeah that makes it not attempt at all
23:36:56 melwitt I have a feeling that RetryFilter is required if scheduler_max_attempts > 1, otherwise it wouldn't know how to skip already tried hosts
23:37:29 melwitt and chance scheduler I think can't work with more than one compute host and scheduler_max_attempts > 1 maybe?
23:37:31 mriedem max_attempts defaults to 3
23:37:47 mriedem throwing the ComputeFilter into the list
23:37:51 dansmith melwitt: so, I just split the third patch into these: http://pastebin.com/raw/SXVqNSZm
23:37:52 mriedem since that's a pretty dumb one

Earlier   Later