| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-21 | |||
| 13:55:41 | bauzas | EmilienM: is the above the undercloud nova.conf ? | |
| 13:55:56 | mriedem | bauzas: jroll's fix isn't any different from how it worked before get_inventory was added | |
| 13:55:59 | mriedem | unless i'm missing something | |
| 13:56:07 | mriedem | so if you're worried about an allocation bug, it's latent | |
| 13:56:34 | bauzas | mriedem: jroll's fix tries to address the fact that operators could ask the scheduler to return hosts that have more space than asked | |
| 13:56:53 | jroll | mriedem: hm, you might be right | |
| 13:56:53 | EmilienM | bauzas: yes | |
| 13:57:08 | bauzas | okay, so the problem is there | |
| 13:57:23 | bauzas | lemme restate | |
| 13:57:48 | mriedem | jaypipes: in case you haven't seen yet, jroll found the cause of the bug with the get_inventory stuff | |
| 13:57:55 | bauzas | Ironic pre-jroll's fix is only accepting the default config (ie. having Exact* filters) | |
| 13:57:59 | jaypipes | mriedem: sorry, meeting hell... | |
| 13:58:06 | jaypipes | mriedem: what was the cause, jroll? | |
| 13:58:16 | mriedem | min_unit was equal to max_unit | |
| 13:58:19 | bauzas | but TripleO isn't using those specific filtrers | |
| 13:58:22 | mriedem | plus bm filters | |
| 13:58:22 | jroll | jaypipes: min_unit==max_unit, tripleo uses flavors smaller than the node | |
| 13:58:29 | EmilienM | scheduler_use_baremetal_filters is set to False by default in puppet-nova, I believe it's the default in Nova, but let me check | |
| 13:58:33 | jroll | mriedem: not related to baremetal filters | |
| 13:58:36 | jroll | EmilienM: that's correct | |
| 13:58:44 | jaypipes | jroll: ahhh. | |
| 13:59:00 | jaypipes | jroll: good. we can blame edleafe then. :P | |
| 13:59:00 | bauzas | so it hits the bug that jroll tries to fix, but which shouldn't be visible unless you modify the specific default values for the list of baremetal filters | |
| 13:59:01 | EmilienM | right https://git.openstack.org/cgit/openstack/nova/tree/nova/conf/scheduler.py#n333 | |
| 13:59:17 | EmilienM | (i'm unavailable the next hour, tripleo meeting) | |
| 13:59:38 | tdurakov | hi folks, live-migration meeting in a minute, on #openstack-meeting-3 | |
| 14:00:14 | bauzas | so, I would suggest at least for TripleO to set scheduler_use_baremetal_filters to True for the undercloud | |
| 14:00:21 | jroll | bauzas: or don't enable baremetal filters at all | |
| 14:00:30 | jroll | bauzas: well, AIUI, tripleo does this intentionally | |
| 14:00:41 | mriedem | i didn't even know baremetal_scheduler_enabled_filters was a thing | |
| 14:00:47 | jroll | bauzas: tripleo uses capabilities hacks to schedule whatever the hell it wants | |
| 14:00:55 | mriedem | or scheduler_use_baremetal_filters | |
| 14:01:12 | bauzas | jroll: well, if you don't enable those exact* filters, then you could end up picking a host that has free space | |
| 14:01:16 | jroll | mriedem: yeah, it's a shortcut to make deployers lives easier, I know that's usually not a goal | |
| 14:01:27 | jroll | bauzas: which is true before this change, as well | |
| 14:01:37 | jroll | bauzas: before the get_inventory change, I mean | |
| 14:01:51 | bauzas | jroll: having the placement returning those hosts wasn't a problem because the scheduler was only returning the right hosts if Exact* filters were present | |
| 14:02:03 | mriedem | jroll: just seems odd we need config options to say do this for bm scheduling, when we already know if we're doing ironic b/c of the host_manager | |
| 14:02:11 | jroll | bauzas: fundamentally I agree with you, but this is a change to tripleo's architecture :P | |
| 14:02:15 | bauzas | mriedem: it's a hack | |
| 14:02:16 | openstackgerrit | Béla Vancsics proposed openstack/nova master: Using max api version in notificationt sample test https://review.openstack.org/443686 | |
| 14:02:17 | edleafe | jaypipes: you can always blame me | |
| 14:02:25 | bauzas | a very long-lived hack | |
| 14:02:44 | bauzas | I guess only a few of us were knowing that :/ | |
| 14:02:51 | jroll | mriedem: idk, tripleo and yahoo very much drive a "things don't need to be exact" use case :/ | |
| 14:03:18 | jroll | I'm told many deployments of ironic manage the compute graveyard where every machine is a different spec | |
| 14:03:20 | jroll | :| | |
| 14:03:37 | bauzas | jroll: here is the thing, eithezr tripleo switches their conf, or we need to both merge your fix *and* consume all the space when allocating | |
| 14:03:50 | bauzas | the former should be quick | |
| 14:03:56 | bauzas | the latter could take a while | |
| 14:04:02 | jroll | bauzas: slow down | |
| 14:04:51 | jroll | bauzas: tripleo was not using exact filters before the get_inventory change | |
| 14:05:02 | jroll | bauzas: we also had broken allocations before that change | |
| 14:05:27 | jroll | bauzas: so with this fix only, we assume they won't be broken (and I believe their flavor uses all of the RAM, which is why) | |
| 14:05:50 | jroll | anyone else not using exact filters was already broken, and will continue to be broken | |
| 14:05:52 | bauzas | *tripleo* won't be broken | |
| 14:06:16 | bauzas | but people using ironic the normal way will be broken if we merge your fix without the allocation side | |
| 14:06:30 | jroll | yes, the same way they were broken on ocata, newton, mitaka, etc | |
| 14:06:52 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Ironic: hardcode min_unit for standard resources to 1 https://review.openstack.org/448098 | |
| 14:06:59 | jroll | ^ with func test fixes | |
| 14:07:20 | bauzas | jroll: they were broken in what sense ? | |
| 14:07:43 | bauzas | the fact that we were underperformant with scheduling, leading to races ? | |
| 14:07:44 | jroll | bauzas: the same way. we weren't allocating all of the resources | |
| 14:07:52 | bauzas | wait | |
| 14:08:07 | bauzas | we weren't allocating all the resources ? I'm surprised | |
| 14:08:22 | jroll | we were not. because people didn't want ironic hacks in there. | |
| 14:08:26 | jroll | (AIUI, of course) | |
| 14:08:38 | jroll | and when people complained, we said "don't be dumb, use the exact filters" | |
| 14:08:40 | bauzas | sec, verifying the RT code and the claims | |
| 14:08:49 | jroll | please do, you know that code better than I do | |
| 14:09:10 | jroll | ironic reports everything used in get_available_resource, but the RT corrects it based on instance.flavor | |
| 14:09:56 | bauzas | jroll: self.driver.estimate_instance_overhead(instance) | |
| 14:10:22 | bauzas | mmm, nevermind | |
| 14:10:29 | bauzas | that one is not what I thought | |
| 14:11:35 | bauzas | looking at the limits now | |
| 14:13:09 | bauzas | yeah bingo | |
| 14:13:41 | bauzas | jroll: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/exact_ram_filter.py#L38-L41 | |
| 14:15:01 | bauzas | jroll: which ends up in https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L247 | |
| 14:15:06 | bauzas | on the compute side | |
| 14:15:37 | jroll | bauzas: neat, but that means people not running exact filters were still broken before this change | |
| 14:15:49 | jroll | bauzas: and means people running exact filters are still not broken after this change | |
| 14:16:05 | bauzas | jroll: so the filter returns a limit that is passed down to the conductor and then the compute which is used there https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L155-L156 | |
| 14:16:13 | jroll | meaning, my change does not introduce more broken behavior, just gets us back to consistent with old behavior | |
| 14:16:21 | mriedem | rfolco: congratulations you are the first pike bp implemented https://blueprints.launchpad.net/nova/+spec/delete-inventories-placement-api | |
| 14:16:31 | mriedem | you get a root beer flavored scratch and sniff sticker | |
| 14:16:38 | bauzas | jroll: lemme consider the impact | |
| 14:16:59 | mriedem | jroll: that's what i said earlier right? | |
| 14:17:11 | mriedem | your patch makes get_inventory the same as it was working for everything before | |
| 14:17:28 | mriedem | the "consume all allocation" is a latent problem | |
| 14:17:32 | mriedem | not introduced by get_inventory | |
| 14:17:39 | bauzas | mriedem: post-placement use yeah | |
| 14:17:48 | cdent | mriedem: I am _so_ motivated now, now that I know there might be scratch and sniff stickers | |
| 14:17:55 | jroll | mriedem: yes, I agree with you | |
| 14:18:09 | bauzas | okay, jroll and mriedem, you're right | |
| 14:18:11 | edleafe | rfolco: w00t! | |
| 14:18:28 | jroll | zomg, that never happens | |
| 14:18:38 | bauzas | we just leave placement returning more hosts | |
| 14:19:00 | bauzas | the allocation consumption problem is only there when you don't use the Exact | |
| 14:19:04 | bauzas | Exact* filters | |
| 14:19:09 | bauzas | okay, so... | |