| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-21 | |||
| 13:50:07 | bauzas | jroll: so the CI could be happy tho | |
| 13:50:24 | jroll | bauzas: let me be clear, I don't want this merged without fixing allocations as well | |
| 13:50:36 | bauzas | a-ah ok, then I agree | |
| 13:50:46 | jroll | bauzas: and if tripleo can boot at least one instance, it's progress | |
| 13:50:55 | jroll | or rather proves that was the issue | |
| 13:51:24 | bauzas | jroll: AFAIK, the Exact* filters are not default in config right? | |
| 13:51:32 | bauzas | jroll: speaking of the IronicHostManager config | |
| 13:51:39 | bauzas | I can verify | |
| 13:51:48 | bauzas | but I'm pretty sure TripleO hasn't modified that | |
| 13:52:12 | jroll | bauzas: it is a default, I think, if CONF.filter_scheduler.use_baremetal_filters=true | |
| 13:52:35 | bauzas | jroll: I know we don't have the same enabled filters if that conf opt is in use, yep | |
| 13:52:49 | bauzas | jroll: I just don't remember if the Exact* filters are part of those | |
| 13:52:56 | jroll | yeah, confirmed, it isn't default but is part of the baremetal set | |
| 13:53:04 | jroll | conf/scheduler.py line 298 | |
| 13:54:20 | bauzas | jroll: so then the scheduler should get the exact same number of nodes with or without that fix you wrote | |
| 13:54:32 | bauzas | oops | |
| 13:54:41 | bauzas | the scheduler should *return* | |
| 13:55:00 | bauzas | but lemme see the 3O config | |
| 13:55:17 | jroll | bauzas: http://logs.openstack.org/22/447522/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7563947/logs/etc/nova/nova.conf.txt.gz | |
| 13:55:20 | bauzas | yup | |
| 13:55:21 | jroll | use_baremetal_filters=False | |
| 13:55:41 | bauzas | EmilienM: is the above the undercloud nova.conf ? | |
| 13:55:56 | mriedem | bauzas: jroll's fix isn't any different from how it worked before get_inventory was added | |
| 13:55:59 | mriedem | unless i'm missing something | |
| 13:56:07 | mriedem | so if you're worried about an allocation bug, it's latent | |
| 13:56:34 | bauzas | mriedem: jroll's fix tries to address the fact that operators could ask the scheduler to return hosts that have more space than asked | |
| 13:56:53 | jroll | mriedem: hm, you might be right | |
| 13:56:53 | EmilienM | bauzas: yes | |
| 13:57:08 | bauzas | okay, so the problem is there | |
| 13:57:23 | bauzas | lemme restate | |
| 13:57:48 | mriedem | jaypipes: in case you haven't seen yet, jroll found the cause of the bug with the get_inventory stuff | |
| 13:57:55 | bauzas | Ironic pre-jroll's fix is only accepting the default config (ie. having Exact* filters) | |
| 13:57:59 | jaypipes | mriedem: sorry, meeting hell... | |
| 13:58:06 | jaypipes | mriedem: what was the cause, jroll? | |
| 13:58:16 | mriedem | min_unit was equal to max_unit | |
| 13:58:19 | bauzas | but TripleO isn't using those specific filtrers | |
| 13:58:22 | mriedem | plus bm filters | |
| 13:58:22 | jroll | jaypipes: min_unit==max_unit, tripleo uses flavors smaller than the node | |
| 13:58:29 | EmilienM | scheduler_use_baremetal_filters is set to False by default in puppet-nova, I believe it's the default in Nova, but let me check | |
| 13:58:33 | jroll | mriedem: not related to baremetal filters | |
| 13:58:36 | jroll | EmilienM: that's correct | |
| 13:58:44 | jaypipes | jroll: ahhh. | |
| 13:59:00 | jaypipes | jroll: good. we can blame edleafe then. :P | |
| 13:59:00 | bauzas | so it hits the bug that jroll tries to fix, but which shouldn't be visible unless you modify the specific default values for the list of baremetal filters | |
| 13:59:01 | EmilienM | right https://git.openstack.org/cgit/openstack/nova/tree/nova/conf/scheduler.py#n333 | |
| 13:59:17 | EmilienM | (i'm unavailable the next hour, tripleo meeting) | |
| 13:59:38 | tdurakov | hi folks, live-migration meeting in a minute, on #openstack-meeting-3 | |
| 14:00:14 | bauzas | so, I would suggest at least for TripleO to set scheduler_use_baremetal_filters to True for the undercloud | |
| 14:00:21 | jroll | bauzas: or don't enable baremetal filters at all | |
| 14:00:30 | jroll | bauzas: well, AIUI, tripleo does this intentionally | |
| 14:00:41 | mriedem | i didn't even know baremetal_scheduler_enabled_filters was a thing | |
| 14:00:47 | jroll | bauzas: tripleo uses capabilities hacks to schedule whatever the hell it wants | |
| 14:00:55 | mriedem | or scheduler_use_baremetal_filters | |
| 14:01:12 | bauzas | jroll: well, if you don't enable those exact* filters, then you could end up picking a host that has free space | |
| 14:01:16 | jroll | mriedem: yeah, it's a shortcut to make deployers lives easier, I know that's usually not a goal | |
| 14:01:27 | jroll | bauzas: which is true before this change, as well | |
| 14:01:37 | jroll | bauzas: before the get_inventory change, I mean | |
| 14:01:51 | bauzas | jroll: having the placement returning those hosts wasn't a problem because the scheduler was only returning the right hosts if Exact* filters were present | |
| 14:02:03 | mriedem | jroll: just seems odd we need config options to say do this for bm scheduling, when we already know if we're doing ironic b/c of the host_manager | |
| 14:02:11 | jroll | bauzas: fundamentally I agree with you, but this is a change to tripleo's architecture :P | |
| 14:02:15 | bauzas | mriedem: it's a hack | |
| 14:02:16 | openstackgerrit | Béla Vancsics proposed openstack/nova master: Using max api version in notificationt sample test https://review.openstack.org/443686 | |
| 14:02:17 | edleafe | jaypipes: you can always blame me | |
| 14:02:25 | bauzas | a very long-lived hack | |
| 14:02:44 | bauzas | I guess only a few of us were knowing that :/ | |
| 14:02:51 | jroll | mriedem: idk, tripleo and yahoo very much drive a "things don't need to be exact" use case :/ | |
| 14:03:18 | jroll | I'm told many deployments of ironic manage the compute graveyard where every machine is a different spec | |
| 14:03:20 | jroll | :| | |
| 14:03:37 | bauzas | jroll: here is the thing, eithezr tripleo switches their conf, or we need to both merge your fix *and* consume all the space when allocating | |
| 14:03:50 | bauzas | the former should be quick | |
| 14:03:56 | bauzas | the latter could take a while | |
| 14:04:02 | jroll | bauzas: slow down | |
| 14:04:51 | jroll | bauzas: tripleo was not using exact filters before the get_inventory change | |
| 14:05:02 | jroll | bauzas: we also had broken allocations before that change | |
| 14:05:27 | jroll | bauzas: so with this fix only, we assume they won't be broken (and I believe their flavor uses all of the RAM, which is why) | |
| 14:05:50 | jroll | anyone else not using exact filters was already broken, and will continue to be broken | |
| 14:05:52 | bauzas | *tripleo* won't be broken | |
| 14:06:16 | bauzas | but people using ironic the normal way will be broken if we merge your fix without the allocation side | |
| 14:06:30 | jroll | yes, the same way they were broken on ocata, newton, mitaka, etc | |
| 14:06:52 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Ironic: hardcode min_unit for standard resources to 1 https://review.openstack.org/448098 | |
| 14:06:59 | jroll | ^ with func test fixes | |
| 14:07:20 | bauzas | jroll: they were broken in what sense ? | |
| 14:07:43 | bauzas | the fact that we were underperformant with scheduling, leading to races ? | |
| 14:07:44 | jroll | bauzas: the same way. we weren't allocating all of the resources | |
| 14:07:52 | bauzas | wait | |
| 14:08:07 | bauzas | we weren't allocating all the resources ? I'm surprised | |
| 14:08:22 | jroll | we were not. because people didn't want ironic hacks in there. | |
| 14:08:26 | jroll | (AIUI, of course) | |
| 14:08:38 | jroll | and when people complained, we said "don't be dumb, use the exact filters" | |
| 14:08:40 | bauzas | sec, verifying the RT code and the claims | |
| 14:08:49 | jroll | please do, you know that code better than I do | |
| 14:09:10 | jroll | ironic reports everything used in get_available_resource, but the RT corrects it based on instance.flavor | |
| 14:09:56 | bauzas | jroll: self.driver.estimate_instance_overhead(instance) | |
| 14:10:22 | bauzas | mmm, nevermind | |
| 14:10:29 | bauzas | that one is not what I thought | |
| 14:11:35 | bauzas | looking at the limits now | |
| 14:13:09 | bauzas | yeah bingo | |
| 14:13:41 | bauzas | jroll: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/exact_ram_filter.py#L38-L41 | |
| 14:15:01 | bauzas | jroll: which ends up in https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L247 | |
| 14:15:06 | bauzas | on the compute side | |