| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-11 | |||
| 20:16:52 | dansmith | so before we were doing allocations wrong, we were doing RT claiming wrong and failing super late | |
| 20:16:53 | mriedem | RT == claim? | |
| 20:16:56 | mriedem | well, | |
| 20:16:58 | dansmith | I mean old school claims | |
| 20:17:02 | mriedem | if you dropped the DiskFilter you wouldn't hit the claim | |
| 20:17:06 | dansmith | pfft :) | |
| 20:17:09 | mriedem | so in ocata, if you're using placement and remove DiskFilter, | |
| 20:17:10 | mriedem | fixed | |
| 20:17:39 | dansmith | where did we not allocate for volume-backed disks in RT? | |
| 20:18:44 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L169 | |
| 20:18:53 | mriedem | this comment in the bug is a pretty good one https://bugs.launchpad.net/nova/+bug/1469179/comments/37 | |
| 20:18:55 | openstack | Launchpad bug 1469179 in OpenStack Compute (nova) "instance.root_gb should be 0 for volume-backed instances" [Medium,In progress] - Assigned to Dan Smith (danms) | |
| 20:19:02 | mriedem | "After disabling the RamFilter and DiskFilter, and instead relying on the Placement API for scheduling, most of the "scheduling" aspect of this problem has been addressed. I was able to drop a local patch to handle this root_gb!=0 for EBS volumes, and for most real-life use cases it is working. The placement API is not recording disk allocations for EBS volumes." | |
| 20:19:44 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L351 | |
| 20:20:07 | dansmith | so it kinda worked almost by accident for a release? | |
| 20:20:12 | mriedem | so in ocata we wouldn't report root_gb for bfv instance allocations, you could still fail scheduling due to the old school RT disk claim if DiskFilter was enabled | |
| 20:20:40 | mriedem | i agree with the 'by accident' part | |
| 20:21:05 | mriedem | and i don't think it was until at least pike that we said to disable the DiskFilter if you're using placement and remove it from the default enabled filters | |
| 20:21:33 | dansmith | right, I was thinking ocata still required that | |
| 20:21:34 | dansmith | well, | |
| 20:21:41 | dansmith | didn't recommend placement as a complete replacement | |
| 20:22:46 | mriedem | yeah dropped CPU/Ram/DiskFilters from default enabled filters in pike | |
| 20:23:26 | mriedem | we'd been saying since at least newton that placement was the eventual fix for this problem :) that's why we kept putting off melwitt's workaround / RT patch | |
| 20:23:35 | mriedem | and sure now it is | |
| 20:23:46 | dansmith | aye | |
| 20:24:12 | dansmith | well, to be fair, we could have done this in queens at least too | |
| 20:24:32 | dansmith | meaning, it could have been the fix already | |
| 20:24:36 | mriedem | sure | |
| 20:24:51 | mriedem | and people could hack in a point-of-scheduling time fix in conductor if they need it.. | |
| 20:27:32 | melwitt | does all of the "fixed" situations require that a flavor with root_gb=0 has been used? I'm reading through the comments trying to understand | |
| 20:27:49 | dansmith | no, the right fix does not | |
| 20:28:20 | melwitt | because I thought, as of ocata, if flavor has root_gb > 0 placement would filter it out of consideration and removing DiskFilter would not help you | |
| 20:28:30 | dansmith | the middle solution requires sending a hacked flavor with root=0 to the scheduler as the signal to avoid a real signal | |
| 20:29:08 | dansmith | right I think mriedem is saying that if you align all the planets on ocata, including set root=0, you can get it to kinda work if you don't sneeze near it | |
| 20:29:16 | melwitt | got it | |
| 20:29:48 | mriedem | we actually don't want people using root_gb=0 in flavors | |
| 20:29:52 | dansmith | right | |
| 20:29:53 | mriedem | there is a security issue for that | |
| 20:30:05 | dansmith | and it's broken because it requires using specific flavors for bfv | |
| 20:30:15 | mriedem | https://github.com/openstack/nova/commit/763fd62464e9a0753e061171cc1fd826055bbc01 | |
| 20:30:22 | melwitt | yeah, just wanted to make sure I understood what the definition of "fixed" was in the convo | |
| 20:30:30 | mriedem | i have to run, be back in a few | |
| 20:30:44 | mriedem | dansmith: let's add the release note and change related-bug to partial-bug (for my ocd) and then i'm +2 | |
| 20:30:53 | dansmith | ack | |
| 20:33:57 | dansmith | mriedem: I'll remind you I added related-bug for you :) | |
| 20:36:27 | dansmith | damn, you said "link" not specifically related | |
| 20:36:30 | dansmith | I' | |
| 20:36:33 | dansmith | I'll get you next time | |
| 20:39:40 | openstackgerrit | Dan Smith proposed openstack/nova master: Avoid requesting DISK_GB allocation for root_gb on BFV instances https://review.openstack.org/580720 | |
| 21:02:25 | mriedem | muwahahaha | |
| 21:08:15 | mriedem | dansmith: see my hedge in the release note | |
| 21:08:38 | mriedem | i wonder if we should be clear that if you're not using the FilterScheduler (and/or using the DiskFilter) this is still a problem | |
| 21:08:42 | dansmith | sure makes sense | |
| 21:08:52 | mriedem | but, CachingScheduler is deprecated and DiskFilter should be disabled if using FilterScheduler | |
| 21:10:43 | dansmith | how's this: | |
| 21:10:45 | openstackgerrit | Dan Smith proposed openstack/nova master: Avoid requesting DISK_GB allocation for root_gb on BFV instances https://review.openstack.org/580720 | |
| 21:11:17 | mriedem | lgtm | |
| 21:11:20 | mriedem | pew pew | |
| 21:12:43 | mriedem | at some point we should drop the ironic flavor migration stuff, since then we can drop the remaining allocation stuff in the report client for the RT | |
| 21:13:05 | mriedem | the only driver that reports allocations in the compute is the ironic driver | |
| 21:14:29 | mriedem | requires_allocation_refresh = True | |
| 21:14:29 | mriedem | # migration has been completed. | |
| 21:14:29 | mriedem | # TODO(johngarbutt) we should remove this once the resource class | |
| 21:14:47 | mriedem | what's our signal that the resource class migration has been completed? | |
| 21:15:05 | dansmith | time and I think it's passed now | |
| 21:15:27 | mriedem | we do have a nova-status check for the ironic instance embedded flavor migration | |
| 21:15:33 | mriedem | and the online data migration | |
| 21:15:35 | mriedem | online/offline | |
| 21:15:55 | mriedem | # baremetal node class in an atomic, singular unit. | |
| 21:15:55 | mriedem | # and DISK_GB resource classes in early Queens when Ironic nodes will | |
| 21:15:55 | mriedem | # TODO(jaypipes): Completely remove the reporting of VCPU, MEMORY_MB, | |
| 21:15:55 | mriedem | and | |
| 21:15:55 | mriedem | # *always* return the custom resource class that represents the | |
| 21:16:32 | mriedem | so we could dump all of that now, or wait until early stein | |
| 21:16:49 | mriedem | i'd like to start dropping compat code that's hung around since newton/ocata times | |
| 21:17:21 | mriedem | jroll: penick: do you guys have your ironic/nova deployment fully migrated to using resource classes? | |
| 21:17:39 | jroll | mriedem: I did that patch already: https://review.openstack.org/#/c/565841/ | |
| 21:17:44 | jroll | just haven't taken the time to rebase | |
| 21:17:58 | jroll | we're still on juno, so... no. but will do in ocata afaik | |
| 21:18:16 | mriedem | the bm cloud is juno and the vm cloud is ocata? | |
| 21:18:25 | jroll | yeah | |
| 21:18:36 | jroll | still working on ocata for bm | |
| 21:18:42 | mriedem | ok, | |
| 21:18:55 | mriedem | would be nice to know people have rolled bm deployments through this transition and there aren't any major issues | |
| 21:18:59 | mriedem | before we drop all of that code | |
| 21:19:44 | mriedem | everyone seems to want an 18-24+ month window of support on transition code... | |
| 21:20:06 | jroll | TheJulia or dtantsur|afk might have people that have been through that | |
| 21:20:20 | mriedem | cern has bm and they are on queens now, | |
| 21:20:39 | mriedem | tssurya: do you know if the cern baremetal deployment has gone through the custom resource class transition for ironic flavors? | |
| 21:21:18 | tssurya | mriedem: yes | |
| 21:21:31 | jroll | \o/ | |
| 21:22:20 | jroll | mriedem: I can get back to 565841 next week maybe, or someone can take it over if they want | |
| 21:22:42 | tssurya | meaning we use resource classes for ironic flavors (reading the convo to find out what transition) | |
| 21:23:21 | jroll | tssurya: there's an online data migration for existing instances | |
| 21:23:23 | edmondsw | mriedem I think your changes in https://review.openstack.org/#/c/528835/ are related to a bug I just opened... take a look when you get a chance? https://bugs.launchpad.net/nova/+bug/1781286 | |
| 21:23:25 | openstack | Launchpad bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during rebuild" [Undecided,New] | |
| 21:23:40 | TheJulia | jroll: mriedem: I'm sure some folks have in our org... That being said unless it is a stated policy I've noticed people upstream tend to try and push the envelope... and downstream seems to pay for it... I guess I'm just rambling that long, but not too long is preferable to short windows | |
| 21:24:18 | jroll | TheJulia: dropping it now would be two cycles | |
| 21:25:07 | mriedem | TheJulia: downstream consumers would have upstream maintain everything forever if possible | |
| 21:25:09 | TheJulia | to support skip-leveling, two would be necessary, right now most people seem to be focused on fast forwarding since that is essentially what we end up testing | |