| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-11 | |||
| 16:30:09 | gibi | mriedem: left comments in the complex-anti-affinity-policies series but I haven't finished yet. I run out of time. I can check the API patch tomorrow if needed | |
| 16:33:44 | openstackgerrit | Matt Rabe proposed openstack/nova master: Add destination MSP IP address to PowerVM migrate data https://review.openstack.org/579676 | |
| 17:25:09 | melwitt | mriedem: yeah, probably a good idea to start one. I've seen others posted to the ML already, at least for cinder | |
| 17:45:02 | melwitt | dansmith: I don't have anything for a cells meeting | |
| 17:45:12 | dansmith | ack | |
| 17:50:03 | melwitt | mriedem: comment on https://review.openstack.org/580755 , I've wondered why we don't use the CastAsCall fixture anymore in functional tests where we want to GET servers after we create them. everything else looks good in the tests | |
| 17:56:40 | mriedem | melwitt: because CastAsCall isn't what you'd really get as a real user | |
| 17:56:49 | mriedem | so unless there is a good reason to use, it don't know why we would | |
| 17:58:47 | mriedem | melwitt: replied inline | |
| 17:58:49 | melwitt | mriedem: I was thinking because of the assumption of the servers before they're active, but in this case "active" is not needed or relevant in the test | |
| 17:59:27 | mriedem | right we don't care about status in these tests | |
| 17:59:34 | melwitt | I was thinking POST followed by GET without waiting for the status, but if you don't care about the status, then it doesn't matter. maybe that's the difference. none of the assertions you're doing depend on the boot process being further along | |
| 18:00:19 | melwitt | a few weeks ago I was working on a test that kept failing until I used CastAsCall, but it must have been that an assertion I had depended on the boot process getting to a certain point. sorry for the noise | |
| 18:01:14 | mriedem | we have utilities for waiting for the server to be a certain status as well | |
| 18:01:22 | mriedem | so if i needed to wait, i'd use that instead of CastAsCall | |
| 18:01:50 | melwitt | true, I'll keep that in mind for next time | |
| 18:20:55 | openstackgerrit | Merged openstack/nova master: [placement] add error.code on a ConcurrentUpdateDetected https://review.openstack.org/581742 | |
| 18:34:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-status upgrade check for request spec migrations https://review.openstack.org/581813 | |
| 18:41:57 | melwitt | mriedem: ptg etherpad if you had some topic(s) you wanted to jot down https://etherpad.openstack.org/p/nova-ptg-stein | |
| 18:42:52 | mriedem | thanks | |
| 19:06:03 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional regressions tests for server_group_members OverQuota https://review.openstack.org/581845 | |
| 19:06:04 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Fix server_group_members quota check https://review.openstack.org/581846 | |
| 19:17:42 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional regressions tests for server_group_members OverQuota https://review.openstack.org/581866 | |
| 19:17:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Fix server_group_members quota check https://review.openstack.org/581867 | |
| 19:25:22 | mriedem | crazy croats | |
| 19:54:47 | mriedem | dansmith: comments on your bfv req spec change https://review.openstack.org/#/c/580720/ | |
| 20:02:05 | dansmith | ooh | |
| 20:10:03 | mriedem | so this isn't backportable b/c of the version change, but in thinking about this more - persisting the is_bfv flag is good b/c for any new instances created after this, moving them also benefits from the flag being set | |
| 20:10:13 | mriedem | the only gap is existing instances that don't have is_bfv set in the request spec, as noted in the commit message | |
| 20:10:18 | mriedem | which we can easily detect and heal during a move | |
| 20:11:19 | mriedem | there are a few comments in the bug report about how once people got to ocata the allocations for volume-backed instances were fixed, and that's because the resource tracker had the is_bfv logic to not account for flavor.root_gb, | |
| 20:11:45 | mriedem | but then in pike we stopped creating allocations in the RT and based the allocations of the request spec, so we regressed that.... | |
| 20:12:03 | mriedem | so this (re)fixes that | |
| 20:12:13 | mriedem | but we'd have a gap in pike and queens | |
| 20:12:54 | dansmith | sorry, was looking at the review window | |
| 20:12:56 | dansmith | I just replied, | |
| 20:13:14 | dansmith | but basically I didn't even consider this as backportable since it changes what we allocate and stuff and thought it wasn't worth the risk | |
| 20:13:15 | mriedem | almost seems like we could do a hybrid of https://review.openstack.org/#/c/428481/ and your new patch which could be backportable | |
| 20:13:31 | dansmith | I'd rather just roll forward and heal | |
| 20:14:05 | mriedem | i get that, | |
| 20:14:18 | mriedem | but this is a pretty high flame count bug or whatever | |
| 20:14:30 | mriedem | so backportability seems worth at least thinking through | |
| 20:14:42 | dansmith | our downstream requirement for stuff like this is that you have to be able to yum update components in any order on any machine, and having things starting to claim differently because you updated one conductor and didn't do another until a month later | |
| 20:14:59 | dansmith | even still, the scheduler is the thing that needs to know, not conductor like you said, | |
| 20:15:06 | dansmith | so I'm not sure how we could do it without an object or rpc change | |
| 20:15:49 | dansmith | if this was a regression in the last release then I'd care more I guess, but this has _always_ been this way, which makes it nearly a feature | |
| 20:16:06 | mriedem | well, it was fixed in ocata and then regressed again in pike | |
| 20:16:19 | dansmith | eh? this has never worked that I know of | |
| 20:16:20 | mriedem | i think anyway | |
| 20:16:27 | dansmith | pretty sure not, | |
| 20:16:28 | mriedem | in ocata the computes would create the allocations right? | |
| 20:16:37 | dansmith | ah, but, it has always been broken in RT :) | |
| 20:16:48 | mriedem | the RT / report client code that created the allocations in ocata took bfv into account for DISK_GB | |
| 20:16:52 | dansmith | so before we were doing allocations wrong, we were doing RT claiming wrong and failing super late | |
| 20:16:53 | mriedem | RT == claim? | |
| 20:16:56 | mriedem | well, | |
| 20:16:58 | dansmith | I mean old school claims | |
| 20:17:02 | mriedem | if you dropped the DiskFilter you wouldn't hit the claim | |
| 20:17:06 | dansmith | pfft :) | |
| 20:17:09 | mriedem | so in ocata, if you're using placement and remove DiskFilter, | |
| 20:17:10 | mriedem | fixed | |
| 20:17:39 | dansmith | where did we not allocate for volume-backed disks in RT? | |
| 20:18:44 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L169 | |
| 20:18:53 | mriedem | this comment in the bug is a pretty good one https://bugs.launchpad.net/nova/+bug/1469179/comments/37 | |
| 20:18:55 | openstack | Launchpad bug 1469179 in OpenStack Compute (nova) "instance.root_gb should be 0 for volume-backed instances" [Medium,In progress] - Assigned to Dan Smith (danms) | |
| 20:19:02 | mriedem | "After disabling the RamFilter and DiskFilter, and instead relying on the Placement API for scheduling, most of the "scheduling" aspect of this problem has been addressed. I was able to drop a local patch to handle this root_gb!=0 for EBS volumes, and for most real-life use cases it is working. The placement API is not recording disk allocations for EBS volumes." | |
| 20:19:44 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L351 | |
| 20:20:07 | dansmith | so it kinda worked almost by accident for a release? | |
| 20:20:12 | mriedem | so in ocata we wouldn't report root_gb for bfv instance allocations, you could still fail scheduling due to the old school RT disk claim if DiskFilter was enabled | |
| 20:20:40 | mriedem | i agree with the 'by accident' part | |
| 20:21:05 | mriedem | and i don't think it was until at least pike that we said to disable the DiskFilter if you're using placement and remove it from the default enabled filters | |
| 20:21:33 | dansmith | right, I was thinking ocata still required that | |
| 20:21:34 | dansmith | well, | |
| 20:21:41 | dansmith | didn't recommend placement as a complete replacement | |
| 20:22:46 | mriedem | yeah dropped CPU/Ram/DiskFilters from default enabled filters in pike | |
| 20:23:26 | mriedem | we'd been saying since at least newton that placement was the eventual fix for this problem :) that's why we kept putting off melwitt's workaround / RT patch | |
| 20:23:35 | mriedem | and sure now it is | |
| 20:23:46 | dansmith | aye | |
| 20:24:12 | dansmith | well, to be fair, we could have done this in queens at least too | |
| 20:24:32 | dansmith | meaning, it could have been the fix already | |
| 20:24:36 | mriedem | sure | |
| 20:24:51 | mriedem | and people could hack in a point-of-scheduling time fix in conductor if they need it.. | |
| 20:27:32 | melwitt | does all of the "fixed" situations require that a flavor with root_gb=0 has been used? I'm reading through the comments trying to understand | |
| 20:27:49 | dansmith | no, the right fix does not | |
| 20:28:20 | melwitt | because I thought, as of ocata, if flavor has root_gb > 0 placement would filter it out of consideration and removing DiskFilter would not help you | |
| 20:28:30 | dansmith | the middle solution requires sending a hacked flavor with root=0 to the scheduler as the signal to avoid a real signal | |
| 20:29:08 | dansmith | right I think mriedem is saying that if you align all the planets on ocata, including set root=0, you can get it to kinda work if you don't sneeze near it | |
| 20:29:16 | melwitt | got it | |
| 20:29:48 | mriedem | we actually don't want people using root_gb=0 in flavors | |
| 20:29:52 | dansmith | right | |
| 20:29:53 | mriedem | there is a security issue for that | |
| 20:30:05 | dansmith | and it's broken because it requires using specific flavors for bfv | |
| 20:30:15 | mriedem | https://github.com/openstack/nova/commit/763fd62464e9a0753e061171cc1fd826055bbc01 | |
| 20:30:22 | melwitt | yeah, just wanted to make sure I understood what the definition of "fixed" was in the convo | |
| 20:30:30 | mriedem | i have to run, be back in a few | |
| 20:30:44 | mriedem | dansmith: let's add the release note and change related-bug to partial-bug (for my ocd) and then i'm +2 | |
| 20:30:53 | dansmith | ack | |
| 20:33:57 | dansmith | mriedem: I'll remind you I added related-bug for you :) | |
| 20:36:27 | dansmith | damn, you said "link" not specifically related | |
| 20:36:30 | dansmith | I' | |