Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-11
16:09:33 mriedem https://review.openstack.org/#/c/520660/
16:10:58 sapd1 Use this patch I can bypass https://review.openstack.org/#/c/528740/8/nova/compute/api.py
16:10:58 sapd1 :D
16:11:30 mriedem if you want to fork a hack into your product that's up to you
16:11:36 mriedem but i'm not going to spend time helping you do it, sorry
16:11:54 mriedem i'm only interested in https://review.openstack.org/#/c/532407/
16:12:34 sapd1 mriedem: So Has anyone implemented this spec yet?
16:13:08 mriedem sapd1: no, see my -1 on the spec
16:13:14 mriedem need to agree on the design first
16:13:24 mriedem which is why the 20 other hack fixes for this problem have been rejected in the past
16:14:44 sapd1 mriedem: I'm trying :D thanks
16:15:10 mriedem my comment from march 27 is where i think it stalled
16:15:12 mriedem "The cleanest / best solution to this is to add a volume action API to cinder for re-imaging the volume. Once that is available in a new cinder v3 microversion, nova can use it. The reason I think this should be done in Cinder with re-imaging the volume there is (1) it's cleaner from the nova side and (2) then Cinder is in control of how that re-image should happen, along with any details it needs to update, e.g. the volume's
16:15:12 mriedem lume_image_metadata" information would need to be updated.We really don't want to do the volume create/delete/swap orchestration thing since that entails issues with the volume type being gone, going over quota, what to do about deleting the old volume, etc.So please propose a spec to Cinder and start working the API changes there and then nova can depend on a new Cinder API."
16:15:59 mriedem i think of this like shelve offloading and unshelving a volume-backed server but with a new image
16:16:10 mriedem and i'd like cinder to control the api for re-imaging the volume
16:16:19 mriedem because cinder owns the volume
16:17:09 mriedem maybe i need to work with tommylikehu on this in stein
16:18:50 mriedem melwitt: it might not be too early to start a stein ptg etherpad for random items
16:24:06 sapd1 mriedem: Yep. I think It can work. with this patch.
16:24:12 sapd1 mriedem: http://paste.ubuntu.com/p/V3MhY7Hp3X/
16:25:14 sapd1 I change flow to detach old root disk device and create attachment ID for new rootdisk device
16:27:37 openstackgerrit Merged openstack/nova master: Test for unsanitized consumer UUID https://review.openstack.org/581137
16:30:09 gibi mriedem: left comments in the complex-anti-affinity-policies series but I haven't finished yet. I run out of time. I can check the API patch tomorrow if needed
16:33:44 openstackgerrit Matt Rabe proposed openstack/nova master: Add destination MSP IP address to PowerVM migrate data https://review.openstack.org/579676
17:25:09 melwitt mriedem: yeah, probably a good idea to start one. I've seen others posted to the ML already, at least for cinder
17:45:02 melwitt dansmith: I don't have anything for a cells meeting
17:45:12 dansmith ack
17:50:03 melwitt mriedem: comment on https://review.openstack.org/580755 , I've wondered why we don't use the CastAsCall fixture anymore in functional tests where we want to GET servers after we create them. everything else looks good in the tests
17:56:40 mriedem melwitt: because CastAsCall isn't what you'd really get as a real user
17:56:49 mriedem so unless there is a good reason to use, it don't know why we would
17:58:47 mriedem melwitt: replied inline
17:58:49 melwitt mriedem: I was thinking because of the assumption of the servers before they're active, but in this case "active" is not needed or relevant in the test
17:59:27 mriedem right we don't care about status in these tests
17:59:34 melwitt I was thinking POST followed by GET without waiting for the status, but if you don't care about the status, then it doesn't matter. maybe that's the difference. none of the assertions you're doing depend on the boot process being further along
18:00:19 melwitt a few weeks ago I was working on a test that kept failing until I used CastAsCall, but it must have been that an assertion I had depended on the boot process getting to a certain point. sorry for the noise
18:01:14 mriedem we have utilities for waiting for the server to be a certain status as well
18:01:22 mriedem so if i needed to wait, i'd use that instead of CastAsCall
18:01:50 melwitt true, I'll keep that in mind for next time
18:20:55 openstackgerrit Merged openstack/nova master: [placement] add error.code on a ConcurrentUpdateDetected https://review.openstack.org/581742
18:34:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova-status upgrade check for request spec migrations https://review.openstack.org/581813
18:41:57 melwitt mriedem: ptg etherpad if you had some topic(s) you wanted to jot down https://etherpad.openstack.org/p/nova-ptg-stein
18:42:52 mriedem thanks
19:06:03 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional regressions tests for server_group_members OverQuota https://review.openstack.org/581845
19:06:04 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Fix server_group_members quota check https://review.openstack.org/581846
19:17:42 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add functional regressions tests for server_group_members OverQuota https://review.openstack.org/581866
19:17:43 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Fix server_group_members quota check https://review.openstack.org/581867
19:25:22 mriedem crazy croats
19:54:47 mriedem dansmith: comments on your bfv req spec change https://review.openstack.org/#/c/580720/
20:02:05 dansmith ooh
20:10:03 mriedem so this isn't backportable b/c of the version change, but in thinking about this more - persisting the is_bfv flag is good b/c for any new instances created after this, moving them also benefits from the flag being set
20:10:13 mriedem the only gap is existing instances that don't have is_bfv set in the request spec, as noted in the commit message
20:10:18 mriedem which we can easily detect and heal during a move
20:11:19 mriedem there are a few comments in the bug report about how once people got to ocata the allocations for volume-backed instances were fixed, and that's because the resource tracker had the is_bfv logic to not account for flavor.root_gb,
20:11:45 mriedem but then in pike we stopped creating allocations in the RT and based the allocations of the request spec, so we regressed that....
20:12:03 mriedem so this (re)fixes that
20:12:13 mriedem but we'd have a gap in pike and queens
20:12:54 dansmith sorry, was looking at the review window
20:12:56 dansmith I just replied,
20:13:14 dansmith but basically I didn't even consider this as backportable since it changes what we allocate and stuff and thought it wasn't worth the risk
20:13:15 mriedem almost seems like we could do a hybrid of https://review.openstack.org/#/c/428481/ and your new patch which could be backportable
20:13:31 dansmith I'd rather just roll forward and heal
20:14:05 mriedem i get that,
20:14:18 mriedem but this is a pretty high flame count bug or whatever
20:14:30 mriedem so backportability seems worth at least thinking through
20:14:42 dansmith our downstream requirement for stuff like this is that you have to be able to yum update components in any order on any machine, and having things starting to claim differently because you updated one conductor and didn't do another until a month later
20:14:59 dansmith even still, the scheduler is the thing that needs to know, not conductor like you said,
20:15:06 dansmith so I'm not sure how we could do it without an object or rpc change
20:15:49 dansmith if this was a regression in the last release then I'd care more I guess, but this has _always_ been this way, which makes it nearly a feature
20:16:06 mriedem well, it was fixed in ocata and then regressed again in pike
20:16:19 dansmith eh? this has never worked that I know of
20:16:20 mriedem i think anyway
20:16:27 dansmith pretty sure not,
20:16:28 mriedem in ocata the computes would create the allocations right?
20:16:37 dansmith ah, but, it has always been broken in RT :)
20:16:48 mriedem the RT / report client code that created the allocations in ocata took bfv into account for DISK_GB
20:16:52 dansmith so before we were doing allocations wrong, we were doing RT claiming wrong and failing super late
20:16:53 mriedem RT == claim?
20:16:56 mriedem well,
20:16:58 dansmith I mean old school claims
20:17:02 mriedem if you dropped the DiskFilter you wouldn't hit the claim
20:17:06 dansmith pfft :)
20:17:09 mriedem so in ocata, if you're using placement and remove DiskFilter,
20:17:10 mriedem fixed
20:17:39 dansmith where did we not allocate for volume-backed disks in RT?
20:18:44 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L169
20:18:53 mriedem this comment in the bug is a pretty good one https://bugs.launchpad.net/nova/+bug/1469179/comments/37
20:18:55 openstack Launchpad bug 1469179 in OpenStack Compute (nova) "instance.root_gb should be 0 for volume-backed instances" [Medium,In progress] - Assigned to Dan Smith (danms)
20:19:02 mriedem "After disabling the RamFilter and DiskFilter, and instead relying on the Placement API for scheduling, most of the "scheduling" aspect of this problem has been addressed. I was able to drop a local patch to handle this root_gb!=0 for EBS volumes, and for most real-life use cases it is working. The placement API is not recording disk allocations for EBS volumes."
20:19:44 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L351
20:20:07 dansmith so it kinda worked almost by accident for a release?
20:20:12 mriedem so in ocata we wouldn't report root_gb for bfv instance allocations, you could still fail scheduling due to the old school RT disk claim if DiskFilter was enabled
20:20:40 mriedem i agree with the 'by accident' part
20:21:05 mriedem and i don't think it was until at least pike that we said to disable the DiskFilter if you're using placement and remove it from the default enabled filters
20:21:33 dansmith right, I was thinking ocata still required that
20:21:34 dansmith well,
20:21:41 dansmith didn't recommend placement as a complete replacement
20:22:46 mriedem yeah dropped CPU/Ram/DiskFilters from default enabled filters in pike
20:23:26 mriedem we'd been saying since at least newton that placement was the eventual fix for this problem :) that's why we kept putting off melwitt's workaround / RT patch
20:23:35 mriedem and sure now it is

Earlier   Later