Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-07
14:54:07 mriedem when attaching a volume to a shelved offloaded instance, we create the bdm in the api - so cell0
14:54:41 mriedem we must not have a tempest test for this
14:54:51 melwitt rats
14:55:13 mriedem could probably recreate with a functional test, but i'll start by reporting abug
14:56:40 mriedem hmm, AttachVolumeShelveTestJSON in tempest should test this
14:57:38 superdan guh
14:59:35 mriedem so either the tempest test is busted, or i'm missing something that makes this work
15:10:13 superdan mriedem: well, that clearly needs to be targeted
15:11:25 leakypipes cdent: thx for continuing your resource providers weekly summaries. good job catching up on all the change while you were on PTO. look to have gotten everything correct, AFAICT.
15:11:48 cdent leakypipes: it's my UVP
15:11:56 mriedem superdan: yeah, just really odd that the tempest test wouldn't fail, unless the test is broken
15:12:09 cdent thanks for the confirmation that I didn't miss anything obvious
15:12:20 superdan mriedem: yeah
15:12:50 superdan mriedem: does that fail with a constraint error, or do we just create it and then it never attaches on unshelve?
15:12:51 superdan because maybe the tempest test isn't validating via ssh
15:13:30 mriedem the tempest test does do some ssh counting of volumes on the guest
15:13:47 superdan mriedem: actually,
15:14:03 superdan mriedem: until my devstack patch, the control services are not pointing at cell0 for their default
15:14:07 superdan so that's why
15:14:27 mriedem hmm, so why doesn't your devstack patch fail on this though?
15:14:47 superdan mriedem: it'd be the canary against nova that d-o it that would fail
15:15:00 superdan this guy: https://review.openstack.org/#/c/473565/
15:15:08 superdan is that test new or something?
15:15:44 superdan hmm, nope passing in the canary
15:16:17 cdent leakypipes: requirements change for new gabbi if you're inclined to move that along: https://review.openstack.org/481665
15:19:21 mriedem tempest is configured to run ssh validatoin, so it should do this: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/volumes/test_attach_volume.py#n181
15:19:54 mriedem maybe the counter is broken so it always reports 0 and it's comparing 0 to 0 and passes?
15:20:10 leakypipes cdent: done
15:20:14 superdan mriedem: heh
15:20:46 mriedem although this would fail http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/volumes/test_attach_volume.py#n219
15:20:49 mriedem num_vol + 1
15:25:38 mriedem superdan: we need to update your canary patch to depend on I08d7da843d18b426dda8a8a231039d950a4c0ce5 now
15:25:59 mriedem shall i push the button?
15:26:11 superdan mriedem: I'm working on a target change, so I will do that when I'm done here
15:26:16 cdent thanks leakypipes
15:27:07 openstackgerrit Merged openstack/nova master: Support tag instances when boot(3/4) https://review.openstack.org/459593
15:31:01 superdan mriedem: was there a bug for this shelved issue?
15:31:12 mriedem no, i can create one
15:31:22 superdan thanks
15:33:53 mriedem you know,
15:33:54 mriedem another thing,
15:34:16 mriedem when we shelve offload a server, we never clean up it's InstanceMapping,
15:34:30 mriedem and when we unshelve it, conductor asks the scheduler for a new host, which could be in a different cell,
15:34:40 mriedem but we never create a new instance mapping, or update the existing instance mapping, for that new cell
15:35:17 superdan mriedem: we can't move it between cells,
15:35:27 superdan so if that's true, we need to be asking the scheduler to limit to one cell,
15:35:29 superdan like we do for resize
15:35:38 mriedem i don't see anything in conductor that's doing that
15:35:49 mriedem is it in the req spec?
15:35:52 superdan yep
15:35:58 superdan reqspec.destination.cell
15:38:58 mriedem ok this https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/conductor/tasks/migrate.py#L63-L66
15:39:09 mriedem that's the only place we set destination.cell from what i can tell
15:40:11 superdan yep
15:41:08 mriedem ok i'll open a separate bug for that
15:42:20 superdan cool
15:42:26 superdan well, not cool, but..
15:42:48 mriedem about as cool as a kick in the pants
15:42:58 mriedem maybe another nail in the shelve coffin
15:43:26 superdan meh, anyone that uses shelve is probably used to shelve being broken
15:43:39 mriedem "Days since shelve failed: 0"
15:43:43 mriedem ^ in their office
15:44:21 superdan hah yeah
15:48:03 mriedem hmm, in the multi-cell job, the cpu config database is also pointing at cell0
15:48:04 mriedem http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova-cpu.conf.txt.gz
15:48:24 mriedem but it should go through conductor when the compute manager queries for the list of bdms right?
15:48:29 mriedem and cell conductor should be using http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz ?
15:48:50 superdan yeah
15:49:22 superdan really we should unset the db connection for both main and api in that cpu config,
15:49:28 superdan but the way I generate it is kinda weird
15:49:56 mriedem the [database] one for sure,
15:50:07 mriedem but the api_database one is needed for the retry and affinity crap
15:50:14 superdan not in compute
15:50:17 mriedem well, at least affinity
15:50:38 mriedem sure it is, the compute manager pulls server groups from the api db right?
15:51:00 superdan no, it would ask conductor like anything else
15:51:20 superdan and conductor in cell1 is configured to not be able to find the api db: http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz
15:51:33 superdan which means it can't retry in that configuration nor do the affinity check,
15:51:34 mriedem so how does the affinity stuff not all blow up?
15:51:45 superdan which is why that workaround config was added and the devstack patch sets it
15:51:53 mriedem oh
15:52:10 superdan mriedem: got a bug for the offload target thing yet?
15:52:16 mriedem http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz
15:52:22 mriedem oops
15:52:31 mriedem i meant, if not CONF.workarounds.disable_group_policy_check_upcall:
15:52:47 mriedem ok
15:53:00 openstack Launchpad bug 1702932 in OpenStack Compute (nova) "Unshelving an offloaded server with volume attachments may not attach to the guest in multi-cell env" [Undecided,New]
15:53:00 mriedem https://bugs.launchpad.net/nova/+bug/1702932
15:53:24 superdan thanks
15:53:24 mriedem i'll crank out another one for the unshelve limit to same cell thing
15:54:48 openstackgerrit Dan Smith proposed openstack/nova master: Fix targeting cell when attaching a volume to a shelved instance https://review.openstack.org/481683
15:54:48 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565
15:54:56 superdan here's that first fix and updated canary ^
15:56:23 mriedem https://bugs.launchpad.net/nova/+bug/1702935
15:56:24 openstack Launchpad bug 1702935 in OpenStack Compute (nova) "Unshelving an offloaded server does not restrict to the original cell" [Undecided,New]
16:01:14 mriedem cool https://review.openstack.org/#/c/481683/ is easier than i thought, but makes sense. still really wondering how this wasn't caught in tempest
16:12:21 mriedem so normal attach volume works to create the volume bdm in the same cell as the instance because when we lookup the instance, set target the context
16:13:08 mriedem and we create that bdm in the cell compute service
16:34:57 openstack Launchpad bug 1702959 in OpenStack Compute (nova) "MigrationNotFound in multi-cell setup doing server external events processing" [High,Triaged]
16:34:57 mriedem another one https://bugs.launchpad.net/nova/+bug/1702959

Earlier   Later