| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-07 | |||
| 15:11:48 | cdent | leakypipes: it's my UVP | |
| 15:11:56 | mriedem | superdan: yeah, just really odd that the tempest test wouldn't fail, unless the test is broken | |
| 15:12:09 | cdent | thanks for the confirmation that I didn't miss anything obvious | |
| 15:12:20 | superdan | mriedem: yeah | |
| 15:12:50 | superdan | mriedem: does that fail with a constraint error, or do we just create it and then it never attaches on unshelve? | |
| 15:12:51 | superdan | because maybe the tempest test isn't validating via ssh | |
| 15:13:30 | mriedem | the tempest test does do some ssh counting of volumes on the guest | |
| 15:13:47 | superdan | mriedem: actually, | |
| 15:14:03 | superdan | mriedem: until my devstack patch, the control services are not pointing at cell0 for their default | |
| 15:14:07 | superdan | so that's why | |
| 15:14:27 | mriedem | hmm, so why doesn't your devstack patch fail on this though? | |
| 15:14:47 | superdan | mriedem: it'd be the canary against nova that d-o it that would fail | |
| 15:15:00 | superdan | this guy: https://review.openstack.org/#/c/473565/ | |
| 15:15:08 | superdan | is that test new or something? | |
| 15:15:44 | superdan | hmm, nope passing in the canary | |
| 15:16:17 | cdent | leakypipes: requirements change for new gabbi if you're inclined to move that along: https://review.openstack.org/481665 | |
| 15:19:21 | mriedem | tempest is configured to run ssh validatoin, so it should do this: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/volumes/test_attach_volume.py#n181 | |
| 15:19:54 | mriedem | maybe the counter is broken so it always reports 0 and it's comparing 0 to 0 and passes? | |
| 15:20:10 | leakypipes | cdent: done | |
| 15:20:14 | superdan | mriedem: heh | |
| 15:20:46 | mriedem | although this would fail http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/volumes/test_attach_volume.py#n219 | |
| 15:20:49 | mriedem | num_vol + 1 | |
| 15:25:38 | mriedem | superdan: we need to update your canary patch to depend on I08d7da843d18b426dda8a8a231039d950a4c0ce5 now | |
| 15:25:59 | mriedem | shall i push the button? | |
| 15:26:11 | superdan | mriedem: I'm working on a target change, so I will do that when I'm done here | |
| 15:26:16 | cdent | thanks leakypipes | |
| 15:27:07 | openstackgerrit | Merged openstack/nova master: Support tag instances when boot(3/4) https://review.openstack.org/459593 | |
| 15:31:01 | superdan | mriedem: was there a bug for this shelved issue? | |
| 15:31:12 | mriedem | no, i can create one | |
| 15:31:22 | superdan | thanks | |
| 15:33:53 | mriedem | you know, | |
| 15:33:54 | mriedem | another thing, | |
| 15:34:16 | mriedem | when we shelve offload a server, we never clean up it's InstanceMapping, | |
| 15:34:30 | mriedem | and when we unshelve it, conductor asks the scheduler for a new host, which could be in a different cell, | |
| 15:34:40 | mriedem | but we never create a new instance mapping, or update the existing instance mapping, for that new cell | |
| 15:35:17 | superdan | mriedem: we can't move it between cells, | |
| 15:35:27 | superdan | so if that's true, we need to be asking the scheduler to limit to one cell, | |
| 15:35:29 | superdan | like we do for resize | |
| 15:35:38 | mriedem | i don't see anything in conductor that's doing that | |
| 15:35:49 | mriedem | is it in the req spec? | |
| 15:35:52 | superdan | yep | |
| 15:35:58 | superdan | reqspec.destination.cell | |
| 15:38:58 | mriedem | ok this https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/conductor/tasks/migrate.py#L63-L66 | |
| 15:39:09 | mriedem | that's the only place we set destination.cell from what i can tell | |
| 15:40:11 | superdan | yep | |
| 15:41:08 | mriedem | ok i'll open a separate bug for that | |
| 15:42:20 | superdan | cool | |
| 15:42:26 | superdan | well, not cool, but.. | |
| 15:42:48 | mriedem | about as cool as a kick in the pants | |
| 15:42:58 | mriedem | maybe another nail in the shelve coffin | |
| 15:43:26 | superdan | meh, anyone that uses shelve is probably used to shelve being broken | |
| 15:43:39 | mriedem | "Days since shelve failed: 0" | |
| 15:43:43 | mriedem | ^ in their office | |
| 15:44:21 | superdan | hah yeah | |
| 15:48:03 | mriedem | hmm, in the multi-cell job, the cpu config database is also pointing at cell0 | |
| 15:48:04 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 15:48:24 | mriedem | but it should go through conductor when the compute manager queries for the list of bdms right? | |
| 15:48:29 | mriedem | and cell conductor should be using http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz ? | |
| 15:48:50 | superdan | yeah | |
| 15:49:22 | superdan | really we should unset the db connection for both main and api in that cpu config, | |
| 15:49:28 | superdan | but the way I generate it is kinda weird | |
| 15:49:56 | mriedem | the [database] one for sure, | |
| 15:50:07 | mriedem | but the api_database one is needed for the retry and affinity crap | |
| 15:50:14 | superdan | not in compute | |
| 15:50:17 | mriedem | well, at least affinity | |
| 15:50:38 | mriedem | sure it is, the compute manager pulls server groups from the api db right? | |
| 15:51:00 | superdan | no, it would ask conductor like anything else | |
| 15:51:20 | superdan | and conductor in cell1 is configured to not be able to find the api db: http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:51:33 | superdan | which means it can't retry in that configuration nor do the affinity check, | |
| 15:51:34 | mriedem | so how does the affinity stuff not all blow up? | |
| 15:51:45 | superdan | which is why that workaround config was added and the devstack patch sets it | |
| 15:51:53 | mriedem | oh | |
| 15:52:10 | superdan | mriedem: got a bug for the offload target thing yet? | |
| 15:52:16 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:52:22 | mriedem | oops | |
| 15:52:31 | mriedem | i meant, if not CONF.workarounds.disable_group_policy_check_upcall: | |
| 15:52:47 | mriedem | ok | |
| 15:53:00 | openstack | Launchpad bug 1702932 in OpenStack Compute (nova) "Unshelving an offloaded server with volume attachments may not attach to the guest in multi-cell env" [Undecided,New] | |
| 15:53:00 | mriedem | https://bugs.launchpad.net/nova/+bug/1702932 | |
| 15:53:24 | superdan | thanks | |
| 15:53:24 | mriedem | i'll crank out another one for the unshelve limit to same cell thing | |
| 15:54:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix targeting cell when attaching a volume to a shelved instance https://review.openstack.org/481683 | |
| 15:54:48 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565 | |
| 15:54:56 | superdan | here's that first fix and updated canary ^ | |
| 15:56:23 | mriedem | https://bugs.launchpad.net/nova/+bug/1702935 | |
| 15:56:24 | openstack | Launchpad bug 1702935 in OpenStack Compute (nova) "Unshelving an offloaded server does not restrict to the original cell" [Undecided,New] | |
| 16:01:14 | mriedem | cool https://review.openstack.org/#/c/481683/ is easier than i thought, but makes sense. still really wondering how this wasn't caught in tempest | |
| 16:12:21 | mriedem | so normal attach volume works to create the volume bdm in the same cell as the instance because when we lookup the instance, set target the context | |
| 16:13:08 | mriedem | and we create that bdm in the cell compute service | |
| 16:34:57 | openstack | Launchpad bug 1702959 in OpenStack Compute (nova) "MigrationNotFound in multi-cell setup doing server external events processing" [High,Triaged] | |
| 16:34:57 | mriedem | another one https://bugs.launchpad.net/nova/+bug/1702959 | |
| 16:38:13 | mriedem | i'll take a crack at that one | |
| 16:38:54 | dfisher | anybody know of attach/detach issues with current nova/cinder (via devstack) bits? I can't seem to get a simple volume to attach to a simple Ubuntu or Cirros VM | |
| 16:39:16 | mriedem | no known issues | |
| 16:39:22 | dfisher | hmm, ok. | |
| 16:39:28 | dfisher | thanks | |
| 16:43:21 | superdan | mriedem: is that one resolved by the patch that is up already? | |
| 16:43:42 | superdan | mriedem: https://review.openstack.org/#/c/445142/ | |
| 16:46:00 | clarkb | mriedem: http://logs.openstack.org/31/476631/18/gate/gate-nova-tox-functional-ubuntu-xenial/2cafb17/console.html#_2017-07-07_15_29_18_416631 known issue? that reset the gate not too long ago. Looks like test is attempting admin request and failing without admin rights | |
| 16:50:23 | mriedem | clarkb: i'm not aware of that one | |