| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-07 | |||
| 13:56:55 | leakypipes | heh | |
| 13:56:57 | fried_rice | +1 | |
| 13:58:52 | mriedem | o/ | |
| 13:59:26 | leakypipes | mriedem: mornin. | |
| 14:10:50 | mriedem | gmann: looking at https://review.openstack.org/#/c/480815/1/specs/pike/approved/fix-to-return-server-groups-quota-in-quota-class.rst@51 | |
| 14:11:04 | mriedem | did we also miss those other resources in the quota class api, like fixed ips, floating ips, etc? | |
| 14:11:08 | mriedem | or was it just 'networks'? | |
| 14:18:07 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Update quota class spec to filter out networks quota https://review.openstack.org/480815 | |
| 14:26:44 | openstackgerrit | Merged openstack/nova-specs master: Update quota class spec to filter out networks quota https://review.openstack.org/480815 | |
| 14:31:04 | mriedem | so who wants to review this change to use the cinder 3.27 attachments API flow in swap volume? https://review.openstack.org/#/c/456971/ | |
| 14:31:17 | mriedem | tempest has already passed on it from higher in the stack which triggers the new behavior | |
| 14:37:24 | leakypipes | mriedem: do you want me to move those two lines in the allocation candidates patch or do it in a followup patch? | |
| 14:37:35 | mriedem | leakypipes: follow up is fine | |
| 14:37:40 | leakypipes | k | |
| 14:53:03 | mriedem | ruh roh | |
| 14:53:44 | mriedem | superdan: melwitt: i know you're on a call, but after that, let me ruin your day https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/compute/api.py#L3723 | |
| 14:54:07 | mriedem | when attaching a volume to a shelved offloaded instance, we create the bdm in the api - so cell0 | |
| 14:54:41 | mriedem | we must not have a tempest test for this | |
| 14:54:51 | melwitt | rats | |
| 14:55:13 | mriedem | could probably recreate with a functional test, but i'll start by reporting abug | |
| 14:56:40 | mriedem | hmm, AttachVolumeShelveTestJSON in tempest should test this | |
| 14:57:38 | superdan | guh | |
| 14:59:35 | mriedem | so either the tempest test is busted, or i'm missing something that makes this work | |
| 15:10:13 | superdan | mriedem: well, that clearly needs to be targeted | |
| 15:11:25 | leakypipes | cdent: thx for continuing your resource providers weekly summaries. good job catching up on all the change while you were on PTO. look to have gotten everything correct, AFAICT. | |
| 15:11:48 | cdent | leakypipes: it's my UVP | |
| 15:11:56 | mriedem | superdan: yeah, just really odd that the tempest test wouldn't fail, unless the test is broken | |
| 15:12:09 | cdent | thanks for the confirmation that I didn't miss anything obvious | |
| 15:12:20 | superdan | mriedem: yeah | |
| 15:12:50 | superdan | mriedem: does that fail with a constraint error, or do we just create it and then it never attaches on unshelve? | |
| 15:12:51 | superdan | because maybe the tempest test isn't validating via ssh | |
| 15:13:30 | mriedem | the tempest test does do some ssh counting of volumes on the guest | |
| 15:13:47 | superdan | mriedem: actually, | |
| 15:14:03 | superdan | mriedem: until my devstack patch, the control services are not pointing at cell0 for their default | |
| 15:14:07 | superdan | so that's why | |
| 15:14:27 | mriedem | hmm, so why doesn't your devstack patch fail on this though? | |
| 15:14:47 | superdan | mriedem: it'd be the canary against nova that d-o it that would fail | |
| 15:15:00 | superdan | this guy: https://review.openstack.org/#/c/473565/ | |
| 15:15:08 | superdan | is that test new or something? | |
| 15:15:44 | superdan | hmm, nope passing in the canary | |
| 15:16:17 | cdent | leakypipes: requirements change for new gabbi if you're inclined to move that along: https://review.openstack.org/481665 | |
| 15:19:21 | mriedem | tempest is configured to run ssh validatoin, so it should do this: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/volumes/test_attach_volume.py#n181 | |
| 15:19:54 | mriedem | maybe the counter is broken so it always reports 0 and it's comparing 0 to 0 and passes? | |
| 15:20:10 | leakypipes | cdent: done | |
| 15:20:14 | superdan | mriedem: heh | |
| 15:20:46 | mriedem | although this would fail http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/volumes/test_attach_volume.py#n219 | |
| 15:20:49 | mriedem | num_vol + 1 | |
| 15:25:38 | mriedem | superdan: we need to update your canary patch to depend on I08d7da843d18b426dda8a8a231039d950a4c0ce5 now | |
| 15:25:59 | mriedem | shall i push the button? | |
| 15:26:11 | superdan | mriedem: I'm working on a target change, so I will do that when I'm done here | |
| 15:26:16 | cdent | thanks leakypipes | |
| 15:27:07 | openstackgerrit | Merged openstack/nova master: Support tag instances when boot(3/4) https://review.openstack.org/459593 | |
| 15:31:01 | superdan | mriedem: was there a bug for this shelved issue? | |
| 15:31:12 | mriedem | no, i can create one | |
| 15:31:22 | superdan | thanks | |
| 15:33:53 | mriedem | you know, | |
| 15:33:54 | mriedem | another thing, | |
| 15:34:16 | mriedem | when we shelve offload a server, we never clean up it's InstanceMapping, | |
| 15:34:30 | mriedem | and when we unshelve it, conductor asks the scheduler for a new host, which could be in a different cell, | |
| 15:34:40 | mriedem | but we never create a new instance mapping, or update the existing instance mapping, for that new cell | |
| 15:35:17 | superdan | mriedem: we can't move it between cells, | |
| 15:35:27 | superdan | so if that's true, we need to be asking the scheduler to limit to one cell, | |
| 15:35:29 | superdan | like we do for resize | |
| 15:35:38 | mriedem | i don't see anything in conductor that's doing that | |
| 15:35:49 | mriedem | is it in the req spec? | |
| 15:35:52 | superdan | yep | |
| 15:35:58 | superdan | reqspec.destination.cell | |
| 15:38:58 | mriedem | ok this https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/conductor/tasks/migrate.py#L63-L66 | |
| 15:39:09 | mriedem | that's the only place we set destination.cell from what i can tell | |
| 15:40:11 | superdan | yep | |
| 15:41:08 | mriedem | ok i'll open a separate bug for that | |
| 15:42:20 | superdan | cool | |
| 15:42:26 | superdan | well, not cool, but.. | |
| 15:42:48 | mriedem | about as cool as a kick in the pants | |
| 15:42:58 | mriedem | maybe another nail in the shelve coffin | |
| 15:43:26 | superdan | meh, anyone that uses shelve is probably used to shelve being broken | |
| 15:43:39 | mriedem | "Days since shelve failed: 0" | |
| 15:43:43 | mriedem | ^ in their office | |
| 15:44:21 | superdan | hah yeah | |
| 15:48:03 | mriedem | hmm, in the multi-cell job, the cpu config database is also pointing at cell0 | |
| 15:48:04 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 15:48:24 | mriedem | but it should go through conductor when the compute manager queries for the list of bdms right? | |
| 15:48:29 | mriedem | and cell conductor should be using http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz ? | |
| 15:48:50 | superdan | yeah | |
| 15:49:22 | superdan | really we should unset the db connection for both main and api in that cpu config, | |
| 15:49:28 | superdan | but the way I generate it is kinda weird | |
| 15:49:56 | mriedem | the [database] one for sure, | |
| 15:50:07 | mriedem | but the api_database one is needed for the retry and affinity crap | |
| 15:50:14 | superdan | not in compute | |
| 15:50:17 | mriedem | well, at least affinity | |
| 15:50:38 | mriedem | sure it is, the compute manager pulls server groups from the api db right? | |
| 15:51:00 | superdan | no, it would ask conductor like anything else | |
| 15:51:20 | superdan | and conductor in cell1 is configured to not be able to find the api db: http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:51:33 | superdan | which means it can't retry in that configuration nor do the affinity check, | |
| 15:51:34 | mriedem | so how does the affinity stuff not all blow up? | |
| 15:51:45 | superdan | which is why that workaround config was added and the devstack patch sets it | |
| 15:51:53 | mriedem | oh | |
| 15:52:10 | superdan | mriedem: got a bug for the offload target thing yet? | |
| 15:52:16 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:52:22 | mriedem | oops | |