| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-28 | |||
| 17:13:45 | jaypipes | okey dokey | |
| 17:14:24 | dansmith | jaypipes: I haven't gotten to how we tell placement about our parent, but presumably we're forbidden from trying to reparent a first-level provider? | |
| 17:15:05 | jaypipes | dansmith: haven't gotten to that either. | |
| 17:15:54 | dansmith | jaypipes: also, in case you're wondering | |
| 17:16:11 | dansmith | jaypipes: yes, it's amazingly beautiful out here on the deck... 71F and clear skies | |
| 17:16:20 | jaypipes | dansmith: lol | |
| 17:26:57 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Return Alternate Hosts https://review.openstack.org/504275 | |
| 17:32:52 | mriedem | stvnoyes: aha | |
| 17:32:58 | mriedem | it's a ci job configuration issue | |
| 17:33:06 | mriedem | the nodes are configured differently for block migration | |
| 17:36:05 | stvnoyes | mriedem: that's good to hear. much better than a code/upgrade issue. | |
| 17:36:46 | mriedem | i'm pretty sure i've had to fix this before... | |
| 17:44:37 | eandersson | Is versioned notifications properly implemented in Mitaka? We don't see any versioned notifications being sent. | |
| 17:44:40 | openstackgerrit | Dan Smith proposed openstack/nova master: Use improved instance_list module in compute API https://review.openstack.org/505418 | |
| 17:44:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix minor input items from previous patches https://review.openstack.org/506416 | |
| 17:44:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix CellDatabases fixture swallowing exceptions https://review.openstack.org/506312 | |
| 17:56:40 | mriedem | eandersson: probably not at that point | |
| 17:57:11 | mriedem | introduced in newton https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/versioned-notification-transformation-newton.html | |
| 17:57:37 | mriedem | the framework code was in mitaka | |
| 17:58:03 | eandersson | I see - thanks mriedem | |
| 18:11:13 | mriedem | mtreinish: do you know anything about how tempest is upgraded in grenade? | |
| 18:12:13 | mtreinish | mriedem: it's not, the code and config should be the same between versions | |
| 18:12:20 | mtreinish | everything should be master for tempest | |
| 18:12:20 | sean-k-mooney | mriedem: i taught tempest was not upgreaed in grenade because it was not ment to be version specific | |
| 18:12:33 | mriedem | mtreinish: the config is different between pike and queens | |
| 18:12:43 | mtreinish | mriedem: links? | |
| 18:12:47 | mriedem | at least in this multinode grenade job | |
| 18:12:48 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/old/tempest_conf.txt.gz | |
| 18:12:51 | mriedem | ^ pike | |
| 18:12:58 | mriedem | note: block_migration_for_live_migration = True | |
| 18:13:06 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/new/tempest_conf.txt.gz | |
| 18:13:08 | mriedem | queens ^ | |
| 18:13:11 | sean-k-mooney | mriedem: the config is likely different yes but it should be useing the same tempest version | |
| 18:13:13 | mriedem | block_migration_for_live_migration = False | |
| 18:13:34 | mriedem | the config is supposed to be copied from old to new https://github.com/openstack-dev/grenade/blob/master/upgrade-tempest#L82 | |
| 18:14:07 | mriedem | looks like that is happening here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_14_23_46_489 | |
| 18:14:11 | mtreinish | mriedem: yeah I don't know why that's being switched or where that's happening | |
| 18:15:43 | sean-k-mooney | is devstack regenerating the config on the second run and overriting it? | |
| 18:15:48 | mriedem | old tempest.conf gets it set here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_13_58_48_782 | |
| 18:16:09 | mtreinish | sean-k-mooney: config should just be copied from old devstack | |
| 18:16:14 | mtreinish | that's how grenade is supposed to work | |
| 18:16:28 | mtreinish | unless it's explicitly changed in the upgrade script | |
| 18:16:45 | mtreinish | which is not something we approve lightly because that means a manual upgrade step | |
| 18:17:22 | sean-k-mooney | mtreinish: yes but devstack generate a new tempest config by default when run so if grenage is reusing that logic it may be chagining it just a guess. you know more about this then i | |
| 18:17:37 | mtreinish | sean-k-mooney: we don't run new devstack | |
| 18:17:47 | sean-k-mooney | ah ok cool | |
| 18:18:10 | mriedem | mtreinish: note this is multinode | |
| 18:18:15 | mriedem | so devstack does run on the subnode | |
| 18:18:40 | mriedem | ENABLED_SERVICES=c-bak,c-vol,ceilometer-acompute,dstat,g-api,n-cpu,peakmem_tracker,placement-client,q-agt | |
| 18:18:40 | mriedem | but the subnode is just | |
| 18:18:59 | mtreinish | mriedem: right and we don't run tempest on the subnode | |
| 18:19:01 | dansmith | mriedem: is there some reason tonyb didn't +W this? https://review.openstack.org/#/c/506760/ | |
| 18:19:03 | dansmith | if not, I'll do it | |
| 18:19:11 | mriedem | dansmith: i assume b/c the change wasn't merged yet on master | |
| 18:19:12 | mtreinish | so even if it was generating a config that shouldn't be coming into play | |
| 18:19:18 | mriedem | er pike | |
| 18:19:23 | dansmith | mriedem: okay but it is now, so good I think? | |
| 18:19:30 | mriedem | yes it's merged on pike | |
| 18:19:58 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Fix wording of debug message for future releases https://review.openstack.org/508261 | |
| 18:19:59 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Only log not correcting allocation once per period https://review.openstack.org/508262 | |
| 18:22:49 | mriedem | mtreinish: found it | |
| 18:22:50 | mriedem | shite | |
| 18:23:03 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/devstack-gate-post_test_hook.txt.gz#_2017-09-26_14_26_36_364 | |
| 18:23:13 | mriedem | it's the gd post-test hook for the live migration job setup in nova | |
| 18:23:53 | mtreinish | mriedem: hah, ok I was looking for plugins and there weren't any. But that makes more sense | |
| 18:24:06 | mriedem | https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L33 | |
| 18:24:43 | mriedem | aha https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L52 | |
| 18:24:46 | mriedem | gdi | |
| 18:24:54 | mriedem | oh f me | |
| 18:24:55 | mriedem | # TODO(mriedem): Remove this in Queens if we haven't fixed the bug yet. | |
| 18:25:05 | mtreinish | haha, nice | |
| 18:27:28 | mriedem | well i think that's for live migration with ceph shared storage, so probably not the actual thing i'm trying to fix, but still | |
| 18:28:29 | mtreinish | mriedem: do you want that block_migration flag to be true or false? | |
| 18:28:53 | mriedem | it needs to be true | |
| 18:28:58 | mriedem | but hold up | |
| 18:29:32 | mriedem | i'm confused as to where the post-test-hook is called | |
| 18:29:41 | mtreinish | mriedem: it's called in devstack gate | |
| 18:30:16 | mtreinish | the second tempest run is done by devstack gate instead of grenade | |
| 18:30:37 | mtreinish | and looking at that post test hook code you're telling devstack gate to not run tempest, so the hook can modify the tempest config and run it itself | |
| 18:31:24 | mriedem | the reason the hook is setting block_migration=False is because the tests that come after that are for shared storage (nfs and ceph) | |
| 18:31:40 | mriedem | so i'm confused as to why that's configuring tempest before the tests are run | |
| 18:32:03 | mriedem | because i can see from the failed tempest log, that tempest is passing block_migration=False b/c that's what's in the config | |
| 18:32:59 | mriedem | starts running here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_25_30_596259 | |
| 18:33:32 | mriedem | at that point, things pass | |
| 18:34:05 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_26_36_296539 | |
| 18:34:37 | mriedem | oh ffs | |
| 18:34:41 | mriedem | it is the branch thing | |
| 18:34:54 | mriedem | it passes the first run with live block migration, and then fails the ceph one | |
| 18:35:00 | mriedem | presumably because we just suck with ceph still | |
| 18:36:52 | sean-k-mooney | mriedem: right so it failing later here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_28_50_096937 | |
| 18:39:04 | mriedem | well, so bug 1691769 isn't a problem anymore | |
| 18:39:05 | openstack | bug 1691769 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode-live-migration-nv fails in pike: "Failed to restart |
|
| 18:42:08 | mtreinish | email address hidden? | |
| 18:42:30 | mtreinish | hah systemd units | |
| 18:43:30 | mriedem | aha | |
| 18:43:33 | mriedem | another discovery | |
| 18:43:44 | mriedem | which dansmith might remember | |
| 18:43:51 | mriedem | grenade runs in singleconductor more | |
| 18:43:52 | mriedem | *mode | |
| 18:43:54 | mriedem | but, | |