| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-28 | |||
| 16:57:39 | mriedem | ^ predates the spike in failures though, which started around 8/18 | |
| 16:57:48 | efried | ...And that's out of the control of ironic-specific code... | |
| 16:58:28 | efried | which is why my comment #5 is a non-starter | |
| 16:58:42 | efried | until, as you say, there's some coordination between RT and virt to manage the RPs. | |
| 16:59:00 | stvnoyes | mriedem: brb- grabbing lunch | |
| 17:00:53 | mriedem | stvnoyes: ok. it also fails on the pike compute... | |
| 17:04:22 | johnthetubaguy | efried: I have to head out now and cook my dinner, but I think this maps nicely to how we setup traits, I am leaning towards ironic doing things in placement | |
| 17:05:02 | efried | johnthetubaguy Okay. I've got a passel of draft comments on your spec, which should be ready for your perusal next time you're on. | |
| 17:05:13 | johnthetubaguy | cool, thanks | |
| 17:06:12 | dansmith | jaypipes: in the db migration, why aren't we referencing the parent provider by id instead of uuid? | |
| 17:06:36 | dansmith | jaypipes: and, isn't recording the root just going to limit us later when we need to restructure a tree? | |
| 17:06:39 | efried | johnthetubaguy (Looking back over 'em, removing the ones we've talked about in here, the only things left are typos ): | |
| 17:07:39 | jaypipes | dansmith: a good question on the parent provider UUID thing. not entirely sure why I did that. | |
| 17:09:22 | dansmith | jaypipes: I can't tell you how elated I feel -1ing a db schema change of yours for performance reasons | |
| 17:09:34 | jaypipes | dansmith: recording the root is an optimization to avoid needing to do hierarchical queries. and we don't restructure the root, only potentially children within the tree (in other words, root_provider_id won't change. | |
| 17:09:41 | jaypipes | dansmith: :) | |
| 17:10:05 | dansmith | jaypipes: right, it won't for compute nodes, but it could for other types of resources | |
| 17:10:22 | dansmith | jaypipes: like you move a disk shelf from one NAS device to another | |
| 17:11:08 | jaypipes | dansmith: possibly, sure, but those kinds of moves are few and far between in comparison to the 10 or 100 times as many read requests for tree data | |
| 17:11:44 | dansmith | hmm | |
| 17:13:33 | dansmith | jaypipes: well, I commented for later | |
| 17:13:45 | jaypipes | okey dokey | |
| 17:14:24 | dansmith | jaypipes: I haven't gotten to how we tell placement about our parent, but presumably we're forbidden from trying to reparent a first-level provider? | |
| 17:15:05 | jaypipes | dansmith: haven't gotten to that either. | |
| 17:15:54 | dansmith | jaypipes: also, in case you're wondering | |
| 17:16:11 | dansmith | jaypipes: yes, it's amazingly beautiful out here on the deck... 71F and clear skies | |
| 17:16:20 | jaypipes | dansmith: lol | |
| 17:26:57 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Return Alternate Hosts https://review.openstack.org/504275 | |
| 17:32:52 | mriedem | stvnoyes: aha | |
| 17:32:58 | mriedem | it's a ci job configuration issue | |
| 17:33:06 | mriedem | the nodes are configured differently for block migration | |
| 17:36:05 | stvnoyes | mriedem: that's good to hear. much better than a code/upgrade issue. | |
| 17:36:46 | mriedem | i'm pretty sure i've had to fix this before... | |
| 17:44:37 | eandersson | Is versioned notifications properly implemented in Mitaka? We don't see any versioned notifications being sent. | |
| 17:44:40 | openstackgerrit | Dan Smith proposed openstack/nova master: Use improved instance_list module in compute API https://review.openstack.org/505418 | |
| 17:44:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix minor input items from previous patches https://review.openstack.org/506416 | |
| 17:44:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix CellDatabases fixture swallowing exceptions https://review.openstack.org/506312 | |
| 17:56:40 | mriedem | eandersson: probably not at that point | |
| 17:57:11 | mriedem | introduced in newton https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/versioned-notification-transformation-newton.html | |
| 17:57:37 | mriedem | the framework code was in mitaka | |
| 17:58:03 | eandersson | I see - thanks mriedem | |
| 18:11:13 | mriedem | mtreinish: do you know anything about how tempest is upgraded in grenade? | |
| 18:12:13 | mtreinish | mriedem: it's not, the code and config should be the same between versions | |
| 18:12:20 | mtreinish | everything should be master for tempest | |
| 18:12:20 | sean-k-mooney | mriedem: i taught tempest was not upgreaed in grenade because it was not ment to be version specific | |
| 18:12:33 | mriedem | mtreinish: the config is different between pike and queens | |
| 18:12:43 | mtreinish | mriedem: links? | |
| 18:12:47 | mriedem | at least in this multinode grenade job | |
| 18:12:48 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/old/tempest_conf.txt.gz | |
| 18:12:51 | mriedem | ^ pike | |
| 18:12:58 | mriedem | note: block_migration_for_live_migration = True | |
| 18:13:06 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/new/tempest_conf.txt.gz | |
| 18:13:08 | mriedem | queens ^ | |
| 18:13:11 | sean-k-mooney | mriedem: the config is likely different yes but it should be useing the same tempest version | |
| 18:13:13 | mriedem | block_migration_for_live_migration = False | |
| 18:13:34 | mriedem | the config is supposed to be copied from old to new https://github.com/openstack-dev/grenade/blob/master/upgrade-tempest#L82 | |
| 18:14:07 | mriedem | looks like that is happening here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_14_23_46_489 | |
| 18:14:11 | mtreinish | mriedem: yeah I don't know why that's being switched or where that's happening | |
| 18:15:43 | sean-k-mooney | is devstack regenerating the config on the second run and overriting it? | |
| 18:15:48 | mriedem | old tempest.conf gets it set here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_13_58_48_782 | |
| 18:16:09 | mtreinish | sean-k-mooney: config should just be copied from old devstack | |
| 18:16:14 | mtreinish | that's how grenade is supposed to work | |
| 18:16:28 | mtreinish | unless it's explicitly changed in the upgrade script | |
| 18:16:45 | mtreinish | which is not something we approve lightly because that means a manual upgrade step | |
| 18:17:22 | sean-k-mooney | mtreinish: yes but devstack generate a new tempest config by default when run so if grenage is reusing that logic it may be chagining it just a guess. you know more about this then i | |
| 18:17:37 | mtreinish | sean-k-mooney: we don't run new devstack | |
| 18:17:47 | sean-k-mooney | ah ok cool | |
| 18:18:10 | mriedem | mtreinish: note this is multinode | |
| 18:18:15 | mriedem | so devstack does run on the subnode | |
| 18:18:40 | mriedem | ENABLED_SERVICES=c-bak,c-vol,ceilometer-acompute,dstat,g-api,n-cpu,peakmem_tracker,placement-client,q-agt | |
| 18:18:40 | mriedem | but the subnode is just | |
| 18:18:59 | mtreinish | mriedem: right and we don't run tempest on the subnode | |
| 18:19:01 | dansmith | mriedem: is there some reason tonyb didn't +W this? https://review.openstack.org/#/c/506760/ | |
| 18:19:03 | dansmith | if not, I'll do it | |
| 18:19:11 | mriedem | dansmith: i assume b/c the change wasn't merged yet on master | |
| 18:19:12 | mtreinish | so even if it was generating a config that shouldn't be coming into play | |
| 18:19:18 | mriedem | er pike | |
| 18:19:23 | dansmith | mriedem: okay but it is now, so good I think? | |
| 18:19:30 | mriedem | yes it's merged on pike | |
| 18:19:58 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Fix wording of debug message for future releases https://review.openstack.org/508261 | |
| 18:19:59 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Only log not correcting allocation once per period https://review.openstack.org/508262 | |
| 18:22:49 | mriedem | mtreinish: found it | |
| 18:22:50 | mriedem | shite | |
| 18:23:03 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/devstack-gate-post_test_hook.txt.gz#_2017-09-26_14_26_36_364 | |
| 18:23:13 | mriedem | it's the gd post-test hook for the live migration job setup in nova | |
| 18:23:53 | mtreinish | mriedem: hah, ok I was looking for plugins and there weren't any. But that makes more sense | |
| 18:24:06 | mriedem | https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L33 | |
| 18:24:43 | mriedem | aha https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L52 | |
| 18:24:46 | mriedem | gdi | |
| 18:24:54 | mriedem | oh f me | |
| 18:24:55 | mriedem | # TODO(mriedem): Remove this in Queens if we haven't fixed the bug yet. | |
| 18:25:05 | mtreinish | haha, nice | |
| 18:27:28 | mriedem | well i think that's for live migration with ceph shared storage, so probably not the actual thing i'm trying to fix, but still | |
| 18:28:29 | mtreinish | mriedem: do you want that block_migration flag to be true or false? | |
| 18:28:53 | mriedem | it needs to be true | |
| 18:28:58 | mriedem | but hold up | |
| 18:29:32 | mriedem | i'm confused as to where the post-test-hook is called | |
| 18:29:41 | mtreinish | mriedem: it's called in devstack gate | |
| 18:30:16 | mtreinish | the second tempest run is done by devstack gate instead of grenade | |
| 18:30:37 | mtreinish | and looking at that post test hook code you're telling devstack gate to not run tempest, so the hook can modify the tempest config and run it itself | |