| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-28 | |||
| 16:55:02 | jaypipes | efried: we want to get to where there is a sort of mediation/negotiation between the RT and the virt driver and the generic device manager as each of them "processes" the ProviderTree and sets inventory records and traits based on their own information. | |
| 16:55:03 | mriedem | tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration [10.052345s] ... FAILED | |
| 16:55:17 | mriedem | so it works with the 2.25 microversion, where we pass block_migration=auto | |
| 16:55:21 | mriedem | but fails before that | |
| 16:55:22 | jaypipes | efried: in ironic, each baremetal Ironic node is its own RP in placementy. | |
| 16:55:48 | mriedem | so i bet something with the auto changes broke how we do live block migration between n-1 hosts | |
| 16:56:25 | stvnoyes | what were the auto changes? | |
| 16:56:26 | efried | jaypipes Aha. Then bug 1714248 exists because... the compute host is creating its own separate RP for some reason? | |
| 16:56:28 | openstack | bug 1714248 in OpenStack Compute (nova) "Compute node HA for ironic doesn't work due to the name duplication of Resource Provider " [High,Confirmed] https://launchpad.net/bugs/1714248 | |
| 16:56:53 | jaypipes | efried: not sure, lemme read | |
| 16:57:18 | efried | oh, or because the RT is trying to create a new RP for the node it's taking over... | |
| 16:57:29 | jaypipes | efried: note that nova-compute was *never* intended for "HA setups"... | |
| 16:57:29 | mriedem | stvnoyes: https://docs.openstack.org/nova/pike/reference/api-microversion-history.html#maximum-in-mitaka | |
| 16:57:39 | mriedem | ^ predates the spike in failures though, which started around 8/18 | |
| 16:57:48 | efried | ...And that's out of the control of ironic-specific code... | |
| 16:58:28 | efried | which is why my comment #5 is a non-starter | |
| 16:58:42 | efried | until, as you say, there's some coordination between RT and virt to manage the RPs. | |
| 16:59:00 | stvnoyes | mriedem: brb- grabbing lunch | |
| 17:00:53 | mriedem | stvnoyes: ok. it also fails on the pike compute... | |
| 17:04:22 | johnthetubaguy | efried: I have to head out now and cook my dinner, but I think this maps nicely to how we setup traits, I am leaning towards ironic doing things in placement | |
| 17:05:02 | efried | johnthetubaguy Okay. I've got a passel of draft comments on your spec, which should be ready for your perusal next time you're on. | |
| 17:05:13 | johnthetubaguy | cool, thanks | |
| 17:06:12 | dansmith | jaypipes: in the db migration, why aren't we referencing the parent provider by id instead of uuid? | |
| 17:06:36 | dansmith | jaypipes: and, isn't recording the root just going to limit us later when we need to restructure a tree? | |
| 17:06:39 | efried | johnthetubaguy (Looking back over 'em, removing the ones we've talked about in here, the only things left are typos ): | |
| 17:07:39 | jaypipes | dansmith: a good question on the parent provider UUID thing. not entirely sure why I did that. | |
| 17:09:22 | dansmith | jaypipes: I can't tell you how elated I feel -1ing a db schema change of yours for performance reasons | |
| 17:09:34 | jaypipes | dansmith: recording the root is an optimization to avoid needing to do hierarchical queries. and we don't restructure the root, only potentially children within the tree (in other words, root_provider_id won't change. | |
| 17:09:41 | jaypipes | dansmith: :) | |
| 17:10:05 | dansmith | jaypipes: right, it won't for compute nodes, but it could for other types of resources | |
| 17:10:22 | dansmith | jaypipes: like you move a disk shelf from one NAS device to another | |
| 17:11:08 | jaypipes | dansmith: possibly, sure, but those kinds of moves are few and far between in comparison to the 10 or 100 times as many read requests for tree data | |
| 17:11:44 | dansmith | hmm | |
| 17:13:33 | dansmith | jaypipes: well, I commented for later | |
| 17:13:45 | jaypipes | okey dokey | |
| 17:14:24 | dansmith | jaypipes: I haven't gotten to how we tell placement about our parent, but presumably we're forbidden from trying to reparent a first-level provider? | |
| 17:15:05 | jaypipes | dansmith: haven't gotten to that either. | |
| 17:15:54 | dansmith | jaypipes: also, in case you're wondering | |
| 17:16:11 | dansmith | jaypipes: yes, it's amazingly beautiful out here on the deck... 71F and clear skies | |
| 17:16:20 | jaypipes | dansmith: lol | |
| 17:26:57 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Return Alternate Hosts https://review.openstack.org/504275 | |
| 17:32:52 | mriedem | stvnoyes: aha | |
| 17:32:58 | mriedem | it's a ci job configuration issue | |
| 17:33:06 | mriedem | the nodes are configured differently for block migration | |
| 17:36:05 | stvnoyes | mriedem: that's good to hear. much better than a code/upgrade issue. | |
| 17:36:46 | mriedem | i'm pretty sure i've had to fix this before... | |
| 17:44:37 | eandersson | Is versioned notifications properly implemented in Mitaka? We don't see any versioned notifications being sent. | |
| 17:44:40 | openstackgerrit | Dan Smith proposed openstack/nova master: Use improved instance_list module in compute API https://review.openstack.org/505418 | |
| 17:44:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix CellDatabases fixture swallowing exceptions https://review.openstack.org/506312 | |
| 17:44:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix minor input items from previous patches https://review.openstack.org/506416 | |
| 17:56:40 | mriedem | eandersson: probably not at that point | |
| 17:57:11 | mriedem | introduced in newton https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/versioned-notification-transformation-newton.html | |
| 17:57:37 | mriedem | the framework code was in mitaka | |
| 17:58:03 | eandersson | I see - thanks mriedem | |
| 18:11:13 | mriedem | mtreinish: do you know anything about how tempest is upgraded in grenade? | |
| 18:12:13 | mtreinish | mriedem: it's not, the code and config should be the same between versions | |
| 18:12:20 | sean-k-mooney | mriedem: i taught tempest was not upgreaed in grenade because it was not ment to be version specific | |
| 18:12:20 | mtreinish | everything should be master for tempest | |
| 18:12:33 | mriedem | mtreinish: the config is different between pike and queens | |
| 18:12:43 | mtreinish | mriedem: links? | |
| 18:12:47 | mriedem | at least in this multinode grenade job | |
| 18:12:48 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/old/tempest_conf.txt.gz | |
| 18:12:51 | mriedem | ^ pike | |
| 18:12:58 | mriedem | note: block_migration_for_live_migration = True | |
| 18:13:06 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/new/tempest_conf.txt.gz | |
| 18:13:08 | mriedem | queens ^ | |
| 18:13:11 | sean-k-mooney | mriedem: the config is likely different yes but it should be useing the same tempest version | |
| 18:13:13 | mriedem | block_migration_for_live_migration = False | |
| 18:13:34 | mriedem | the config is supposed to be copied from old to new https://github.com/openstack-dev/grenade/blob/master/upgrade-tempest#L82 | |
| 18:14:07 | mriedem | looks like that is happening here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_14_23_46_489 | |
| 18:14:11 | mtreinish | mriedem: yeah I don't know why that's being switched or where that's happening | |
| 18:15:43 | sean-k-mooney | is devstack regenerating the config on the second run and overriting it? | |
| 18:15:48 | mriedem | old tempest.conf gets it set here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_13_58_48_782 | |
| 18:16:09 | mtreinish | sean-k-mooney: config should just be copied from old devstack | |
| 18:16:14 | mtreinish | that's how grenade is supposed to work | |
| 18:16:28 | mtreinish | unless it's explicitly changed in the upgrade script | |
| 18:16:45 | mtreinish | which is not something we approve lightly because that means a manual upgrade step | |
| 18:17:22 | sean-k-mooney | mtreinish: yes but devstack generate a new tempest config by default when run so if grenage is reusing that logic it may be chagining it just a guess. you know more about this then i | |
| 18:17:37 | mtreinish | sean-k-mooney: we don't run new devstack | |
| 18:17:47 | sean-k-mooney | ah ok cool | |
| 18:18:10 | mriedem | mtreinish: note this is multinode | |
| 18:18:15 | mriedem | so devstack does run on the subnode | |
| 18:18:40 | mriedem | but the subnode is just | |
| 18:18:40 | mriedem | ENABLED_SERVICES=c-bak,c-vol,ceilometer-acompute,dstat,g-api,n-cpu,peakmem_tracker,placement-client,q-agt | |
| 18:18:59 | mtreinish | mriedem: right and we don't run tempest on the subnode | |
| 18:19:01 | dansmith | mriedem: is there some reason tonyb didn't +W this? https://review.openstack.org/#/c/506760/ | |
| 18:19:03 | dansmith | if not, I'll do it | |
| 18:19:11 | mriedem | dansmith: i assume b/c the change wasn't merged yet on master | |
| 18:19:12 | mtreinish | so even if it was generating a config that shouldn't be coming into play | |
| 18:19:18 | mriedem | er pike | |
| 18:19:23 | dansmith | mriedem: okay but it is now, so good I think? | |
| 18:19:30 | mriedem | yes it's merged on pike | |
| 18:19:58 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Fix wording of debug message for future releases https://review.openstack.org/508261 | |
| 18:19:59 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Only log not correcting allocation once per period https://review.openstack.org/508262 | |
| 18:22:49 | mriedem | mtreinish: found it | |
| 18:22:50 | mriedem | shite | |
| 18:23:03 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/devstack-gate-post_test_hook.txt.gz#_2017-09-26_14_26_36_364 | |
| 18:23:13 | mriedem | it's the gd post-test hook for the live migration job setup in nova | |
| 18:23:53 | mtreinish | mriedem: hah, ok I was looking for plugins and there weren't any. But that makes more sense | |
| 18:24:06 | mriedem | https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L33 | |