| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-28 | |||
| 18:14:07 | mriedem | looks like that is happening here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_14_23_46_489 | |
| 18:14:11 | mtreinish | mriedem: yeah I don't know why that's being switched or where that's happening | |
| 18:15:43 | sean-k-mooney | is devstack regenerating the config on the second run and overriting it? | |
| 18:15:48 | mriedem | old tempest.conf gets it set here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/grenade.sh.txt.gz#_2017-09-26_13_58_48_782 | |
| 18:16:09 | mtreinish | sean-k-mooney: config should just be copied from old devstack | |
| 18:16:14 | mtreinish | that's how grenade is supposed to work | |
| 18:16:28 | mtreinish | unless it's explicitly changed in the upgrade script | |
| 18:16:45 | mtreinish | which is not something we approve lightly because that means a manual upgrade step | |
| 18:17:22 | sean-k-mooney | mtreinish: yes but devstack generate a new tempest config by default when run so if grenage is reusing that logic it may be chagining it just a guess. you know more about this then i | |
| 18:17:37 | mtreinish | sean-k-mooney: we don't run new devstack | |
| 18:17:47 | sean-k-mooney | ah ok cool | |
| 18:18:10 | mriedem | mtreinish: note this is multinode | |
| 18:18:15 | mriedem | so devstack does run on the subnode | |
| 18:18:40 | mriedem | ENABLED_SERVICES=c-bak,c-vol,ceilometer-acompute,dstat,g-api,n-cpu,peakmem_tracker,placement-client,q-agt | |
| 18:18:40 | mriedem | but the subnode is just | |
| 18:18:59 | mtreinish | mriedem: right and we don't run tempest on the subnode | |
| 18:19:01 | dansmith | mriedem: is there some reason tonyb didn't +W this? https://review.openstack.org/#/c/506760/ | |
| 18:19:03 | dansmith | if not, I'll do it | |
| 18:19:11 | mriedem | dansmith: i assume b/c the change wasn't merged yet on master | |
| 18:19:12 | mtreinish | so even if it was generating a config that shouldn't be coming into play | |
| 18:19:18 | mriedem | er pike | |
| 18:19:23 | dansmith | mriedem: okay but it is now, so good I think? | |
| 18:19:30 | mriedem | yes it's merged on pike | |
| 18:19:58 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Fix wording of debug message for future releases https://review.openstack.org/508261 | |
| 18:19:59 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Only log not correcting allocation once per period https://review.openstack.org/508262 | |
| 18:22:49 | mriedem | mtreinish: found it | |
| 18:22:50 | mriedem | shite | |
| 18:23:03 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/devstack-gate-post_test_hook.txt.gz#_2017-09-26_14_26_36_364 | |
| 18:23:13 | mriedem | it's the gd post-test hook for the live migration job setup in nova | |
| 18:23:53 | mtreinish | mriedem: hah, ok I was looking for plugins and there weren't any. But that makes more sense | |
| 18:24:06 | mriedem | https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L33 | |
| 18:24:43 | mriedem | aha https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/tests/live_migration/hooks/run_tests.sh#L52 | |
| 18:24:46 | mriedem | gdi | |
| 18:24:54 | mriedem | oh f me | |
| 18:24:55 | mriedem | # TODO(mriedem): Remove this in Queens if we haven't fixed the bug yet. | |
| 18:25:05 | mtreinish | haha, nice | |
| 18:27:28 | mriedem | well i think that's for live migration with ceph shared storage, so probably not the actual thing i'm trying to fix, but still | |
| 18:28:29 | mtreinish | mriedem: do you want that block_migration flag to be true or false? | |
| 18:28:53 | mriedem | it needs to be true | |
| 18:28:58 | mriedem | but hold up | |
| 18:29:32 | mriedem | i'm confused as to where the post-test-hook is called | |
| 18:29:41 | mtreinish | mriedem: it's called in devstack gate | |
| 18:30:16 | mtreinish | the second tempest run is done by devstack gate instead of grenade | |
| 18:30:37 | mtreinish | and looking at that post test hook code you're telling devstack gate to not run tempest, so the hook can modify the tempest config and run it itself | |
| 18:31:24 | mriedem | the reason the hook is setting block_migration=False is because the tests that come after that are for shared storage (nfs and ceph) | |
| 18:31:40 | mriedem | so i'm confused as to why that's configuring tempest before the tests are run | |
| 18:32:03 | mriedem | because i can see from the failed tempest log, that tempest is passing block_migration=False b/c that's what's in the config | |
| 18:32:59 | mriedem | starts running here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_25_30_596259 | |
| 18:33:32 | mriedem | at that point, things pass | |
| 18:34:05 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_26_36_296539 | |
| 18:34:37 | mriedem | oh ffs | |
| 18:34:41 | mriedem | it is the branch thing | |
| 18:34:54 | mriedem | it passes the first run with live block migration, and then fails the ceph one | |
| 18:35:00 | mriedem | presumably because we just suck with ceph still | |
| 18:36:52 | sean-k-mooney | mriedem: right so it failing later here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_28_50_096937 | |
| 18:39:04 | mriedem | well, so bug 1691769 isn't a problem anymore | |
| 18:39:05 | openstack | bug 1691769 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode-live-migration-nv fails in pike: "Failed to restart |
|
| 18:42:08 | mtreinish | email address hidden? | |
| 18:42:30 | mtreinish | hah systemd units | |
| 18:43:30 | mriedem | aha | |
| 18:43:33 | mriedem | another discovery | |
| 18:43:44 | mriedem | which dansmith might remember | |
| 18:43:51 | mriedem | grenade runs in singleconductor more | |
| 18:43:52 | mriedem | *mode | |
| 18:43:54 | mriedem | but, | |
| 18:44:09 | mriedem | this live migration test hook is configuring /etc/nova/nova-cpu.conf | |
| 18:44:10 | mriedem | http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/subnode-2/etc/nova/nova-cpu.conf.txt.gz | |
| 18:44:45 | mriedem | and it's not running with that one | |
| 18:44:46 | mriedem | ['--config-file', '/etc/nova/nova.conf'] | |
| 18:46:35 | mriedem | Ibc4b21089ef86ab2430874c39d63174528c9a83e | |
| 18:47:18 | sean-k-mooney | right its doing that here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_27_39_368886 and running and here its using /etc/nova/nova.conf http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_27_56_301409 | |
| 18:48:07 | mriedem | yup, the issue is that https://review.openstack.org/#/c/458537/ isn't aware of grenade being singleconductor | |
| 18:48:27 | mriedem | because, jesus, live migration + grenade + superconductor...crazy | |
| 18:52:05 | sean-k-mooney | so the fix is the run nova-compute with /etc/nova/nova-cpu.conf ? | |
| 18:52:38 | mriedem | no | |
| 18:55:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271 | |
| 18:55:30 | mriedem | i think this is the fix ^ | |
| 18:58:04 | sean-k-mooney | hum that should work however it may be better to always use a seperate config for nova compute so that it does not chage regardless of the config. | |
| 18:58:09 | mriedem | i don't think zuul is running any jobs right now though so, i guess we'll find out later | |
| 18:58:33 | mriedem | sean-k-mooney: oh i'm sure this is air tight and will never require changes again :) | |
| 18:58:52 | sean-k-mooney | :) | |
| 18:59:25 | sean-k-mooney | by the way https://github.com/openstack/nova/search?utf8=%E2%9C%93&q=TODO%28+Queens&type= that was not the only todo with remove in queens ... | |
| 19:00:37 | sean-k-mooney | stephenfin: i belive this one is all yours https://github.com/openstack/nova/blob/762c89f85a4aeceb7bed5e80edc70a239872b757/nova/cmd/manage.py#L141 | |
| 19:01:04 | tssurya | mriedem, dansmith, melwitt : random off-topic : saw the nova cellv2 meeting logs from yesterday, so sorry I could not attend (was afk due to bad time zone timings) it and for wasting some precious seconds during the meeting; but thanks for the ping. Also from next time I will keep you posted earlier on if I am not attending. | |
| 19:01:31 | sean-k-mooney | wait i just confused myself queens comes after pike... | |
| 19:01:44 | dansmith | tssurya: not a problem and no need to keep us notified of your schedule, I just made a point of pinging you because we had fooled you with our hangout the week prior | |
| 19:02:05 | melwitt | no worries tssurya. yep, what dansmith said | |
| 19:02:55 | tssurya | dansmith, melwitt : cool :) thanks! | |
| 19:05:03 | mriedem | tssurya: i got to thinking, | |
| 19:05:09 | mriedem | cern is upgrading to pike right now right? | |
| 19:05:14 | mriedem | and cells v2 was required in ocata, | |
| 19:05:34 | mriedem | so it would be nice if you could give us some info on how cern upgraded to ocata and rolled out cellsv2 even though they are using cells v1 | |
| 19:05:47 | mriedem | like, was cern in ocata just a single giant cells v2 cell? | |
| 19:06:03 | tssurya | mriedem : no cern is still on newton, it is going to upgrade to ocata | |
| 19:06:04 | mriedem | or, is cern actually migrating to cells v2 | |
| 19:06:16 | mriedem | i saw something the other day saying they were upgrading glance to pike | |
| 19:06:19 | mriedem | maybe that's just glance> | |
| 19:06:20 | mriedem | ? | |
| 19:06:27 | tssurya | mriedem : yes its just glance | |
| 19:06:31 | mriedem | gah! | |