| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 15:13:35 | sdague | hint | |
| 15:13:48 | ftersin | mdbooth: this patch does not bring a lot of changes for driver.py. Moreover, some changed might be separated out of this patch. E.g. see a series started with https://review.openstack.org/#/c/411942 | |
| 15:13:55 | sdague | then you are dealing with things that wrote their own service management logic outside of devstack | |
| 15:14:06 | sdague | which reacted poorly | |
| 15:14:36 | sdague | because their start/stop/restart calls aren't going through devstack code with vars that would have done the right things | |
| 15:14:40 | openstackgerrit | Chris Friesen proposed openstack/nova master: correctly log port id in neutron api https://review.openstack.org/466003 | |
| 15:14:54 | sdague | so options are: 1) replicate special logic | |
| 15:15:01 | sdague | 2) redo them to use devstack functions | |
| 15:15:17 | sdague | 3) not care about this one job that's in a tarpit | |
| 15:15:45 | sdague | given that general live migration post upgrade is already tested, and multi backend live migration on master is already tested | |
| 15:15:49 | sdague | just not the combo of the 2 | |
| 15:16:34 | mriedem | general live migration post upgrade being the grenade multinode job? | |
| 15:16:39 | sdague | yes | |
| 15:17:05 | sdague | so the missing test coverage right now is post upgrade live migration with ceph / nfs | |
| 15:17:16 | mriedem | only ceph actually | |
| 15:17:19 | mriedem | the nfs stuff was never fixed | |
| 15:17:24 | sdague | ok, so only ceph | |
| 15:17:31 | mriedem | what's multibackend live migration? | |
| 15:17:34 | mriedem | the back and forth thing? | |
| 15:17:34 | sdague | also, note, it's not a vm that predates the migration | |
| 15:17:42 | sdague | mriedem: the normal livemigration job | |
| 15:17:58 | sdague | that tests local disk, and ceph, and I guess not nfs any more | |
| 15:18:05 | mriedem | the back and forth thing was meant to test live migrating from n to n-1 and back | |
| 15:18:24 | sdague | oh, right, this is mixed computes? | |
| 15:18:35 | sdague | right so mixed computes with ceph, not tested | |
| 15:18:48 | sdague | that's what's currently lost | |
| 15:19:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform aggregate.remove_host notification https://review.openstack.org/396211 | |
| 15:20:28 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 15:20:30 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 15:20:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 15:20:42 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 15:20:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 15:21:07 | mriedem | sdague: i'm confused as to why i don't see live migration tempest tests running in the normal grenade multinode job then http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/22cb09f/console.html | |
| 15:21:10 | mriedem | oh b/c smoke | |
| 15:21:37 | mriedem | yeah, that was another thing the grenade live miration job did, was bypass the smoke test stuff | |
| 15:21:51 | mriedem | so the normal grenade multinode job is not running live migration tests since they aren't smoke tests | |
| 15:21:59 | sdague | um... really? | |
| 15:22:04 | sdague | I thought we added those back in tehre | |
| 15:22:49 | mriedem | yeah it's why we added this https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1698 | |
| 15:22:52 | mriedem | to the grenade live migration job | |
| 15:22:56 | sdague | ah | |
| 15:23:12 | mriedem | so we don't have mixed compute live migration back and forth from n to n-1 and back testing | |
| 15:23:14 | sean-k-mooney | mriedem: QQ what is the hosting requirements(days to keep logs) for the 3rd party ci logs? the systemd change has tripled the log size so we are running out of space. | |
| 15:23:26 | mriedem | sean-k-mooney: no idea | |
| 15:23:42 | mriedem | sean-k-mooney: i'd ask in infra probably | |
| 15:24:01 | sdague | mriedem: ok, I honestly though we still had basic live migration in multinode grenade, maybe it wasn't reliable enough | |
| 15:24:04 | mriedem | sean-k-mooney: https://docs.openstack.org/infra/system-config/third_party.html | |
| 15:24:08 | mriedem | Note All test artifacts must be retained for one month. | |
| 15:24:10 | sean-k-mooney | mriedem: we were aiming for 90 days before im looking into geting more storage but we might have to reduce for a bit while we get that set up | |
| 15:24:27 | mriedem | sean-k-mooney: sounds like 1 month minimum | |
| 15:24:30 | sdague | mriedem: ok, so option 4) don't do the special ceph live migration job | |
| 15:24:59 | sdague | and just make that job run live migration grenade tests | |
| 15:25:08 | sean-k-mooney | mriedem: cool thats what i taught. we might reduce to 45 days while we see if we can upgrade form 500GB to 1TB of hosting | |
| 15:25:23 | sdague | because it's the post_hook that's a bit hacking and a problem | |
| 15:25:42 | sean-k-mooney | the nova jobs are eating ~ 13.5GB per day post the systemd change | |
| 15:26:29 | mriedem | sdague: the special ceph live migration job being the non-grenade live migration job that runs with ceph too? that one isn't a problem. | |
| 15:26:49 | sdague | mriedem: I'm saying it all got too compled | |
| 15:26:54 | sdague | there is grenade-multi | |
| 15:27:07 | mriedem | gate-grenade-dsvm-neutron-multinode-ubuntu-xenial | |
| 15:27:08 | mriedem | yes | |
| 15:27:11 | sdague | sorry, let me blast this out | |
| 15:27:27 | sdague | grenade-multi - no ceph, no live migration | |
| 15:27:39 | sdague | dsvm-multi - no ceph, live migration | |
| 15:27:56 | sdague | live-migration - ceph, live migration (custom post_hook, fixed) | |
| 15:28:31 | sdague | grenade-live-migration - ceph, live migration (custom post_hook as above, doesn't work across systemd transition) | |
| 15:28:45 | sdague | there is no grenade-multi-full | |
| 15:28:53 | sdague | that would give you working live migration testing today | |
| 15:29:15 | sdague | the equiv of dsvm-multi | |
| 15:30:49 | mriedem | yeah, now i'm trying to see what's setting live_migrate_back_and_forth in tempest.conf | |
| 15:30:59 | mriedem | because the dsvm-multi job isn't doing it | |
| 15:31:28 | mriedem | ah only set in the grenade-dsvm-neutron-multinode-live-migration job | |
| 15:33:28 | mriedem | but i don't actually see that set in tempest.conf in the job, probably because of the hackery in the hooks in nova | |
| 15:33:29 | mriedem | for that job | |
| 15:34:14 | mriedem | man i don't even know anymore, i might just give up | |
| 15:47:29 | openstackgerrit | Tao Li proposed openstack/nova master: Fix the display of updated_at time when using memcache driver. https://review.openstack.org/414463 | |
| 15:52:57 | mriedem | ildikov: johnthetubaguy: i can't make the nova/cinder meeting today | |
| 15:53:07 | mriedem | stvnoyes: ^ | |
| 15:53:16 | mriedem | i also haven't had a chance to run through steve's live migration wip changes yet | |
| 15:54:59 | ildikov | mriedem: thanks for the heads up | |
| 15:55:40 | sean-k-mooney | mriedem: were you looking for where in the code the live_migration flag is set in the gate-grenade-dsvm-neutron-multinode-live-migration-nv or just what file it was set in | |
| 15:56:42 | mriedem | sean-k-mooney: it's set in local.conf https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1703 | |
| 15:57:12 | mriedem | and should be set in tempest.conf during devstack setup here https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L364 | |
| 15:57:13 | kashyap | cfriesen: Looked at your patch this morning, responded on it | |
| 15:57:16 | sean-k-mooney | mriedem: ah you found it so i was goint to take a look and see if i could find it | |
| 15:57:22 | mriedem | but i'm not actually seeing it set in tempest.conf in that job | |
| 15:57:23 | kashyap | cfriesen: It looks fine to me, nice catch. I presume you stumbled on it by testing | |
| 15:57:32 | sean-k-mooney | i am | |
| 15:57:34 | kashyap | (I mean, in real env) | |
| 15:57:52 | sean-k-mooney | mriedem: http://logs.openstack.org/60/465160/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/c34cb8b/logs/new/tempest_conf.txt.gz | |
| 15:58:20 | mriedem | sean-k-mooney: i don't see live_migrate_back_and_forth in there | |
| 15:58:44 | sean-k-mooney | or sorry i taught you were just looking for live_migration = True sorry | |
| 15:58:52 | mriedem | it's in the loca.conf though http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/new/local.conf.txt.gz | |
| 15:58:57 | mriedem | LIVE_MIGRATE_BACK_AND_FORTH=True | |
| 16:00:15 | dansmith | mriedem: omg, is that really a flag? | |
| 16:00:34 | mriedem | it is | |
| 16:00:38 | dansmith | love. it. | |
| 16:01:00 | mriedem | i must be missing something | |
| 16:01:01 | cfriesen | kashyap: thanks for the review...we did indeed trip over it | |
| 16:01:04 | sean-k-mooney | mriedem: hum maybe it would be better to skip the macro and update the tempest conf via setting [[test-config|/opt/stack/new/tempest/etc/tempest.conf]] ... section in the local.conf | |
| 16:01:22 | mriedem | sean-k-mooney: that didn't exist when this was added | |