| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 15:04:44 | mriedem | ftersin: mdbooth | |
| 15:05:11 | ftersin | mriedem: thx | |
| 15:07:02 | ftersin | mdbooth: hi. are you around? | |
| 15:07:12 | mdbooth | ftersin: Yep | |
| 15:07:57 | ftersin | mdbooth: could you look at https://review.openstack.org/#/c/407440/ again? | |
| 15:08:05 | mdbooth | Yeah, sure. | |
| 15:08:32 | mdbooth | So, my issue with that patch has always been the number of changes required in driver.py | |
| 15:08:57 | mdbooth | Which is obvious an artifact of the imagebackend interface being completely borked | |
| 15:08:59 | ftersin | mdbooth: i addressed some your comments there. so, it'd be good to get new ones :) | |
| 15:09:06 | mdbooth | Somebody should clean that up... ;) | |
| 15:09:28 | mdbooth | Anyway, we also agreed at some point to move on with it anyway, so I'll try to look again in that light | |
| 15:09:47 | mriedem | sdague: ok, so maybe i didn't ask the right question, | |
| 15:10:01 | mriedem | i'm basically wondering what i can key off of to fix this grenade live migration issue | |
| 15:10:07 | sdague | mriedem: no | |
| 15:10:18 | sdague | you are going to need more complex logic | |
| 15:10:25 | openstackgerrit | Chris Friesen proposed openstack/nova master: correctly log port id in neutron api https://review.openstack.org/466003 | |
| 15:10:36 | cfriesen | ^ trivial fix to logging | |
| 15:11:08 | sdague | mriedem: https://github.com/openstack-dev/devstack/blob/6a62619cbc28625755a76eae08b529438eb4f281/stackrc#L166-L173 | |
| 15:11:17 | sdague | that's the gate level work around, and why grenade works | |
| 15:11:18 | cfriesen | crap, let me respin that...an extra file snuck in | |
| 15:12:47 | sdague | mriedem: USE_SCREEN=False is always the case for the gate | |
| 15:13:02 | sdague | it's just that we decided to make the transition simpler by making that mean something else | |
| 15:13:13 | sdague | so 95% of things just worked | |
| 15:13:14 | dpawlik | mriedem: thanks for review. DocImpact-Group: openstack/glance is because someone on this channel told me that docs are related to Openstack Glance. I will create a new PS soon. Thanks! | |
| 15:13:19 | mriedem | sdague: ok i see that in devstack-gate localrc_set "$localrc_file" "USE_SCREEN" "False" | |
| 15:13:24 | sdague | the grenade cases needed this one extra hit | |
| 15:13:35 | sdague | hint | |
| 15:13:48 | ftersin | mdbooth: this patch does not bring a lot of changes for driver.py. Moreover, some changed might be separated out of this patch. E.g. see a series started with https://review.openstack.org/#/c/411942 | |
| 15:13:55 | sdague | then you are dealing with things that wrote their own service management logic outside of devstack | |
| 15:14:06 | sdague | which reacted poorly | |
| 15:14:36 | sdague | because their start/stop/restart calls aren't going through devstack code with vars that would have done the right things | |
| 15:14:40 | openstackgerrit | Chris Friesen proposed openstack/nova master: correctly log port id in neutron api https://review.openstack.org/466003 | |
| 15:14:54 | sdague | so options are: 1) replicate special logic | |
| 15:15:01 | sdague | 2) redo them to use devstack functions | |
| 15:15:17 | sdague | 3) not care about this one job that's in a tarpit | |
| 15:15:45 | sdague | given that general live migration post upgrade is already tested, and multi backend live migration on master is already tested | |
| 15:15:49 | sdague | just not the combo of the 2 | |
| 15:16:34 | mriedem | general live migration post upgrade being the grenade multinode job? | |
| 15:16:39 | sdague | yes | |
| 15:17:05 | sdague | so the missing test coverage right now is post upgrade live migration with ceph / nfs | |
| 15:17:16 | mriedem | only ceph actually | |
| 15:17:19 | mriedem | the nfs stuff was never fixed | |
| 15:17:24 | sdague | ok, so only ceph | |
| 15:17:31 | mriedem | what's multibackend live migration? | |
| 15:17:34 | sdague | also, note, it's not a vm that predates the migration | |
| 15:17:34 | mriedem | the back and forth thing? | |
| 15:17:42 | sdague | mriedem: the normal livemigration job | |
| 15:17:58 | sdague | that tests local disk, and ceph, and I guess not nfs any more | |
| 15:18:05 | mriedem | the back and forth thing was meant to test live migrating from n to n-1 and back | |
| 15:18:24 | sdague | oh, right, this is mixed computes? | |
| 15:18:35 | sdague | right so mixed computes with ceph, not tested | |
| 15:18:48 | sdague | that's what's currently lost | |
| 15:19:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform aggregate.remove_host notification https://review.openstack.org/396211 | |
| 15:20:28 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 15:20:30 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 15:20:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 15:20:42 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 15:20:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 15:21:07 | mriedem | sdague: i'm confused as to why i don't see live migration tempest tests running in the normal grenade multinode job then http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/22cb09f/console.html | |
| 15:21:10 | mriedem | oh b/c smoke | |
| 15:21:37 | mriedem | yeah, that was another thing the grenade live miration job did, was bypass the smoke test stuff | |
| 15:21:51 | mriedem | so the normal grenade multinode job is not running live migration tests since they aren't smoke tests | |
| 15:21:59 | sdague | um... really? | |
| 15:22:04 | sdague | I thought we added those back in tehre | |
| 15:22:49 | mriedem | yeah it's why we added this https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1698 | |
| 15:22:52 | mriedem | to the grenade live migration job | |
| 15:22:56 | sdague | ah | |
| 15:23:12 | mriedem | so we don't have mixed compute live migration back and forth from n to n-1 and back testing | |
| 15:23:14 | sean-k-mooney | mriedem: QQ what is the hosting requirements(days to keep logs) for the 3rd party ci logs? the systemd change has tripled the log size so we are running out of space. | |
| 15:23:26 | mriedem | sean-k-mooney: no idea | |
| 15:23:42 | mriedem | sean-k-mooney: i'd ask in infra probably | |
| 15:24:01 | sdague | mriedem: ok, I honestly though we still had basic live migration in multinode grenade, maybe it wasn't reliable enough | |
| 15:24:04 | mriedem | sean-k-mooney: https://docs.openstack.org/infra/system-config/third_party.html | |
| 15:24:08 | mriedem | Note All test artifacts must be retained for one month. | |
| 15:24:10 | sean-k-mooney | mriedem: we were aiming for 90 days before im looking into geting more storage but we might have to reduce for a bit while we get that set up | |
| 15:24:27 | mriedem | sean-k-mooney: sounds like 1 month minimum | |
| 15:24:30 | sdague | mriedem: ok, so option 4) don't do the special ceph live migration job | |
| 15:24:59 | sdague | and just make that job run live migration grenade tests | |
| 15:25:08 | sean-k-mooney | mriedem: cool thats what i taught. we might reduce to 45 days while we see if we can upgrade form 500GB to 1TB of hosting | |
| 15:25:23 | sdague | because it's the post_hook that's a bit hacking and a problem | |
| 15:25:42 | sean-k-mooney | the nova jobs are eating ~ 13.5GB per day post the systemd change | |
| 15:26:29 | mriedem | sdague: the special ceph live migration job being the non-grenade live migration job that runs with ceph too? that one isn't a problem. | |
| 15:26:49 | sdague | mriedem: I'm saying it all got too compled | |
| 15:26:54 | sdague | there is grenade-multi | |
| 15:27:07 | mriedem | gate-grenade-dsvm-neutron-multinode-ubuntu-xenial | |
| 15:27:08 | mriedem | yes | |
| 15:27:11 | sdague | sorry, let me blast this out | |
| 15:27:27 | sdague | grenade-multi - no ceph, no live migration | |
| 15:27:39 | sdague | dsvm-multi - no ceph, live migration | |
| 15:27:56 | sdague | live-migration - ceph, live migration (custom post_hook, fixed) | |
| 15:28:31 | sdague | grenade-live-migration - ceph, live migration (custom post_hook as above, doesn't work across systemd transition) | |
| 15:28:45 | sdague | there is no grenade-multi-full | |
| 15:28:53 | sdague | that would give you working live migration testing today | |
| 15:29:15 | sdague | the equiv of dsvm-multi | |
| 15:30:49 | mriedem | yeah, now i'm trying to see what's setting live_migrate_back_and_forth in tempest.conf | |
| 15:30:59 | mriedem | because the dsvm-multi job isn't doing it | |
| 15:31:28 | mriedem | ah only set in the grenade-dsvm-neutron-multinode-live-migration job | |
| 15:33:28 | mriedem | but i don't actually see that set in tempest.conf in the job, probably because of the hackery in the hooks in nova | |
| 15:33:29 | mriedem | for that job | |
| 15:34:14 | mriedem | man i don't even know anymore, i might just give up | |