| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 14:02:27 | sdague | mriedem: grenade completed fine | |
| 14:02:46 | sdague | the scripts after that are making assumptions | |
| 14:03:34 | mriedem | sdague: will dig after the nova meeting | |
| 14:15:29 | mriedem | https://bugs.launchpad.net/nova/+bug/1691769 | |
| 14:15:29 | openstack | Launchpad bug 1691769 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode-live-migration-nv fails in pike: "Failed to restart <email address hidden>: Unit <email address hidden> not found."" [Medium,Triaged] | |
| 14:40:30 | takashin | dansmith: Would you review https://review.openstack.org/#/c/408955/ again? | |
| 14:44:41 | mriedem | takashin: i'm confused, | |
| 14:44:51 | mriedem | https://review.openstack.org/#/c/408964/49/nova/api/openstack/api_version_request.py and https://review.openstack.org/#/c/459483/3/nova/api/openstack/api_version_request.py are competing for the same microversion, | |
| 14:45:05 | mriedem | but i thought the abort cold migration blueprint dependent on being able to list cold migrations? | |
| 14:45:26 | mriedem | oh nvm | |
| 14:45:30 | mriedem | i'm thinking of a different blueprint | |
| 14:45:37 | mriedem | this is cold migration with a target host | |
| 14:51:25 | takashin | mriedem: As you said, there is no dependency between "cold migration with target host" and "List/show all migration types" blueprint. | |
| 14:55:35 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix uuid replacement in aggregate notification test https://review.openstack.org/465896 | |
| 14:55:42 | mriedem | sdague: i'm confused that USE_SCREEN=False on the old side of this live migration grenade job http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/old/local_conf.txt.gz | |
| 14:55:43 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix uuid replacement in aggregate notification test https://review.openstack.org/465896 | |
| 14:55:48 | mriedem | i thought screen was still the default in devstack in ocata? | |
| 14:56:43 | mriedem | i guess it is https://github.com/openstack-dev/devstack/blob/stable/ocata/stackrc#L88 | |
| 14:58:24 | sdague | mriedem: the confusion is there are 3 operating modes right now | |
| 14:59:10 | sdague | USE_SCREEN=False before a month ago, means direct exec everything in bash and hope you captured the pid correctly to kill it later | |
| 14:59:32 | sdague | we've used that in the gate for the last 3 years because screen races | |
| 14:59:54 | sdague | and in the non repeat state, it's fine. | |
| 15:00:21 | sdague | it is a different home grown init system with different bad trade offs than screen | |
| 15:01:58 | dpawlik | Hi.If you have some time pls review my PS https://review.openstack.org/#/c/459753/ | |
| 15:02:00 | dpawlik | thanks a lot | |
| 15:02:19 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform aggregate.add_host notification https://review.openstack.org/396210 | |
| 15:04:04 | ftersin | mriedem: hi. who could review https://review.openstack.org/#/c/407440/ ? | |
| 15:04:35 | ftersin | mriedem: who of you guys is more familiar with that area? | |
| 15:04:44 | mriedem | ftersin: mdbooth | |
| 15:05:11 | ftersin | mriedem: thx | |
| 15:07:02 | ftersin | mdbooth: hi. are you around? | |
| 15:07:12 | mdbooth | ftersin: Yep | |
| 15:07:57 | ftersin | mdbooth: could you look at https://review.openstack.org/#/c/407440/ again? | |
| 15:08:05 | mdbooth | Yeah, sure. | |
| 15:08:32 | mdbooth | So, my issue with that patch has always been the number of changes required in driver.py | |
| 15:08:57 | mdbooth | Which is obvious an artifact of the imagebackend interface being completely borked | |
| 15:08:59 | ftersin | mdbooth: i addressed some your comments there. so, it'd be good to get new ones :) | |
| 15:09:06 | mdbooth | Somebody should clean that up... ;) | |
| 15:09:28 | mdbooth | Anyway, we also agreed at some point to move on with it anyway, so I'll try to look again in that light | |
| 15:09:47 | mriedem | sdague: ok, so maybe i didn't ask the right question, | |
| 15:10:01 | mriedem | i'm basically wondering what i can key off of to fix this grenade live migration issue | |
| 15:10:07 | sdague | mriedem: no | |
| 15:10:18 | sdague | you are going to need more complex logic | |
| 15:10:25 | openstackgerrit | Chris Friesen proposed openstack/nova master: correctly log port id in neutron api https://review.openstack.org/466003 | |
| 15:10:36 | cfriesen | ^ trivial fix to logging | |
| 15:11:08 | sdague | mriedem: https://github.com/openstack-dev/devstack/blob/6a62619cbc28625755a76eae08b529438eb4f281/stackrc#L166-L173 | |
| 15:11:17 | sdague | that's the gate level work around, and why grenade works | |
| 15:11:18 | cfriesen | crap, let me respin that...an extra file snuck in | |
| 15:12:47 | sdague | mriedem: USE_SCREEN=False is always the case for the gate | |
| 15:13:02 | sdague | it's just that we decided to make the transition simpler by making that mean something else | |
| 15:13:13 | sdague | so 95% of things just worked | |
| 15:13:14 | dpawlik | mriedem: thanks for review. DocImpact-Group: openstack/glance is because someone on this channel told me that docs are related to Openstack Glance. I will create a new PS soon. Thanks! | |
| 15:13:19 | mriedem | sdague: ok i see that in devstack-gate localrc_set "$localrc_file" "USE_SCREEN" "False" | |
| 15:13:24 | sdague | the grenade cases needed this one extra hit | |
| 15:13:35 | sdague | hint | |
| 15:13:48 | ftersin | mdbooth: this patch does not bring a lot of changes for driver.py. Moreover, some changed might be separated out of this patch. E.g. see a series started with https://review.openstack.org/#/c/411942 | |
| 15:13:55 | sdague | then you are dealing with things that wrote their own service management logic outside of devstack | |
| 15:14:06 | sdague | which reacted poorly | |
| 15:14:36 | sdague | because their start/stop/restart calls aren't going through devstack code with vars that would have done the right things | |
| 15:14:40 | openstackgerrit | Chris Friesen proposed openstack/nova master: correctly log port id in neutron api https://review.openstack.org/466003 | |
| 15:14:54 | sdague | so options are: 1) replicate special logic | |
| 15:15:01 | sdague | 2) redo them to use devstack functions | |
| 15:15:17 | sdague | 3) not care about this one job that's in a tarpit | |
| 15:15:45 | sdague | given that general live migration post upgrade is already tested, and multi backend live migration on master is already tested | |
| 15:15:49 | sdague | just not the combo of the 2 | |
| 15:16:34 | mriedem | general live migration post upgrade being the grenade multinode job? | |
| 15:16:39 | sdague | yes | |
| 15:17:05 | sdague | so the missing test coverage right now is post upgrade live migration with ceph / nfs | |
| 15:17:16 | mriedem | only ceph actually | |
| 15:17:19 | mriedem | the nfs stuff was never fixed | |
| 15:17:24 | sdague | ok, so only ceph | |
| 15:17:31 | mriedem | what's multibackend live migration? | |
| 15:17:34 | sdague | also, note, it's not a vm that predates the migration | |
| 15:17:34 | mriedem | the back and forth thing? | |
| 15:17:42 | sdague | mriedem: the normal livemigration job | |
| 15:17:58 | sdague | that tests local disk, and ceph, and I guess not nfs any more | |
| 15:18:05 | mriedem | the back and forth thing was meant to test live migrating from n to n-1 and back | |
| 15:18:24 | sdague | oh, right, this is mixed computes? | |
| 15:18:35 | sdague | right so mixed computes with ceph, not tested | |
| 15:18:48 | sdague | that's what's currently lost | |
| 15:19:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform aggregate.remove_host notification https://review.openstack.org/396211 | |
| 15:20:28 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 15:20:30 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 15:20:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 15:20:42 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 15:20:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 15:21:07 | mriedem | sdague: i'm confused as to why i don't see live migration tempest tests running in the normal grenade multinode job then http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/22cb09f/console.html | |
| 15:21:10 | mriedem | oh b/c smoke | |
| 15:21:37 | mriedem | yeah, that was another thing the grenade live miration job did, was bypass the smoke test stuff | |
| 15:21:51 | mriedem | so the normal grenade multinode job is not running live migration tests since they aren't smoke tests | |
| 15:21:59 | sdague | um... really? | |
| 15:22:04 | sdague | I thought we added those back in tehre | |
| 15:22:49 | mriedem | yeah it's why we added this https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1698 | |
| 15:22:52 | mriedem | to the grenade live migration job | |
| 15:22:56 | sdague | ah | |
| 15:23:12 | mriedem | so we don't have mixed compute live migration back and forth from n to n-1 and back testing | |
| 15:23:14 | sean-k-mooney | mriedem: QQ what is the hosting requirements(days to keep logs) for the 3rd party ci logs? the systemd change has tripled the log size so we are running out of space. | |
| 15:23:26 | mriedem | sean-k-mooney: no idea | |
| 15:23:42 | mriedem | sean-k-mooney: i'd ask in infra probably | |
| 15:24:01 | sdague | mriedem: ok, I honestly though we still had basic live migration in multinode grenade, maybe it wasn't reliable enough | |