| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 15:20:42 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 15:20:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 15:21:07 | mriedem | sdague: i'm confused as to why i don't see live migration tempest tests running in the normal grenade multinode job then http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/22cb09f/console.html | |
| 15:21:10 | mriedem | oh b/c smoke | |
| 15:21:37 | mriedem | yeah, that was another thing the grenade live miration job did, was bypass the smoke test stuff | |
| 15:21:51 | mriedem | so the normal grenade multinode job is not running live migration tests since they aren't smoke tests | |
| 15:21:59 | sdague | um... really? | |
| 15:22:04 | sdague | I thought we added those back in tehre | |
| 15:22:49 | mriedem | yeah it's why we added this https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1698 | |
| 15:22:52 | mriedem | to the grenade live migration job | |
| 15:22:56 | sdague | ah | |
| 15:23:12 | mriedem | so we don't have mixed compute live migration back and forth from n to n-1 and back testing | |
| 15:23:14 | sean-k-mooney | mriedem: QQ what is the hosting requirements(days to keep logs) for the 3rd party ci logs? the systemd change has tripled the log size so we are running out of space. | |
| 15:23:26 | mriedem | sean-k-mooney: no idea | |
| 15:23:42 | mriedem | sean-k-mooney: i'd ask in infra probably | |
| 15:24:01 | sdague | mriedem: ok, I honestly though we still had basic live migration in multinode grenade, maybe it wasn't reliable enough | |
| 15:24:04 | mriedem | sean-k-mooney: https://docs.openstack.org/infra/system-config/third_party.html | |
| 15:24:08 | mriedem | Note All test artifacts must be retained for one month. | |
| 15:24:10 | sean-k-mooney | mriedem: we were aiming for 90 days before im looking into geting more storage but we might have to reduce for a bit while we get that set up | |
| 15:24:27 | mriedem | sean-k-mooney: sounds like 1 month minimum | |
| 15:24:30 | sdague | mriedem: ok, so option 4) don't do the special ceph live migration job | |
| 15:24:59 | sdague | and just make that job run live migration grenade tests | |
| 15:25:08 | sean-k-mooney | mriedem: cool thats what i taught. we might reduce to 45 days while we see if we can upgrade form 500GB to 1TB of hosting | |
| 15:25:23 | sdague | because it's the post_hook that's a bit hacking and a problem | |
| 15:25:42 | sean-k-mooney | the nova jobs are eating ~ 13.5GB per day post the systemd change | |
| 15:26:29 | mriedem | sdague: the special ceph live migration job being the non-grenade live migration job that runs with ceph too? that one isn't a problem. | |
| 15:26:49 | sdague | mriedem: I'm saying it all got too compled | |
| 15:26:54 | sdague | there is grenade-multi | |
| 15:27:07 | mriedem | gate-grenade-dsvm-neutron-multinode-ubuntu-xenial | |
| 15:27:08 | mriedem | yes | |
| 15:27:11 | sdague | sorry, let me blast this out | |
| 15:27:27 | sdague | grenade-multi - no ceph, no live migration | |
| 15:27:39 | sdague | dsvm-multi - no ceph, live migration | |
| 15:27:56 | sdague | live-migration - ceph, live migration (custom post_hook, fixed) | |
| 15:28:31 | sdague | grenade-live-migration - ceph, live migration (custom post_hook as above, doesn't work across systemd transition) | |
| 15:28:45 | sdague | there is no grenade-multi-full | |
| 15:28:53 | sdague | that would give you working live migration testing today | |
| 15:29:15 | sdague | the equiv of dsvm-multi | |
| 15:30:49 | mriedem | yeah, now i'm trying to see what's setting live_migrate_back_and_forth in tempest.conf | |
| 15:30:59 | mriedem | because the dsvm-multi job isn't doing it | |
| 15:31:28 | mriedem | ah only set in the grenade-dsvm-neutron-multinode-live-migration job | |
| 15:33:28 | mriedem | but i don't actually see that set in tempest.conf in the job, probably because of the hackery in the hooks in nova | |
| 15:33:29 | mriedem | for that job | |
| 15:34:14 | mriedem | man i don't even know anymore, i might just give up | |
| 15:47:29 | openstackgerrit | Tao Li proposed openstack/nova master: Fix the display of updated_at time when using memcache driver. https://review.openstack.org/414463 | |
| 15:52:57 | mriedem | ildikov: johnthetubaguy: i can't make the nova/cinder meeting today | |
| 15:53:07 | mriedem | stvnoyes: ^ | |
| 15:53:16 | mriedem | i also haven't had a chance to run through steve's live migration wip changes yet | |
| 15:54:59 | ildikov | mriedem: thanks for the heads up | |
| 15:55:40 | sean-k-mooney | mriedem: were you looking for where in the code the live_migration flag is set in the gate-grenade-dsvm-neutron-multinode-live-migration-nv or just what file it was set in | |
| 15:56:42 | mriedem | sean-k-mooney: it's set in local.conf https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1703 | |
| 15:57:12 | mriedem | and should be set in tempest.conf during devstack setup here https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L364 | |
| 15:57:13 | kashyap | cfriesen: Looked at your patch this morning, responded on it | |
| 15:57:16 | sean-k-mooney | mriedem: ah you found it so i was goint to take a look and see if i could find it | |
| 15:57:22 | mriedem | but i'm not actually seeing it set in tempest.conf in that job | |
| 15:57:23 | kashyap | cfriesen: It looks fine to me, nice catch. I presume you stumbled on it by testing | |
| 15:57:32 | sean-k-mooney | i am | |
| 15:57:34 | kashyap | (I mean, in real env) | |
| 15:57:52 | sean-k-mooney | mriedem: http://logs.openstack.org/60/465160/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/c34cb8b/logs/new/tempest_conf.txt.gz | |
| 15:58:20 | mriedem | sean-k-mooney: i don't see live_migrate_back_and_forth in there | |
| 15:58:44 | sean-k-mooney | or sorry i taught you were just looking for live_migration = True sorry | |
| 15:58:52 | mriedem | it's in the loca.conf though http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/new/local.conf.txt.gz | |
| 15:58:57 | mriedem | LIVE_MIGRATE_BACK_AND_FORTH=True | |
| 16:00:15 | dansmith | mriedem: omg, is that really a flag? | |
| 16:00:34 | mriedem | it is | |
| 16:00:38 | dansmith | love. it. | |
| 16:01:00 | mriedem | i must be missing something | |
| 16:01:01 | cfriesen | kashyap: thanks for the review...we did indeed trip over it | |
| 16:01:04 | sean-k-mooney | mriedem: hum maybe it would be better to skip the macro and update the tempest conf via setting [[test-config|/opt/stack/new/tempest/etc/tempest.conf]] ... section in the local.conf | |
| 16:01:22 | mriedem | sean-k-mooney: that didn't exist when this was added | |
| 16:01:27 | mriedem | and this used to work, i'm pretty sure | |
| 16:01:33 | mriedem | i see the devstack flag in the grenade log | |
| 16:01:43 | mriedem | http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/grenade.sh.txt.gz#_2017-05-18_00_34_34_956 | |
| 16:02:29 | kashyap | cfriesen: Should that be documented the format-domain for libvirt upstream, too? | |
| 16:02:40 | dansmith | mriedem: probably unintentional, but just thinking of the potential "me and you and everyone we know" reference | |
| 16:02:43 | sean-k-mooney | mriedem: well its only used once in devstack and its pretty clear what its trying to do so that is strange that its not working https://github.com/openstack-dev/devstack/blob/ac042d41222eea83bf360b4db89c4c7bf415457d/lib/tempest#L364 | |
| 16:02:47 | mriedem | wtf it should happen right here http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/grenade.sh.txt.gz#_2017-05-18_00_21_27_068 | |
| 16:02:58 | kashyap | cfriesen: Okay, currently I see it documented under 'vcpu' -> 'cpuset' there | |
| 16:03:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: remove pagesize from __init__ of InstanceNUMATopology https://review.openstack.org/408241 | |
| 16:03:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: remove related pinning from __init__ of InstanceNUMATopology https://review.openstack.org/408242 | |
| 16:03:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: remove cpuset_reserved from __init__ of InstanceNUMATopology https://review.openstack.org/466030 | |
| 16:03:20 | kashyap | cfriesen: Probably that's good enough: "Each element in that list is either a single CPU number, a range of CPU numbers, or a caret followed by a CPU number to be excluded from a previous range." | |
| 16:03:26 | sfinucan | dansmith: Finally got one of those working ^ | |
| 16:03:50 | dansmith | sfinucan: cool | |
| 16:03:50 | mriedem | sean-k-mooney: i see the problem | |
| 16:03:51 | mriedem | https://github.com/openstack-dev/devstack/blob/stable/ocata/lib/tempest | |
| 16:03:57 | mriedem | the devstack flag didn't get into devstack stable/ocata | |
| 16:04:05 | mriedem | and tempest.conf is configured on the old side in the grenade run | |
| 16:04:15 | sfinucan | dansmith: ...however, 'nova.tests.unit.virt.test_hardware.HelperMethodsTestCase.test_dict_numa_topology_to_obj' is failing for the other two and I can't figure out why | |
| 16:04:37 | sean-k-mooney | mriedem: ah ok that makes sense | |
| 16:04:39 | sfinucan | If you have a chance to look at that at some point, could you? | |
| 16:04:53 | sfinucan | For some reason, a dict is not behaving like a dict :S | |
| 16:05:11 | sfinucan | stupid ovo | |
| 16:05:41 | cfriesen | kashyap: yes, I think that's okay. the implicit all-cpus was just in the nova code. | |
| 16:05:52 | mriedem | sean-k-mooney: so i just needed to backport the fix to devstack https://review.openstack.org/#/c/466033/ | |
| 16:06:07 | mriedem | now having done that, the grenade live migration job is still broken, as discussed with sdague earlier | |
| 16:06:25 | mriedem | and i'm not sure i have the energy atm to untangle that mess | |
| 16:07:06 | sean-k-mooney | mriedem: yes backport or swap to useing the test-config update to the file which is supported on ocata already https://github.com/openstack-dev/devstack/blob/stable/ocata/stack.sh#L1405 | |
| 16:07:14 | kashyap | cfriesen: True. Okay, I'll go cook food, and head to my Dutch class, then :-) | |
| 16:08:57 | sean-k-mooney | mriedem: oh the flag is new. backporting makes sense then | |