Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-18
15:25:08 sean-k-mooney mriedem: cool thats what i taught. we might reduce to 45 days while we see if we can upgrade form 500GB to 1TB of hosting
15:25:23 sdague because it's the post_hook that's a bit hacking and a problem
15:25:42 sean-k-mooney the nova jobs are eating ~ 13.5GB per day post the systemd change
15:26:29 mriedem sdague: the special ceph live migration job being the non-grenade live migration job that runs with ceph too? that one isn't a problem.
15:26:49 sdague mriedem: I'm saying it all got too compled
15:26:54 sdague there is grenade-multi
15:27:07 mriedem gate-grenade-dsvm-neutron-multinode-ubuntu-xenial
15:27:08 mriedem yes
15:27:11 sdague sorry, let me blast this out
15:27:27 sdague grenade-multi - no ceph, no live migration
15:27:39 sdague dsvm-multi - no ceph, live migration
15:27:56 sdague live-migration - ceph, live migration (custom post_hook, fixed)
15:28:31 sdague grenade-live-migration - ceph, live migration (custom post_hook as above, doesn't work across systemd transition)
15:28:45 sdague there is no grenade-multi-full
15:28:53 sdague that would give you working live migration testing today
15:29:15 sdague the equiv of dsvm-multi
15:30:49 mriedem yeah, now i'm trying to see what's setting live_migrate_back_and_forth in tempest.conf
15:30:59 mriedem because the dsvm-multi job isn't doing it
15:31:28 mriedem ah only set in the grenade-dsvm-neutron-multinode-live-migration job
15:33:28 mriedem but i don't actually see that set in tempest.conf in the job, probably because of the hackery in the hooks in nova
15:33:29 mriedem for that job
15:34:14 mriedem man i don't even know anymore, i might just give up
15:47:29 openstackgerrit Tao Li proposed openstack/nova master: Fix the display of updated_at time when using memcache driver. https://review.openstack.org/414463
15:52:57 mriedem ildikov: johnthetubaguy: i can't make the nova/cinder meeting today
15:53:07 mriedem stvnoyes: ^
15:53:16 mriedem i also haven't had a chance to run through steve's live migration wip changes yet
15:54:59 ildikov mriedem: thanks for the heads up
15:55:40 sean-k-mooney mriedem: were you looking for where in the code the live_migration flag is set in the gate-grenade-dsvm-neutron-multinode-live-migration-nv or just what file it was set in
15:56:42 mriedem sean-k-mooney: it's set in local.conf https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1703
15:57:12 mriedem and should be set in tempest.conf during devstack setup here https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L364
15:57:13 kashyap cfriesen: Looked at your patch this morning, responded on it
15:57:16 sean-k-mooney mriedem: ah you found it so i was goint to take a look and see if i could find it
15:57:22 mriedem but i'm not actually seeing it set in tempest.conf in that job
15:57:23 kashyap cfriesen: It looks fine to me, nice catch. I presume you stumbled on it by testing
15:57:32 sean-k-mooney i am
15:57:34 kashyap (I mean, in real env)
15:57:52 sean-k-mooney mriedem: http://logs.openstack.org/60/465160/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/c34cb8b/logs/new/tempest_conf.txt.gz
15:58:20 mriedem sean-k-mooney: i don't see live_migrate_back_and_forth in there
15:58:44 sean-k-mooney or sorry i taught you were just looking for live_migration = True sorry
15:58:52 mriedem it's in the loca.conf though http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/new/local.conf.txt.gz
15:58:57 mriedem LIVE_MIGRATE_BACK_AND_FORTH=True
16:00:15 dansmith mriedem: omg, is that really a flag?
16:00:34 mriedem it is
16:00:38 dansmith love. it.
16:01:00 mriedem i must be missing something
16:01:01 cfriesen kashyap: thanks for the review...we did indeed trip over it
16:01:04 sean-k-mooney mriedem: hum maybe it would be better to skip the macro and update the tempest conf via setting [[test-config|/opt/stack/new/tempest/etc/tempest.conf]] ... section in the local.conf
16:01:22 mriedem sean-k-mooney: that didn't exist when this was added
16:01:27 mriedem and this used to work, i'm pretty sure
16:01:33 mriedem i see the devstack flag in the grenade log
16:01:43 mriedem http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/grenade.sh.txt.gz#_2017-05-18_00_34_34_956
16:02:29 kashyap cfriesen: Should that be documented the format-domain for libvirt upstream, too?
16:02:40 dansmith mriedem: probably unintentional, but just thinking of the potential "me and you and everyone we know" reference
16:02:43 sean-k-mooney mriedem: well its only used once in devstack and its pretty clear what its trying to do so that is strange that its not working https://github.com/openstack-dev/devstack/blob/ac042d41222eea83bf360b4db89c4c7bf415457d/lib/tempest#L364
16:02:47 mriedem wtf it should happen right here http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/grenade.sh.txt.gz#_2017-05-18_00_21_27_068
16:02:58 kashyap cfriesen: Okay, currently I see it documented under 'vcpu' -> 'cpuset' there
16:03:13 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: remove pagesize from __init__ of InstanceNUMATopology https://review.openstack.org/408241
16:03:13 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: remove related pinning from __init__ of InstanceNUMATopology https://review.openstack.org/408242
16:03:14 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: remove cpuset_reserved from __init__ of InstanceNUMATopology https://review.openstack.org/466030
16:03:20 kashyap cfriesen: Probably that's good enough: "Each element in that list is either a single CPU number, a range of CPU numbers, or a caret followed by a CPU number to be excluded from a previous range."
16:03:26 sfinucan dansmith: Finally got one of those working ^
16:03:50 dansmith sfinucan: cool
16:03:50 mriedem sean-k-mooney: i see the problem
16:03:51 mriedem https://github.com/openstack-dev/devstack/blob/stable/ocata/lib/tempest
16:03:57 mriedem the devstack flag didn't get into devstack stable/ocata
16:04:05 mriedem and tempest.conf is configured on the old side in the grenade run
16:04:15 sfinucan dansmith: ...however, 'nova.tests.unit.virt.test_hardware.HelperMethodsTestCase.test_dict_numa_topology_to_obj' is failing for the other two and I can't figure out why
16:04:37 sean-k-mooney mriedem: ah ok that makes sense
16:04:39 sfinucan If you have a chance to look at that at some point, could you?
16:04:53 sfinucan For some reason, a dict is not behaving like a dict :S
16:05:11 sfinucan stupid ovo
16:05:41 cfriesen kashyap: yes, I think that's okay. the implicit all-cpus was just in the nova code.
16:05:52 mriedem sean-k-mooney: so i just needed to backport the fix to devstack https://review.openstack.org/#/c/466033/
16:06:07 mriedem now having done that, the grenade live migration job is still broken, as discussed with sdague earlier
16:06:25 mriedem and i'm not sure i have the energy atm to untangle that mess
16:07:06 sean-k-mooney mriedem: yes backport or swap to useing the test-config update to the file which is supported on ocata already https://github.com/openstack-dev/devstack/blob/stable/ocata/stack.sh#L1405
16:07:14 kashyap cfriesen: True. Okay, I'll go cook food, and head to my Dutch class, then :-)
16:08:57 sean-k-mooney mriedem: oh the flag is new. backporting makes sense then
16:13:03 sean-k-mooney sfinucan: how dare you disbarage ovo :P
16:13:53 sfinucan sean-k-mooney: I'm sure it's lovely, but I've just burnt through a full day looking at ovo-related issues. I ain't feeling it today ;)
16:14:49 sean-k-mooney there a child of nova object you have to treat them kindly but ya they can be a pain.
16:16:04 alex_xu mriedem: sdague sfinucan oomichi there are some nova bug fix in the etherpad from bugsmash https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Pike-Suzhou-Bug-List, appreciate if you can review few of them when you have free time, thanks in advance!
16:17:03 openstackgerrit Alex Xu proposed openstack/nova master: Correct the error message for query parameter validation https://review.openstack.org/465785
16:19:35 dansmith mriedem: so, I'm getting this on my cells test: http://logs.openstack.org/37/458537/20/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/ab364da/logs/testr_results.html.gz
16:19:48 dansmith mriedem: but I'm failing because of lack of shared storage, which shouldn't be affected by what I'm doing
16:19:54 dansmith oh, um, hmm..
16:20:10 dansmith I wonder if it's because I'm mucking with the config for compute..
16:23:17 dansmith mriedem: isn't there some late hooky stuff going on from the nova tree for the live migration job?
16:23:18 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-manage: Deprecate 'host' commands https://review.openstack.org/453175
16:23:18 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-manage: Deprecate 'log' commands https://review.openstack.org/453176
16:23:18 dansmith sdague: ^
16:23:19 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-manage: Deprecate 'project', 'account' commands https://review.openstack.org/453174
16:23:42 sfinucan alex_xu: I'll trade you some of yours for some of mine ;) ^^^
16:24:05 sfinucan artom: Addressed your comment in above change, if you're about ^
16:24:26 zzzeek sdague: what is a "5 byte utf8" character ?
16:25:18 artom sfinucan, I'm about
16:28:41 sfinucan zzzeek: UTF-8 chars can vary in size from 1 to 6 bytes, it appears https://stackoverflow.com/q/5290182/
16:28:47 sdague zzzeek: there was a nova bug on it at some point
16:29:01 zzzeek sdague sfinucan OK but unicode itself only needs four bytes
16:29:15 zzzeek sdague sfinucan MySQL / Posgresql / anything only goes out to 4 bytes max

Earlier   Later