Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-08
16:21:51 mriedem ci job shit is always a minefield
16:21:59 finucannot sean-k-mooney: Those patches simply copy in the legacy ones. I haven't wrapped my head around translating a job to zuulv3 yet
16:22:04 finucannot despite mordred's best efforts
16:22:28 melwitt would appreciate reviews on this relatively simple bug fix to set the right cache mode during image conversion. most of the patch is moving a function that checks for O_DIRECT support to nova/utils so it can be used by both virt/images and libvirt/driver. the actual fix is in https://review.openstack.org/#/c/523554/6/nova/virt/images.py
16:22:34 mordred I didn't do it
16:23:14 sean-k-mooney finucannot: ya just too a look at your patches now.
16:23:38 mordred finucannot: oh - well - I'm working on converting glanceclient jobs to be native at the moment (and finding a weird issue) - once I'm done with that, I'll circle back around and help update yours
16:24:11 finucannot mriedem: The migration made it especially fun https://review.openstack.org/#/c/514310/3/zuul.d/zuul-legacy-jobs.yaml@6092
16:24:12 sean-k-mooney finucannot: there is a sports and social Q4 party tonight in the stand so ime going to head home soon but ill take a look at what you have started on monday
16:24:24 finucannot mordred: I'd appreciate that. Will do whatever I can to help
16:24:45 finucannot sean-k-mooney: Enjoy. We've the Dublin office party tonight too
16:25:22 sean-k-mooney oh fun dublin has more interesting venues then the strand to choose form
16:25:58 openstackgerrit Merged openstack/python-novaclient master: inject file: add method of showing quota value of injecting files for 'rebuild' command https://review.openstack.org/524588
16:26:28 mordred finucannot: it should actually be *fairly* easy ... once I figure out why the glanceclient job is somehow getting a broken mysql - but shade and python-openstackclient work fine
16:27:13 finucannot Aye. The biggest issue I had was figuring out who was using what (and why). That's probably general infra- inexperience though
16:27:15 sean-k-mooney mordred: is it insatlling in a venv and therefor may be using a different pymysql package
16:27:43 finucannot and I haven't gone further down the zuulv3 rabbit hole that attempting to diff jobs before and after (I didn't get far)
16:29:38 mriedem dmsimard: are the centos or opensuse jobs voting anywhere?
16:30:39 dmsimard mriedem: the devstack ones ? I'm not sure, but it's impacting non-devstack stuff like packstack
16:31:07 dmsimard mriedem: tripleo doesn't run disk injection tests in tempest so they're not running into the issue
16:31:23 mriedem dmsimard: so, has anyone tried downgrading the libguestfs package on centos to the version that we have in the UCA jobs and see if they still fail?
16:31:34 dmsimard I haven't really tried reaching out to OSA and kolla if they've been noticing it too.
16:32:25 dmsimard mriedem: no, not that I am aware of -- I'm not sure if that version is even available on EL7, I'd need to check.
16:34:00 mordred sean-k-mooney: well - eventually it should - but it's actually failing in the devstack layer before it even gets there. it's super SUPER strange, but I'm certain I just left out a comma or something
16:36:17 dmsimard mriedem: yeah we don't have 1.32.2, closest I see is libguestfs-1.32.7-3.el7.centos.x86_64.rpm and that's in an archived release
16:36:59 dmsimard mriedem: it might be complicated to downgrade the package, it probably doesn't come on its own
16:37:41 finucannot melwitt: I'll take a look now
16:38:15 finucannot leakypipes: Could you take a look at https://review.openstack.org/#/c/390520/ again at some point in the next few days?
16:38:33 melwitt thanks finucannot
16:38:57 leakypipes finucannot: yes, wil try my best
16:39:22 finucannot leakypipes: Excellent. Appreciate it
16:41:44 mriedem sdague: superdan: methinks we should consider deprecating the libvirt.inject_key option
16:41:57 mriedem (1) depends on file injection
16:42:02 mriedem (2) doesn't work for volume-backed instances
16:42:06 mriedem (3) use config drive
16:42:54 superdan i
16:42:56 superdan am down
16:43:00 mriedem same with inject_password
16:44:37 mriedem oh fun we pass down the libvirt virt_type when injecting network info because of lxc
16:45:40 finucannot I've got a rather simply bugfix here that resolves an issue we've seen with dodgy glance configurations, if anyone's bored https://review.openstack.org/#/c/526329
16:46:53 finucannot mriedem: Do we still support lxc?
16:47:31 mriedem finucannot: where does https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html say something about 0 data?
16:47:51 finucannot mriedem: https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data
16:48:06 finucannot I trimmed the URL because I didn't want to pollute everyone's git-log
16:48:14 finucannot (more than I'd already done)
16:54:45 finucannot Ha! python-glanceclient is broken itself
16:56:21 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
16:56:21 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
16:56:22 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
16:56:22 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
17:04:06 melwitt mriedem: I noticed you abandoned these https://review.openstack.org/#/c/508163 did we not want to keep them and not skip that test now that test_volume_backed_live_migration works? to guard against regression?
17:04:29 mriedem fewer double negatives please
17:05:28 melwitt sorry. like, we were using those DNM patches to test a fix for volume backed live migration
17:05:58 melwitt now that volume backed live migration is fixed, shouldn't we have removed DNM from those to unskip the test?
17:07:23 melwitt don't we want to unskip the test now that it will pass, that is
17:07:47 mriedem now that volume backed live migration is fixed....
17:07:50 mriedem it is?
17:08:09 mriedem you mean https://review.openstack.org/#/c/507202/ ?
17:08:35 melwitt yeah, I thought that fixed it. you had those DNM patches depend on it and the unskipped tests passed
17:08:46 melwitt did I miss something?
17:08:52 mriedem melwitt: i believe those tests will intermittently pass regardless of that patch
17:09:06 melwitt okay, that's what I was missing then
17:09:09 mriedem the volume-backed and iscsi live migration tests were skipped in tempest long ago due to random failures
17:09:17 mriedem that no one could sleuth because qemu
17:09:29 mriedem using UCA now might have changed that
17:09:31 melwitt I see
17:09:43 mriedem bug 1524898
17:09:44 openstack bug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,In progress] https://launchpad.net/bugs/1524898
17:10:11 melwitt okay, I thought maybe that was related to the recent fix we had
17:10:20 mriedem even if we enable the test in tempest again,
17:10:33 mriedem tempest is branchless so it would run on stable branches and could fail there if we're using older packages that had a problem
17:10:37 mriedem hard to tell
17:10:53 mriedem i would very much like to get volume-backed live migration testing enabled in the gate though yes
17:10:54 melwitt oh, right.
17:11:08 mriedem we should probably change the global skip on the test in tempest to a config option,
17:11:11 mriedem so that we can enable the test per-branch
17:11:24 melwitt I can't recall if we skip certain tests for stable-only, or if that's even possible
17:11:36 mriedem we can configure tempest per-branch, per-job
17:11:43 mriedem we can configure tempest in our in-tree live migration scripts
17:11:46 mriedem the post_test_hook
17:12:07 mriedem https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh#L33
17:12:19 mriedem with the magic of ansible
17:13:18 melwitt okay, so we'd add another option volume_backed_live_migration or such
17:13:25 mriedem so if you wanted to be cool, propose a change to tempest to add a config option for volume-backed live migration to replace that skip line, default to false for stable, and then set it to true in our live migration job post_test_hook script
17:13:44 mriedem and then be available when it melts the gate
17:14:09 melwitt can do
17:14:56 melwitt thanks
17:16:18 mriedem it is my pleasure
17:16:27 mriedem that's a weird phrase
17:18:18 mriedem dmsimard: interesting, even when it appears that we do set force_tcg mode, guestfs returns None rather than an int
17:18:19 mriedem http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full-centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_13_43_19_923098
17:18:21 melwitt lots of zvm:recheck spam going on
17:18:25 mriedem Failed to force guestfs TCG mode. guestfs_set_backend_settings returned: None
17:18:45 dmsimard weird
17:19:43 dmsimard mriedem: it seems like there's two set_backend_settings happening there
17:19:57 dmsimard event=trace eh=0 buf='set_backend_settings "force_tcg"' array=[]
17:20:05 dmsimard and event=trace eh=0 buf='set_backend_settings = 0' array=[] ?
17:21:21 mriedem i think one is the value being set?
17:21:32 mriedem or the return value?
17:21:41 mriedem i don't know what the array is

Earlier   Later