Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-08
16:09:15 dmsimard mriedem, kashyap: would love if we could iron out https://bugs.launchpad.net/nova/+bug/1735823 soon, it's making a non-negligible amount of jobs flap
16:09:16 openstack Launchpad bug 1735823 in OpenStack Compute (nova) "Nova can hang when creating a VM with disk injection" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
16:09:17 mriedem cdent: very carefully
16:09:31 cdent as is so often the case
16:09:33 superdan cdent: that's my point
16:09:44 melwitt finucannot: been meaning to do that, yes I will
16:09:44 kashyap dmsimard: Hey, I've seen it; but a bit overwhelmed today
16:09:49 kashyap dmsimard: I haven't forgotten it
16:09:58 superdan cdent: if they're doing this thing, its going to leave residue we likely don't have a cleanup command for
16:09:58 dmsimard kashyap: I know the feeling, just a friendly reminder :)
16:09:58 mriedem dmsimard: are these only centos jobs?
16:10:12 kashyap Yeah, was wondering the same.
16:10:20 cdent superdan: ah okay, I think I’m _finally_ caught up then. I was way way back there in the weeds.
16:10:20 mriedem i'm assuming the problem is some difference in the guestfs package versions
16:10:43 cdent I blame society. Or my head cold. Or just being stupid.
16:10:46 mriedem dmsimard: because https://review.openstack.org/#/c/524748/ has been busted for a long time, and we know it doesn't fix the issue
16:10:46 dmsimard mriedem: the data that we have has problems on centos and opensuse, I couldn't find any on Ubuntu but that doesn't mean it doesn't happen there
16:11:05 mriedem i think the lock issue is the fact we're using eventlet.tpool
16:11:17 dmsimard mriedem: well, it kind of fixes the issue but then generates new ones ? right ? the patch with the depends-on fails horribly
16:11:55 mriedem https://review.openstack.org/#/c/524750/ ?
16:12:27 kashyap Yeah, that seems to be the test
16:12:29 finucannot sean-k-mooney: What's this about functional tests?
16:12:39 finucannot I assume you mean enabling ralonsoh's patches?
16:13:19 mriedem oh boy http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full/eb8d805/logs/screen-n-api.txt.gz?level=TRACE#_Dec_04_13_34_20_635874
16:13:20 sean-k-mooney finucannot: yes im going to add a job to the experimental pipline for os-vif so we can run rodolfos tests
16:13:34 sean-k-mooney finucannot: once they are merged i can move it to the gate pipline
16:13:45 finucannot sean-k-mooney: Ideal
16:14:05 finucannot gate for os-vif or nova too?
16:14:18 sean-k-mooney check and gate for os-vif
16:14:40 superdan leakypipes: finucannot: melwitt: either one of you can mindlessly approve this trivial thing without worry: https://review.openstack.org/#/c/507874/1
16:14:54 leakypipes superdan: BFV with BDMs?
16:14:59 finucannot superdan: If there's one thing I know how to do...
16:15:05 superdan leakypipes: not even
16:15:09 leakypipes :)
16:15:10 superdan finucannot: I knew I could count on you
16:15:35 finucannot sean-k-mooney: Cool. We already run oslo.versionedobjects tests in the experimental pipeline for nova. Might be worth adding os-vif there too
16:15:41 finucannot Eventually, anyway
16:15:55 sean-k-mooney i would also like to move all our os-vif jobs to zuul v3 in repo config and add a linux-bridge job for os-vif but it will be a few week before i have time to do that
16:16:11 sean-k-mooney finucannot: os-vif is already in the nova experimental pipline
16:16:47 sean-k-mooney check experimtal on a nova patch will test that patch with the head of the os-vif master branch in a seperate job
16:16:57 finucannot superdan: I know you asked for a mindless approval but...no bug? https://review.openstack.org/#/c/507874/
16:17:14 superdan finucannot: it's not a bug yet
16:17:20 sean-k-mooney finucannot: the one thing it does not do is work with depend-on where the depends-on is an unmerged os-vif change
16:17:34 superdan finucannot: it will be after my live migrations patch merges, it's just an omission right now, the last of a few fixes ahead of time
16:17:54 finucannot superdan: Yup, just saw jianghuaw_'s comment now. Cool, +Wd
16:18:03 superdan finucannot: thanks
16:18:27 mriedem well it might be a bug
16:18:33 finucannot sean-k-mooney: All the os-vif jobs or just the unit tests?
16:18:40 finucannot *unit test jobs
16:18:40 mriedem if the driver doesn't pass migrate_data, we don't set the migration.status to 'error'
16:19:02 sean-k-mooney finucannot: ill try and move them all i think. we will see what infra advise
16:19:09 sean-k-mooney finucannot: what has nova done?
16:19:24 mriedem nothing
16:19:27 mriedem novaclient on the other hand https://review.openstack.org/#/q/I43a8435485751748ca6228f67d401945cb32652e
16:19:42 finucannot mriedem: Well https://review.openstack.org/#/q/status:open+branch:master+topic:zuulv3-nova
16:19:48 mriedem sean-k-mooney: there are very few ci jobs that nova runs which are just run against nova
16:19:51 finucannot We don't have to move much, thankfully
16:19:53 mriedem i think the nova-next job is the only one
16:19:54 finucannot Because of that, yes
16:20:25 mriedem oh lxc/lvm
16:20:39 mriedem umm, we can drop the wsgi-full job, unless we run that on stable
16:20:52 mriedem wsgi is the default in devstack since pike
16:20:59 finucannot We don't run any of them on stable. Drop that entirely so?
16:21:37 sean-k-mooney ah cool well i dont think it will be that hard to move os-vif. is there a plan to move nova eventually or just rely on the infra maintained jobs
16:21:37 mriedem idk, i'd have to go through this with a comb
16:21:51 mriedem ci job shit is always a minefield
16:21:59 finucannot sean-k-mooney: Those patches simply copy in the legacy ones. I haven't wrapped my head around translating a job to zuulv3 yet
16:22:04 finucannot despite mordred's best efforts
16:22:28 melwitt would appreciate reviews on this relatively simple bug fix to set the right cache mode during image conversion. most of the patch is moving a function that checks for O_DIRECT support to nova/utils so it can be used by both virt/images and libvirt/driver. the actual fix is in https://review.openstack.org/#/c/523554/6/nova/virt/images.py
16:22:34 mordred I didn't do it
16:23:14 sean-k-mooney finucannot: ya just too a look at your patches now.
16:23:38 mordred finucannot: oh - well - I'm working on converting glanceclient jobs to be native at the moment (and finding a weird issue) - once I'm done with that, I'll circle back around and help update yours
16:24:11 finucannot mriedem: The migration made it especially fun https://review.openstack.org/#/c/514310/3/zuul.d/zuul-legacy-jobs.yaml@6092
16:24:12 sean-k-mooney finucannot: there is a sports and social Q4 party tonight in the stand so ime going to head home soon but ill take a look at what you have started on monday
16:24:24 finucannot mordred: I'd appreciate that. Will do whatever I can to help
16:24:45 finucannot sean-k-mooney: Enjoy. We've the Dublin office party tonight too
16:25:22 sean-k-mooney oh fun dublin has more interesting venues then the strand to choose form
16:25:58 openstackgerrit Merged openstack/python-novaclient master: inject file: add method of showing quota value of injecting files for 'rebuild' command https://review.openstack.org/524588
16:26:28 mordred finucannot: it should actually be *fairly* easy ... once I figure out why the glanceclient job is somehow getting a broken mysql - but shade and python-openstackclient work fine
16:27:13 finucannot Aye. The biggest issue I had was figuring out who was using what (and why). That's probably general infra- inexperience though
16:27:15 sean-k-mooney mordred: is it insatlling in a venv and therefor may be using a different pymysql package
16:27:43 finucannot and I haven't gone further down the zuulv3 rabbit hole that attempting to diff jobs before and after (I didn't get far)
16:29:38 mriedem dmsimard: are the centos or opensuse jobs voting anywhere?
16:30:39 dmsimard mriedem: the devstack ones ? I'm not sure, but it's impacting non-devstack stuff like packstack
16:31:07 dmsimard mriedem: tripleo doesn't run disk injection tests in tempest so they're not running into the issue
16:31:23 mriedem dmsimard: so, has anyone tried downgrading the libguestfs package on centos to the version that we have in the UCA jobs and see if they still fail?
16:31:34 dmsimard I haven't really tried reaching out to OSA and kolla if they've been noticing it too.
16:32:25 dmsimard mriedem: no, not that I am aware of -- I'm not sure if that version is even available on EL7, I'd need to check.
16:34:00 mordred sean-k-mooney: well - eventually it should - but it's actually failing in the devstack layer before it even gets there. it's super SUPER strange, but I'm certain I just left out a comma or something
16:36:17 dmsimard mriedem: yeah we don't have 1.32.2, closest I see is libguestfs-1.32.7-3.el7.centos.x86_64.rpm and that's in an archived release
16:36:59 dmsimard mriedem: it might be complicated to downgrade the package, it probably doesn't come on its own
16:37:41 finucannot melwitt: I'll take a look now
16:38:15 finucannot leakypipes: Could you take a look at https://review.openstack.org/#/c/390520/ again at some point in the next few days?
16:38:33 melwitt thanks finucannot
16:38:57 leakypipes finucannot: yes, wil try my best
16:39:22 finucannot leakypipes: Excellent. Appreciate it
16:41:44 mriedem sdague: superdan: methinks we should consider deprecating the libvirt.inject_key option
16:41:57 mriedem (1) depends on file injection
16:42:02 mriedem (2) doesn't work for volume-backed instances
16:42:06 mriedem (3) use config drive

Earlier   Later