| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-08 | |||
| 16:31:34 | dmsimard | I haven't really tried reaching out to OSA and kolla if they've been noticing it too. | |
| 16:32:25 | dmsimard | mriedem: no, not that I am aware of -- I'm not sure if that version is even available on EL7, I'd need to check. | |
| 16:34:00 | mordred | sean-k-mooney: well - eventually it should - but it's actually failing in the devstack layer before it even gets there. it's super SUPER strange, but I'm certain I just left out a comma or something | |
| 16:36:17 | dmsimard | mriedem: yeah we don't have 1.32.2, closest I see is libguestfs-1.32.7-3.el7.centos.x86_64.rpm and that's in an archived release | |
| 16:36:59 | dmsimard | mriedem: it might be complicated to downgrade the package, it probably doesn't come on its own | |
| 16:37:41 | finucannot | melwitt: I'll take a look now | |
| 16:38:15 | finucannot | leakypipes: Could you take a look at https://review.openstack.org/#/c/390520/ again at some point in the next few days? | |
| 16:38:33 | melwitt | thanks finucannot | |
| 16:38:57 | leakypipes | finucannot: yes, wil try my best | |
| 16:39:22 | finucannot | leakypipes: Excellent. Appreciate it | |
| 16:41:44 | mriedem | sdague: superdan: methinks we should consider deprecating the libvirt.inject_key option | |
| 16:41:57 | mriedem | (1) depends on file injection | |
| 16:42:02 | mriedem | (2) doesn't work for volume-backed instances | |
| 16:42:06 | mriedem | (3) use config drive | |
| 16:42:54 | superdan | i | |
| 16:42:56 | superdan | am down | |
| 16:43:00 | mriedem | same with inject_password | |
| 16:44:37 | mriedem | oh fun we pass down the libvirt virt_type when injecting network info because of lxc | |
| 16:45:40 | finucannot | I've got a rather simply bugfix here that resolves an issue we've seen with dodgy glance configurations, if anyone's bored https://review.openstack.org/#/c/526329 | |
| 16:46:53 | finucannot | mriedem: Do we still support lxc? | |
| 16:47:31 | mriedem | finucannot: where does https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html say something about 0 data? | |
| 16:47:51 | finucannot | mriedem: https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data | |
| 16:48:06 | finucannot | I trimmed the URL because I didn't want to pollute everyone's git-log | |
| 16:48:14 | finucannot | (more than I'd already done) | |
| 16:54:45 | finucannot | Ha! python-glanceclient is broken itself | |
| 16:56:21 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707 | |
| 16:56:21 | openstackgerrit | Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357 | |
| 16:56:22 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 16:56:22 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 17:04:06 | melwitt | mriedem: I noticed you abandoned these https://review.openstack.org/#/c/508163 did we not want to keep them and not skip that test now that test_volume_backed_live_migration works? to guard against regression? | |
| 17:04:29 | mriedem | fewer double negatives please | |
| 17:05:28 | melwitt | sorry. like, we were using those DNM patches to test a fix for volume backed live migration | |
| 17:05:58 | melwitt | now that volume backed live migration is fixed, shouldn't we have removed DNM from those to unskip the test? | |
| 17:07:23 | melwitt | don't we want to unskip the test now that it will pass, that is | |
| 17:07:47 | mriedem | now that volume backed live migration is fixed.... | |
| 17:07:50 | mriedem | it is? | |
| 17:08:09 | mriedem | you mean https://review.openstack.org/#/c/507202/ ? | |
| 17:08:35 | melwitt | yeah, I thought that fixed it. you had those DNM patches depend on it and the unskipped tests passed | |
| 17:08:46 | melwitt | did I miss something? | |
| 17:08:52 | mriedem | melwitt: i believe those tests will intermittently pass regardless of that patch | |
| 17:09:06 | melwitt | okay, that's what I was missing then | |
| 17:09:09 | mriedem | the volume-backed and iscsi live migration tests were skipped in tempest long ago due to random failures | |
| 17:09:17 | mriedem | that no one could sleuth because qemu | |
| 17:09:29 | mriedem | using UCA now might have changed that | |
| 17:09:31 | melwitt | I see | |
| 17:09:43 | mriedem | bug 1524898 | |
| 17:09:44 | openstack | bug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,In progress] https://launchpad.net/bugs/1524898 | |
| 17:10:11 | melwitt | okay, I thought maybe that was related to the recent fix we had | |
| 17:10:20 | mriedem | even if we enable the test in tempest again, | |
| 17:10:33 | mriedem | tempest is branchless so it would run on stable branches and could fail there if we're using older packages that had a problem | |
| 17:10:37 | mriedem | hard to tell | |
| 17:10:53 | mriedem | i would very much like to get volume-backed live migration testing enabled in the gate though yes | |
| 17:10:54 | melwitt | oh, right. | |
| 17:11:08 | mriedem | we should probably change the global skip on the test in tempest to a config option, | |
| 17:11:11 | mriedem | so that we can enable the test per-branch | |
| 17:11:24 | melwitt | I can't recall if we skip certain tests for stable-only, or if that's even possible | |
| 17:11:36 | mriedem | we can configure tempest per-branch, per-job | |
| 17:11:43 | mriedem | we can configure tempest in our in-tree live migration scripts | |
| 17:11:46 | mriedem | the post_test_hook | |
| 17:12:07 | mriedem | https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh#L33 | |
| 17:12:19 | mriedem | with the magic of ansible | |
| 17:13:18 | melwitt | okay, so we'd add another option volume_backed_live_migration or such | |
| 17:13:25 | mriedem | so if you wanted to be cool, propose a change to tempest to add a config option for volume-backed live migration to replace that skip line, default to false for stable, and then set it to true in our live migration job post_test_hook script | |
| 17:13:44 | mriedem | and then be available when it melts the gate | |
| 17:14:09 | melwitt | can do | |
| 17:14:56 | melwitt | thanks | |
| 17:16:18 | mriedem | it is my pleasure | |
| 17:16:27 | mriedem | that's a weird phrase | |
| 17:18:18 | mriedem | dmsimard: interesting, even when it appears that we do set force_tcg mode, guestfs returns None rather than an int | |
| 17:18:19 | mriedem | http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full-centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_13_43_19_923098 | |
| 17:18:21 | melwitt | lots of zvm:recheck spam going on | |
| 17:18:25 | mriedem | Failed to force guestfs TCG mode. guestfs_set_backend_settings returned: None | |
| 17:18:45 | dmsimard | weird | |
| 17:19:43 | dmsimard | mriedem: it seems like there's two set_backend_settings happening there | |
| 17:19:57 | dmsimard | event=trace eh=0 buf='set_backend_settings "force_tcg"' array=[] | |
| 17:20:05 | dmsimard | and event=trace eh=0 buf='set_backend_settings = 0' array=[] ? | |
| 17:21:21 | mriedem | i think one is the value being set? | |
| 17:21:32 | mriedem | or the return value? | |
| 17:21:41 | mriedem | i don't know what the array is | |
| 17:21:46 | mriedem | but like "add_drive = 0" | |
| 17:24:27 | mriedem | http://libguestfs.org/guestfs.3.html#guestfs_set_event_callback | |
| 17:33:32 | mriedem | ok this is where we have the eventlet switch that blows up http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full-centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_13_43_19_941869 | |
| 17:33:39 | mriedem | that's right in the middle of the guestfs.launch call | |
| 17:33:48 | mriedem | error: cannot switch to a different thread | |
| 17:34:44 | mriedem | heh http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full-centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_15_25_29_320753 | |
| 17:34:54 | mriedem | so, i think the logging is what's maybe making it switch | |
| 17:35:13 | mriedem | so you enable guestfs debug to get callbacks from guestfs to log them, | |
| 17:35:19 | mriedem | meanwhile, guestfs is running in a thread | |
| 17:36:18 | mriedem | that doesn't explain why we were hanging before we enabled debug though | |
| 17:36:32 | mriedem | but i can push a patch to only launch in a thread if guestfs.debug is False | |
| 17:38:34 | fried_rice | cdent You going to be around in a couple hours? For a "warm handoff"? | |
| 17:38:41 | mriedem | gross | |
| 17:39:25 | cdent | fried_rice: yessir, I should be around, or at least nearby, make a ping and I’ll (eventually) come running | |
| 17:40:35 | fried_rice | cdent ack | |
| 17:40:59 | fried_rice | cdent Leaving a bit more of a mess than anticipated, I'm afraid. But nothing insurmountable. | |
| 17:49:26 | mnaser | is there anyone here who worked on glance image signature verification? | |
| 17:50:03 | mnaser | we have a change in puppet-nova to add the ability to configure it, but i'd like to know if its a global nova config (aka, api/etc need it) or nova-compute only | |
| 17:50:20 | mnaser | from my understanding of code, the option seems to be used in the glance image download code which probably only happens on computes? | |
| 17:52:09 | mriedem | peter-hamilton: ^ | |
| 17:52:24 | mriedem | mnaser: dane-fichter is who i'd usually go to but haven't seem him in channel in a long time | |