| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-20 | |||
| 17:43:58 | robcresswell | sdague: Thanks, you too | |
| 17:44:05 | diana_clarke | About the horizon/python-novaclient thing: the tests all pass but it doesn't work in real life (aka devstack), correct? | |
| 17:46:43 | robcresswell | diana_clarke: Seems that way, yep | |
| 17:47:48 | robcresswell | diana_clarke: Haven't done much investigation, mainly just marched over here with my pitchfork. I'll be around again Monday if there are some suggestions, otherwise I'll just poke at Horizon until it works. | |
| 17:49:14 | robcresswell | Primarily a time constraint thing though so far, will have more time to dedicate next week. PTl isn't around much right now, so I'm probably the easiest person to contact :) | |
| 17:49:53 | openstackgerrit | Matthew Booth proposed openstack/nova: libvirt: Remove more imagecache info file code https://review.openstack.org/423392 | |
| 17:50:43 | diana_clarke | robcresswell: If I had to guess (without much digging), I suspect it has to do with the deprecations that were done here, specially if it's project id related: https://review.openstack.org/#/q/project:openstack/python-novaclient+branch:master+topic:session | |
| 17:51:05 | diana_clarke | *especially | |
| 17:51:11 | diana_clarke | (but I could be wrong) | |
| 17:52:27 | diana_clarke | perhaps this, in particular: https://review.openstack.org/#/c/350106/14/releasenotes/notes/clarify-project-id-variable-5832698315000438.yaml | |
| 17:52:34 | robcresswell | diana_clarke: Would you mind leaving suggestions on https://bugs.launchpad.net/python-novaclient/+bug/1658148 ? I'm happy to spend the time looking, but this is a busy channel so I'll likely lose track if suggestions are posted here. | |
| 17:52:34 | openstack | Launchpad bug 1658148 in OpenStack Dashboard (Horizon) "Horizon/Nova interactions broken " [Critical,New] - Assigned to Rob Cresswell (robcresswell) | |
| 17:53:23 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 17:53:35 | diana_clarke | robcresswell: I'm mostly just talking out loud at this point. If I find something legit I'll be sure to update that bug, I'm on PTO this afternoon though, so I probably won't look much until Monday either. | |
| 17:54:55 | robcresswell | diana_clarke: Ah cool, not a problem. Even educated guesses are probably more valuable than my blundering around novaclient code though. Have a good weekend, and thanks for the help. | |
| 18:15:33 | openstackgerrit | Matthew Booth proposed openstack/nova: libvirt: Fix races in glusterfs mount/umount https://review.openstack.org/423402 | |
| 18:16:33 | mdbooth | johnthetubaguy: For context with the nfs volume driver, with that in place GlusterFS took me about 30 minutes ^^^ | |
| 18:23:47 | openstackgerrit | Anusha Unnam proposed openstack/nova: Add keypairs field to InstancePayload https://review.openstack.org/419730 | |
| 18:30:10 | openstackgerrit | Matthew Booth proposed openstack/nova: libvirt: Remove more imagecache info file code https://review.openstack.org/423392 | |
| 18:48:28 | sdague | the fail rate on one of the notification tests in the functional bucket is pretty problematic - http://logs.openstack.org/29/421529/1/gate/gate-nova-tox-db-functional-ubuntu-xenial/20c5e9c/console.html#_2017-01-20_18_13_50_140452 | |
| 18:48:59 | sdague | we've got 2 gate resets right now on that | |
| 18:49:03 | sdague | http://logs.openstack.org/09/407309/16/gate/gate-nova-tox-db-functional-ubuntu-xenial/5680644/console.html#_2017-01-20_17_53_41_192031 as well | |
| 18:56:22 | melwitt | sdague: mriedem_afk has this change up to add logging to help with investigating that https://review.openstack.org/#/c/421221/ | |
| 18:59:21 | sdague | melwitt: +A | |
| 18:59:53 | melwitt | sweet | |
| 19:05:02 | melwitt | mriedem: ohay, I just commented on your d-g change, I think maybe I found what's wrong | |
| 19:05:11 | mriedem | melwitt: i just updated it for that thanks | |
| 19:05:20 | melwitt | oh, cool | |
| 19:05:24 | mriedem | that doesn't really explain why it's trying to run things on stable though | |
| 19:05:34 | mriedem | since: test -f discover_hosts.sh won't exist on stable | |
| 19:05:38 | mriedem | so it shouldn't try the sudo part | |
| 19:06:38 | melwitt | mriedem: oh, yeah sorry I found something else that I just commented on a sec ago, the permissions on the /logs dir | |
| 19:06:54 | melwitt | I dunno about the stable thing though | |
| 19:08:15 | mriedem | the logs permissions are changed before we run discover_host | |
| 19:08:23 | mriedem | so i'm thinking that's not it | |
| 19:09:09 | melwitt | I thought it's only doing that in the other code path | |
| 19:09:26 | melwitt | in the "else" | |
| 19:12:12 | mriedem | gdi | |
| 19:12:20 | mriedem | melwitt: do you ever get tired of being right? | |
| 19:13:00 | sdague | mriedem: is that the t-shirt slogan for Boston? | |
| 19:13:48 | melwitt | haha, well, we'll see if that actually works. d-g looks like magic to me | |
| 19:14:29 | oomichi | alex_xu: johnthetubaguy: gmann: Can you take a look at https://review.openstack.org/#/c/406377 ? | |
| 19:15:18 | mriedem | sdague: i think the slogan is going to be "wait, what if i upgrade to ocata THIS way" | |
| 19:16:08 | melwitt | haha, yes that | |
| 19:16:11 | sdague | people didn't realize that ocata is actually a puzzle room | |
| 19:16:18 | melwitt | lol | |
| 19:23:03 | mriedem | bauzas: ok +W on the placement fixture | |
| 19:23:47 | mriedem | sdague: so before laura gets home and explodes her travel anxiety on me, can you take a quick peek at https://review.openstack.org/#/c/423005/ and let me know if that's an OK track for this thing with cells v2 in d-g | |
| 19:24:18 | mriedem | the point of putting discover_hosts into a script in devstack is to avoid needing ugly branch logic in d-g itself | |
| 19:34:45 | sdague | mriedem: I think so... | |
| 19:36:03 | openstackgerrit | Francesco Santoro proposed openstack/nova: os-vif: convert libvirt driver to use os-vif for fast path vhostuser https://review.openstack.org/385061 | |
| 19:36:16 | melwitt | something blowing up on the bottom change | |
| 19:38:23 | openstackgerrit | STEW TY proposed openstack/nova: Transform instance.rescue notifications https://review.openstack.org/385644 | |
| 19:39:42 | melwitt | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 288: localrc: Permission denied | |
| 19:40:10 | melwitt | I wonder if those commands have to be after setup_localrc or something? | |
| 19:43:56 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323 | |
| 19:47:47 | mriedem | maybe but i'm not sure why | |
| 19:50:48 | mriedem | sdague: ^ do we need to run setup_localrc before we can change permissions on the files in the workspace? | |
| 19:51:22 | sdague | is there a full log somewhere? | |
| 19:52:37 | melwitt | sdague: http://logs.openstack.org/17/423417/1/check/gate-grenade-dsvm-neutron-ubuntu-xenial/479e3c9/console.html#_2017-01-20_19_25_44_044049 I don't know how to see more detail than that | |
| 19:55:10 | mriedem | yeah that's all i see | |
| 19:55:17 | melwitt | and this is the change https://review.openstack.org/#/c/423417/ | |
| 19:55:18 | mriedem | it calls setup_localrc and things blow up | |
| 19:55:53 | mriedem | oh i think i know | |
| 19:56:03 | mriedem | because d-g can't write to the files because stack user owns then now? | |
| 19:56:12 | mriedem | so, zuul user can't write to them in d-g | |
| 19:56:19 | melwitt | ohh | |
| 19:57:59 | melwitt | so I guess normally it sets up things first, then changes things over to the stack user | |
| 20:00:17 | sdague | we do definitely drop privs completely for safety reasons, and remove files from sudoers | |
| 20:02:37 | mriedem | ok, so i'll redo this and just move that stuff into a common function | |
| 20:02:41 | mriedem | so i can also call from the grenade branch | |
| 20:06:51 | sdague | yeh, your analysis that you are changing permissions too early is right, sorry it took a minute for me to gather all the context | |
| 20:07:21 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 20:08:33 | melwitt | thanks sdague | |
| 20:08:43 | mriedem | pushed | |
| 20:08:52 | mriedem | just in time for laura to walk in | |
| 20:08:59 | sdague | the permission changing is done for the stack user right before the stack.sh hand off. It starts as being under the jenkins user. The stack user can itself sudo up until this point - https://github.com/openstack-infra/devstack-gate/blob/5883a8269a741d77c1853150c56d6d90fe01b83e/devstack-vm-gate.sh#L720-L724 | |
| 20:09:16 | sdague | it's sudo privs are taken away before tempest runs, so that tempest doesn't cheat | |
| 20:09:19 | dims | mriedem :) | |
| 20:09:46 | dims | sdague : gate is 39 deep, something got busted? | |
| 20:10:02 | sdague | dims: as far as I can, it's just a ton of paper cuts | |
| 20:10:20 | sdague | nova notification tests are sometimes failing | |
| 20:10:23 | dims | ouch ok. at least we have weekend to catch up on | |
| 20:10:26 | sdague | I saw some db things failing on cinder | |
| 20:10:31 | dims | will watch a bit as well | |
| 20:10:42 | sdague | the keystone os-api-ref one, which I pushed a fix (and email on) | |
| 20:11:29 | dims | ack y i saw that. thanks sdague | |
| 20:14:50 | dims | ugh. just spotted this in the gate queue - http://logs.openstack.org/11/423111/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/60c1d57/logs/screen-n-cpu.txt.gz?level=ERROR#_2017-01-20_19_36_46_607 | |
| 20:15:00 | dims | libvirtError: internal error: client socket is closed | |
| 20:16:42 | mriedem | dims: that's known | |
| 20:16:50 | mriedem | libvirt crashes intermittently on xenial | |
| 20:17:10 | mriedem | http://status.openstack.org//elastic-recheck/index.html#1643911 | |
| 20:17:18 | mriedem | http://status.openstack.org//elastic-recheck/index.html#1646779 | |
| 20:17:25 | mriedem | http://status.openstack.org//elastic-recheck/index.html#1638982 | |
| 20:18:36 | dims | ack mriedem | |
| 20:21:54 | mriedem | jroll: vdrok: sambetts|afk: can i mark this as complete too? https://blueprints.launchpad.net/nova/+spec/ironic-plug-unplug-vifs-update | |
| 20:22:00 | mriedem | the vifs plug/unplug patch is merged, | |
| 20:22:05 | mriedem | the interface attach/detach thing isn't | |
| 20:22:21 | jroll | mriedem: yes, that doesn't mention the latter, that should be a separate blueprint imo | |
| 20:22:24 | mriedem | which is https://review.openstack.org/#/c/419975/ | |