| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-20 | |||
| 18:48:28 | sdague | the fail rate on one of the notification tests in the functional bucket is pretty problematic - http://logs.openstack.org/29/421529/1/gate/gate-nova-tox-db-functional-ubuntu-xenial/20c5e9c/console.html#_2017-01-20_18_13_50_140452 | |
| 18:48:59 | sdague | we've got 2 gate resets right now on that | |
| 18:49:03 | sdague | http://logs.openstack.org/09/407309/16/gate/gate-nova-tox-db-functional-ubuntu-xenial/5680644/console.html#_2017-01-20_17_53_41_192031 as well | |
| 18:56:22 | melwitt | sdague: mriedem_afk has this change up to add logging to help with investigating that https://review.openstack.org/#/c/421221/ | |
| 18:59:21 | sdague | melwitt: +A | |
| 18:59:53 | melwitt | sweet | |
| 19:05:02 | melwitt | mriedem: ohay, I just commented on your d-g change, I think maybe I found what's wrong | |
| 19:05:11 | mriedem | melwitt: i just updated it for that thanks | |
| 19:05:20 | melwitt | oh, cool | |
| 19:05:24 | mriedem | that doesn't really explain why it's trying to run things on stable though | |
| 19:05:34 | mriedem | since: test -f discover_hosts.sh won't exist on stable | |
| 19:05:38 | mriedem | so it shouldn't try the sudo part | |
| 19:06:38 | melwitt | mriedem: oh, yeah sorry I found something else that I just commented on a sec ago, the permissions on the /logs dir | |
| 19:06:54 | melwitt | I dunno about the stable thing though | |
| 19:08:15 | mriedem | the logs permissions are changed before we run discover_host | |
| 19:08:23 | mriedem | so i'm thinking that's not it | |
| 19:09:09 | melwitt | I thought it's only doing that in the other code path | |
| 19:09:26 | melwitt | in the "else" | |
| 19:12:12 | mriedem | gdi | |
| 19:12:20 | mriedem | melwitt: do you ever get tired of being right? | |
| 19:13:00 | sdague | mriedem: is that the t-shirt slogan for Boston? | |
| 19:13:48 | melwitt | haha, well, we'll see if that actually works. d-g looks like magic to me | |
| 19:14:29 | oomichi | alex_xu: johnthetubaguy: gmann: Can you take a look at https://review.openstack.org/#/c/406377 ? | |
| 19:15:18 | mriedem | sdague: i think the slogan is going to be "wait, what if i upgrade to ocata THIS way" | |
| 19:16:08 | melwitt | haha, yes that | |
| 19:16:11 | sdague | people didn't realize that ocata is actually a puzzle room | |
| 19:16:18 | melwitt | lol | |
| 19:23:03 | mriedem | bauzas: ok +W on the placement fixture | |
| 19:23:47 | mriedem | sdague: so before laura gets home and explodes her travel anxiety on me, can you take a quick peek at https://review.openstack.org/#/c/423005/ and let me know if that's an OK track for this thing with cells v2 in d-g | |
| 19:24:18 | mriedem | the point of putting discover_hosts into a script in devstack is to avoid needing ugly branch logic in d-g itself | |
| 19:34:45 | sdague | mriedem: I think so... | |
| 19:36:03 | openstackgerrit | Francesco Santoro proposed openstack/nova: os-vif: convert libvirt driver to use os-vif for fast path vhostuser https://review.openstack.org/385061 | |
| 19:36:16 | melwitt | something blowing up on the bottom change | |
| 19:38:23 | openstackgerrit | STEW TY proposed openstack/nova: Transform instance.rescue notifications https://review.openstack.org/385644 | |
| 19:39:42 | melwitt | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 288: localrc: Permission denied | |
| 19:40:10 | melwitt | I wonder if those commands have to be after setup_localrc or something? | |
| 19:43:56 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323 | |
| 19:47:47 | mriedem | maybe but i'm not sure why | |
| 19:50:48 | mriedem | sdague: ^ do we need to run setup_localrc before we can change permissions on the files in the workspace? | |
| 19:51:22 | sdague | is there a full log somewhere? | |
| 19:52:37 | melwitt | sdague: http://logs.openstack.org/17/423417/1/check/gate-grenade-dsvm-neutron-ubuntu-xenial/479e3c9/console.html#_2017-01-20_19_25_44_044049 I don't know how to see more detail than that | |
| 19:55:10 | mriedem | yeah that's all i see | |
| 19:55:17 | melwitt | and this is the change https://review.openstack.org/#/c/423417/ | |
| 19:55:18 | mriedem | it calls setup_localrc and things blow up | |
| 19:55:53 | mriedem | oh i think i know | |
| 19:56:03 | mriedem | because d-g can't write to the files because stack user owns then now? | |
| 19:56:12 | mriedem | so, zuul user can't write to them in d-g | |
| 19:56:19 | melwitt | ohh | |
| 19:57:59 | melwitt | so I guess normally it sets up things first, then changes things over to the stack user | |
| 20:00:17 | sdague | we do definitely drop privs completely for safety reasons, and remove files from sudoers | |
| 20:02:37 | mriedem | ok, so i'll redo this and just move that stuff into a common function | |
| 20:02:41 | mriedem | so i can also call from the grenade branch | |
| 20:06:51 | sdague | yeh, your analysis that you are changing permissions too early is right, sorry it took a minute for me to gather all the context | |
| 20:07:21 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 20:08:33 | melwitt | thanks sdague | |
| 20:08:43 | mriedem | pushed | |
| 20:08:52 | mriedem | just in time for laura to walk in | |
| 20:08:59 | sdague | the permission changing is done for the stack user right before the stack.sh hand off. It starts as being under the jenkins user. The stack user can itself sudo up until this point - https://github.com/openstack-infra/devstack-gate/blob/5883a8269a741d77c1853150c56d6d90fe01b83e/devstack-vm-gate.sh#L720-L724 | |
| 20:09:16 | sdague | it's sudo privs are taken away before tempest runs, so that tempest doesn't cheat | |
| 20:09:19 | dims | mriedem :) | |
| 20:09:46 | dims | sdague : gate is 39 deep, something got busted? | |
| 20:10:02 | sdague | dims: as far as I can, it's just a ton of paper cuts | |
| 20:10:20 | sdague | nova notification tests are sometimes failing | |
| 20:10:23 | dims | ouch ok. at least we have weekend to catch up on | |
| 20:10:26 | sdague | I saw some db things failing on cinder | |
| 20:10:31 | dims | will watch a bit as well | |
| 20:10:42 | sdague | the keystone os-api-ref one, which I pushed a fix (and email on) | |
| 20:11:29 | dims | ack y i saw that. thanks sdague | |
| 20:14:50 | dims | ugh. just spotted this in the gate queue - http://logs.openstack.org/11/423111/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/60c1d57/logs/screen-n-cpu.txt.gz?level=ERROR#_2017-01-20_19_36_46_607 | |
| 20:15:00 | dims | libvirtError: internal error: client socket is closed | |
| 20:16:42 | mriedem | dims: that's known | |
| 20:16:50 | mriedem | libvirt crashes intermittently on xenial | |
| 20:17:10 | mriedem | http://status.openstack.org//elastic-recheck/index.html#1643911 | |
| 20:17:18 | mriedem | http://status.openstack.org//elastic-recheck/index.html#1646779 | |
| 20:17:25 | mriedem | http://status.openstack.org//elastic-recheck/index.html#1638982 | |
| 20:18:36 | dims | ack mriedem | |
| 20:21:54 | mriedem | jroll: vdrok: sambetts|afk: can i mark this as complete too? https://blueprints.launchpad.net/nova/+spec/ironic-plug-unplug-vifs-update | |
| 20:22:00 | mriedem | the vifs plug/unplug patch is merged, | |
| 20:22:05 | mriedem | the interface attach/detach thing isn't | |
| 20:22:21 | jroll | mriedem: yes, that doesn't mention the latter, that should be a separate blueprint imo | |
| 20:22:24 | mriedem | which is https://review.openstack.org/#/c/419975/ | |
| 20:22:26 | mriedem | ok | |
| 20:22:27 | mriedem | cool | |
| 20:23:01 | mriedem | nice both of those bps are merged then (portgroups and vifs update) | |
| 20:28:35 | bauzas | oh man, the gate... | |
| 20:37:26 | mriedem | ew we unconditionally run osprofiler on all unit test runs | |
| 20:37:30 | mriedem | even if i just want to test a specific path | |
| 20:50:40 | openstackgerrit | Julian proposed openstack/nova: Create cmd/script that prints JSON Schema of Notification Payload Classes https://review.openstack.org/411836 | |
| 20:52:30 | openstackgerrit | Matt Riedemann proposed openstack/nova: os-vif: convert libvirt driver to use os-vif for fast path vhostuser https://review.openstack.org/385061 | |
| 20:52:55 | mriedem | sean-k-mooney: leakypipes: i'm happy with ^ now | |
| 20:53:53 | mriedem | actually, hmm | |
| 20:54:18 | leakypipes | mriedem: need to boot into Windows VM... biab. (yep, Webex calls) | |
| 21:00:58 | bauzas | mriedem: sdague: I'm currently writing a patch for adding the microversion header in the report client | |
| 21:01:12 | bauzas | mriedem: sdague: do you think we should default to the min version or the max one ? | |
| 21:01:18 | bauzas | I think the latter | |
| 21:01:39 | bauzas | but maybe lemme upload the change | |
| 21:01:49 | sdague | bauzas: I thought we always default to min unless specified | |
| 21:02:03 | bauzas | sdague: I first thought about that | |
| 21:02:46 | bauzas | sdague: okay, lemme try to do this | |
| 21:02:56 | bauzas | and upload it so we can disucss | |