| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-02 | |||
| 19:40:45 | openstack | Launchpad bug 1669468 in OpenStack Compute (nova) "tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently in neutron multinode nv job" [Medium,Confirmed] | |
| 19:40:45 | mriedem | ihrachys: re-opened https://bugs.launchpad.net/nova/+bug/1669468 | |
| 19:41:07 | dansmith | mriedem: sdague: so I would love to lead one of you through an exercise to prove that something is broken or that I'm crazy | |
| 19:41:38 | sdague | dansmith: I'm kind of borked today, as the TC meeting is in 20, and I've got a hard stop after that. Tomorrow I'd be fine | |
| 19:42:00 | mriedem | dansmith: i can be your huckleberry here in a bit | |
| 19:42:01 | ihrachys | mriedem, thanks | |
| 19:42:17 | ihrachys | mriedem, is it intermittent or total stop? | |
| 19:42:24 | ihrachys | I mean for ocata | |
| 19:43:26 | mriedem | ihrachys: intermittent i think | |
| 19:43:32 | mriedem | 21 hits in 7 days | |
| 19:43:36 | mriedem | and we've been merging ocata changes | |
| 19:43:56 | ihrachys | ok, will recheck then | |
| 19:48:12 | melwitt | dansmith: yes, I did the scatter-gather here https://review.openstack.org/#/c/416521/29/nova/quota.py@1903 | |
| 19:49:01 | dansmith | melwitt: cool thanks | |
| 19:56:32 | cdent | mriedem: I will be at summit, and when I found out I updated the etherpad that had that list of names of who will or won't be there. The foundation requested my presence as api-wg dood gave me a room. | |
| 19:56:44 | mriedem | cdent: ooo | |
| 19:56:50 | mriedem | that's news to me, obviously | |
| 19:57:03 | cdent | I am, however, off work tomorrow, walking in the woods | |
| 19:57:09 | EmilienM | iiuc I should not do recheck on nova patches right? live migration is broken in the gate? | |
| 19:57:23 | mriedem | EmilienM: fixed a few hours ago | |
| 19:57:26 | EmilienM | ack | |
| 19:57:50 | mriedem | cdent: that's allowed as long as you promise to ponder nova onboarding and other work-related issues while in nature | |
| 19:57:56 | cdent | mriedem: i intend to attend all the placement related sessions (forum and conference) and in general be around | |
| 19:58:01 | cdent | mriedem: thanks so much, I'll do that | |
| 19:58:25 | cdent | I shall think upon things that made me go WTF | |
| 20:00:23 | openstackgerrit | Steve Noyes proposed openstack/nova master: Add Cinder v3 detach to shutdown_instance https://review.openstack.org/456877 | |
| 20:33:33 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: get providers sharing capacity https://review.openstack.org/460798 | |
| 20:33:33 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: shared resources when finding providers https://review.openstack.org/461574 | |
| 20:33:34 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: test non-shared out of inventory https://review.openstack.org/461900 | |
| 20:33:34 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: tests for non-shared with shared https://review.openstack.org/461899 | |
| 20:33:35 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: test for agg association not sharing https://review.openstack.org/461901 | |
| 20:33:51 | jaypipes | cdent: added those additional tests you were looking for ^^ | |
| 20:34:15 | cdent | you are a star, will look tomorrow | |
| 20:34:16 | cdent | actually no I won't | |
| 20:34:18 | cdent | I will look Thursday. I'm gone tomorrow | |
| 20:34:51 | jaypipes | cdent: cool beans, thanks! | |
| 20:37:03 | jaypipes | mriedem: this onboarding thing is for potential new Nova contributors, yes? | |
| 20:37:21 | mriedem | jaypipes: that's my understanding | |
| 20:37:25 | jaypipes | kk | |
| 20:37:35 | jaypipes | mriedem: just wanted to make sure... | |
| 20:43:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992 | |
| 20:43:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676 | |
| 20:43:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Transform instance.volume_attach.error notification https://review.openstack.org/455801 | |
| 20:46:56 | mriedem | ^ +2 on those 3 now | |
| 20:47:09 | mriedem | dansmith: did you figure out your dilemma yet? | |
| 20:57:16 | dansmith | mriedem: no | |
| 20:58:06 | dansmith | mriedem: I want to trace this delete call: http://logs.openstack.org/94/436094/23/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/83b5f81/console.html#_2017-05-02_18_17_34_903731 | |
| 20:58:16 | dansmith | here's the api log: http://logs.openstack.org/94/436094/23/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/83b5f81/logs/screen-n-api.txt.gz | |
| 20:59:01 | dansmith | the request-id for that is req-bc378dc5-41f0-4df8-95cf-4226ae088007 right? | |
| 20:59:06 | dansmith | so search for that in the api log, there is one hit | |
| 20:59:11 | dansmith | it's the return of the http call | |
| 20:59:23 | dansmith | but it should have been logged a bunch of times before that doing the actual delete | |
| 20:59:44 | dansmith | so, grab the instance uuid: 121af6e1-981e-4b5b-96c4-c9c19cd8b955 and search for that in the api log | |
| 21:00:14 | dansmith | the delete of that instance is here: http://logs.openstack.org/94/436094/23/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/83b5f81/logs/screen-n-api.txt.gz#_May_02_17_55_49_123287 | |
| 21:00:24 | dansmith | four minutes after the http call returned | |
| 21:00:33 | dansmith | and with a different req-id | |
| 21:01:01 | dansmith | (brb) | |
| 21:08:28 | mriedem | dansmith: my guess would be the first delete request is ignored/lost somewhere in the api because the instance isn't in a cell yet, and the 2nd delete request is when the test tears down after a timeout | |
| 21:08:46 | mriedem | so when the first delete request comes in, i'm guessing we're building the instance in conductor | |
| 21:09:10 | dansmith | mriedem: it doesn't run any of the debugging messages that I have sprinkled throughout the delete-while-building code | |
| 21:09:46 | mriedem | do we know what the nova-osapi_compute is at all? | |
| 21:09:51 | mriedem | version i mean | |
| 21:10:10 | dansmith | what do you mean? that's not an upgrade test | |
| 21:10:32 | mriedem | i mean this check https://review.openstack.org/#/c/461513/2/nova/compute/api.py@1762 | |
| 21:11:15 | sdague | req-bc378dc5-41f0-4df8-95cf-4226ae088007 is the req id on the metadata server, right? | |
| 21:11:19 | dansmith | hmm, good point, I thought we removed that | |
| 21:11:49 | dansmith | sdague: why do you say that? | |
| 21:12:01 | mriedem | req-bc378dc5-41f0-4df8-95cf-4226ae088007 is the request id in the original response for DELETE | |
| 21:12:04 | mriedem | from what i can tell | |
| 21:12:11 | dansmith | yeah | |
| 21:12:40 | mriedem | dansmith: it's also odd that we don't see a debug message about fetching the instance for the original request | |
| 21:12:45 | mriedem | original DELETE request | |
| 21:12:46 | dansmith | right | |
| 21:12:59 | mriedem | so it seems that the compute api.get() method didn't find it | |
| 21:13:01 | sdague | oh, maybe, not the double host thing I guess is new? | |
| 21:13:01 | dansmith | or any of the other stuff I have added in my dependent patch | |
| 21:13:10 | sdague | that's weird, I wonder where that's coming from | |
| 21:13:13 | sdague | anyway, got torun | |
| 21:13:18 | mriedem | dansmith: but then we should get a 404 | |
| 21:13:19 | mriedem | not a 204 | |
| 21:13:27 | dansmith | mriedem: right | |
| 21:14:20 | mriedem | Fetching instance by UUID | |
| 21:14:22 | mriedem | is what i'd expect to see | |
| 21:15:46 | dansmith | yep | |
| 21:16:33 | mriedem | where did all of the "from (pid=16617)" and "from (pid=16616)" come from in the debug messages? | |
| 21:16:39 | mriedem | those aren't on the INFO messages | |
| 21:17:39 | dansmith | I think this must be from the systemd change | |
| 21:17:45 | dansmith | because those weren't in there a few days ago | |
| 21:17:48 | mriedem | https://github.com/openstack/oslo.log/commit/a5ee482ceeee78cc801b3419b9bc3efae04d0f26 | |
| 21:17:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/461446 | |
| 21:18:52 | mriedem | oslo.log 3.26.0 came into upper-constraints on 4/24 | |
| 21:19:34 | openstackgerrit | Abhishek Sharma M proposed openstack/nova master: Catching OverQuota Exception https://review.openstack.org/460305 | |
| 21:21:42 | dansmith | mriedem: sdague is pinging me through another channel whilst elevating the temperature of dead animal flesh exterior to his home | |
| 21:22:03 | dansmith | mriedem: about the possibility of an interaction between the journal logging calls when we're using eventlet | |
| 21:22:07 | dansmith | like blocking the process | |
| 21:22:11 | dansmith | which is kindof interesting | |
| 21:22:27 | mriedem | ok | |
| 21:22:30 | mriedem | that sounds super fun | |
| 21:22:39 | dansmith | not so much | |
| 21:22:52 | mriedem | also, these giant journal zips produced in the jobs now, how does one read those? | |