| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-02 | |||
| 14:15:21 | mriedem | sdague: plus if it's in the list then others can weigh in | |
| 14:17:37 | mriedem | sileht: i feel like manila had the exact same type of issue you're talking about in the ML with the broken telemetry job | |
| 14:17:45 | mriedem | and it had something to do with the order of their devstack plugin setup, | |
| 14:17:53 | mriedem | such that it was trying to list instances before the instances were in a cell | |
| 14:17:54 | mriedem | something like that | |
| 14:18:17 | sileht | We don't do anything special | |
| 14:18:20 | mriedem | sileht: so need to know when simple_cell_setup is called relative to when the openstack server list returns nothing | |
| 14:19:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: fix missing instance.delete notification https://review.openstack.org/437222 | |
| 14:19:35 | mdbooth | johnthetubaguy: Nova re-fetching the list with the lock held would work, but only if cinder also didn't update the list until Nova had confirmed the detach. | |
| 14:19:43 | sileht | mriedem, we call "openstack server list" after ./stack.sh says everything is ready | |
| 14:19:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | |
| 14:19:59 | sileht | mriedem, and then we polls nova every second during 10 minutes | |
| 14:20:59 | mdbooth | johnthetubaguy: It's starting to sound to me like a nova-side implementation is going to be simpler. | |
| 14:21:19 | mdbooth | After all, it really is an implementation detail of the driver. | |
| 14:21:41 | johnthetubaguy | mdbooth: right, its assuming the attachments that are shared on the nova host are only modified by Nova (which we can enforce now, using service tokens) | |
| 14:22:00 | mriedem | sileht: ok simple_cell_setup is run well before that http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_09_08_19_738 | |
| 14:22:04 | johnthetubaguy | mdbooth: generally thats always the case anyways, but it can be enforced too | |
| 14:22:11 | sileht | mriedem, yes | |
| 14:23:17 | mriedem | sileht: is it listing servers with a filter? | |
| 14:23:23 | mriedem | or just listing servers and checking for one result? | |
| 14:23:43 | sileht | mriedem, just GET /servers/detail HTTP/1.1 | |
| 14:23:48 | mriedem | when did this start failing? | |
| 14:24:21 | sileht | I would says some days ago | |
| 14:25:16 | sileht | We didn't notice the exact time because we fall into an oslo.messaging from time to time and people recheck a lot this job | |
| 14:25:32 | mriedem | markus_z: looks like we have a novnc test failure in tempest http://logs.openstack.org/63/435563/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/20e15f9/console.html#_2017-03-01_20_30_52_369683 | |
| 14:25:36 | mriedem | intermittent i mean | |
| 14:25:44 | sileht | since yesterday or the day before, we seen that the error is no more the oslo.messaging bug | |
| 14:25:55 | sileht | and it's always failing | |
| 14:25:58 | openstackgerrit | Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580 | |
| 14:26:01 | mriedem | sileht: we should be able to identify when the failure started using logstash | |
| 14:26:18 | sileht | mriedem, i'm suspecting a policies bug, because when ceilometer polls nova with ceilometer service account and 'all_tenants=True', it see the instances | |
| 14:26:39 | sileht | instances owner are admin/admin | |
| 14:26:49 | mriedem | sileht: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20json%20path%20%24.servers%5B0%5D.status%20cannot%20match%20%7Bu'servers'%3A%20%5B%5D%7D%5C%22&from=7d | |
| 14:26:53 | sileht | and we do the 'openstack server list' as admin/admin | |
| 14:27:33 | markus_z | mriedem: I haven't seen that test before, I can take a look. | |
| 14:28:06 | mriedem | markus_z: pretty high failure rate | |
| 14:28:10 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20_validate_rfb_negotiation%5C%22&from=7d | |
| 14:28:12 | mriedem | sdague: ^ | |
| 14:28:17 | mriedem | i don't think we're tracking that - mtreinish ^ | |
| 14:28:53 | mriedem | sileht: looks like it started on 2/28 | |
| 14:29:04 | mriedem | so should dig through related looking changes https://github.com/openstack/nova/commits/master | |
| 14:29:06 | mriedem | i have to run for a bit | |
| 14:29:52 | mtreinish | mriedem: I don't think we're tracking it yet | |
| 14:34:08 | markus_z | the first hits start on 02/21, the tempest test merged 02/03 | |
| 14:34:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: fix missing instance.delete notification https://review.openstack.org/437222 | |
| 14:34:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | |
| 14:36:15 | markus_z | That's the first one: http://logs.openstack.org/38/431038/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/5e1d485/console.html | |
| 14:39:29 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting https://review.openstack.org/428199 | |
| 14:48:37 | bhagyashris | johnthetubaguy: Hi, i am waiting for your opinion regarding: which response code should i return if rotation==0? | |
| 14:49:14 | johnthetubaguy | bhagyashris: on that patch, I basically said there should be no API change, I believe? | |
| 14:49:44 | johnthetubaguy | bhagyashris: I suspect that wasn't very clear though, sorry. | |
| 14:50:00 | johnthetubaguy | bhagyashris: I would just upload what you have so far, or did you already do that? | |
| 14:50:20 | bhagyashris | johnthetubaguy: ok. | |
| 14:50:53 | johnthetubaguy | bhagyashris: sorry, I seem to have missed that you reply to me earlier, for some reason | |
| 14:51:31 | johnthetubaguy | bhagyashris: basically I think we should allow 0, and make that just delete snapshots, rather than create one it deletes right away after having just created it | |
| 14:52:22 | bhagyashris | johnthetubaguy: ok. | |
| 14:53:34 | johnthetubaguy | bhagyashris: that seems to solve the bug, and keep the API behaviour we had before, but I could be missing something | |
| 14:57:41 | bhagyashris | johnthetubaguy: in that case image_meta will be empty | |
| 14:58:35 | johnthetubaguy | bhagyashris: sorry, I am not sure what you mean? | |
| 15:00:01 | bhagyashris | johnthetubaguy: I mean the https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2697 will be empty if rotation==0 | |
| 15:01:07 | mriedem | sileht: i'm back, any news? | |
| 15:01:07 | johnthetubaguy | bhagyashris: ah, very good point | |
| 15:01:31 | johnthetubaguy | bhagyashris: which would totally be a change in the API | |
| 15:03:47 | johnthetubaguy | bhagyashris: it feels like this needs more discussion / a spec | |
| 15:05:47 | mriedem | oy | |
| 15:06:21 | mriedem | how do you keep calm when working with a customer for 2 weeks to apply some patches and they complain the patches don't work and you see they were patching with the --dry-run option the entire time | |
| 15:06:27 | mriedem | i.e. not actually applying any patches | |
| 15:06:30 | mriedem | :( | |
| 15:07:30 | sileht | mriedem, I'm seen the call returning something here: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_25_647 | |
| 15:07:49 | sileht | mriedem, and later nothing: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_43_150 | |
| 15:08:51 | bauzas | mriedem: fun story | |
| 15:09:12 | bhagyashris | johnthetubaguy: i will think on it. | |
| 15:09:13 | mriedem | bauzas: not only is the customer not applying them correctly, but there are at least 4 L3 people on the email thread who couldn't figure this out | |
| 15:09:27 | bauzas | :) | |
| 15:10:02 | mriedem | sileht: hmm, and nothing was deleted right? | |
| 15:10:14 | sileht | mriedem, yes one of the two instances have been deleted | |
| 15:10:18 | mriedem | markus_z: mtreinish: did a bug get created for that novnc issue? | |
| 15:10:30 | johnthetubaguy | bhagyashris: on a call, but just digging on the API | |
| 15:10:34 | kashyap | mriedem: Oh dear, not fun at all. | |
| 15:10:34 | sileht | mriedem, and I see only one instance delete on n-cpu | |
| 15:10:35 | markus_z | mriedem: I didn't. Here's the only novnc related patch I found https://github.com/openstack/nova/commit/8bbc1ee but it merged a few hours after the first issue :/ | |
| 15:10:50 | kashyap | mriedem: But hey, on the plus side, it is (hopefully) not some gory bug with packet loss :-) | |
| 15:10:50 | johnthetubaguy | bhagyashris: I am not sure the REST API actually changes, so an empty dict might be a fine return value from that function | |
| 15:11:07 | mriedem | kashyap: yeah if they said they did apply the fixes and the issue was still there, then that would be much worse | |
| 15:11:11 | markus_z | mriedem: I'm going to check other places too. | |
| 15:11:53 | kashyap | mriedem: Yeah. Glad you noticed it | |
| 15:12:02 | mriedem | markus_z: if we can just get a bug reported then we can track it in e-r | |
| 15:12:05 | mriedem | that's what i'm looking for now | |
| 15:12:12 | bhagyashris | johnthetubaguy: ok. Thank you for information | |
| 15:12:14 | markus_z | mriedem: yep, I'm writing it right now | |
| 15:13:40 | johnthetubaguy | bhagyashris: it looks like we already don't return the header if rotation == 0: https://github.com/openstack/nova/blob/e75d5f9d531dddd6ce0ada47d64bfcdce14f93cf/nova/api/openstack/compute/create_backup.py#L83 | |
| 15:15:00 | johnthetubaguy | bhagyashris: I quite strongly think we should keep allowing 0, but just stop creating the snapshot for that case, and only do the deletes | |
| 15:15:32 | openstack | Launchpad bug 1669468 in OpenStack Compute (nova) "tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently" [Undecided,New] | |
| 15:15:32 | markus_z | mriedem: https://bugs.launchpad.net/nova/+bug/1669468 | |
| 15:16:00 | mriedem | sileht: looks like at this point we have 1 cell and it's cell0 http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_07_05_528 | |
| 15:16:02 | bhagyashris | johnthetubaguy: ok. | |
| 15:16:16 | mriedem | sileht: at this point we have 2 cells http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_13_54_642 | |
| 15:16:20 | mriedem | cell0 and the main cell | |
| 15:16:25 | bhagyashris | johnthetubaguy: Thank you for information | |
| 15:16:34 | johnthetubaguy | bhagyashris: no problem | |