| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-02 | |||
| 14:50:20 | bhagyashris | johnthetubaguy: ok. | |
| 14:50:53 | johnthetubaguy | bhagyashris: sorry, I seem to have missed that you reply to me earlier, for some reason | |
| 14:51:31 | johnthetubaguy | bhagyashris: basically I think we should allow 0, and make that just delete snapshots, rather than create one it deletes right away after having just created it | |
| 14:52:22 | bhagyashris | johnthetubaguy: ok. | |
| 14:53:34 | johnthetubaguy | bhagyashris: that seems to solve the bug, and keep the API behaviour we had before, but I could be missing something | |
| 14:57:41 | bhagyashris | johnthetubaguy: in that case image_meta will be empty | |
| 14:58:35 | johnthetubaguy | bhagyashris: sorry, I am not sure what you mean? | |
| 15:00:01 | bhagyashris | johnthetubaguy: I mean the https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2697 will be empty if rotation==0 | |
| 15:01:07 | johnthetubaguy | bhagyashris: ah, very good point | |
| 15:01:07 | mriedem | sileht: i'm back, any news? | |
| 15:01:31 | johnthetubaguy | bhagyashris: which would totally be a change in the API | |
| 15:03:47 | johnthetubaguy | bhagyashris: it feels like this needs more discussion / a spec | |
| 15:05:47 | mriedem | oy | |
| 15:06:21 | mriedem | how do you keep calm when working with a customer for 2 weeks to apply some patches and they complain the patches don't work and you see they were patching with the --dry-run option the entire time | |
| 15:06:27 | mriedem | i.e. not actually applying any patches | |
| 15:06:30 | mriedem | :( | |
| 15:07:30 | sileht | mriedem, I'm seen the call returning something here: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_25_647 | |
| 15:07:49 | sileht | mriedem, and later nothing: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_43_150 | |
| 15:08:51 | bauzas | mriedem: fun story | |
| 15:09:12 | bhagyashris | johnthetubaguy: i will think on it. | |
| 15:09:13 | mriedem | bauzas: not only is the customer not applying them correctly, but there are at least 4 L3 people on the email thread who couldn't figure this out | |
| 15:09:27 | bauzas | :) | |
| 15:10:02 | mriedem | sileht: hmm, and nothing was deleted right? | |
| 15:10:14 | sileht | mriedem, yes one of the two instances have been deleted | |
| 15:10:18 | mriedem | markus_z: mtreinish: did a bug get created for that novnc issue? | |
| 15:10:30 | johnthetubaguy | bhagyashris: on a call, but just digging on the API | |
| 15:10:34 | sileht | mriedem, and I see only one instance delete on n-cpu | |
| 15:10:34 | kashyap | mriedem: Oh dear, not fun at all. | |
| 15:10:35 | markus_z | mriedem: I didn't. Here's the only novnc related patch I found https://github.com/openstack/nova/commit/8bbc1ee but it merged a few hours after the first issue :/ | |
| 15:10:50 | johnthetubaguy | bhagyashris: I am not sure the REST API actually changes, so an empty dict might be a fine return value from that function | |
| 15:10:50 | kashyap | mriedem: But hey, on the plus side, it is (hopefully) not some gory bug with packet loss :-) | |
| 15:11:07 | mriedem | kashyap: yeah if they said they did apply the fixes and the issue was still there, then that would be much worse | |
| 15:11:11 | markus_z | mriedem: I'm going to check other places too. | |
| 15:11:53 | kashyap | mriedem: Yeah. Glad you noticed it | |
| 15:12:02 | mriedem | markus_z: if we can just get a bug reported then we can track it in e-r | |
| 15:12:05 | mriedem | that's what i'm looking for now | |
| 15:12:12 | bhagyashris | johnthetubaguy: ok. Thank you for information | |
| 15:12:14 | markus_z | mriedem: yep, I'm writing it right now | |
| 15:13:40 | johnthetubaguy | bhagyashris: it looks like we already don't return the header if rotation == 0: https://github.com/openstack/nova/blob/e75d5f9d531dddd6ce0ada47d64bfcdce14f93cf/nova/api/openstack/compute/create_backup.py#L83 | |
| 15:15:00 | johnthetubaguy | bhagyashris: I quite strongly think we should keep allowing 0, but just stop creating the snapshot for that case, and only do the deletes | |
| 15:15:32 | markus_z | mriedem: https://bugs.launchpad.net/nova/+bug/1669468 | |
| 15:15:32 | openstack | Launchpad bug 1669468 in OpenStack Compute (nova) "tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently" [Undecided,New] | |
| 15:16:00 | mriedem | sileht: looks like at this point we have 1 cell and it's cell0 http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_07_05_528 | |
| 15:16:02 | bhagyashris | johnthetubaguy: ok. | |
| 15:16:16 | mriedem | sileht: at this point we have 2 cells http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_13_54_642 | |
| 15:16:20 | mriedem | cell0 and the main cell | |
| 15:16:25 | bhagyashris | johnthetubaguy: Thank you for information | |
| 15:16:34 | johnthetubaguy | bhagyashris: no problem | |
| 15:16:42 | mriedem | sileht: looks like both of those are before you're listing instances | |
| 15:17:25 | mriedem | markus_z: thanks | |
| 15:18:00 | mriedem | markus_z: note that logstash only goes back 10 days | |
| 15:18:04 | mriedem | so it might have been an issue before 2/21 | |
| 15:18:39 | markus_z | mriedem: ohh, ok, I didn't know that this is a hard limit. I'll check earlier changes too. | |
| 15:19:00 | markus_z | mriedem: I'm writing the e-r entry, takes a minute, haven't done it in a while. | |
| 15:19:48 | mriedem | markus_z: actually we probably don't want an e-r query for this, | |
| 15:19:53 | mriedem | it's mostly in the neutron multinode nv job | |
| 15:19:57 | mriedem | we don't have e-r queries for nv jobs | |
| 15:20:07 | mriedem | so it's probably something that just fails in multinode | |
| 15:20:56 | markus_z | mriedem: OK, got it. | |
| 15:23:01 | mriedem | sileht: so something is weird, looking at http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_43_150 where we don't get anything back | |
| 15:23:16 | mriedem | i see "Skipping already-collected cell0" just before that | |
| 15:23:27 | mriedem | which is this https://github.com/openstack/nova/blob/68e808623e42442482e92f0d61a6667876f29e19/nova/compute/api.py#L2538 | |
| 15:23:36 | mriedem | but i don't see this message after it | |
| 15:23:37 | mriedem | https://github.com/openstack/nova/blob/68e808623e42442482e92f0d61a6667876f29e19/nova/compute/api.py#L2540 | |
| 15:23:47 | mriedem | which is where it should be listing instances from the main cell | |
| 15:24:23 | mriedem | sileht: do you have a bug tracking the issue? i wanted to push some debug patches | |
| 15:24:32 | sileht | mriedem, I open it | |
| 15:25:35 | mriedem | so at this point we should have 2 cells | |
| 15:25:35 | mriedem | http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_13_54_642 | |
| 15:25:38 | mriedem | cell0 and cell1 | |
| 15:26:16 | mriedem | but 3 minutes later, we don't process cell1 http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_43_150 | |
| 15:26:26 | mriedem | which is where the instance would be | |
| 15:27:09 | mriedem | dansmith: melwitt: just thought of something, but on the CELLS cache in the api https://github.com/openstack/nova/blob/68e808623e42442482e92f0d61a6667876f29e19/nova/compute/api.py#L2524 | |
| 15:27:22 | mriedem | once we set that, we don't refresh it w/o restarting nova-api, | |
| 15:27:44 | mriedem | so you could map_cell0, start nova-api, and then create (or add) another cell | |
| 15:27:49 | mriedem | i think that's a known issue | |
| 15:27:52 | dansmith | yeah true | |
| 15:28:03 | dansmith | I was literally just thinking about updating those things | |
| 15:28:24 | dansmith | however, like the service catalog.. that's once per setup | |
| 15:28:28 | mriedem | seems we should re-fetch if len(CELLS) == 1 and CELLS[0].uuid == cell0.uuid | |
| 15:29:31 | mriedem | this is really odd actually | |
| 15:29:35 | mriedem | at this point we have cell0 http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_07_05_528 | |
| 15:29:52 | mriedem | at this point we have 2 cells http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_13_54_642 | |
| 15:30:01 | mriedem | that could only happen if nova-api was restarted in between | |
| 15:31:18 | mriedem | i don't see that happening though | |
| 15:31:51 | sileht | mriedem, https://bugs.launchpad.net/nova/+bug/1669473 | |
| 15:31:51 | openstack | Launchpad bug 1669473 in OpenStack Compute (nova) "openstack server list returns an empty list" [Undecided,New] | |
| 15:41:00 | mdbooth | BobBall: Are you the citrix xenserver guy? | |
| 15:41:10 | mriedem | sileht: ok dumped some debug details in the bug, i'm going to work on adding some debug to nova | |
| 15:41:14 | mriedem | mdbooth: he is | |
| 15:41:22 | sileht | mriedem, cool thx | |
| 15:41:24 | mdbooth | Thanks. | |
| 15:41:34 | mdbooth | BobBall: CI looks sad. Are you aware? | |
| 15:43:05 | mdbooth | mriedem: After reviewing your patch this morning, btw, I decided to 2 a cleanup of check_instance_state and found 2 bugs: https://review.openstack.org/#/c/440536/ | |
| 15:43:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log with cell.uuid if cell.name is not set https://review.openstack.org/440644 | |
| 15:43:34 | mriedem | mdbooth: which patch? | |
| 15:43:40 | mriedem | dansmith: ^ easy debug fix patch | |
| 15:43:49 | mdbooth | Nothing earth shattering, but still bugs caused by readability | |
| 15:44:19 | mdbooth | mriedem: https://review.openstack.org/#/c/429476/ | |
| 15:47:11 | mriedem | ah | |