| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 13:38:12 | archies | JOIN /#openstack-meeting | |
| 13:40:21 | sambetts | johnthetubaguy: I'm still not sure I get what passing host_id=None through would do, we want to force it to None if it is already set, because otherwise it would prevent Ironic binding its provisoning network port | |
| 13:40:46 | johnthetubaguy | sambetts: but if its already set, someone else owns the port | |
| 13:41:17 | johnthetubaguy | sambetts: we should fail earlier on if thats already set | |
| 13:41:33 | johnthetubaguy | (hence the race is a really tiny window, if at all) | |
| 13:41:50 | sambetts | johnthetubaguy: +1 surly you can't pass a port that has host_id set into --nic port-id right?? | |
| 13:42:24 | sambetts | johnthetubaguy: that would be dangerous for normal cases | |
| 13:42:26 | johnthetubaguy | right, but you can pass the same port into two instances, and they race each other to apply the binding, the last one "wins" | |
| 13:42:57 | johnthetubaguy | I would just rather we didn't send bind_host_id=None, we just don't update bind_host_id | |
| 13:43:10 | johnthetubaguy | its a niggle though, not a blocker | |
| 13:43:28 | sambetts | johnthetubaguy: but you are OK writing host_id=<the last one> over the one already set? | |
| 13:43:53 | sambetts | johnthetubaguy: surely we should fail over if at that point the host_id is already set on that port | |
| 13:43:57 | johnthetubaguy | sambetts: no, there is a fix something in the neutron API soon, so we don't do that any more | |
| 13:44:17 | johnthetubaguy | basically, the neutron API isn't atomic, and they are adding a generation id we can use to make sure the update is atomic | |
| 13:44:33 | johnthetubaguy | thats a fix for next cycle, with a bit of luck | |
| 13:45:43 | johnthetubaguy | i.e we can check generation7 doesn't have the bind_host_id set, and fail if there have been updates to it since that check | |
| 13:45:53 | sambetts | I think for our case then we'd need a if get_port(<port_id>).binding_profile['host_id'] is not None: raise PortAlreadyBound() because we need the port to be unbound | |
| 13:47:09 | sambetts | we'd still run into issues were if we were racing another deploy we won't catch it if a port binds after that point | |
| 13:47:22 | sambetts | after we've been through it | |
| 13:47:23 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 13:48:31 | johnthetubaguy | sambetts: I am trying to find that check, I thought we had that already | |
| 13:48:59 | sambetts | johnthetubaguy: isn't there a part of the port that ties it to an instance not just a host_id? | |
| 13:49:13 | johnthetubaguy | https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L555 | |
| 13:49:15 | sambetts | johnthetubaguy: surely we should be looking at that to know if its already used not the host id | |
| 13:49:17 | johnthetubaguy | device_id does that | |
| 13:49:37 | johnthetubaguy | so... we should back up | |
| 13:49:52 | johnthetubaguy | not questioning the overall approach here, thats just fine | |
| 13:50:10 | sambetts | yeah, I'm just trying to work out if the race is possible or not :/ | |
| 13:50:33 | sambetts | I would have expected it to catch if the port was in use really early in the process | |
| 13:50:58 | johnthetubaguy | we do, we check twice, because of the racyness | |
| 13:51:04 | johnthetubaguy | still can happen | |
| 13:51:22 | _gryf | EmilienM, did you find out the culprit for the failing placement? | |
| 13:51:33 | johnthetubaguy | once in API, once in compute before binding, but its racey still (just) | |
| 13:52:01 | johnthetubaguy | we have a way to fix that for next cycle | |
| 13:53:08 | bauzas | _gryf: it seems so | |
| 13:53:42 | bauzas | _gryf: EmilienM pinged me late yesterday night for telling me it was due to the HAProxy service used by TripleO | |
| 13:53:48 | archies | @mriedem, @nova core, sorry for interrupting in between. Wanted to check your views on the https://bugs.launchpad.net/nova/+bug/1636489 | |
| 13:53:48 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 13:54:35 | archies | we are stuck on this defect and wants to check the your views on it, to get the importance etc | |
| 13:54:45 | _gryf | bauzas, is there any patch for that? | |
| 13:54:58 | johnthetubaguy | sambetts: OK, so I think this bit brings me around to your approach: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L497 | |
| 13:55:14 | bauzas | _gryf: dunno | |
| 13:56:31 | _gryf | bauzas, k. besides that seems like without keystone we do have failing like this | |
| 13:59:07 | EmilienM | bauzas: yes, I think we spot the issue in tripleo | |
| 13:59:30 | EmilienM | bauzas: the haproxy config was not done correctly for this service, the binding was on the wrong network | |
| 13:59:42 | EmilienM | so haproxy thought placement api was down while it was up on another network | |
| 13:59:51 | bauzas | ok good to know | |
| 13:59:52 | openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Scheduler calling the Placement API https://review.openstack.org/417961 | |
| 13:59:54 | EmilienM | we are fixing it now but everything should work | |
| 13:59:58 | EmilienM | bauzas: thanks again for your help | |
| 14:00:09 | bauzas | EmilienM: np | |
| 14:00:21 | bauzas | EmilienM: FWIW, still working on the scheduler change above ^ | |
| 14:00:31 | bauzas | EmilienM: this one could be hitting TripleO | |
| 14:00:38 | bauzas | unless the placement service is there | |
| 14:00:53 | bauzas | so I'll run check experimental once I'm done with the UTs | |
| 14:01:04 | EmilienM | ok cool | |
| 14:01:33 | bauzas | _gryf: FWIW, in case you don't know, we can test placement changes against TripleO by running check experimental :) | |
| 14:01:48 | sambetts | johnthetubaguy: yeah :/ not sure what the best approach for now is, it sounds like we need the atomic thing, IMO we shouldn't ever manage to set the binding profile for a port not for this instance and if its for this instance then it should be unbound because ironic will bind it | |
| 14:01:49 | bauzas | jaypipes: in case you don't know too ^ | |
| 14:02:30 | bauzas | jaypipes: FWIW, do you think Fuel or other CDs would be impacted by https://review.openstack.org/417961 ? | |
| 14:03:01 | bauzas | perhaps Kolla | |
| 14:03:56 | johnthetubaguy | sambetts: I should find the spec that has the solution for that race, I think its mostly Nova's issue, and shouldn't worry the ironic phase, if we consider the device_id the thing that claims the ownership (beyond ironic checking the device_id is set correctly) | |
| 14:06:21 | jroll | johnthetubaguy: sambetts: so I don't think what we're changing here changes the situation at all, right, it's the same problem before and after the change | |
| 14:07:26 | johnthetubaguy | jroll: right, thats a better way of putting what I said | |
| 14:07:34 | johnthetubaguy | well, what I meant to say | |
| 14:08:24 | jroll | heh, ok cool :) | |
| 14:08:26 | jroll | thanks | |
| 14:08:59 | jroll | ah, I see your +Ws now, thank you! :) | |
| 14:11:40 | Sudha | Hi all , ANy thoughts on https://bugs.launchpad.net/nova/+bug/1636489 ? | |
| 14:11:40 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 14:12:33 | sambetts | johnthetubaguy, jroll: awesome, I hope we can fix it with the atomic stuff next cycle :) its really edge casey anyway | |
| 14:12:43 | jroll | yeah, +1 | |
| 14:14:07 | johnthetubaguy | sambetts: jroll: sorry for the firedrill, mostly getting a heap of stuff re-discovered and straightened out in my head | |
| 14:14:20 | jroll | johnthetubaguy: no worries, I learned a thing today \o/ | |
| 14:14:41 | sambetts | johnthetubaguy: \o/ yeah big TIL moment | |
| 14:22:48 | bauzas | do we have something wonkey with the py35 unit tests ? | |
| 14:23:26 | bauzas | that looks weirdo | |
| 14:24:02 | bauzas | dims: have we recently merged something for py35 ? | |
| 14:24:11 | archies | @Nova team, I have a case where I need your help to understand better. | |
| 14:24:13 | bauzas | it seems we have a critical issue | |
| 14:24:23 | dims | nope bauzas | |
| 14:24:33 | archies | @all, In one of my case, Nova's "detach_volume" fires the detach method into libvirt, which claims success, but the device is still attached as per the guest XML file. | |
| 14:24:33 | bauzas | lemme look at logstash | |
| 14:24:40 | sdague | bauzas: is there a link to a fail? | |
| 14:24:50 | bauzas | sdague: eg. http://logs.openstack.org/34/418134/7/check/gate-nova-python35-db/578aaff/testr_results.html.gz | |
| 14:25:00 | bauzas | zuul tells me all nova checks are red | |
| 14:25:44 | sdague | bauzas: https://pypi.python.org/pypi/mox3/0.19.0 ... | |
| 14:26:00 | bauzas | sdague: opening a bug | |
| 14:28:21 | sdague | bauzas: there are only a couple of commits - http://git.openstack.org/cgit/openstack/mox3/commit/?id=4a85ff0ec4ceb98ad87bec6002f6e67788b06d00 | |
| 14:28:51 | bauzas | sdague: https://bugs.launchpad.net/nova/+bug/1657757 | |
| 14:28:51 | openstack | Launchpad bug 1657757 in OpenStack Compute (nova) "py35 unit tests are hugely failing" [Critical,New] | |
| 14:29:04 | jroll | s/hugely/bigly/ | |
| 14:29:19 | sdague | bauzas: there are only 4 patches since the last release, the inspect one is the only of consequence | |
| 14:29:34 | mriedem | new release broke everything? | |
| 14:30:04 | bauzas | mriedem: sdague: trying to bisect | |
| 14:30:30 | sdague | bauzas: it has to be that patch I posted | |
| 14:32:19 | bauzas | sdague: probably, I just wanted to see the logstash query | |
| 14:32:55 | bauzas | mriedem: sdague: tried with the query "project:openstack/nova AND build_status:FAILURE AND build_name:gate-nova-python35-db" but not sure that one is good | |
| 14:36:09 | bauzas | sdague: mriedem: trying to run the job locally with an older mox release | |
| 14:36:24 | sdague | https://review.openstack.org/422651 is the revert | |