| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 13:50:10 | sambetts | yeah, I'm just trying to work out if the race is possible or not :/ | |
| 13:50:33 | sambetts | I would have expected it to catch if the port was in use really early in the process | |
| 13:50:58 | johnthetubaguy | we do, we check twice, because of the racyness | |
| 13:51:04 | johnthetubaguy | still can happen | |
| 13:51:22 | _gryf | EmilienM, did you find out the culprit for the failing placement? | |
| 13:51:33 | johnthetubaguy | once in API, once in compute before binding, but its racey still (just) | |
| 13:52:01 | johnthetubaguy | we have a way to fix that for next cycle | |
| 13:53:08 | bauzas | _gryf: it seems so | |
| 13:53:42 | bauzas | _gryf: EmilienM pinged me late yesterday night for telling me it was due to the HAProxy service used by TripleO | |
| 13:53:48 | archies | @mriedem, @nova core, sorry for interrupting in between. Wanted to check your views on the https://bugs.launchpad.net/nova/+bug/1636489 | |
| 13:53:48 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 13:54:35 | archies | we are stuck on this defect and wants to check the your views on it, to get the importance etc | |
| 13:54:45 | _gryf | bauzas, is there any patch for that? | |
| 13:54:58 | johnthetubaguy | sambetts: OK, so I think this bit brings me around to your approach: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L497 | |
| 13:55:14 | bauzas | _gryf: dunno | |
| 13:56:31 | _gryf | bauzas, k. besides that seems like without keystone we do have failing like this | |
| 13:59:07 | EmilienM | bauzas: yes, I think we spot the issue in tripleo | |
| 13:59:30 | EmilienM | bauzas: the haproxy config was not done correctly for this service, the binding was on the wrong network | |
| 13:59:42 | EmilienM | so haproxy thought placement api was down while it was up on another network | |
| 13:59:51 | bauzas | ok good to know | |
| 13:59:52 | openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Scheduler calling the Placement API https://review.openstack.org/417961 | |
| 13:59:54 | EmilienM | we are fixing it now but everything should work | |
| 13:59:58 | EmilienM | bauzas: thanks again for your help | |
| 14:00:09 | bauzas | EmilienM: np | |
| 14:00:21 | bauzas | EmilienM: FWIW, still working on the scheduler change above ^ | |
| 14:00:31 | bauzas | EmilienM: this one could be hitting TripleO | |
| 14:00:38 | bauzas | unless the placement service is there | |
| 14:00:53 | bauzas | so I'll run check experimental once I'm done with the UTs | |
| 14:01:04 | EmilienM | ok cool | |
| 14:01:33 | bauzas | _gryf: FWIW, in case you don't know, we can test placement changes against TripleO by running check experimental :) | |
| 14:01:48 | sambetts | johnthetubaguy: yeah :/ not sure what the best approach for now is, it sounds like we need the atomic thing, IMO we shouldn't ever manage to set the binding profile for a port not for this instance and if its for this instance then it should be unbound because ironic will bind it | |
| 14:01:49 | bauzas | jaypipes: in case you don't know too ^ | |
| 14:02:30 | bauzas | jaypipes: FWIW, do you think Fuel or other CDs would be impacted by https://review.openstack.org/417961 ? | |
| 14:03:01 | bauzas | perhaps Kolla | |
| 14:03:56 | johnthetubaguy | sambetts: I should find the spec that has the solution for that race, I think its mostly Nova's issue, and shouldn't worry the ironic phase, if we consider the device_id the thing that claims the ownership (beyond ironic checking the device_id is set correctly) | |
| 14:06:21 | jroll | johnthetubaguy: sambetts: so I don't think what we're changing here changes the situation at all, right, it's the same problem before and after the change | |
| 14:07:26 | johnthetubaguy | jroll: right, thats a better way of putting what I said | |
| 14:07:34 | johnthetubaguy | well, what I meant to say | |
| 14:08:24 | jroll | heh, ok cool :) | |
| 14:08:26 | jroll | thanks | |
| 14:08:59 | jroll | ah, I see your +Ws now, thank you! :) | |
| 14:11:40 | Sudha | Hi all , ANy thoughts on https://bugs.launchpad.net/nova/+bug/1636489 ? | |
| 14:11:40 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 14:12:33 | sambetts | johnthetubaguy, jroll: awesome, I hope we can fix it with the atomic stuff next cycle :) its really edge casey anyway | |
| 14:12:43 | jroll | yeah, +1 | |
| 14:14:07 | johnthetubaguy | sambetts: jroll: sorry for the firedrill, mostly getting a heap of stuff re-discovered and straightened out in my head | |
| 14:14:20 | jroll | johnthetubaguy: no worries, I learned a thing today \o/ | |
| 14:14:41 | sambetts | johnthetubaguy: \o/ yeah big TIL moment | |
| 14:22:48 | bauzas | do we have something wonkey with the py35 unit tests ? | |
| 14:23:26 | bauzas | that looks weirdo | |
| 14:24:02 | bauzas | dims: have we recently merged something for py35 ? | |
| 14:24:11 | archies | @Nova team, I have a case where I need your help to understand better. | |
| 14:24:13 | bauzas | it seems we have a critical issue | |
| 14:24:23 | dims | nope bauzas | |
| 14:24:33 | archies | @all, In one of my case, Nova's "detach_volume" fires the detach method into libvirt, which claims success, but the device is still attached as per the guest XML file. | |
| 14:24:33 | bauzas | lemme look at logstash | |
| 14:24:40 | sdague | bauzas: is there a link to a fail? | |
| 14:24:50 | bauzas | sdague: eg. http://logs.openstack.org/34/418134/7/check/gate-nova-python35-db/578aaff/testr_results.html.gz | |
| 14:25:00 | bauzas | zuul tells me all nova checks are red | |
| 14:25:44 | sdague | bauzas: https://pypi.python.org/pypi/mox3/0.19.0 ... | |
| 14:26:00 | bauzas | sdague: opening a bug | |
| 14:28:21 | sdague | bauzas: there are only a couple of commits - http://git.openstack.org/cgit/openstack/mox3/commit/?id=4a85ff0ec4ceb98ad87bec6002f6e67788b06d00 | |
| 14:28:51 | bauzas | sdague: https://bugs.launchpad.net/nova/+bug/1657757 | |
| 14:28:51 | openstack | Launchpad bug 1657757 in OpenStack Compute (nova) "py35 unit tests are hugely failing" [Critical,New] | |
| 14:29:04 | jroll | s/hugely/bigly/ | |
| 14:29:19 | sdague | bauzas: there are only 4 patches since the last release, the inspect one is the only of consequence | |
| 14:29:34 | mriedem | new release broke everything? | |
| 14:30:04 | bauzas | mriedem: sdague: trying to bisect | |
| 14:30:30 | sdague | bauzas: it has to be that patch I posted | |
| 14:32:19 | bauzas | sdague: probably, I just wanted to see the logstash query | |
| 14:32:55 | bauzas | mriedem: sdague: tried with the query "project:openstack/nova AND build_status:FAILURE AND build_name:gate-nova-python35-db" but not sure that one is good | |
| 14:36:09 | bauzas | sdague: mriedem: trying to run the job locally with an older mox release | |
| 14:36:24 | sdague | https://review.openstack.org/422651 is the revert | |
| 14:37:11 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient: Pass relevant parameters to Token based authentication https://review.openstack.org/419441 | |
| 14:41:58 | mriedem | sdague: https://review.openstack.org/#/c/422653/ | |
| 14:42:05 | mriedem | should stem the bleeding | |
| 14:42:16 | mriedem | i need to follow up with a g-r blacklist | |
| 14:43:21 | sdague | mriedem: I was doing that right now | |
| 14:44:29 | sdague | mriedem: you need the block, otherwise this is going to be broken within an hour with updates | |
| 14:44:41 | sdague | mriedem: https://review.openstack.org/422656 instead | |
| 14:46:24 | bhagyashris | jaypipes: cdent: Hi | |
| 14:49:58 | mriedem | need a +W on this https://review.openstack.org/#/c/421482/ so oslo.serialization can get their final release in for ocata today | |
| 14:51:01 | mriedem | harlowja: ^ it is done | |
| 14:52:26 | vladikr | mriedem, could you please remove your -2 from https://review.openstack.org/#/c/405951 ? - It's all rewritten now | |
| 14:52:51 | mriedem | done | |
| 14:53:00 | vladikr | mriedem, thanks | |
| 14:53:08 | mriedem | vladikr: so you were able to condense that whole series into 2 changes now? | |
| 14:53:53 | vladikr | mriedem, looks like it. removing all the classes in the middle, there is not much left | |
| 14:54:15 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 14:57:48 | bauzas | sdague: got a sec for a problem related to the placement call from the scheduler ? | |
| 14:58:14 | bauzas | sdague: it seems we default calling the placement service from the reportclient by not using the latest microversion | |
| 14:58:16 | bauzas | http://logs.openstack.org/61/417961/10/check/gate-tempest-dsvm-cells-ubuntu-xenial/8a974f7/logs/screen-n-sch.txt.gz?level=TRACE#_2017-01-19_14_17_38_980 | |
| 14:58:47 | bauzas | sdague: since cdent is on PTO today, do you know how could I add this header ? | |
| 15:00:12 | bauzas | sdague: I mean, in https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L172 | |
| 15:00:57 | mriedem | bauzas: jaypipes has a patch that does that, sec | |
| 15:01:10 | bauzas | mriedem: oh cool | |
| 15:01:25 | mriedem | https://review.openstack.org/#/c/407309/13/nova/scheduler/client/report.py | |
| 15:01:32 | mriedem | the problem there is he's requesting 'latest', | |
| 15:01:49 | mriedem | we don't want the report client to request 'latest', we want a specific version that we're opting into | |
| 15:02:02 | mriedem | because at some point latest in pike might break an ocata client | |