| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 12:40:11 | bhagyashris | Hi all, I am facing issue after hitting any nova commands eg. nova list and got error at n-api as DBNonExistentDatabase: (pymysql.err.InternalError) (1049, u"Unknown database 'nova_cell0'"). Can anyone have any idea? | |
| 12:40:36 | openstackgerrit | Béla Vancsics proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 12:44:08 | sambetts | johnthetubaguy, jroll: ping can we go through your comment here please, we're struggling to understand why that would cause the races https://review.openstack.org/#/c/364413/24/nova/virt/ironic/driver.py@1242 | |
| 12:44:40 | jroll | (and/or how said race would cause a port to be stolen) | |
| 12:49:40 | _gryf | EmilienM, around? | |
| 12:51:21 | openstackgerrit | Wang Qing wu proposed openstack/nova: WIP(6): Introduce base PowerVM driver (ssp drv) https://review.openstack.org/422594 | |
| 12:58:43 | openstackgerrit | Abhishek Kekane proposed openstack/python-novaclient: x-openstack-request-id logged twice in logs https://review.openstack.org/422602 | |
| 12:58:46 | EmilienM | _gryf: what's up? | |
| 13:00:05 | openstackgerrit | sahid proposed openstack/nova: numa: add numa constraints for emulator threads policy https://review.openstack.org/385023 | |
| 13:00:08 | openstackgerrit | sahid proposed openstack/nova: libvirt: return a CPU overhead if isolate emulator threads requested https://review.openstack.org/385364 | |
| 13:00:09 | openstackgerrit | sahid proposed openstack/nova: objects: add new field cpus_reserved in NUMACell https://review.openstack.org/400874 | |
| 13:00:09 | openstackgerrit | sahid proposed openstack/nova: numa: fit instance topology with cpus reserved onto cores https://review.openstack.org/400875 | |
| 13:00:12 | openstackgerrit | sahid proposed openstack/nova: virt: update overhead to take into account vCPUs https://review.openstack.org/385024 | |
| 13:00:17 | openstackgerrit | sahid proposed openstack/nova: libvirt: make emulator threads to run on the reserved pCPU https://review.openstack.org/400878 | |
| 13:00:18 | openstackgerrit | sahid proposed openstack/nova: numa: take into account cpus reserved https://review.openstack.org/400876 | |
| 13:00:18 | openstackgerrit | sahid proposed openstack/nova: numa: update numa usage incording the cpu reserved https://review.openstack.org/400877 | |
| 13:10:30 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V PCI Passthrough https://review.openstack.org/420614 | |
| 13:25:08 | johnthetubaguy | sambetts: jroll: sorry, was having lunch | |
| 13:26:01 | johnthetubaguy | sambetts: jroll: I think its the using two ports for the same instance, and we would reset the bind host, but its very theoretical | |
| 13:26:31 | johnthetubaguy | sambetts: jroll: do we need to set the bind_host to None, or could we just skip that? | |
| 13:29:11 | johnthetubaguy | sambetts: jroll: I am basically good with this as it is, probably just leaning towards a quick follow up patch to for that one line change I suggested, if that would be helpful, also just making sure I am not missing something | |
| 13:37:57 | archies | JOIN #openstack-meeting | |
| 13:38:12 | archies | JOIN /#openstack-meeting | |
| 13:40:21 | sambetts | johnthetubaguy: I'm still not sure I get what passing host_id=None through would do, we want to force it to None if it is already set, because otherwise it would prevent Ironic binding its provisoning network port | |
| 13:40:46 | johnthetubaguy | sambetts: but if its already set, someone else owns the port | |
| 13:41:17 | johnthetubaguy | sambetts: we should fail earlier on if thats already set | |
| 13:41:33 | johnthetubaguy | (hence the race is a really tiny window, if at all) | |
| 13:41:50 | sambetts | johnthetubaguy: +1 surly you can't pass a port that has host_id set into --nic port-id right?? | |
| 13:42:24 | sambetts | johnthetubaguy: that would be dangerous for normal cases | |
| 13:42:26 | johnthetubaguy | right, but you can pass the same port into two instances, and they race each other to apply the binding, the last one "wins" | |
| 13:42:57 | johnthetubaguy | I would just rather we didn't send bind_host_id=None, we just don't update bind_host_id | |
| 13:43:10 | johnthetubaguy | its a niggle though, not a blocker | |
| 13:43:28 | sambetts | johnthetubaguy: but you are OK writing host_id=<the last one> over the one already set? | |
| 13:43:53 | sambetts | johnthetubaguy: surely we should fail over if at that point the host_id is already set on that port | |
| 13:43:57 | johnthetubaguy | sambetts: no, there is a fix something in the neutron API soon, so we don't do that any more | |
| 13:44:17 | johnthetubaguy | basically, the neutron API isn't atomic, and they are adding a generation id we can use to make sure the update is atomic | |
| 13:44:33 | johnthetubaguy | thats a fix for next cycle, with a bit of luck | |
| 13:45:43 | johnthetubaguy | i.e we can check generation7 doesn't have the bind_host_id set, and fail if there have been updates to it since that check | |
| 13:45:53 | sambetts | I think for our case then we'd need a if get_port(<port_id>).binding_profile['host_id'] is not None: raise PortAlreadyBound() because we need the port to be unbound | |
| 13:47:09 | sambetts | we'd still run into issues were if we were racing another deploy we won't catch it if a port binds after that point | |
| 13:47:22 | sambetts | after we've been through it | |
| 13:47:23 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 13:48:31 | johnthetubaguy | sambetts: I am trying to find that check, I thought we had that already | |
| 13:48:59 | sambetts | johnthetubaguy: isn't there a part of the port that ties it to an instance not just a host_id? | |
| 13:49:13 | johnthetubaguy | https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L555 | |
| 13:49:15 | sambetts | johnthetubaguy: surely we should be looking at that to know if its already used not the host id | |
| 13:49:17 | johnthetubaguy | device_id does that | |
| 13:49:37 | johnthetubaguy | so... we should back up | |
| 13:49:52 | johnthetubaguy | not questioning the overall approach here, thats just fine | |
| 13:50:10 | sambetts | yeah, I'm just trying to work out if the race is possible or not :/ | |
| 13:50:33 | sambetts | I would have expected it to catch if the port was in use really early in the process | |
| 13:50:58 | johnthetubaguy | we do, we check twice, because of the racyness | |
| 13:51:04 | johnthetubaguy | still can happen | |
| 13:51:22 | _gryf | EmilienM, did you find out the culprit for the failing placement? | |
| 13:51:33 | johnthetubaguy | once in API, once in compute before binding, but its racey still (just) | |
| 13:52:01 | johnthetubaguy | we have a way to fix that for next cycle | |
| 13:53:08 | bauzas | _gryf: it seems so | |
| 13:53:42 | bauzas | _gryf: EmilienM pinged me late yesterday night for telling me it was due to the HAProxy service used by TripleO | |
| 13:53:48 | archies | @mriedem, @nova core, sorry for interrupting in between. Wanted to check your views on the https://bugs.launchpad.net/nova/+bug/1636489 | |
| 13:53:48 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 13:54:35 | archies | we are stuck on this defect and wants to check the your views on it, to get the importance etc | |
| 13:54:45 | _gryf | bauzas, is there any patch for that? | |
| 13:54:58 | johnthetubaguy | sambetts: OK, so I think this bit brings me around to your approach: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L497 | |
| 13:55:14 | bauzas | _gryf: dunno | |
| 13:56:31 | _gryf | bauzas, k. besides that seems like without keystone we do have failing like this | |
| 13:59:07 | EmilienM | bauzas: yes, I think we spot the issue in tripleo | |
| 13:59:30 | EmilienM | bauzas: the haproxy config was not done correctly for this service, the binding was on the wrong network | |
| 13:59:42 | EmilienM | so haproxy thought placement api was down while it was up on another network | |
| 13:59:51 | bauzas | ok good to know | |
| 13:59:52 | openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Scheduler calling the Placement API https://review.openstack.org/417961 | |
| 13:59:54 | EmilienM | we are fixing it now but everything should work | |
| 13:59:58 | EmilienM | bauzas: thanks again for your help | |
| 14:00:09 | bauzas | EmilienM: np | |
| 14:00:21 | bauzas | EmilienM: FWIW, still working on the scheduler change above ^ | |
| 14:00:31 | bauzas | EmilienM: this one could be hitting TripleO | |
| 14:00:38 | bauzas | unless the placement service is there | |
| 14:00:53 | bauzas | so I'll run check experimental once I'm done with the UTs | |
| 14:01:04 | EmilienM | ok cool | |
| 14:01:33 | bauzas | _gryf: FWIW, in case you don't know, we can test placement changes against TripleO by running check experimental :) | |
| 14:01:48 | sambetts | johnthetubaguy: yeah :/ not sure what the best approach for now is, it sounds like we need the atomic thing, IMO we shouldn't ever manage to set the binding profile for a port not for this instance and if its for this instance then it should be unbound because ironic will bind it | |
| 14:01:49 | bauzas | jaypipes: in case you don't know too ^ | |
| 14:02:30 | bauzas | jaypipes: FWIW, do you think Fuel or other CDs would be impacted by https://review.openstack.org/417961 ? | |
| 14:03:01 | bauzas | perhaps Kolla | |
| 14:03:56 | johnthetubaguy | sambetts: I should find the spec that has the solution for that race, I think its mostly Nova's issue, and shouldn't worry the ironic phase, if we consider the device_id the thing that claims the ownership (beyond ironic checking the device_id is set correctly) | |
| 14:06:21 | jroll | johnthetubaguy: sambetts: so I don't think what we're changing here changes the situation at all, right, it's the same problem before and after the change | |
| 14:07:26 | johnthetubaguy | jroll: right, thats a better way of putting what I said | |
| 14:07:34 | johnthetubaguy | well, what I meant to say | |
| 14:08:24 | jroll | heh, ok cool :) | |
| 14:08:26 | jroll | thanks | |
| 14:08:59 | jroll | ah, I see your +Ws now, thank you! :) | |
| 14:11:40 | Sudha | Hi all , ANy thoughts on https://bugs.launchpad.net/nova/+bug/1636489 ? | |
| 14:11:40 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 14:12:33 | sambetts | johnthetubaguy, jroll: awesome, I hope we can fix it with the atomic stuff next cycle :) its really edge casey anyway | |
| 14:12:43 | jroll | yeah, +1 | |
| 14:14:07 | johnthetubaguy | sambetts: jroll: sorry for the firedrill, mostly getting a heap of stuff re-discovered and straightened out in my head | |
| 14:14:20 | jroll | johnthetubaguy: no worries, I learned a thing today \o/ | |
| 14:14:41 | sambetts | johnthetubaguy: \o/ yeah big TIL moment | |
| 14:22:48 | bauzas | do we have something wonkey with the py35 unit tests ? | |
| 14:23:26 | bauzas | that looks weirdo | |
| 14:24:02 | bauzas | dims: have we recently merged something for py35 ? | |