| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 10:23:51 | Kevin_Zheng | johnthetubaguy: so we are good with https://review.openstack.org/#/c/415142/ ? | |
| 11:13:40 | BobBall | snikitin: Awesome - thanks for your changes. Just to check, did you have some testing with a real XS host? | |
| 11:15:52 | BobBall | snikitin: I suspect the CI will run the code, but not verify that the answers are right (e.g. does the add_memory_usage work or always just return 0 used) | |
| 11:18:01 | snikitin | BobBall: I'm deploying env at this moment. But I already got bad news from xen CI job :( the message say "ValueError: cannot convert float NaN to integer". That is because of 'utulisation' field. | |
| 11:18:01 | snikitin | http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/06/413606/11/check/dsvm-tempest-neutron-network/54a6132/ | |
| 11:18:44 | snikitin | BobBall: yep I will check all fields when my env will be deployed | |
| 11:19:06 | BobBall | snikitin: ahhh - this is the updated tempest test - awesome. | |
| 11:19:11 | BobBall | snikitin: I hadn't been tracking that. | |
| 11:20:39 | snikitin | BobBall: also I have a dirty trick in my mind :) I can create a fake tempest test with self.assertEqual(1, diagnostics). When it will fail on Cen Ci job we will get whole diagnostics responce | |
| 11:20:52 | BobBall | snikitin: I like that trick :) | |
| 11:37:12 | dims | jaypipes : bauzas : this one is good to go. tiny fixes. helps fix up several tempest scenarios - https://review.openstack.org/#/c/421911/ | |
| 11:45:05 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V PCI Passthrough https://review.openstack.org/420614 | |
| 12:03:51 | blinky_ghost | hi all can anybody tell me what this error means: "error: Index 'allocations_resource_provider_class_used_idx' already exists!" when I migrate nova db schema from liberty to mitaka. Thanks | |
| 12:07:22 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient: Pass relevant parameters to Token based authentication https://review.openstack.org/419441 | |
| 12:30:30 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.live_migration_rollback notification https://review.openstack.org/402124 | |
| 12:40:11 | bhagyashris | Hi all, I am facing issue after hitting any nova commands eg. nova list and got error at n-api as DBNonExistentDatabase: (pymysql.err.InternalError) (1049, u"Unknown database 'nova_cell0'"). Can anyone have any idea? | |
| 12:40:36 | openstackgerrit | Béla Vancsics proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 12:44:08 | sambetts | johnthetubaguy, jroll: ping can we go through your comment here please, we're struggling to understand why that would cause the races https://review.openstack.org/#/c/364413/24/nova/virt/ironic/driver.py@1242 | |
| 12:44:40 | jroll | (and/or how said race would cause a port to be stolen) | |
| 12:49:40 | _gryf | EmilienM, around? | |
| 12:51:21 | openstackgerrit | Wang Qing wu proposed openstack/nova: WIP(6): Introduce base PowerVM driver (ssp drv) https://review.openstack.org/422594 | |
| 12:58:43 | openstackgerrit | Abhishek Kekane proposed openstack/python-novaclient: x-openstack-request-id logged twice in logs https://review.openstack.org/422602 | |
| 12:58:46 | EmilienM | _gryf: what's up? | |
| 13:00:05 | openstackgerrit | sahid proposed openstack/nova: numa: add numa constraints for emulator threads policy https://review.openstack.org/385023 | |
| 13:00:08 | openstackgerrit | sahid proposed openstack/nova: libvirt: return a CPU overhead if isolate emulator threads requested https://review.openstack.org/385364 | |
| 13:00:09 | openstackgerrit | sahid proposed openstack/nova: objects: add new field cpus_reserved in NUMACell https://review.openstack.org/400874 | |
| 13:00:09 | openstackgerrit | sahid proposed openstack/nova: numa: fit instance topology with cpus reserved onto cores https://review.openstack.org/400875 | |
| 13:00:12 | openstackgerrit | sahid proposed openstack/nova: virt: update overhead to take into account vCPUs https://review.openstack.org/385024 | |
| 13:00:17 | openstackgerrit | sahid proposed openstack/nova: libvirt: make emulator threads to run on the reserved pCPU https://review.openstack.org/400878 | |
| 13:00:18 | openstackgerrit | sahid proposed openstack/nova: numa: take into account cpus reserved https://review.openstack.org/400876 | |
| 13:00:18 | openstackgerrit | sahid proposed openstack/nova: numa: update numa usage incording the cpu reserved https://review.openstack.org/400877 | |
| 13:10:30 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V PCI Passthrough https://review.openstack.org/420614 | |
| 13:25:08 | johnthetubaguy | sambetts: jroll: sorry, was having lunch | |
| 13:26:01 | johnthetubaguy | sambetts: jroll: I think its the using two ports for the same instance, and we would reset the bind host, but its very theoretical | |
| 13:26:31 | johnthetubaguy | sambetts: jroll: do we need to set the bind_host to None, or could we just skip that? | |
| 13:29:11 | johnthetubaguy | sambetts: jroll: I am basically good with this as it is, probably just leaning towards a quick follow up patch to for that one line change I suggested, if that would be helpful, also just making sure I am not missing something | |
| 13:37:57 | archies | JOIN #openstack-meeting | |
| 13:38:12 | archies | JOIN /#openstack-meeting | |
| 13:40:21 | sambetts | johnthetubaguy: I'm still not sure I get what passing host_id=None through would do, we want to force it to None if it is already set, because otherwise it would prevent Ironic binding its provisoning network port | |
| 13:40:46 | johnthetubaguy | sambetts: but if its already set, someone else owns the port | |
| 13:41:17 | johnthetubaguy | sambetts: we should fail earlier on if thats already set | |
| 13:41:33 | johnthetubaguy | (hence the race is a really tiny window, if at all) | |
| 13:41:50 | sambetts | johnthetubaguy: +1 surly you can't pass a port that has host_id set into --nic port-id right?? | |
| 13:42:24 | sambetts | johnthetubaguy: that would be dangerous for normal cases | |
| 13:42:26 | johnthetubaguy | right, but you can pass the same port into two instances, and they race each other to apply the binding, the last one "wins" | |
| 13:42:57 | johnthetubaguy | I would just rather we didn't send bind_host_id=None, we just don't update bind_host_id | |
| 13:43:10 | johnthetubaguy | its a niggle though, not a blocker | |
| 13:43:28 | sambetts | johnthetubaguy: but you are OK writing host_id=<the last one> over the one already set? | |
| 13:43:53 | sambetts | johnthetubaguy: surely we should fail over if at that point the host_id is already set on that port | |
| 13:43:57 | johnthetubaguy | sambetts: no, there is a fix something in the neutron API soon, so we don't do that any more | |
| 13:44:17 | johnthetubaguy | basically, the neutron API isn't atomic, and they are adding a generation id we can use to make sure the update is atomic | |
| 13:44:33 | johnthetubaguy | thats a fix for next cycle, with a bit of luck | |
| 13:45:43 | johnthetubaguy | i.e we can check generation7 doesn't have the bind_host_id set, and fail if there have been updates to it since that check | |
| 13:45:53 | sambetts | I think for our case then we'd need a if get_port(<port_id>).binding_profile['host_id'] is not None: raise PortAlreadyBound() because we need the port to be unbound | |
| 13:47:09 | sambetts | we'd still run into issues were if we were racing another deploy we won't catch it if a port binds after that point | |
| 13:47:22 | sambetts | after we've been through it | |
| 13:47:23 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 13:48:31 | johnthetubaguy | sambetts: I am trying to find that check, I thought we had that already | |
| 13:48:59 | sambetts | johnthetubaguy: isn't there a part of the port that ties it to an instance not just a host_id? | |
| 13:49:13 | johnthetubaguy | https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L555 | |
| 13:49:15 | sambetts | johnthetubaguy: surely we should be looking at that to know if its already used not the host id | |
| 13:49:17 | johnthetubaguy | device_id does that | |
| 13:49:37 | johnthetubaguy | so... we should back up | |
| 13:49:52 | johnthetubaguy | not questioning the overall approach here, thats just fine | |
| 13:50:10 | sambetts | yeah, I'm just trying to work out if the race is possible or not :/ | |
| 13:50:33 | sambetts | I would have expected it to catch if the port was in use really early in the process | |
| 13:50:58 | johnthetubaguy | we do, we check twice, because of the racyness | |
| 13:51:04 | johnthetubaguy | still can happen | |
| 13:51:22 | _gryf | EmilienM, did you find out the culprit for the failing placement? | |
| 13:51:33 | johnthetubaguy | once in API, once in compute before binding, but its racey still (just) | |
| 13:52:01 | johnthetubaguy | we have a way to fix that for next cycle | |
| 13:53:08 | bauzas | _gryf: it seems so | |
| 13:53:42 | bauzas | _gryf: EmilienM pinged me late yesterday night for telling me it was due to the HAProxy service used by TripleO | |
| 13:53:48 | archies | @mriedem, @nova core, sorry for interrupting in between. Wanted to check your views on the https://bugs.launchpad.net/nova/+bug/1636489 | |
| 13:53:48 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 13:54:35 | archies | we are stuck on this defect and wants to check the your views on it, to get the importance etc | |
| 13:54:45 | _gryf | bauzas, is there any patch for that? | |
| 13:54:58 | johnthetubaguy | sambetts: OK, so I think this bit brings me around to your approach: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L497 | |
| 13:55:14 | bauzas | _gryf: dunno | |
| 13:56:31 | _gryf | bauzas, k. besides that seems like without keystone we do have failing like this | |
| 13:59:07 | EmilienM | bauzas: yes, I think we spot the issue in tripleo | |
| 13:59:30 | EmilienM | bauzas: the haproxy config was not done correctly for this service, the binding was on the wrong network | |
| 13:59:42 | EmilienM | so haproxy thought placement api was down while it was up on another network | |
| 13:59:51 | bauzas | ok good to know | |
| 13:59:52 | openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Scheduler calling the Placement API https://review.openstack.org/417961 | |
| 13:59:54 | EmilienM | we are fixing it now but everything should work | |
| 13:59:58 | EmilienM | bauzas: thanks again for your help | |
| 14:00:09 | bauzas | EmilienM: np | |
| 14:00:21 | bauzas | EmilienM: FWIW, still working on the scheduler change above ^ | |
| 14:00:31 | bauzas | EmilienM: this one could be hitting TripleO | |
| 14:00:38 | bauzas | unless the placement service is there | |
| 14:00:53 | bauzas | so I'll run check experimental once I'm done with the UTs | |
| 14:01:04 | EmilienM | ok cool | |
| 14:01:33 | bauzas | _gryf: FWIW, in case you don't know, we can test placement changes against TripleO by running check experimental :) | |
| 14:01:48 | sambetts | johnthetubaguy: yeah :/ not sure what the best approach for now is, it sounds like we need the atomic thing, IMO we shouldn't ever manage to set the binding profile for a port not for this instance and if its for this instance then it should be unbound because ironic will bind it | |
| 14:01:49 | bauzas | jaypipes: in case you don't know too ^ | |
| 14:02:30 | bauzas | jaypipes: FWIW, do you think Fuel or other CDs would be impacted by https://review.openstack.org/417961 ? | |
| 14:03:01 | bauzas | perhaps Kolla | |
| 14:03:56 | johnthetubaguy | sambetts: I should find the spec that has the solution for that race, I think its mostly Nova's issue, and shouldn't worry the ironic phase, if we consider the device_id the thing that claims the ownership (beyond ironic checking the device_id is set correctly) | |
| 14:06:21 | jroll | johnthetubaguy: sambetts: so I don't think what we're changing here changes the situation at all, right, it's the same problem before and after the change | |