| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-17 | |||
| 16:44:09 | mriedem | because when you move the instance, the source host RT might overwrite the allocations for the instance which are meant for the dest host | |
| 16:44:27 | mriedem | because of periodic task race window wackiness | |
| 16:45:25 | sean-k-mooney | mriedem: yes in the future we defiently want to move the allocation out of the compute node the sepc was assuming the workflow we have today where the compute node does the allocation but i dont think it would break if the allocation was done earlier | |
| 16:46:19 | mriedem | sean-k-mooney: if today is >= pike, we don't create allocatoins in the computes anymore | |
| 16:46:31 | mriedem | allocations in the computes is soooooo 2 months ago :) | |
| 16:46:38 | sean-k-mooney | mriedem: oh did that land in pike cool | |
| 16:46:47 | mriedem | yeah i linked to the change in the spec | |
| 16:47:05 | mriedem | as a result of that change we've been doing a lot of cleanup of places where we dont cleanup allocations | |
| 16:47:10 | sean-k-mooney | mriedem: so is that now done in the conductor/scheduler | |
| 16:47:10 | mriedem | still working on getting those fixes into stable/pike | |
| 16:47:18 | mriedem | the scheduler claims yes | |
| 16:47:27 | mriedem | for VCPUS/MEMORY_MB/DISK_GB | |
| 16:47:43 | mriedem | if you have NUMATopologyLimits, those get claimed in the compute still, | |
| 16:47:53 | mriedem | since we don't model NUMA via nested RPs in placement yet | |
| 16:48:18 | sean-k-mooney | pci device im guessing are still on compute node too for same reason | |
| 16:48:40 | mriedem | yeah | |
| 16:49:08 | sean-k-mooney | to do the claim for the bandwidth in the scheduler we would also have to bind the port in the scheduler | |
| 16:49:09 | mriedem | pci request claims are handled differently from numa limits somehow, i can't remember the details | |
| 16:49:37 | mriedem | sean-k-mooney: why? | |
| 16:50:09 | mriedem | scheduler picks a host, makes the bandwidth allocation against that host, and the port is later bound to the host in the compute | |
| 16:50:18 | sean-k-mooney | to know which network backend to claim the bandwitdh form | |
| 16:50:19 | mriedem | if we fail on the compute, we deallocate | |
| 16:50:35 | sean-k-mooney | e.g. sriov and ovs on same host | |
| 16:50:50 | mriedem | so we don't know the network backend until we bind the host and get a binding:profile from neutron? | |
| 16:51:07 | sean-k-mooney | correct. | |
| 16:51:12 | mriedem | well shit balls | |
| 16:51:17 | mriedem | that's what i call a catch-2 | |
| 16:51:19 | mriedem | *catch-22 | |
| 16:51:43 | sean-k-mooney | that is why we assumed the current workflow in the spec | |
| 16:52:01 | mriedem | yeah... | |
| 16:53:15 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Reproduce bug 1721652 in the functional test env https://review.openstack.org/511759 | |
| 16:53:17 | openstack | bug 1721652 in OpenStack Compute (nova) "Evacuate cleanup fails at _delete_allocation_for_moved_instance" [High,In progress] https://launchpad.net/bugs/1721652 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 16:53:17 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: fix cleaning up evacuated instances https://review.openstack.org/512716 | |
| 16:53:41 | sean-k-mooney | before ovs offload we could have used the vnic type to differenciate sriov and ovs/linuxbirdge/vpp... but now we cant since vnic type direct is valid for both | |
| 16:53:59 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: fix cleaning up evacuated instances https://review.openstack.org/512716 | |
| 16:54:29 | gibi | mriedem, bauzas: sorry I was away from the computer. I can do the suggested refactor of https://review.openstack.org/#/c/510938/ in a follow up | |
| 16:58:13 | sean-k-mooney | mriedem: anyway thanks for reviewing it. one spec that is hopefully less contoversial since it was approved last cycle is https://review.openstack.org/#/c/504895/ i think all the code for that has been up for a few months also | |
| 17:00:10 | sean-k-mooney | mriedem: johnthetubaguy oh by the way was this https://review.openstack.org/#/c/499777/5 the cinder attach work you were suggesting emultating for the multiple port binding in relation to how upgrades should work | |
| 17:00:27 | mriedem | sean-k-mooney: no | |
| 17:00:35 | mriedem | sec | |
| 17:00:47 | mriedem | sean-k-mooney: https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/cinder-new-attach-apis.html | |
| 17:01:39 | sean-k-mooney | mriedem: ah ok i have basically added johnthetubaguy suggestions to the latest version but ill go read that now | |
| 17:01:42 | sean-k-mooney | mriedem: thanks | |
| 17:02:14 | mriedem | i need to read the updates on the portbinding spec too | |
| 17:02:39 | edleafe | dansmith: Here's the scenario: cell conductor gets a list of alternates after the initial build attempt fails. Assume that all of these alternates are stale, and no longer have enough resources. Should the cell conductor just log it and do nothing, or should it raise an exception? | |
| 17:02:44 | edleafe | And if an exception, what kind? | |
| 17:03:09 | mriedem | NoValidHost? | |
| 17:03:27 | mriedem | cell conductor raising NoValidHost is something that can already happen today when you run out of alternatives | |
| 17:03:30 | mriedem | it will log a fault | |
| 17:03:33 | mriedem | instance goes to ERROR state | |
| 17:03:34 | mriedem | easy peasy | |
| 17:03:49 | dansmith | edleafe: it should do whatever would happen if we had no retries left in the current path | |
| 17:03:50 | dansmith | yeah | |
| 17:03:51 | dansmith | that | |
| 17:03:58 | edleafe | mriedem: that was my assumption, but I wanted to be sure there wasn't some other action | |
| 17:04:07 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Fix wording of debug message for future releases https://review.openstack.org/508261 | |
| 17:04:31 | edleafe | ah, so that should be the same as what happens in compute when there are no retries left. | |
| 17:04:37 | dansmith | right | |
| 17:08:59 | openstackgerrit | Elod Illes proposed openstack/nova master: WIP: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 17:12:35 | dansmith | edleafe: so, looking at your third patch, | |
| 17:12:43 | dansmith | edleafe: we have limits in the reqspec, | |
| 17:13:03 | dansmith | oh, | |
| 17:13:22 | dansmith | the selection limits are really just for the return value from the rpc call to continue to include them is that right? | |
| 17:13:35 | dansmith | I was thinking we had to have them for communicating to the compute, but that is in reqspec | |
| 17:13:37 | dansmith | is that right? | |
| 17:13:57 | edleafe | dansmith: the caching scheduler needs them | |
| 17:14:16 | dansmith | the caching scheduler _sets_ them | |
| 17:14:22 | dansmith | the compute _needs_ them | |
| 17:14:44 | edleafe | ok, I thought it was needed in the filters | |
| 17:15:02 | dansmith | my point is, we really just need them in Selection so that the return from select_destinations(), which is now a bunch of Selections, can include the limits back to conductor, | |
| 17:15:06 | edleafe | but yeah, the compute will need them too | |
| 17:15:11 | dansmith | which will populate the reqspec | |
| 17:15:22 | dansmith | the filters don't need them, the filters are what put things into limits | |
| 17:15:42 | edleafe | ok, I guess I misunderstood mriedem's concerns | |
| 17:15:49 | dansmith | compute can get them from the reqspec | |
| 17:18:44 | mriedem | the limits are stored in the reqspec? | |
| 17:19:01 | mriedem | tbc, we pass a reqspec down to build_and_run_instance in compute and it's *totally* ignored | |
| 17:19:05 | mriedem | we pass the limits dict separately | |
| 17:19:57 | mriedem | ok i see the SchedulerLimits field in the request spec | |
| 17:20:02 | mriedem | i just don't think compute is using that today | |
| 17:20:14 | dansmith | mriedem: right, so this will be passing limits N more times than we currently pass it | |
| 17:20:39 | dansmith | I'm thinking maybe we just have select_destinations return (reqspec, alternates) with reqspec updated with limits, | |
| 17:20:53 | dansmith | and then we can avoid putting them into the selection | |
| 17:21:10 | dansmith | aside from numa, they'll be the same for num_instances, right? | |
| 17:21:14 | mriedem | need to see what actually populates RequestSpec.limits today | |
| 17:21:26 | dansmith | I think it's populate_filter_properties | |
| 17:23:48 | dansmith | https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L367 | |
| 17:24:01 | mriedem | yeah looking at what calls that | |
| 17:24:12 | dansmith | conductor | |
| 17:24:39 | dansmith | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1110-L1110 | |
| 17:25:03 | mriedem | god what a weird mess | |
| 17:25:11 | dansmith | yeah | |
| 17:25:32 | dansmith | it's also still in the half-uncleaned-up reqspec object | |
| 17:25:42 | dansmith | the only problem is, | |
| 17:25:49 | dansmith | I think numa limits will be per-host, | |
| 17:25:58 | dansmith | although,hmm | |
| 17:26:08 | dansmith | maybe the regular limits won't because of the allocation ratios or something? | |
| 17:26:49 | dansmith | yeah, toast | |
| 17:26:50 | mriedem | we also aren't doing this trick for evacuate | |
| 17:26:51 | dansmith | gdi | |