| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |
| 19:21:41 | dansmith | sdeath: you'll have to delete it from sql I guess | |
| 19:21:48 | sdeath | DS: is that safe, then? | |
| 19:21:54 | sdeath | if I remove from the nova.cells table? | |
| 19:22:11 | dansmith | sdeath: from nova_api.cell_mappings | |
| 19:22:22 | jaypipes | cfriesen: gross. | |
| 19:22:22 | sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | |
| 19:22:34 | mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | |
| 19:22:39 | cfriesen | sean-k-mooney: except for the final allocations part at the end | |
| 19:22:44 | rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | |
| 19:22:48 | dansmith | mriedem: that will delete all the hosts | |
| 19:22:52 | dansmith | mriedem: which he doesn't want | |
| 19:22:53 | jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | |
| 19:22:59 | sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | |
| 19:23:00 | mriedem | the mappings, right | |
| 19:23:01 | mriedem | nvm | |
| 19:23:30 | sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | |
| 19:24:07 | sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | |
| 19:24:22 | dansmith | sdeath: figure out which one is right right one and delete the other | |
| 19:24:23 | cfriesen | sean-k-mooney: isnt' that already in the host infomration? | |
| 19:24:28 | mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | |
| 19:24:30 | jaypipes | sean-k-mooney: no. I'm not willing to change the design of the placement service (and the allocation request atomicity/guarantees) just to meet these wack-o requirements. | |
| 19:24:34 | dansmith | sdeath: hopefully all your instance and host mappings refer to one of the two cell mappings | |
| 19:24:36 | melwitt | rybridges: yeah, so that implies the nova API call to add the host must be failing https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L224 else segment_host_mappings is empty there | |
| 19:24:42 | sean-k-mooney | cfriesen: what how many ht the host has | |
| 19:24:48 | sdeath | m: let's see if —force exists... | |
| 19:25:01 | sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | |
| 19:25:03 | dansmith | sdeath: you do not want --force | |
| 19:25:18 | mlavalle | rybridges: in other words. Have you created a routed network yet? | |
| 19:25:18 | dansmith | sdeath: --force on that is actually --recursive | |
| 19:25:25 | cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | |
| 19:25:29 | sdeath | ds: so maybe we won't do that then | |
| 19:25:48 | sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | |
| 19:25:50 | dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | |
| 19:25:52 | melwitt | heh | |
| 19:26:05 | cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | |
| 19:26:08 | sdeath | adium, evidently it does | |
| 19:26:35 | sdeath | so, backtrcing: nova_api.cell_mappings | |
| 19:26:44 | cfriesen | jaypipes: I think I see what you mean, we can't retroactively change the size of an earlier allocation request | |
| 19:27:00 | sean-k-mooney | cfriesen: ya so in this case we would have to make another allocation_candiates request and then find the host n that again and consume that | |
| 19:27:49 | cfriesen | sean-k-mooney: right. Or else we throw out the cpu_thread_policy option in the flavor, or else we model siblings in placement. | |
| 19:28:02 | jaypipes | cfriesen: right. the whole point of the allocation candidates list was "these are the nodes that met your request for resources". if we then just modify that resource amount, we invalidate the result of allocation candidates. :( | |
| 19:28:27 | sdeath | dansmith: OK, rows delete (from host_mappings and from cell_mappings; host_mappings due to foreign_key constraint) | |
| 19:28:30 | sdeath | restarting nova… let's see if this work. | |
| 19:28:33 | sdeath | *wrks | |
| 19:28:51 | sdeath | hey, how about that. only one row per hypervisor! | |
| 19:28:55 | sdeath | OK, so... | |
| 19:29:00 | dansmith | sdeath: so you'll need to do discover_hosts again, but it probably won't find them all because they've been marked as already mapped | |
| 19:29:17 | dansmith | sdeath: you should have changed the id on the others to be the cell you kept | |
| 19:30:06 | sdeath | will it barf if I manually reinsert? | |
| 19:30:13 | dansmith | sdeath: no | |
| 19:30:20 | sean-k-mooney | cfriesen: i dont really see a good way to keep cpu_thread_policy=isolate with out having an allocation on the compute host | |
| 19:30:29 | tblakes | mriedmen: I responded to his comment in bug 1756360. He was seeing that issue because he wasn't passing in a kwarg that nova passes in. | |
| 19:30:31 | jaypipes | cfriesen: or... we just go with my spec and just track dedicated CPUs as their own thing and deal with the workloads that can't tolerate having their dedicated CPUs be hyperthreads as the snowflakey thing that it is when we get to the destination host the scheduler selected. | |
| 19:30:31 | sdeath | dansmith: actually, map_cell_and_hosts seems to have picked them back up.. | |
| 19:30:32 | openstack | bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] https://launchpad.net/bugs/1756360 - Assigned to Tyler Blakeslee (tblakes) | |
| 19:30:41 | sdeath | I see fresh IDs and happiness | |
| 19:30:51 | dansmith | sdeath: that's not what you wanted I think | |
| 19:31:25 | cfriesen | jaypipes: so we'd get to the compute node and find out "oops, it doesn't have as many PCPU resources as we thought" and fail the claim? | |
| 19:31:28 | dansmith | sdeath: didn't that create a new cell mapping again? | |
| 19:31:34 | sdeath | dansmith: nope | |
| 19:31:49 | sean-k-mooney | cfriesen: well claim would happen in scuduler before getting to compute host | |
| 19:31:58 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614 | |
| 19:32:10 | sean-k-mooney | cfriesen: we could at the very end try to extend the claim and only fail if we could not extend | |
| 19:32:19 | sdeath | cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors. | |
| 19:32:24 | dansmith | sdeath: ah, I see where it went, okay | |
| 19:32:52 | dansmith | sdeath: so you're good now? | |
| 19:32:59 | sdeath | nova-manage cell_v2 list_cells shows right number of cells... | |
| 19:33:02 | sdeath | I think it's good. | |
| 19:33:03 | sdeath | so... | |
| 19:33:09 | cfriesen | sean-k-mooney: I'm talking about ResourceTracker.instance_claim() | |
| 19:33:12 | dansmith | sdeath: cool, now read the channel topic :) | |
| 19:33:27 | jaypipes | cfriesen: no... it *would* have as many pCPUs as we thought... it's just *that snowflake of a workload considers hyperthreads not to be good enough pCPUs for it*. | |
| 19:34:05 | cfriesen | jaypipes: heh...that's one way of thinking about it. :) | |
| 19:34:33 | sean-k-mooney | jaypipes: well one exampel of that is realtime cpus and hyperthread dont mix | |
| 19:34:33 | jaypipes | cfriesen: it's the only way to think about it. but I digress.. | |
| 19:34:34 | sdeath | Apologies; I tried #openstack two days running, no soap… wasn't sure where to head from there. There a better "general guidance" channel? | |
| 19:34:47 | jaypipes | sean-k-mooney: I don't care? | |
| 19:34:55 | tblakes | mriedem: I responded to the comment in https://bugs.launchpad.net/nova/+bug/1756360. They were seeing that issue because they were not passing in a kwarg that nova passes in. | |
| 19:34:57 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:34:58 | cfriesen | jaypipes: I think that'd work, it wouldn't be any more racy than it is now. | |
| 19:35:07 | dansmith | sdeath: nope, and I feel your pain, but still.. normally we run people out of here _before_ answering their questions | |
| 19:35:13 | sean-k-mooney | jaypipes: similar feeling... | |
| 19:35:26 | cfriesen | sean-k-mooney: the numatopologyfilter would still catch stuff like that I think | |
| 19:35:27 | jaypipes | sean-k-mooney: no, in all seriousness, it doesn't impact the accounting of CPU resources. | |
| 19:35:38 | jaypipes | cfriesen: yes, it would indeed. | |
| 19:35:57 | sean-k-mooney | jaypipes: yes that is true i shoudl be placement | |
| 19:36:24 | sean-k-mooney | it does not care becase it doesn't impact the accounting of CPU resources. numa topology filter might | |
| 19:37:21 | sdeath | dansmith: then thanks for the assistance! there any objection to lurking? | |
| 19:37:37 | dansmith | sdeath: nope | |
| 19:37:46 | sean-k-mooney | jaypipes: so jay for the snowflake. it lands on a host with HT off and boots fine. it lands on a host with HT on. what happens. retry? | |
| 19:37:49 | melwitt | sdeath: fwiw the openstack@lists.openstack.org is better than the channel for getting responses about deployment/config issues, IMHO | |
| 19:38:01 | melwitt | mailing list | |
| 19:38:14 | cfriesen | jaypipes: so if we ignore ht in placement we don't get the atomicity of reservation benefits, but it's no worse than it is currently. we might fail ResourceTracker.instance_claim() or rebuild_claim() or resize_claim() once we get to the compute node. | |
| 19:38:20 | jaypipes | sean-k-mooney: NUMATopologyFilter has no impact on resource accounting whatsoever, for two reasons: a) the results of numa_fit_instance_to_host() are immediately discarded and b) it looks at *assignment*, not allocation. | |
| 19:38:29 | melwitt | gah, more POST_FAILURE on zuul >:| | |
| 19:38:48 | sdeath | melwitt: will give that one a shot if/when needed... thanks! | |
| 19:39:01 | cfriesen | sean-k-mooney: if it lands on a host with HT on, but there's enough free host CPUs, then we behave as now and we're good | |