| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:20:13 | dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | |
| 19:20:31 | jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | |
| 19:20:32 | dansmith | sdeath: pastebin, yo | |
| 19:20:34 | melwitt | use paste.openstack.org | |
| 19:20:46 | melwitt | or pastebin | |
| 19:20:51 | sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | |
| 19:21:20 | melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | |
| 19:21:21 | openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |
| 19:21:41 | dansmith | sdeath: you'll have to delete it from sql I guess | |
| 19:21:48 | sdeath | DS: is that safe, then? | |
| 19:21:54 | sdeath | if I remove from the nova.cells table? | |
| 19:22:11 | dansmith | sdeath: from nova_api.cell_mappings | |
| 19:22:22 | sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | |
| 19:22:22 | jaypipes | cfriesen: gross. | |
| 19:22:34 | mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | |
| 19:22:39 | cfriesen | sean-k-mooney: except for the final allocations part at the end | |
| 19:22:44 | rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | |
| 19:22:48 | dansmith | mriedem: that will delete all the hosts | |
| 19:22:52 | dansmith | mriedem: which he doesn't want | |
| 19:22:53 | jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | |
| 19:22:59 | sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | |
| 19:23:00 | mriedem | the mappings, right | |
| 19:23:01 | mriedem | nvm | |
| 19:23:30 | sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | |
| 19:24:07 | sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | |
| 19:24:22 | dansmith | sdeath: figure out which one is right right one and delete the other | |
| 19:24:23 | cfriesen | sean-k-mooney: isnt' that already in the host infomration? | |
| 19:24:28 | mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | |
| 19:24:30 | jaypipes | sean-k-mooney: no. I'm not willing to change the design of the placement service (and the allocation request atomicity/guarantees) just to meet these wack-o requirements. | |
| 19:24:34 | dansmith | sdeath: hopefully all your instance and host mappings refer to one of the two cell mappings | |
| 19:24:36 | melwitt | rybridges: yeah, so that implies the nova API call to add the host must be failing https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L224 else segment_host_mappings is empty there | |
| 19:24:42 | sean-k-mooney | cfriesen: what how many ht the host has | |
| 19:24:48 | sdeath | m: let's see if —force exists... | |
| 19:25:01 | sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | |
| 19:25:03 | dansmith | sdeath: you do not want --force | |
| 19:25:18 | dansmith | sdeath: --force on that is actually --recursive | |
| 19:25:18 | mlavalle | rybridges: in other words. Have you created a routed network yet? | |
| 19:25:25 | cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | |
| 19:25:29 | sdeath | ds: so maybe we won't do that then | |
| 19:25:48 | sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | |
| 19:25:50 | dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | |
| 19:25:52 | melwitt | heh | |
| 19:26:05 | cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | |
| 19:26:08 | sdeath | adium, evidently it does | |
| 19:26:35 | sdeath | so, backtrcing: nova_api.cell_mappings | |
| 19:26:44 | cfriesen | jaypipes: I think I see what you mean, we can't retroactively change the size of an earlier allocation request | |
| 19:27:00 | sean-k-mooney | cfriesen: ya so in this case we would have to make another allocation_candiates request and then find the host n that again and consume that | |
| 19:27:49 | cfriesen | sean-k-mooney: right. Or else we throw out the cpu_thread_policy option in the flavor, or else we model siblings in placement. | |
| 19:28:02 | jaypipes | cfriesen: right. the whole point of the allocation candidates list was "these are the nodes that met your request for resources". if we then just modify that resource amount, we invalidate the result of allocation candidates. :( | |
| 19:28:27 | sdeath | dansmith: OK, rows delete (from host_mappings and from cell_mappings; host_mappings due to foreign_key constraint) | |
| 19:28:30 | sdeath | restarting nova… let's see if this work. | |
| 19:28:33 | sdeath | *wrks | |
| 19:28:51 | sdeath | hey, how about that. only one row per hypervisor! | |
| 19:28:55 | sdeath | OK, so... | |
| 19:29:00 | dansmith | sdeath: so you'll need to do discover_hosts again, but it probably won't find them all because they've been marked as already mapped | |
| 19:29:17 | dansmith | sdeath: you should have changed the id on the others to be the cell you kept | |
| 19:30:06 | sdeath | will it barf if I manually reinsert? | |
| 19:30:13 | dansmith | sdeath: no | |
| 19:30:20 | sean-k-mooney | cfriesen: i dont really see a good way to keep cpu_thread_policy=isolate with out having an allocation on the compute host | |
| 19:30:29 | tblakes | mriedmen: I responded to his comment in bug 1756360. He was seeing that issue because he wasn't passing in a kwarg that nova passes in. | |
| 19:30:31 | sdeath | dansmith: actually, map_cell_and_hosts seems to have picked them back up.. | |
| 19:30:31 | jaypipes | cfriesen: or... we just go with my spec and just track dedicated CPUs as their own thing and deal with the workloads that can't tolerate having their dedicated CPUs be hyperthreads as the snowflakey thing that it is when we get to the destination host the scheduler selected. | |
| 19:30:32 | openstack | bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] https://launchpad.net/bugs/1756360 - Assigned to Tyler Blakeslee (tblakes) | |
| 19:30:41 | sdeath | I see fresh IDs and happiness | |
| 19:30:51 | dansmith | sdeath: that's not what you wanted I think | |
| 19:31:25 | cfriesen | jaypipes: so we'd get to the compute node and find out "oops, it doesn't have as many PCPU resources as we thought" and fail the claim? | |
| 19:31:28 | dansmith | sdeath: didn't that create a new cell mapping again? | |
| 19:31:34 | sdeath | dansmith: nope | |
| 19:31:49 | sean-k-mooney | cfriesen: well claim would happen in scuduler before getting to compute host | |
| 19:31:58 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614 | |
| 19:32:10 | sean-k-mooney | cfriesen: we could at the very end try to extend the claim and only fail if we could not extend | |
| 19:32:19 | sdeath | cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors. | |
| 19:32:24 | dansmith | sdeath: ah, I see where it went, okay | |
| 19:32:52 | dansmith | sdeath: so you're good now? | |
| 19:32:59 | sdeath | nova-manage cell_v2 list_cells shows right number of cells... | |
| 19:33:02 | sdeath | I think it's good. | |
| 19:33:03 | sdeath | so... | |
| 19:33:09 | cfriesen | sean-k-mooney: I'm talking about ResourceTracker.instance_claim() | |
| 19:33:12 | dansmith | sdeath: cool, now read the channel topic :) | |
| 19:33:27 | jaypipes | cfriesen: no... it *would* have as many pCPUs as we thought... it's just *that snowflake of a workload considers hyperthreads not to be good enough pCPUs for it*. | |
| 19:34:05 | cfriesen | jaypipes: heh...that's one way of thinking about it. :) | |
| 19:34:33 | jaypipes | cfriesen: it's the only way to think about it. but I digress.. | |
| 19:34:33 | sean-k-mooney | jaypipes: well one exampel of that is realtime cpus and hyperthread dont mix | |
| 19:34:34 | sdeath | Apologies; I tried #openstack two days running, no soap… wasn't sure where to head from there. There a better "general guidance" channel? | |
| 19:34:47 | jaypipes | sean-k-mooney: I don't care? | |
| 19:34:55 | tblakes | mriedem: I responded to the comment in https://bugs.launchpad.net/nova/+bug/1756360. They were seeing that issue because they were not passing in a kwarg that nova passes in. | |
| 19:34:57 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:34:58 | cfriesen | jaypipes: I think that'd work, it wouldn't be any more racy than it is now. | |
| 19:35:07 | dansmith | sdeath: nope, and I feel your pain, but still.. normally we run people out of here _before_ answering their questions | |
| 19:35:13 | sean-k-mooney | jaypipes: similar feeling... | |
| 19:35:26 | cfriesen | sean-k-mooney: the numatopologyfilter would still catch stuff like that I think | |
| 19:35:27 | jaypipes | sean-k-mooney: no, in all seriousness, it doesn't impact the accounting of CPU resources. | |
| 19:35:38 | jaypipes | cfriesen: yes, it would indeed. | |
| 19:35:57 | sean-k-mooney | jaypipes: yes that is true i shoudl be placement | |
| 19:36:24 | sean-k-mooney | it does not care becase it doesn't impact the accounting of CPU resources. numa topology filter might | |
| 19:37:21 | sdeath | dansmith: then thanks for the assistance! there any objection to lurking? | |
| 19:37:37 | dansmith | sdeath: nope | |