| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:24:48 | sdeath | m: let's see if —force exists... | |
| 19:25:01 | sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | |
| 19:25:03 | dansmith | sdeath: you do not want --force | |
| 19:25:18 | dansmith | sdeath: --force on that is actually --recursive | |
| 19:25:18 | mlavalle | rybridges: in other words. Have you created a routed network yet? | |
| 19:25:25 | cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | |
| 19:25:29 | sdeath | ds: so maybe we won't do that then | |
| 19:25:48 | sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | |
| 19:25:50 | dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | |
| 19:25:52 | melwitt | heh | |
| 19:26:05 | cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | |
| 19:26:08 | sdeath | adium, evidently it does | |
| 19:26:35 | sdeath | so, backtrcing: nova_api.cell_mappings | |
| 19:26:44 | cfriesen | jaypipes: I think I see what you mean, we can't retroactively change the size of an earlier allocation request | |
| 19:27:00 | sean-k-mooney | cfriesen: ya so in this case we would have to make another allocation_candiates request and then find the host n that again and consume that | |
| 19:27:49 | cfriesen | sean-k-mooney: right. Or else we throw out the cpu_thread_policy option in the flavor, or else we model siblings in placement. | |
| 19:28:02 | jaypipes | cfriesen: right. the whole point of the allocation candidates list was "these are the nodes that met your request for resources". if we then just modify that resource amount, we invalidate the result of allocation candidates. :( | |
| 19:28:27 | sdeath | dansmith: OK, rows delete (from host_mappings and from cell_mappings; host_mappings due to foreign_key constraint) | |
| 19:28:30 | sdeath | restarting nova… let's see if this work. | |
| 19:28:33 | sdeath | *wrks | |
| 19:28:51 | sdeath | hey, how about that. only one row per hypervisor! | |
| 19:28:55 | sdeath | OK, so... | |
| 19:29:00 | dansmith | sdeath: so you'll need to do discover_hosts again, but it probably won't find them all because they've been marked as already mapped | |
| 19:29:17 | dansmith | sdeath: you should have changed the id on the others to be the cell you kept | |
| 19:30:06 | sdeath | will it barf if I manually reinsert? | |
| 19:30:13 | dansmith | sdeath: no | |
| 19:30:20 | sean-k-mooney | cfriesen: i dont really see a good way to keep cpu_thread_policy=isolate with out having an allocation on the compute host | |
| 19:30:29 | tblakes | mriedmen: I responded to his comment in bug 1756360. He was seeing that issue because he wasn't passing in a kwarg that nova passes in. | |
| 19:30:31 | sdeath | dansmith: actually, map_cell_and_hosts seems to have picked them back up.. | |
| 19:30:31 | jaypipes | cfriesen: or... we just go with my spec and just track dedicated CPUs as their own thing and deal with the workloads that can't tolerate having their dedicated CPUs be hyperthreads as the snowflakey thing that it is when we get to the destination host the scheduler selected. | |
| 19:30:32 | openstack | bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] https://launchpad.net/bugs/1756360 - Assigned to Tyler Blakeslee (tblakes) | |
| 19:30:41 | sdeath | I see fresh IDs and happiness | |
| 19:30:51 | dansmith | sdeath: that's not what you wanted I think | |
| 19:31:25 | cfriesen | jaypipes: so we'd get to the compute node and find out "oops, it doesn't have as many PCPU resources as we thought" and fail the claim? | |
| 19:31:28 | dansmith | sdeath: didn't that create a new cell mapping again? | |
| 19:31:34 | sdeath | dansmith: nope | |
| 19:31:49 | sean-k-mooney | cfriesen: well claim would happen in scuduler before getting to compute host | |
| 19:31:58 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614 | |
| 19:32:10 | sean-k-mooney | cfriesen: we could at the very end try to extend the claim and only fail if we could not extend | |
| 19:32:19 | sdeath | cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors. | |
| 19:32:24 | dansmith | sdeath: ah, I see where it went, okay | |
| 19:32:52 | dansmith | sdeath: so you're good now? | |
| 19:32:59 | sdeath | nova-manage cell_v2 list_cells shows right number of cells... | |
| 19:33:02 | sdeath | I think it's good. | |
| 19:33:03 | sdeath | so... | |
| 19:33:09 | cfriesen | sean-k-mooney: I'm talking about ResourceTracker.instance_claim() | |
| 19:33:12 | dansmith | sdeath: cool, now read the channel topic :) | |
| 19:33:27 | jaypipes | cfriesen: no... it *would* have as many pCPUs as we thought... it's just *that snowflake of a workload considers hyperthreads not to be good enough pCPUs for it*. | |
| 19:34:05 | cfriesen | jaypipes: heh...that's one way of thinking about it. :) | |
| 19:34:33 | jaypipes | cfriesen: it's the only way to think about it. but I digress.. | |
| 19:34:33 | sean-k-mooney | jaypipes: well one exampel of that is realtime cpus and hyperthread dont mix | |
| 19:34:34 | sdeath | Apologies; I tried #openstack two days running, no soap… wasn't sure where to head from there. There a better "general guidance" channel? | |
| 19:34:47 | jaypipes | sean-k-mooney: I don't care? | |
| 19:34:55 | tblakes | mriedem: I responded to the comment in https://bugs.launchpad.net/nova/+bug/1756360. They were seeing that issue because they were not passing in a kwarg that nova passes in. | |
| 19:34:57 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:34:58 | cfriesen | jaypipes: I think that'd work, it wouldn't be any more racy than it is now. | |
| 19:35:07 | dansmith | sdeath: nope, and I feel your pain, but still.. normally we run people out of here _before_ answering their questions | |
| 19:35:13 | sean-k-mooney | jaypipes: similar feeling... | |
| 19:35:26 | cfriesen | sean-k-mooney: the numatopologyfilter would still catch stuff like that I think | |
| 19:35:27 | jaypipes | sean-k-mooney: no, in all seriousness, it doesn't impact the accounting of CPU resources. | |
| 19:35:38 | jaypipes | cfriesen: yes, it would indeed. | |
| 19:35:57 | sean-k-mooney | jaypipes: yes that is true i shoudl be placement | |
| 19:36:24 | sean-k-mooney | it does not care becase it doesn't impact the accounting of CPU resources. numa topology filter might | |
| 19:37:21 | sdeath | dansmith: then thanks for the assistance! there any objection to lurking? | |
| 19:37:37 | dansmith | sdeath: nope | |
| 19:37:46 | sean-k-mooney | jaypipes: so jay for the snowflake. it lands on a host with HT off and boots fine. it lands on a host with HT on. what happens. retry? | |
| 19:37:49 | melwitt | sdeath: fwiw the openstack@lists.openstack.org is better than the channel for getting responses about deployment/config issues, IMHO | |
| 19:38:01 | melwitt | mailing list | |
| 19:38:14 | cfriesen | jaypipes: so if we ignore ht in placement we don't get the atomicity of reservation benefits, but it's no worse than it is currently. we might fail ResourceTracker.instance_claim() or rebuild_claim() or resize_claim() once we get to the compute node. | |
| 19:38:20 | jaypipes | sean-k-mooney: NUMATopologyFilter has no impact on resource accounting whatsoever, for two reasons: a) the results of numa_fit_instance_to_host() are immediately discarded and b) it looks at *assignment*, not allocation. | |
| 19:38:29 | melwitt | gah, more POST_FAILURE on zuul >:| | |
| 19:38:48 | sdeath | melwitt: will give that one a shot if/when needed... thanks! | |
| 19:39:01 | cfriesen | sean-k-mooney: if it lands on a host with HT on, but there's enough free host CPUs, then we behave as now and we're good | |
| 19:39:18 | sean-k-mooney | jaypipes: yes i know. im aske about the usecase boot me a vm with 2 pinned cpus and tread policy isolate. will that still work with out a retry if the host has hypertreads on | |
| 19:39:18 | jaypipes | sean-k-mooney: see what cfriesen just said... the NUMATopologyFilter will catch the HT-snowflake stuff in the scheduler before going to the compute host. | |
| 19:39:20 | cfriesen | sean-k-mooney: actually no, we'd need to account in placement for the extra CPUs we consume | |
| 19:40:05 | cfriesen | jaypipes: given that the snowflake instance will actually consume more CPUs, we need to account for those extra ones in placement | |
| 19:40:17 | sean-k-mooney | cfriesen: jaypipes yes so there are two options at that point, 1 we fail to boot and retry, or 2 we ask placement to extend the allocation and fail if it would not fit | |
| 19:40:31 | jaypipes | cfriesen, sean-k-mooney: the fundamental problem with the cpu_thread_policy is that it leads to non-deterministic amounts of requested resources. | |
| 19:40:50 | cfriesen | jaypipes: it's deterministic, but it depends on the host | |
| 19:40:55 | jaypipes | sean-k-mooney: I'm not going to change the allocation request. | |
| 19:41:04 | jaypipes | cfriesen: omg, I'm gonna slap you. | |
| 19:41:09 | jaypipes | :) | |
| 19:41:12 | cfriesen | jaypipes: what about a whole new allocation request | |
| 19:41:22 | jaypipes | cfriesen: it's non-deterministic from the viewpoint of the scheduler | |
| 19:41:33 | sean-k-mooney | jaypipes: ok so we just define it a retry and maybe you could use a weigher to minimies the change it would happen | |
| 19:42:17 | sean-k-mooney | e.g. you land on a host. figure out you need more resouce to run on that host then you asked for and retry on next host | |
| 19:42:24 | jaypipes | cfriesen: for a whole new allocation request, we'd need to re-submit to GET /allocation_candidates with a new requested resource amount, which would give us back a different set of compute hosts, which we would send to the NUMATopologyFilter, which would re-work the allocation request again, and we'd end up in an infinite loop of sadnsees. | |
| 19:42:24 | cfriesen | sean-k-mooney: so now we're saying that ISOLATE can't possible run on a host with HT enabled. how is this different from an aggregate? | |
| 19:42:59 | cfriesen | jaypipes: can we specify a particular compute node when doing the allocation request? | |
| 19:43:04 | sean-k-mooney | cfriesen: actully you can make it work but it doubles the amount of flavors | |
| 19:43:14 | jaypipes | cfriesen: no. | |
| 19:43:31 | cfriesen | jaypipes: how do we handle specifying the compute node when doing a migration? | |
| 19:43:33 | sean-k-mooney | cfriesen: you set flavor.vcpu=4 and resouce[vcpu]=8 and it will work only on ht systems | |
| 19:43:56 | cfriesen | sean-k-mooney: that'd work on non-ht as well, technically | |
| 19:43:56 | jaypipes | cfriesen: you can ask for an aggregate, an amount of resources, required traits, but not a specific provider (since that would defeat the entire purpose of the GET /allocation_candidates endpoint. | |
| 19:44:30 | jaypipes | cfriesen: we don't call GET /allocation_candidates when specifying a compute node (force_host). | |
| 19:44:37 | sean-k-mooney | cfriesen: yes but it really expecive maybe add trait:HT_COUNT_2=required to avoid that | |
| 19:45:03 | rybridges | mlavalle: This is how I am creating the network / segment / subnet: http://paste.openstack.org/show/708997/ As you can see the aggregate host list is empty. I was under the impression that this process creates a routed network. Is there something I am missing? | |
| 19:45:05 | jaypipes | sean-k-mooney: that's a trait that looks suspiciously like a quantity of resources. | |