| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:30:13 | dansmith | sdeath: no | |
| 19:30:20 | sean-k-mooney | cfriesen: i dont really see a good way to keep cpu_thread_policy=isolate with out having an allocation on the compute host | |
| 19:30:29 | tblakes | mriedmen: I responded to his comment in bug 1756360. He was seeing that issue because he wasn't passing in a kwarg that nova passes in. | |
| 19:30:31 | sdeath | dansmith: actually, map_cell_and_hosts seems to have picked them back up.. | |
| 19:30:31 | jaypipes | cfriesen: or... we just go with my spec and just track dedicated CPUs as their own thing and deal with the workloads that can't tolerate having their dedicated CPUs be hyperthreads as the snowflakey thing that it is when we get to the destination host the scheduler selected. | |
| 19:30:32 | openstack | bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] https://launchpad.net/bugs/1756360 - Assigned to Tyler Blakeslee (tblakes) | |
| 19:30:41 | sdeath | I see fresh IDs and happiness | |
| 19:30:51 | dansmith | sdeath: that's not what you wanted I think | |
| 19:31:25 | cfriesen | jaypipes: so we'd get to the compute node and find out "oops, it doesn't have as many PCPU resources as we thought" and fail the claim? | |
| 19:31:28 | dansmith | sdeath: didn't that create a new cell mapping again? | |
| 19:31:34 | sdeath | dansmith: nope | |
| 19:31:49 | sean-k-mooney | cfriesen: well claim would happen in scuduler before getting to compute host | |
| 19:31:58 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614 | |
| 19:32:10 | sean-k-mooney | cfriesen: we could at the very end try to extend the claim and only fail if we could not extend | |
| 19:32:19 | sdeath | cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors. | |
| 19:32:24 | dansmith | sdeath: ah, I see where it went, okay | |
| 19:32:52 | dansmith | sdeath: so you're good now? | |
| 19:32:59 | sdeath | nova-manage cell_v2 list_cells shows right number of cells... | |
| 19:33:02 | sdeath | I think it's good. | |
| 19:33:03 | sdeath | so... | |
| 19:33:09 | cfriesen | sean-k-mooney: I'm talking about ResourceTracker.instance_claim() | |
| 19:33:12 | dansmith | sdeath: cool, now read the channel topic :) | |
| 19:33:27 | jaypipes | cfriesen: no... it *would* have as many pCPUs as we thought... it's just *that snowflake of a workload considers hyperthreads not to be good enough pCPUs for it*. | |
| 19:34:05 | cfriesen | jaypipes: heh...that's one way of thinking about it. :) | |
| 19:34:33 | jaypipes | cfriesen: it's the only way to think about it. but I digress.. | |
| 19:34:33 | sean-k-mooney | jaypipes: well one exampel of that is realtime cpus and hyperthread dont mix | |
| 19:34:34 | sdeath | Apologies; I tried #openstack two days running, no soap… wasn't sure where to head from there. There a better "general guidance" channel? | |
| 19:34:47 | jaypipes | sean-k-mooney: I don't care? | |
| 19:34:55 | tblakes | mriedem: I responded to the comment in https://bugs.launchpad.net/nova/+bug/1756360. They were seeing that issue because they were not passing in a kwarg that nova passes in. | |
| 19:34:57 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:34:58 | cfriesen | jaypipes: I think that'd work, it wouldn't be any more racy than it is now. | |
| 19:35:07 | dansmith | sdeath: nope, and I feel your pain, but still.. normally we run people out of here _before_ answering their questions | |
| 19:35:13 | sean-k-mooney | jaypipes: similar feeling... | |
| 19:35:26 | cfriesen | sean-k-mooney: the numatopologyfilter would still catch stuff like that I think | |
| 19:35:27 | jaypipes | sean-k-mooney: no, in all seriousness, it doesn't impact the accounting of CPU resources. | |
| 19:35:38 | jaypipes | cfriesen: yes, it would indeed. | |
| 19:35:57 | sean-k-mooney | jaypipes: yes that is true i shoudl be placement | |
| 19:36:24 | sean-k-mooney | it does not care becase it doesn't impact the accounting of CPU resources. numa topology filter might | |
| 19:37:21 | sdeath | dansmith: then thanks for the assistance! there any objection to lurking? | |
| 19:37:37 | dansmith | sdeath: nope | |
| 19:37:46 | sean-k-mooney | jaypipes: so jay for the snowflake. it lands on a host with HT off and boots fine. it lands on a host with HT on. what happens. retry? | |
| 19:37:49 | melwitt | sdeath: fwiw the openstack@lists.openstack.org is better than the channel for getting responses about deployment/config issues, IMHO | |
| 19:38:01 | melwitt | mailing list | |
| 19:38:14 | cfriesen | jaypipes: so if we ignore ht in placement we don't get the atomicity of reservation benefits, but it's no worse than it is currently. we might fail ResourceTracker.instance_claim() or rebuild_claim() or resize_claim() once we get to the compute node. | |
| 19:38:20 | jaypipes | sean-k-mooney: NUMATopologyFilter has no impact on resource accounting whatsoever, for two reasons: a) the results of numa_fit_instance_to_host() are immediately discarded and b) it looks at *assignment*, not allocation. | |
| 19:38:29 | melwitt | gah, more POST_FAILURE on zuul >:| | |
| 19:38:48 | sdeath | melwitt: will give that one a shot if/when needed... thanks! | |
| 19:39:01 | cfriesen | sean-k-mooney: if it lands on a host with HT on, but there's enough free host CPUs, then we behave as now and we're good | |
| 19:39:18 | sean-k-mooney | jaypipes: yes i know. im aske about the usecase boot me a vm with 2 pinned cpus and tread policy isolate. will that still work with out a retry if the host has hypertreads on | |
| 19:39:18 | jaypipes | sean-k-mooney: see what cfriesen just said... the NUMATopologyFilter will catch the HT-snowflake stuff in the scheduler before going to the compute host. | |
| 19:39:20 | cfriesen | sean-k-mooney: actually no, we'd need to account in placement for the extra CPUs we consume | |
| 19:40:05 | cfriesen | jaypipes: given that the snowflake instance will actually consume more CPUs, we need to account for those extra ones in placement | |
| 19:40:17 | sean-k-mooney | cfriesen: jaypipes yes so there are two options at that point, 1 we fail to boot and retry, or 2 we ask placement to extend the allocation and fail if it would not fit | |
| 19:40:31 | jaypipes | cfriesen, sean-k-mooney: the fundamental problem with the cpu_thread_policy is that it leads to non-deterministic amounts of requested resources. | |
| 19:40:50 | cfriesen | jaypipes: it's deterministic, but it depends on the host | |
| 19:40:55 | jaypipes | sean-k-mooney: I'm not going to change the allocation request. | |
| 19:41:04 | jaypipes | cfriesen: omg, I'm gonna slap you. | |
| 19:41:09 | jaypipes | :) | |
| 19:41:12 | cfriesen | jaypipes: what about a whole new allocation request | |
| 19:41:22 | jaypipes | cfriesen: it's non-deterministic from the viewpoint of the scheduler | |
| 19:41:33 | sean-k-mooney | jaypipes: ok so we just define it a retry and maybe you could use a weigher to minimies the change it would happen | |
| 19:42:17 | sean-k-mooney | e.g. you land on a host. figure out you need more resouce to run on that host then you asked for and retry on next host | |
| 19:42:24 | jaypipes | cfriesen: for a whole new allocation request, we'd need to re-submit to GET /allocation_candidates with a new requested resource amount, which would give us back a different set of compute hosts, which we would send to the NUMATopologyFilter, which would re-work the allocation request again, and we'd end up in an infinite loop of sadnsees. | |
| 19:42:24 | cfriesen | sean-k-mooney: so now we're saying that ISOLATE can't possible run on a host with HT enabled. how is this different from an aggregate? | |
| 19:42:59 | cfriesen | jaypipes: can we specify a particular compute node when doing the allocation request? | |
| 19:43:04 | sean-k-mooney | cfriesen: actully you can make it work but it doubles the amount of flavors | |
| 19:43:14 | jaypipes | cfriesen: no. | |
| 19:43:31 | cfriesen | jaypipes: how do we handle specifying the compute node when doing a migration? | |
| 19:43:33 | sean-k-mooney | cfriesen: you set flavor.vcpu=4 and resouce[vcpu]=8 and it will work only on ht systems | |
| 19:43:56 | cfriesen | sean-k-mooney: that'd work on non-ht as well, technically | |
| 19:43:56 | jaypipes | cfriesen: you can ask for an aggregate, an amount of resources, required traits, but not a specific provider (since that would defeat the entire purpose of the GET /allocation_candidates endpoint. | |
| 19:44:30 | jaypipes | cfriesen: we don't call GET /allocation_candidates when specifying a compute node (force_host). | |
| 19:44:37 | sean-k-mooney | cfriesen: yes but it really expecive maybe add trait:HT_COUNT_2=required to avoid that | |
| 19:45:03 | rybridges | mlavalle: This is how I am creating the network / segment / subnet: http://paste.openstack.org/show/708997/ As you can see the aggregate host list is empty. I was under the impression that this process creates a routed network. Is there something I am missing? | |
| 19:45:05 | jaypipes | sean-k-mooney: that's a trait that looks suspiciously like a quantity of resources. | |
| 19:45:27 | sean-k-mooney | jaypipes: the trait is based on how we said we would do cpu frequency | |
| 19:45:58 | jaypipes | sean-k-mooney: you mean vGPU display heads? | |
| 19:46:05 | sean-k-mooney | e.g. tag it with multipel tratis so 4GHZ cpu would have 1GHZ,2GHZ,3GHZ and 4GHZ traits applied | |
| 19:46:36 | mriedem | melwitt: that's a known issue http://status.openstack.org/elastic-recheck/#1758054 | |
| 19:46:40 | mriedem | everything is blocked until that's merged | |
| 19:46:48 | sean-k-mooney | jaypipes: ya i think that does the same thing but did not look at that that closely | |
| 19:46:59 | jaypipes | sean-k-mooney: but in this case, the amount of resources being *requested* changes depending on which host a workload ends up on :( that's the whole problem with this... | |
| 19:47:06 | melwitt | mriedem: ah, thanks | |
| 19:47:09 | dansmith | is gerrit sucking hard for everyone else? | |
| 19:47:42 | sean-k-mooney | jaypipes: yes so if you wanted an isoleated vm on a host with HT on you would have a flavor like this | |
| 19:48:17 | cfriesen | jaypipes: what do we call when specifying a compute node on a migration? | |
| 19:48:41 | sean-k-mooney | flavor.vcpu=4,resouce[vcpu]=8:traits:HT_count_2=forbid thread_policy=isolate | |
| 19:48:41 | jaypipes | dansmith: I'm too busy wanting to shoot myself in the head with a bazooka to feel any pain from gerrit. | |
| 19:48:52 | dansmith | jaypipes: roger that | |
| 19:49:10 | cfriesen | jaypipes: and why couldn't we do that in the scheduler if we realize after selecting a host that we need to account for some extra PCPU resources? | |
| 19:49:30 | cfriesen | sean-k-mooney: at that point you may as well use a host aggregate | |
| 19:49:45 | jaypipes | cfriesen: we call PUT /allocations/{migration_uuid} to reserve resources on the source host for the migration and PUT /allocations/{instance_uuid} to consume the instance resources on the destination host. | |
| 19:49:57 | jaypipes | cfriesen: we don't go through the shceduler at all when force_host. | |
| 19:50:24 | cfriesen | jaypipes: we do for migrations (when it's really a "suggested host" rather than force) | |
| 19:51:03 | sean-k-mooney | cfriesen: well you can have lavor.vcpu=4,resouce[vcpu]=8:traits:ht_count=require thread_policy=isolate and flavor.vcpu=4,resouce[vcpu]=4:traits:HT_count_2=forbid thread_policy=isolate | |
| 19:51:47 | sean-k-mooney | cfriesen: it should have been HT_count_2=require when resouce[vcpu]=8 not forbid originally | |
| 19:52:08 | jaypipes | sean-k-mooney: that just doesn't seem right to me. | |
| 19:52:10 | cfriesen | I really don't want to have multiple extra-spec keys that depend on the value of other extra-spec keys | |
| 19:52:22 | jaypipes | and certainly isn't very understandable to me. | |
| 19:52:28 | cfriesen | agreed, that's a mess. :) | |