Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
19:31:34 sdeath dansmith: nope
19:31:49 sean-k-mooney cfriesen: well claim would happen in scuduler before getting to compute host
19:31:58 openstackgerrit Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614
19:32:10 sean-k-mooney cfriesen: we could at the very end try to extend the claim and only fail if we could not extend
19:32:19 sdeath cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors.
19:32:24 dansmith sdeath: ah, I see where it went, okay
19:32:52 dansmith sdeath: so you're good now?
19:32:59 sdeath nova-manage cell_v2 list_cells shows right number of cells...
19:33:02 sdeath I think it's good.
19:33:03 sdeath so...
19:33:09 cfriesen sean-k-mooney: I'm talking about ResourceTracker.instance_claim()
19:33:12 dansmith sdeath: cool, now read the channel topic :)
19:33:27 jaypipes cfriesen: no... it *would* have as many pCPUs as we thought... it's just *that snowflake of a workload considers hyperthreads not to be good enough pCPUs for it*.
19:34:05 cfriesen jaypipes: heh...that's one way of thinking about it. :)
19:34:33 sean-k-mooney jaypipes: well one exampel of that is realtime cpus and hyperthread dont mix
19:34:33 jaypipes cfriesen: it's the only way to think about it. but I digress..
19:34:34 sdeath Apologies; I tried #openstack two days running, no soap… wasn't sure where to head from there. There a better "general guidance" channel?
19:34:47 jaypipes sean-k-mooney: I don't care?
19:34:55 tblakes mriedem: I responded to the comment in https://bugs.launchpad.net/nova/+bug/1756360. They were seeing that issue because they were not passing in a kwarg that nova passes in.
19:34:57 openstack Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes)
19:34:58 cfriesen jaypipes: I think that'd work, it wouldn't be any more racy than it is now.
19:35:07 dansmith sdeath: nope, and I feel your pain, but still.. normally we run people out of here _before_ answering their questions
19:35:13 sean-k-mooney jaypipes: similar feeling...
19:35:26 cfriesen sean-k-mooney: the numatopologyfilter would still catch stuff like that I think
19:35:27 jaypipes sean-k-mooney: no, in all seriousness, it doesn't impact the accounting of CPU resources.
19:35:38 jaypipes cfriesen: yes, it would indeed.
19:35:57 sean-k-mooney jaypipes: yes that is true i shoudl be placement
19:36:24 sean-k-mooney it does not care becase it doesn't impact the accounting of CPU resources. numa topology filter might
19:37:21 sdeath dansmith: then thanks for the assistance! there any objection to lurking?
19:37:37 dansmith sdeath: nope
19:37:46 sean-k-mooney jaypipes: so jay for the snowflake. it lands on a host with HT off and boots fine. it lands on a host with HT on. what happens. retry?
19:37:49 melwitt sdeath: fwiw the openstack@lists.openstack.org is better than the channel for getting responses about deployment/config issues, IMHO
19:38:01 melwitt mailing list
19:38:14 cfriesen jaypipes: so if we ignore ht in placement we don't get the atomicity of reservation benefits, but it's no worse than it is currently. we might fail ResourceTracker.instance_claim() or rebuild_claim() or resize_claim() once we get to the compute node.
19:38:20 jaypipes sean-k-mooney: NUMATopologyFilter has no impact on resource accounting whatsoever, for two reasons: a) the results of numa_fit_instance_to_host() are immediately discarded and b) it looks at *assignment*, not allocation.
19:38:29 melwitt gah, more POST_FAILURE on zuul >:|
19:38:48 sdeath melwitt: will give that one a shot if/when needed... thanks!
19:39:01 cfriesen sean-k-mooney: if it lands on a host with HT on, but there's enough free host CPUs, then we behave as now and we're good
19:39:18 jaypipes sean-k-mooney: see what cfriesen just said... the NUMATopologyFilter will catch the HT-snowflake stuff in the scheduler before going to the compute host.
19:39:18 sean-k-mooney jaypipes: yes i know. im aske about the usecase boot me a vm with 2 pinned cpus and tread policy isolate. will that still work with out a retry if the host has hypertreads on
19:39:20 cfriesen sean-k-mooney: actually no, we'd need to account in placement for the extra CPUs we consume
19:40:05 cfriesen jaypipes: given that the snowflake instance will actually consume more CPUs, we need to account for those extra ones in placement
19:40:17 sean-k-mooney cfriesen: jaypipes yes so there are two options at that point, 1 we fail to boot and retry, or 2 we ask placement to extend the allocation and fail if it would not fit
19:40:31 jaypipes cfriesen, sean-k-mooney: the fundamental problem with the cpu_thread_policy is that it leads to non-deterministic amounts of requested resources.
19:40:50 cfriesen jaypipes: it's deterministic, but it depends on the host
19:40:55 jaypipes sean-k-mooney: I'm not going to change the allocation request.
19:41:04 jaypipes cfriesen: omg, I'm gonna slap you.
19:41:09 jaypipes :)
19:41:12 cfriesen jaypipes: what about a whole new allocation request
19:41:22 jaypipes cfriesen: it's non-deterministic from the viewpoint of the scheduler
19:41:33 sean-k-mooney jaypipes: ok so we just define it a retry and maybe you could use a weigher to minimies the change it would happen
19:42:17 sean-k-mooney e.g. you land on a host. figure out you need more resouce to run on that host then you asked for and retry on next host
19:42:24 cfriesen sean-k-mooney: so now we're saying that ISOLATE can't possible run on a host with HT enabled. how is this different from an aggregate?
19:42:24 jaypipes cfriesen: for a whole new allocation request, we'd need to re-submit to GET /allocation_candidates with a new requested resource amount, which would give us back a different set of compute hosts, which we would send to the NUMATopologyFilter, which would re-work the allocation request again, and we'd end up in an infinite loop of sadnsees.
19:42:59 cfriesen jaypipes: can we specify a particular compute node when doing the allocation request?
19:43:04 sean-k-mooney cfriesen: actully you can make it work but it doubles the amount of flavors
19:43:14 jaypipes cfriesen: no.
19:43:31 cfriesen jaypipes: how do we handle specifying the compute node when doing a migration?
19:43:33 sean-k-mooney cfriesen: you set flavor.vcpu=4 and resouce[vcpu]=8 and it will work only on ht systems
19:43:56 jaypipes cfriesen: you can ask for an aggregate, an amount of resources, required traits, but not a specific provider (since that would defeat the entire purpose of the GET /allocation_candidates endpoint.
19:43:56 cfriesen sean-k-mooney: that'd work on non-ht as well, technically
19:44:30 jaypipes cfriesen: we don't call GET /allocation_candidates when specifying a compute node (force_host).
19:44:37 sean-k-mooney cfriesen: yes but it really expecive maybe add trait:HT_COUNT_2=required to avoid that
19:45:03 rybridges mlavalle: This is how I am creating the network / segment / subnet: http://paste.openstack.org/show/708997/ As you can see the aggregate host list is empty. I was under the impression that this process creates a routed network. Is there something I am missing?
19:45:05 jaypipes sean-k-mooney: that's a trait that looks suspiciously like a quantity of resources.
19:45:27 sean-k-mooney jaypipes: the trait is based on how we said we would do cpu frequency
19:45:58 jaypipes sean-k-mooney: you mean vGPU display heads?
19:46:05 sean-k-mooney e.g. tag it with multipel tratis so 4GHZ cpu would have 1GHZ,2GHZ,3GHZ and 4GHZ traits applied
19:46:36 mriedem melwitt: that's a known issue http://status.openstack.org/elastic-recheck/#1758054
19:46:40 mriedem everything is blocked until that's merged
19:46:48 sean-k-mooney jaypipes: ya i think that does the same thing but did not look at that that closely
19:46:59 jaypipes sean-k-mooney: but in this case, the amount of resources being *requested* changes depending on which host a workload ends up on :( that's the whole problem with this...
19:47:06 melwitt mriedem: ah, thanks
19:47:09 dansmith is gerrit sucking hard for everyone else?
19:47:42 sean-k-mooney jaypipes: yes so if you wanted an isoleated vm on a host with HT on you would have a flavor like this
19:48:17 cfriesen jaypipes: what do we call when specifying a compute node on a migration?
19:48:41 jaypipes dansmith: I'm too busy wanting to shoot myself in the head with a bazooka to feel any pain from gerrit.
19:48:41 sean-k-mooney flavor.vcpu=4,resouce[vcpu]=8:traits:HT_count_2=forbid thread_policy=isolate
19:48:52 dansmith jaypipes: roger that
19:49:10 cfriesen jaypipes: and why couldn't we do that in the scheduler if we realize after selecting a host that we need to account for some extra PCPU resources?
19:49:30 cfriesen sean-k-mooney: at that point you may as well use a host aggregate
19:49:45 jaypipes cfriesen: we call PUT /allocations/{migration_uuid} to reserve resources on the source host for the migration and PUT /allocations/{instance_uuid} to consume the instance resources on the destination host.
19:49:57 jaypipes cfriesen: we don't go through the shceduler at all when force_host.
19:50:24 cfriesen jaypipes: we do for migrations (when it's really a "suggested host" rather than force)
19:51:03 sean-k-mooney cfriesen: well you can have lavor.vcpu=4,resouce[vcpu]=8:traits:ht_count=require thread_policy=isolate and flavor.vcpu=4,resouce[vcpu]=4:traits:HT_count_2=forbid thread_policy=isolate
19:51:47 sean-k-mooney cfriesen: it should have been HT_count_2=require when resouce[vcpu]=8 not forbid originally
19:52:08 jaypipes sean-k-mooney: that just doesn't seem right to me.
19:52:10 cfriesen I really don't want to have multiple extra-spec keys that depend on the value of other extra-spec keys
19:52:22 jaypipes and certainly isn't very understandable to me.
19:52:28 cfriesen agreed, that's a mess. :)
19:53:13 mriedem we don't go through the scheduler when a host is forced, but conductor does the resource allocation 'claim'
19:53:22 mriedem for live migrate and evacuate
19:53:41 sean-k-mooney so we keep coming back to we deprecate and remove tread_policy=isolate or extend the allocation on the compute host which breaks the workflow
19:54:29 sean-k-mooney or we change isolate to mean "host with HT off"
19:55:35 cfriesen mriedem: for "cold migrate to this compute node" we need to be going through the scheduler to do cpu pinning, pci, etc.
19:57:29 mriedem cfriesen: we do go through the scheduler for cold migrate
19:57:34 mriedem there is no force for cold migrate
19:57:35 mriedem remember
19:57:37 mriedem -5
19:57:41 cfriesen jaypipes: is there an actual API spec somewhere for placement? https://docs.openstack.org/nova/latest/user/placement.html doesn't seem to document the HTTP calls.

Earlier   Later