Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-04
15:22:38 mriedem bauzas: if we can't reschedule, why do we care about how fast the allocations are deleted?
15:23:17 mriedem that raises the question of when the allocations are deleted, but i think that's just left unchanged, when we delete the instance we remove the allocations via RT in the comptue host
15:23:19 mriedem *compute
15:23:21 bauzas mriedem: it's a separate problem, hence me saying if reschedules are possible, then I'd prefer to delete the allocations sooner than later
15:23:23 mdbooth mriedem: Do you have a spec to delete bdm v1, btw?
15:23:29 mriedem mdbooth: i don't
15:23:36 mriedem bauzas: also, have we taken into account local deletes in the API?
15:23:47 bauzas mriedem: I'm just putting a big hypothesis in my spec that is, reschedules are possible
15:23:47 mriedem the compute api would have to delete the allocations there if the compute is dead
15:23:50 mdbooth mriedem: Why does it linger, btw?
15:24:07 mriedem mdbooth: no good reason probably
15:24:26 mdbooth It's a prime candidate for simplification.
15:24:33 bauzas mriedem: well, that's possibly a good point, if the compute is dead, then the allocation would still be created
15:24:50 bauzas mriedem: so we could possibly clean up "dead" allocations
15:25:07 openstackgerrit Markus Zoeller (markus_z) proposed openstack/nova master: The API defines PUT and POST as HTTP methods which need a request body. It is allowed, that this request body might be of zero length. What was missing is, that the "content-length" of the request also might be None. https://review.openstack.org/453220
15:25:12 bauzas mriedem: that said, the ComputeFilter verifies whether the service is still up
15:25:39 pumarani_ sdague, yes. With neutron changes getting merged, we can also have some code samples on it. we are following up on neutron RFE.
15:25:50 bauzas mriedem: so the real problem is rather, say, a full compute MQ with bad performance
15:26:10 bauzas mriedem: then we would have created an allocation for something not really created
15:26:26 bauzas either way, I need to think more of that
15:26:34 openstackgerrit Markus Zoeller (markus_z) proposed openstack/nova master: API: accept None as content-length in HTTP requests https://review.openstack.org/453220
15:26:47 sdague mriedem: in our "spec review sprint" today, are there particular specs you want eyes on instead of just scatter shotting?
15:29:13 openstackgerrit Peter Hamilton proposed openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/357151
15:31:04 mriedem lyarwood: https://bugs.launchpad.net/cinder/+bug/1679736
15:31:04 openstack Launchpad bug 1679736 in Cinder "VolumeTypeEncryption API extension is not documented in api-ref site" [Undecided,New]
15:31:35 mriedem bauzas: i'm talking about after you've landed an instance on a host, so after scheduling, and the compute is down and you delete the instance
15:31:52 mriedem since we can't remove the allocation via RT on the compute host, where do we deallocate for that resource provider for a local delete?
15:31:59 mriedem hell, is that just a latent bug we already have with placement?
15:32:09 mriedem jaypipes: dansmith: ^?
15:32:26 bauzas mriedem: well, me shrugs
15:32:29 bauzas I guess yes
15:32:30 mriedem bauzas: i don't know what "then we would have created an allocation for something not really created" means
15:32:43 mriedem sdague: full monty
15:33:14 mriedem bauzas: for the comptue isn't down, but the instance is shelved-offloaded,
15:33:14 bauzas mriedem: I was thinking of the scheduler saying "okay, this one" then posting the allocations, then the boot requests goes trampled by the RPC bus
15:33:20 mriedem i guess when we shelve offload, do we deallocate?
15:33:38 cdent I think there's some misconceptions going on here
15:34:31 mordred cdent: people have the wrong idea about something? bah
15:34:45 jaypipes mriedem: yeah, it's a latent bug I think. we don't currently deallocate orphaned instances like we should.
15:34:55 mriedem lyarwood: i think we should add a microversion to cinder v3 to fix some of the volume type encryption api stuff, like remove the need for the full classname in the provider entry, just say 'luks' or 'cryptsetup'
15:35:16 cdent yes, known bug: a dead compute node's allocations persist
15:35:21 mriedem lyarwood: and as part of that microversion change in cinder v3, we enforce the actual providers using schema validation, rather than let it be free-form like it is today
15:35:58 mriedem cdent: jaypipes: how about shelve offload?
15:36:26 cdent there's also this: https://bugs.launchpad.net/nova/+bug/1661312
15:36:26 openstack Launchpad bug 1661312 in OpenStack Compute (nova) "Evacuation will corrupt instance allocations" [Undecided,New]
15:36:27 mriedem because when we unshelve, the instance could end up on a different host
15:36:41 jaypipes mriedem: no idea. would need to check. not sure if the compute manager calls an RT method on shelve_offload().
15:36:46 mriedem and if it's on a different host, we're then counting allocatoins twice for the same instnace
15:38:10 cdent since allocations are keyed by consumer id, and we set them (as in fully replace them) the situation is a bit cleaner than it might initially appear
15:38:55 mriedem but when we do claims in the scheduler,
15:39:04 mriedem aren't we aggregating the total allocations for a given resource provider?
15:39:25 mriedem and if we don't remove the allocations for a shelved offloaded consumer on that compute resource provider, it's a lie about actual resource allocation right?
15:39:32 cdent if we are puting an allocation, it will be for the instance uuid, no matter what it uses
15:39:51 jaypipes mriedem: no.
15:40:07 mriedem i guess when we do claims we're looking at inventory?
15:40:41 mriedem if i have an inventory of 5 apples, and your instance claims 3 apples, i have 2 apples left for sally to claim, right?
15:40:43 jaypipes mriedem: the "claim" is merely the transactional writing of allocation records against one or more resource providers for a single consumer ID.
15:41:16 bauzas mriedem: I think all the move operations need to update the allocations indeed
15:41:18 jaypipes mriedem: yes, that is a check that occurs entirely within the POST /allocations context in the placement API though.
15:41:42 bauzas I need a second pass of thoughts
15:41:42 mriedem jaypipes: don't we POST allocations to a specific resource provider though?
15:42:09 cdent mriedem: no
15:42:09 jaypipes mriedem: you will get a 409 if, say, Alice gets an apple before Sally's attempt to get 2 apples.
15:42:22 jaypipes mriedem: no, allocations are multi-provider.
15:42:25 cdent mriedem: or rather, not quite
15:42:38 gibi mriedem: hi! I saw your ping from yesterday about the snapshot notification and I had some free time to put a patch togheter for that issue https://review.openstack.org/#/c/453077/
15:42:50 mriedem resource_provider_uuid = allocation['resource_provider']['uuid']
15:42:52 jaypipes mriedem: for instance, a compute host provider and a shared storage pool provider might be involved in the same POST /allocations for a consumer (instance UUID)
15:43:13 mriedem but the allocation is still tied to a resource provider
15:43:14 mriedem one or more
15:43:42 jaypipes mriedem: the allocation record is, yes. but the entire "claim" is a *set* of allocation records for a single consumer (the instance)
15:43:43 mriedem the scheduler is asking placement for the list of available resource providers given some vcpu/ram/disk constraints, right?
15:43:55 jaypipes yep
15:43:58 mriedem and then takes that list, gets the compute nodes from the db, and filters them
15:44:03 jaypipes yep
15:44:11 mriedem so then i have maybe 3 available hosts, i pick one, and make my claim
15:44:18 mriedem which is a POST allocation on that resource provider, yes?
15:44:22 jaypipes no
15:44:23 jaypipes :)
15:44:51 jaypipes it would be a POST /allocations and the compute node resource provider would be one of the resource providers involved in the allocation request.
15:45:18 jaypipes mriedem: again, the reason this is like that is because you can claim resources against >1 provider in a single request.
15:45:21 mriedem i think i'm trying to say the same thing
15:45:32 mriedem but in the less formal way
15:45:54 mriedem me ugg, me want 2 apples, you mug have 3 apples, me claim 2 of your 3 apples, yes?
15:46:21 jaypipes yes
15:46:30 mriedem i get the multiple provider thing, that's fine, i need some compute resources, i need some shared storage resources from another provider, whatever
15:46:38 cdent (just to be pedantic the currently available method is PUT /allocations/{consumer_uuid} and I think we can continue using that one for the time being)
15:47:20 mriedem ok, so i think we've established that the scheduler has 'claimed' resources on a provider, or set of providers, by putting an allocation for a given instance (consumer) on those providers,
15:47:24 mriedem and then sends the build request to that host
15:47:26 mriedem to build the instance
15:48:01 mriedem since we can't reschedule, i guess i won't ask about that
15:48:20 mriedem so when we shelve offload an instance, do we deallocate for that consumer on it's resource providers
15:48:41 mriedem or when we delete the instance, and the compute host is down so we can't rpc cast to the compute to do the delete, do we deallocate from the api
15:48:44 mriedem i think the answer to both is no
15:49:16 jaypipes yeah, I suspect you are correct about that. but would be easy enough to fix in a bug report.
15:50:08 mriedem i may be wrong about the shelve offload case, because i remember alaski pointing out a super gross tight coupling of instance vm_state checking in the RT
15:50:21 mriedem yup ALLOW_RESOURCE_REMOVAL = [DELETED, SHELVED_OFFLOADED]
15:50:30 mriedem RT._update_usage_from_instance
15:50:32 mriedem checks that
15:50:57 jaypipes ok

Earlier   Later