| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-04 | |||
| 15:31:59 | mriedem | hell, is that just a latent bug we already have with placement? | |
| 15:32:09 | mriedem | jaypipes: dansmith: ^? | |
| 15:32:26 | bauzas | mriedem: well, me shrugs | |
| 15:32:29 | bauzas | I guess yes | |
| 15:32:30 | mriedem | bauzas: i don't know what "then we would have created an allocation for something not really created" means | |
| 15:32:43 | mriedem | sdague: full monty | |
| 15:33:14 | bauzas | mriedem: I was thinking of the scheduler saying "okay, this one" then posting the allocations, then the boot requests goes trampled by the RPC bus | |
| 15:33:14 | mriedem | bauzas: for the comptue isn't down, but the instance is shelved-offloaded, | |
| 15:33:20 | mriedem | i guess when we shelve offload, do we deallocate? | |
| 15:33:38 | cdent | I think there's some misconceptions going on here | |
| 15:34:31 | mordred | cdent: people have the wrong idea about something? bah | |
| 15:34:45 | jaypipes | mriedem: yeah, it's a latent bug I think. we don't currently deallocate orphaned instances like we should. | |
| 15:34:55 | mriedem | lyarwood: i think we should add a microversion to cinder v3 to fix some of the volume type encryption api stuff, like remove the need for the full classname in the provider entry, just say 'luks' or 'cryptsetup' | |
| 15:35:16 | cdent | yes, known bug: a dead compute node's allocations persist | |
| 15:35:21 | mriedem | lyarwood: and as part of that microversion change in cinder v3, we enforce the actual providers using schema validation, rather than let it be free-form like it is today | |
| 15:35:58 | mriedem | cdent: jaypipes: how about shelve offload? | |
| 15:36:26 | openstack | Launchpad bug 1661312 in OpenStack Compute (nova) "Evacuation will corrupt instance allocations" [Undecided,New] | |
| 15:36:26 | cdent | there's also this: https://bugs.launchpad.net/nova/+bug/1661312 | |
| 15:36:27 | mriedem | because when we unshelve, the instance could end up on a different host | |
| 15:36:41 | jaypipes | mriedem: no idea. would need to check. not sure if the compute manager calls an RT method on shelve_offload(). | |
| 15:36:46 | mriedem | and if it's on a different host, we're then counting allocatoins twice for the same instnace | |
| 15:38:10 | cdent | since allocations are keyed by consumer id, and we set them (as in fully replace them) the situation is a bit cleaner than it might initially appear | |
| 15:38:55 | mriedem | but when we do claims in the scheduler, | |
| 15:39:04 | mriedem | aren't we aggregating the total allocations for a given resource provider? | |
| 15:39:25 | mriedem | and if we don't remove the allocations for a shelved offloaded consumer on that compute resource provider, it's a lie about actual resource allocation right? | |
| 15:39:32 | cdent | if we are puting an allocation, it will be for the instance uuid, no matter what it uses | |
| 15:39:51 | jaypipes | mriedem: no. | |
| 15:40:07 | mriedem | i guess when we do claims we're looking at inventory? | |
| 15:40:41 | mriedem | if i have an inventory of 5 apples, and your instance claims 3 apples, i have 2 apples left for sally to claim, right? | |
| 15:40:43 | jaypipes | mriedem: the "claim" is merely the transactional writing of allocation records against one or more resource providers for a single consumer ID. | |
| 15:41:16 | bauzas | mriedem: I think all the move operations need to update the allocations indeed | |
| 15:41:18 | jaypipes | mriedem: yes, that is a check that occurs entirely within the POST /allocations context in the placement API though. | |
| 15:41:42 | mriedem | jaypipes: don't we POST allocations to a specific resource provider though? | |
| 15:41:42 | bauzas | I need a second pass of thoughts | |
| 15:42:09 | jaypipes | mriedem: you will get a 409 if, say, Alice gets an apple before Sally's attempt to get 2 apples. | |
| 15:42:09 | cdent | mriedem: no | |
| 15:42:22 | jaypipes | mriedem: no, allocations are multi-provider. | |
| 15:42:25 | cdent | mriedem: or rather, not quite | |
| 15:42:38 | gibi | mriedem: hi! I saw your ping from yesterday about the snapshot notification and I had some free time to put a patch togheter for that issue https://review.openstack.org/#/c/453077/ | |
| 15:42:50 | mriedem | resource_provider_uuid = allocation['resource_provider']['uuid'] | |
| 15:42:52 | jaypipes | mriedem: for instance, a compute host provider and a shared storage pool provider might be involved in the same POST /allocations for a consumer (instance UUID) | |
| 15:43:13 | mriedem | but the allocation is still tied to a resource provider | |
| 15:43:14 | mriedem | one or more | |
| 15:43:42 | jaypipes | mriedem: the allocation record is, yes. but the entire "claim" is a *set* of allocation records for a single consumer (the instance) | |
| 15:43:43 | mriedem | the scheduler is asking placement for the list of available resource providers given some vcpu/ram/disk constraints, right? | |
| 15:43:55 | jaypipes | yep | |
| 15:43:58 | mriedem | and then takes that list, gets the compute nodes from the db, and filters them | |
| 15:44:03 | jaypipes | yep | |
| 15:44:11 | mriedem | so then i have maybe 3 available hosts, i pick one, and make my claim | |
| 15:44:18 | mriedem | which is a POST allocation on that resource provider, yes? | |
| 15:44:22 | jaypipes | no | |
| 15:44:23 | jaypipes | :) | |
| 15:44:51 | jaypipes | it would be a POST /allocations and the compute node resource provider would be one of the resource providers involved in the allocation request. | |
| 15:45:18 | jaypipes | mriedem: again, the reason this is like that is because you can claim resources against >1 provider in a single request. | |
| 15:45:21 | mriedem | i think i'm trying to say the same thing | |
| 15:45:32 | mriedem | but in the less formal way | |
| 15:45:54 | mriedem | me ugg, me want 2 apples, you mug have 3 apples, me claim 2 of your 3 apples, yes? | |
| 15:46:21 | jaypipes | yes | |
| 15:46:30 | mriedem | i get the multiple provider thing, that's fine, i need some compute resources, i need some shared storage resources from another provider, whatever | |
| 15:46:38 | cdent | (just to be pedantic the currently available method is PUT /allocations/{consumer_uuid} and I think we can continue using that one for the time being) | |
| 15:47:20 | mriedem | ok, so i think we've established that the scheduler has 'claimed' resources on a provider, or set of providers, by putting an allocation for a given instance (consumer) on those providers, | |
| 15:47:24 | mriedem | and then sends the build request to that host | |
| 15:47:26 | mriedem | to build the instance | |
| 15:48:01 | mriedem | since we can't reschedule, i guess i won't ask about that | |
| 15:48:20 | mriedem | so when we shelve offload an instance, do we deallocate for that consumer on it's resource providers | |
| 15:48:41 | mriedem | or when we delete the instance, and the compute host is down so we can't rpc cast to the compute to do the delete, do we deallocate from the api | |
| 15:48:44 | mriedem | i think the answer to both is no | |
| 15:49:16 | jaypipes | yeah, I suspect you are correct about that. but would be easy enough to fix in a bug report. | |
| 15:50:08 | mriedem | i may be wrong about the shelve offload case, because i remember alaski pointing out a super gross tight coupling of instance vm_state checking in the RT | |
| 15:50:21 | mriedem | yup ALLOW_RESOURCE_REMOVAL = [DELETED, SHELVED_OFFLOADED] | |
| 15:50:30 | mriedem | RT._update_usage_from_instance | |
| 15:50:32 | mriedem | checks that | |
| 15:50:57 | jaypipes | ok | |
| 15:51:18 | mriedem | which passes sign=-1 to update_instance_allocation to call _delete_allocation_for_instance | |
| 15:51:19 | mriedem | cool | |
| 15:51:29 | mriedem | we still fail the local delete case probably | |
| 15:51:39 | mriedem | which is maybe related to dansmith's bug about evacuate | |
| 16:01:04 | dansmith | sorry I was on a call, and have only skimmed the scrollback | |
| 16:01:06 | openstack | Launchpad bug 1661312 in OpenStack Compute (nova) "Evacuation will corrupt instance allocations" [Undecided,New] | |
| 16:01:06 | mriedem | dansmith: in https://bugs.launchpad.net/nova/+bug/1661312 where does step 4 happen? | |
| 16:01:19 | cdent | jaypipes, bauzas, mriedem I left a longer comment on the spec to remind us of some of these things | |
| 16:01:19 | openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Undecided,New] | |
| 16:01:19 | mriedem | dansmith: jaypipes: cdent: created https://bugs.launchpad.net/nova/+bug/1679750 | |
| 16:01:21 | dansmith | jaypipes: if we end up off the compute node due to shelve or evacuate, won't we heal our way to deleting the allocations after a bit? | |
| 16:01:39 | dansmith | jaypipes: also for local delete, I would think | |
| 16:01:51 | cdent | thanks mriedem | |
| 16:01:58 | jaypipes | dansmith: need to look into it. we *may* but wasn't 100% sure. | |
| 16:02:11 | dansmith | jaypipes: I would think we do | |
| 16:02:21 | dansmith | jaypipes: it might take a bit, but.. should happen I think | |
| 16:02:30 | cdent | the periodic job in the RT is always trying to heal if it sees a gap | |
| 16:02:39 | dansmith | cdent: right | |
| 16:02:41 | cdent | so the problem comes up when there is no RT | |
| 16:02:47 | cdent | s/the/a/ | |
| 16:04:58 | dansmith | you mean in a scheduler-claims world? | |
| 16:05:41 | cdent | dansmith: no, in the world where we want the representation of available resources to be correct | |
| 16:06:27 | cdent | it doesn't always have practical impact if a compute nodes is down but in the placement db but it does mean that our global total of eg vcpus is wrong | |
| 16:07:01 | cdent | but since they self register | |
| 16:07:01 | cdent | it would be nice that when a compute node went down , it was removed from the placement db | |
| 16:07:18 | cdent | We don't currently have anything in place to clean up when they die, do we? | |
| 16:07:27 | dansmith | I dunno about that.. for instances that aren't evacuated, those are still valid allocations, yes? | |