| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-04 | |||
| 15:36:41 | jaypipes | mriedem: no idea. would need to check. not sure if the compute manager calls an RT method on shelve_offload(). | |
| 15:36:46 | mriedem | and if it's on a different host, we're then counting allocatoins twice for the same instnace | |
| 15:38:10 | cdent | since allocations are keyed by consumer id, and we set them (as in fully replace them) the situation is a bit cleaner than it might initially appear | |
| 15:38:55 | mriedem | but when we do claims in the scheduler, | |
| 15:39:04 | mriedem | aren't we aggregating the total allocations for a given resource provider? | |
| 15:39:25 | mriedem | and if we don't remove the allocations for a shelved offloaded consumer on that compute resource provider, it's a lie about actual resource allocation right? | |
| 15:39:32 | cdent | if we are puting an allocation, it will be for the instance uuid, no matter what it uses | |
| 15:39:51 | jaypipes | mriedem: no. | |
| 15:40:07 | mriedem | i guess when we do claims we're looking at inventory? | |
| 15:40:41 | mriedem | if i have an inventory of 5 apples, and your instance claims 3 apples, i have 2 apples left for sally to claim, right? | |
| 15:40:43 | jaypipes | mriedem: the "claim" is merely the transactional writing of allocation records against one or more resource providers for a single consumer ID. | |
| 15:41:16 | bauzas | mriedem: I think all the move operations need to update the allocations indeed | |
| 15:41:18 | jaypipes | mriedem: yes, that is a check that occurs entirely within the POST /allocations context in the placement API though. | |
| 15:41:42 | bauzas | I need a second pass of thoughts | |
| 15:41:42 | mriedem | jaypipes: don't we POST allocations to a specific resource provider though? | |
| 15:42:09 | cdent | mriedem: no | |
| 15:42:09 | jaypipes | mriedem: you will get a 409 if, say, Alice gets an apple before Sally's attempt to get 2 apples. | |
| 15:42:22 | jaypipes | mriedem: no, allocations are multi-provider. | |
| 15:42:25 | cdent | mriedem: or rather, not quite | |
| 15:42:38 | gibi | mriedem: hi! I saw your ping from yesterday about the snapshot notification and I had some free time to put a patch togheter for that issue https://review.openstack.org/#/c/453077/ | |
| 15:42:50 | mriedem | resource_provider_uuid = allocation['resource_provider']['uuid'] | |
| 15:42:52 | jaypipes | mriedem: for instance, a compute host provider and a shared storage pool provider might be involved in the same POST /allocations for a consumer (instance UUID) | |
| 15:43:13 | mriedem | but the allocation is still tied to a resource provider | |
| 15:43:14 | mriedem | one or more | |
| 15:43:42 | jaypipes | mriedem: the allocation record is, yes. but the entire "claim" is a *set* of allocation records for a single consumer (the instance) | |
| 15:43:43 | mriedem | the scheduler is asking placement for the list of available resource providers given some vcpu/ram/disk constraints, right? | |
| 15:43:55 | jaypipes | yep | |
| 15:43:58 | mriedem | and then takes that list, gets the compute nodes from the db, and filters them | |
| 15:44:03 | jaypipes | yep | |
| 15:44:11 | mriedem | so then i have maybe 3 available hosts, i pick one, and make my claim | |
| 15:44:18 | mriedem | which is a POST allocation on that resource provider, yes? | |
| 15:44:22 | jaypipes | no | |
| 15:44:23 | jaypipes | :) | |
| 15:44:51 | jaypipes | it would be a POST /allocations and the compute node resource provider would be one of the resource providers involved in the allocation request. | |
| 15:45:18 | jaypipes | mriedem: again, the reason this is like that is because you can claim resources against >1 provider in a single request. | |
| 15:45:21 | mriedem | i think i'm trying to say the same thing | |
| 15:45:32 | mriedem | but in the less formal way | |
| 15:45:54 | mriedem | me ugg, me want 2 apples, you mug have 3 apples, me claim 2 of your 3 apples, yes? | |
| 15:46:21 | jaypipes | yes | |
| 15:46:30 | mriedem | i get the multiple provider thing, that's fine, i need some compute resources, i need some shared storage resources from another provider, whatever | |
| 15:46:38 | cdent | (just to be pedantic the currently available method is PUT /allocations/{consumer_uuid} and I think we can continue using that one for the time being) | |
| 15:47:20 | mriedem | ok, so i think we've established that the scheduler has 'claimed' resources on a provider, or set of providers, by putting an allocation for a given instance (consumer) on those providers, | |
| 15:47:24 | mriedem | and then sends the build request to that host | |
| 15:47:26 | mriedem | to build the instance | |
| 15:48:01 | mriedem | since we can't reschedule, i guess i won't ask about that | |
| 15:48:20 | mriedem | so when we shelve offload an instance, do we deallocate for that consumer on it's resource providers | |
| 15:48:41 | mriedem | or when we delete the instance, and the compute host is down so we can't rpc cast to the compute to do the delete, do we deallocate from the api | |
| 15:48:44 | mriedem | i think the answer to both is no | |
| 15:49:16 | jaypipes | yeah, I suspect you are correct about that. but would be easy enough to fix in a bug report. | |
| 15:50:08 | mriedem | i may be wrong about the shelve offload case, because i remember alaski pointing out a super gross tight coupling of instance vm_state checking in the RT | |
| 15:50:21 | mriedem | yup ALLOW_RESOURCE_REMOVAL = [DELETED, SHELVED_OFFLOADED] | |
| 15:50:30 | mriedem | RT._update_usage_from_instance | |
| 15:50:32 | mriedem | checks that | |
| 15:50:57 | jaypipes | ok | |
| 15:51:18 | mriedem | which passes sign=-1 to update_instance_allocation to call _delete_allocation_for_instance | |
| 15:51:19 | mriedem | cool | |
| 15:51:29 | mriedem | we still fail the local delete case probably | |
| 15:51:39 | mriedem | which is maybe related to dansmith's bug about evacuate | |
| 16:01:04 | dansmith | sorry I was on a call, and have only skimmed the scrollback | |
| 16:01:06 | mriedem | dansmith: in https://bugs.launchpad.net/nova/+bug/1661312 where does step 4 happen? | |
| 16:01:06 | openstack | Launchpad bug 1661312 in OpenStack Compute (nova) "Evacuation will corrupt instance allocations" [Undecided,New] | |
| 16:01:19 | mriedem | dansmith: jaypipes: cdent: created https://bugs.launchpad.net/nova/+bug/1679750 | |
| 16:01:19 | openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Undecided,New] | |
| 16:01:19 | cdent | jaypipes, bauzas, mriedem I left a longer comment on the spec to remind us of some of these things | |
| 16:01:21 | dansmith | jaypipes: if we end up off the compute node due to shelve or evacuate, won't we heal our way to deleting the allocations after a bit? | |
| 16:01:39 | dansmith | jaypipes: also for local delete, I would think | |
| 16:01:51 | cdent | thanks mriedem | |
| 16:01:58 | jaypipes | dansmith: need to look into it. we *may* but wasn't 100% sure. | |
| 16:02:11 | dansmith | jaypipes: I would think we do | |
| 16:02:21 | dansmith | jaypipes: it might take a bit, but.. should happen I think | |
| 16:02:30 | cdent | the periodic job in the RT is always trying to heal if it sees a gap | |
| 16:02:39 | dansmith | cdent: right | |
| 16:02:41 | cdent | so the problem comes up when there is no RT | |
| 16:02:47 | cdent | s/the/a/ | |
| 16:04:58 | dansmith | you mean in a scheduler-claims world? | |
| 16:05:41 | cdent | dansmith: no, in the world where we want the representation of available resources to be correct | |
| 16:06:27 | cdent | it doesn't always have practical impact if a compute nodes is down but in the placement db but it does mean that our global total of eg vcpus is wrong | |
| 16:07:01 | cdent | it would be nice that when a compute node went down , it was removed from the placement db | |
| 16:07:01 | cdent | but since they self register | |
| 16:07:18 | cdent | We don't currently have anything in place to clean up when they die, do we? | |
| 16:07:27 | dansmith | I dunno about that.. for instances that aren't evacuated, those are still valid allocations, yes? | |
| 16:08:01 | dansmith | if you think about when we do quota by counting things in placement, | |
| 16:08:06 | cdent | I was thinking more in the sense of full on node crash | |
| 16:08:13 | dansmith | one compute node going offline shouldn't free up quota for you and then get you in the red once it comes back | |
| 16:08:44 | dansmith | even if the compute node is off and cold, if there are unevacuated (or undeleted) instances there, the allocations should stay, IMHO | |
| 16:08:59 | mriedem | dansmith: i agree with that | |
| 16:09:00 | cdent | the quota case nicely complicates things | |
| 16:09:01 | dansmith | i.e. I wouldn't think placement should know about liveness | |
| 16:09:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1679750 is about when you delete an instance on a dead node | |
| 16:09:05 | openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Undecided,New] | |
| 16:09:17 | dansmith | if nova has an instance, then placement has a record of allocations.. nothing more than that | |
| 16:09:34 | cdent | because if we are using allocations for usage, is a cold node supposed to count against my quota? that's not my (the user's) fault and I want to run some instances | |
| 16:09:45 | cdent | s/usage/quota measurment/ | |
| 16:09:48 | dansmith | cdent: it should IMHO yeah | |
| 16:10:01 | dansmith | cdent: you're free to delete the instance and get the quota back | |
| 16:10:05 | dansmith | the flip side is, | |
| 16:10:13 | dansmith | if a compute node goes down, you get more quota and boot a thing, | |
| 16:10:31 | dansmith | when the compute node comes back, should the provider be free to nuke the instance that was running on that node since you'd now be over quota? | |
| 16:10:33 | cdent | hmmm. yeah. weird. | |
| 16:10:41 | cdent | blech | |