| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-09 | |||
| 12:12:20 | sean-k-mooney | have you filed a bug for this | |
| 12:12:39 | sean-k-mooney | placment should not be made aware of evuacation or other lifecycle operations explcitly | |
| 12:13:20 | songwenping_ | havenot filed bug. | |
| 12:13:25 | sean-k-mooney | so im not sure we could proceed with any apprcoh that required modifciation of placment to make it expcitly aware of evacuation | |
| 12:13:48 | sean-k-mooney | but we might be able to handel the error condition with corupt data | |
| 12:13:57 | sean-k-mooney | so that it woudl not fail | |
| 12:17:56 | gibi | songwenping_: during evacuation the source node allocation of the VM is kept and the dest node allocation is added to it | |
| 12:18:18 | gibi | so for an evacuated VM you will see allocation on the source and dest nodes _until_ the source compute is down | |
| 12:18:40 | gibi | when the source compute node is recovered it will delete the allocation on the source node for the already evacuated VMs | |
| 12:19:27 | sean-k-mooney | gibi: for evacuate should we not be using the migration uuid for the evaucation | |
| 12:19:32 | sean-k-mooney | alocaltions | |
| 12:19:36 | sean-k-mooney | like we do for rezise | |
| 12:20:24 | songwenping_ | gibi: yeah, this is right workflow, but placement check the source node capacity before evacuate. | |
| 12:21:12 | gibi | sean-k-mooney: we never switched the evac workflow to migration allocations | |
| 12:21:15 | gibi | there is todo in the code | |
| 12:21:40 | sean-k-mooney | ah ok | |
| 12:22:02 | sean-k-mooney | i guess that would be the correct way to fix this then | |
| 12:23:08 | gibi | songwenping_: do you have a pointer where nova checks the source node capacity during evac? | |
| 12:23:35 | sean-k-mooney | its in placment https://github.com/openstack/placement/blob/master/placement/objects/allocation.py#L120 | |
| 12:24:43 | songwenping_ | yes, it's placement checking. | |
| 12:25:41 | gibi | what placement does during any kind of allocation update, is to replace the existing allocation of the consumer (VM in this case) with the new requested allocation | |
| 12:26:01 | gibi | so if the node is not overallocated then the replace_all should not fail | |
| 12:26:18 | gibi | if you got your node overallocated then I guess the allocation update can fail | |
| 12:27:07 | gibi | as placement will not allow you to overallocate | |
| 12:27:10 | songwenping_ | yes, i means why we check the source node overallocated as the vm will be evacuated to the dest node. | |
| 12:27:38 | gibi | placement does not know these things, what placement sees is a request to update the allocation of a consumer | |
| 12:27:54 | gibi | and the old allocation has resources on the source node, the new allocation has resource on the source node and on the dest node | |
| 12:28:06 | gibi | placement goes and replaces the whole old allocation with the new allocation | |
| 12:28:33 | gibi | if you got your compute overallocated then simply removing the source allocation and adding it back at this step fails | |
| 12:29:07 | gibi | sean-k-mooney: btw, moving to migration allocation will not solve this as there we move the VM allocation to the migration allocation and that move will fail for the same reason | |
| 12:29:27 | gibi | in short, if you overallocated your compute then placement will reject allocation updates on that compute | |
| 12:29:41 | gibi | you need to resolve the overallocation | |
| 12:29:42 | sean-k-mooney | it should reject new allcoations | |
| 12:29:50 | gibi | no | |
| 12:29:59 | sean-k-mooney | but if we dont change the reosuce in a currnt oen it shoudl work no | |
| 12:30:12 | gibi | placement implements replace_all for allocation update | |
| 12:30:23 | gibi | so it is a delete + create in the same transaction | |
| 12:30:32 | gibi | but after the delte the compute is full | |
| 12:30:36 | gibi | so create fails | |
| 12:30:43 | songwenping_ | on our product env, there are some unknow reasons that vm has allocations on two resource provider. | |
| 12:33:36 | gibi | sean-k-mooney: I mean I got that it would be nice to detect that the source node allocation does not change during the update and dont delete + re-create it, but still placement does not do that logic | |
| 12:34:08 | sean-k-mooney | well i was hoping that a simple uuid update woudl not triger this check | |
| 12:34:28 | gibi | sean-k-mooney: there is no way to update a consumer uuid | |
| 12:35:09 | gibi | sean-k-mooney: you update the allocation of a consumer or you create / delete consumers | |
| 12:35:13 | gibi | there is no rename consumer | |
| 12:35:25 | gibi | and there is no partial allocation update | |
| 12:35:28 | gibi | just total one | |
| 12:35:58 | sean-k-mooney | ack | |
| 12:36:27 | gibi | probably the easyest thing is to implement rename consumer, the partial allocation update feels hard | |
| 12:37:10 | sean-k-mooney | ya so we would move the source allocation to the migration uuid create a new allocation for the vm using its uuid | |
| 12:37:18 | gibi | then we can reimplement the allocation move from VM -> migration with the rename | |
| 12:37:24 | sean-k-mooney | then have the dest delete the migration allocation after evac | |
| 12:38:02 | sean-k-mooney | which will avoid leaking the allcoation in placment if the source compute never comes back | |
| 12:38:13 | sean-k-mooney | well actully no | |
| 12:38:30 | sean-k-mooney | we have to be careful | |
| 12:38:42 | sean-k-mooney | to make sure if the evac fails we can jsut evac again | |
| 12:38:53 | sean-k-mooney | so the dest vm need to have the migration uuid | |
| 12:38:53 | gibi | yeah, I feel there is a reason why we kept the source allocation for the source compute to clean up | |
| 12:39:14 | sean-k-mooney | untill it succeed then we cna remvoe the source vm allcoation and rename the migration allcotion | |
| 12:39:36 | gibi | having the migration uuid to allocate on the test is a surgery as today we just call the scheduler and that always uses the instance uuid to allocate | |
| 12:39:43 | gibi | s/test/dest/ | |
| 12:40:08 | gibi | all the moves are using the migration uuid on the dest so the scheduler don't have to be branched for moves | |
| 12:40:16 | gibi | sorry on the source | |
| 12:40:23 | sean-k-mooney | ok | |
| 12:40:32 | sean-k-mooney | well we can just use our exisign patteren | |
| 12:40:41 | sean-k-mooney | but rename would make it simpler | |
| 12:41:02 | gibi | rename would be needed to solve the above placement-reject-evac-as-source-is-overallocated issue | |
| 12:41:02 | sean-k-mooney | it might also be useful for blazar | |
| 12:42:15 | sean-k-mooney | this would obviously be a api change right | |
| 12:42:21 | gibi | yepp | |
| 12:42:43 | sean-k-mooney | technically there are no filed change and we are just chanign form a 400 to 200 | |
| 12:42:58 | sean-k-mooney | but i assume that still need a microverion bump | |
| 12:43:02 | sean-k-mooney | so not backportable? | |
| 12:43:30 | gibi | as the 400 wasnt caused by a bug, the transformation that to 200 is a microversion bump | |
| 12:44:17 | sean-k-mooney | ok so i dont really see a way to fix this in code for exisitng release then | |
| 12:44:37 | sean-k-mooney | operators will just need to fix the RP inventories | |
| 12:44:49 | sean-k-mooney | e.g. set capasty to max int or something | |
| 12:44:59 | sean-k-mooney | the comptue node would fix it when it started back up | |
| 12:45:12 | gibi | basically the operator needs to resolve the overallocation | |
| 12:45:18 | sean-k-mooney | but while its down you can use osc to manually update it | |
| 12:45:26 | gibi | either by deleting allocations or by increasing inventory | |
| 12:45:35 | sean-k-mooney | gibi: right but if the host is down they cant really do deletes | |
| 12:45:49 | gibi | ture | |
| 12:45:50 | gibi | true | |
| 12:45:57 | gibi | the change the inventory via OSC | |
| 12:46:02 | gibi | then | |
| 12:46:06 | sean-k-mooney | yep | |
| 12:46:10 | gibi | that is the way | |
| 12:46:22 | gibi | and also investigate how you ended up in overallocation | |
| 12:46:28 | gibi | as placement should not allow that | |
| 12:47:48 | sean-k-mooney | it normally happens if you change things like cpu_dedicated_set or the amount of hugepages ectra | |
| 12:48:23 | sean-k-mooney | or actully more commanly the ram/disk/cpu allcoation ratios | |
| 12:49:21 | sean-k-mooney | im sure there are other ways too but i have most often seen it due to operators chanign config such that the current vms nolonger fit | |
| 12:50:25 | gibi | hm, maybe we should add a WARNING for the compute log / placement log if there is overallocation detected so the admin will detect the misconfiguration | |
| 12:50:47 | sean-k-mooney | to the perodic | |
| 12:51:02 | sean-k-mooney | update_avaialable_resouces when we recalulate the placment update | |
| 12:51:24 | sean-k-mooney | ya we could | |
| 12:52:06 | sean-k-mooney | im not sure how spamy that would be but it does indeicate the might need to heal allcoaiton or other wise investigate why | |
| 14:07:00 | gibi | sean-k-mooney: actually placement already has a warning | |
| 14:07:02 | gibi | sean-k-mooney: "WARNING placement.objects.resource_provider [None req-6f2253b9-a195-4bf9-8c7e-2a32271a8c0c admin admin] Resource provider 935b9ad6-d7d1-4b5a-bb49-022acbba7c72 is now over-capacity for VCPU" | |