Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-09
12:28:33 gibi if you got your compute overallocated then simply removing the source allocation and adding it back at this step fails
12:29:07 gibi sean-k-mooney: btw, moving to migration allocation will not solve this as there we move the VM allocation to the migration allocation and that move will fail for the same reason
12:29:27 gibi in short, if you overallocated your compute then placement will reject allocation updates on that compute
12:29:41 gibi you need to resolve the overallocation
12:29:42 sean-k-mooney it should reject new allcoations
12:29:50 gibi no
12:29:59 sean-k-mooney but if we dont change the reosuce in a currnt oen it shoudl work no
12:30:12 gibi placement implements replace_all for allocation update
12:30:23 gibi so it is a delete + create in the same transaction
12:30:32 gibi but after the delte the compute is full
12:30:36 gibi so create fails
12:30:43 songwenping_ on our product env, there are some unknow reasons that vm has allocations on two resource provider.
12:33:36 gibi sean-k-mooney: I mean I got that it would be nice to detect that the source node allocation does not change during the update and dont delete + re-create it, but still placement does not do that logic
12:34:08 sean-k-mooney well i was hoping that a simple uuid update woudl not triger this check
12:34:28 gibi sean-k-mooney: there is no way to update a consumer uuid
12:35:09 gibi sean-k-mooney: you update the allocation of a consumer or you create / delete consumers
12:35:13 gibi there is no rename consumer
12:35:25 gibi and there is no partial allocation update
12:35:28 gibi just total one
12:35:58 sean-k-mooney ack
12:36:27 gibi probably the easyest thing is to implement rename consumer, the partial allocation update feels hard
12:37:10 sean-k-mooney ya so we would move the source allocation to the migration uuid create a new allocation for the vm using its uuid
12:37:18 gibi then we can reimplement the allocation move from VM -> migration with the rename
12:37:24 sean-k-mooney then have the dest delete the migration allocation after evac
12:38:02 sean-k-mooney which will avoid leaking the allcoation in placment if the source compute never comes back
12:38:13 sean-k-mooney well actully no
12:38:30 sean-k-mooney we have to be careful
12:38:42 sean-k-mooney to make sure if the evac fails we can jsut evac again
12:38:53 gibi yeah, I feel there is a reason why we kept the source allocation for the source compute to clean up
12:38:53 sean-k-mooney so the dest vm need to have the migration uuid
12:39:14 sean-k-mooney untill it succeed then we cna remvoe the source vm allcoation and rename the migration allcotion
12:39:36 gibi having the migration uuid to allocate on the test is a surgery as today we just call the scheduler and that always uses the instance uuid to allocate
12:39:43 gibi s/test/dest/
12:40:08 gibi all the moves are using the migration uuid on the dest so the scheduler don't have to be branched for moves
12:40:16 gibi sorry on the source
12:40:23 sean-k-mooney ok
12:40:32 sean-k-mooney well we can just use our exisign patteren
12:40:41 sean-k-mooney but rename would make it simpler
12:41:02 sean-k-mooney it might also be useful for blazar
12:41:02 gibi rename would be needed to solve the above placement-reject-evac-as-source-is-overallocated issue
12:42:15 sean-k-mooney this would obviously be a api change right
12:42:21 gibi yepp
12:42:43 sean-k-mooney technically there are no filed change and we are just chanign form a 400 to 200
12:42:58 sean-k-mooney but i assume that still need a microverion bump
12:43:02 sean-k-mooney so not backportable?
12:43:30 gibi as the 400 wasnt caused by a bug, the transformation that to 200 is a microversion bump
12:44:17 sean-k-mooney ok so i dont really see a way to fix this in code for exisitng release then
12:44:37 sean-k-mooney operators will just need to fix the RP inventories
12:44:49 sean-k-mooney e.g. set capasty to max int or something
12:44:59 sean-k-mooney the comptue node would fix it when it started back up
12:45:12 gibi basically the operator needs to resolve the overallocation
12:45:18 sean-k-mooney but while its down you can use osc to manually update it
12:45:26 gibi either by deleting allocations or by increasing inventory
12:45:35 sean-k-mooney gibi: right but if the host is down they cant really do deletes
12:45:49 gibi ture
12:45:50 gibi true
12:45:57 gibi the change the inventory via OSC
12:46:02 gibi then
12:46:06 sean-k-mooney yep
12:46:10 gibi that is the way
12:46:22 gibi and also investigate how you ended up in overallocation
12:46:28 gibi as placement should not allow that
12:47:48 sean-k-mooney it normally happens if you change things like cpu_dedicated_set or the amount of hugepages ectra
12:48:23 sean-k-mooney or actully more commanly the ram/disk/cpu allcoation ratios
12:49:21 sean-k-mooney im sure there are other ways too but i have most often seen it due to operators chanign config such that the current vms nolonger fit
12:50:25 gibi hm, maybe we should add a WARNING for the compute log / placement log if there is overallocation detected so the admin will detect the misconfiguration
12:50:47 sean-k-mooney to the perodic
12:51:02 sean-k-mooney update_avaialable_resouces when we recalulate the placment update
12:51:24 sean-k-mooney ya we could
12:52:06 sean-k-mooney im not sure how spamy that would be but it does indeicate the might need to heal allcoaiton or other wise investigate why
14:07:00 gibi sean-k-mooney: actually placement already has a warning
14:07:02 gibi sean-k-mooney: "WARNING placement.objects.resource_provider [None req-6f2253b9-a195-4bf9-8c7e-2a32271a8c0c admin admin] Resource provider 935b9ad6-d7d1-4b5a-bb49-022acbba7c72 is now over-capacity for VCPU"
14:07:23 gibi when I set the allocation ratio to lower to induce overallocation
14:15:27 opendevreview Balazs Gibizer proposed openstack/placement master: DNM: extra logs to troubleshoot overcapacity https://review.opendev.org/c/openstack/placement/+/808083
15:33:10 sean-k-mooney ah nice
15:47:17 sean-k-mooney although if the compute agent is down you might not see that
17:03:11 gibi elodilles_pto: there is probably a stable only bug here https://bugs.launchpad.net/nova/+bug/1941819 but as it is probably only affect stein and older which are in EM I don't think I will spend time fixing it. Maybe the bug author try it.
18:40:37 legochen hey nova experts, one question - can someone point me the best practice about configuring nova-scheduler filter to distribute VMs to user-specified cabinets?
18:41:31 dansmith user-specified, meaning "at boot a user specifies where this should go" ?
18:41:35 dansmith or did you mean user-specific?
18:42:05 legochen user-specified, meaning "at boot a user specifies where this should go" ? <= yes
18:42:24 dansmith in general this is not a thing nova allows or intends to allow, with one exception: AZs
18:42:42 dansmith users can choose AZs, so if you want them to be able to choose, make AZs for them to specify
18:43:00 dansmith you can do an AZ per site, or per aisle, or per rack or something
18:43:58 legochen For example, I have multiple cabients in data center, users want to distribute their VMs to different cabinets equally in order to avoid SPOF of tor switch or power stuff.
18:44:15 dansmith that's what AZs are for
18:44:32 legochen hmm, per cabinet per AZ seems not that reasonable to me :(
18:49:23 legochen I was thinking to configure per aggregate per cabinet. And set property - cabient=A for aggregate A, cabinet=B for aggregate B…. then users can specify —hint cabinet=A while creating a VM.
18:49:50 dansmith that's what AZs are for
19:18:09 opendevreview xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126
#openstack-nova - 2021-09-10
02:12:18 opendevreview xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126
02:41:15 opendevreview Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143
06:08:09 opendevreview Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143
07:00:49 admin1 seeking info on what happens during a migration .. say i have h1 ( hypervisor1 ) and h2 ( hypervisor2 ) .. when I want to migrate vm1 on h1 -> h2, what makes the ssh conection .. is it h1 nova -> h2 nova .. or h1 libvirt -> h2 libvirt or h1 root -> h2 root ..
07:24:31 gibi bauzas: o/ friendly reminder that we need to land melwitt's placemnt fix in Xena RC1 https://review.opendev.org/c/openstack/placement/+/807014
07:25:08 gibi melwitt, dansmith: we are out of EU cores for now, so could you please look at a small doc patch before RC1 https://review.opendev.org/c/openstack/nova/+/807564 ?
07:25:38 gibi melwitt, dansmith: also the reno prelude is ready https://review.opendev.org/c/openstack/nova/+/807786
07:56:14 bauzas gibi: yeah I know and I started to review it
07:56:26 bauzas but y'know how it's a bit difficult
07:56:42 bauzas + loving escalations during the last 2 days :(

Earlier   Later