| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-09 | |||
| 12:35:09 | gibi | sean-k-mooney: you update the allocation of a consumer or you create / delete consumers | |
| 12:35:13 | gibi | there is no rename consumer | |
| 12:35:25 | gibi | and there is no partial allocation update | |
| 12:35:28 | gibi | just total one | |
| 12:35:58 | sean-k-mooney | ack | |
| 12:36:27 | gibi | probably the easyest thing is to implement rename consumer, the partial allocation update feels hard | |
| 12:37:10 | sean-k-mooney | ya so we would move the source allocation to the migration uuid create a new allocation for the vm using its uuid | |
| 12:37:18 | gibi | then we can reimplement the allocation move from VM -> migration with the rename | |
| 12:37:24 | sean-k-mooney | then have the dest delete the migration allocation after evac | |
| 12:38:02 | sean-k-mooney | which will avoid leaking the allcoation in placment if the source compute never comes back | |
| 12:38:13 | sean-k-mooney | well actully no | |
| 12:38:30 | sean-k-mooney | we have to be careful | |
| 12:38:42 | sean-k-mooney | to make sure if the evac fails we can jsut evac again | |
| 12:38:53 | sean-k-mooney | so the dest vm need to have the migration uuid | |
| 12:38:53 | gibi | yeah, I feel there is a reason why we kept the source allocation for the source compute to clean up | |
| 12:39:14 | sean-k-mooney | untill it succeed then we cna remvoe the source vm allcoation and rename the migration allcotion | |
| 12:39:36 | gibi | having the migration uuid to allocate on the test is a surgery as today we just call the scheduler and that always uses the instance uuid to allocate | |
| 12:39:43 | gibi | s/test/dest/ | |
| 12:40:08 | gibi | all the moves are using the migration uuid on the dest so the scheduler don't have to be branched for moves | |
| 12:40:16 | gibi | sorry on the source | |
| 12:40:23 | sean-k-mooney | ok | |
| 12:40:32 | sean-k-mooney | well we can just use our exisign patteren | |
| 12:40:41 | sean-k-mooney | but rename would make it simpler | |
| 12:41:02 | gibi | rename would be needed to solve the above placement-reject-evac-as-source-is-overallocated issue | |
| 12:41:02 | sean-k-mooney | it might also be useful for blazar | |
| 12:42:15 | sean-k-mooney | this would obviously be a api change right | |
| 12:42:21 | gibi | yepp | |
| 12:42:43 | sean-k-mooney | technically there are no filed change and we are just chanign form a 400 to 200 | |
| 12:42:58 | sean-k-mooney | but i assume that still need a microverion bump | |
| 12:43:02 | sean-k-mooney | so not backportable? | |
| 12:43:30 | gibi | as the 400 wasnt caused by a bug, the transformation that to 200 is a microversion bump | |
| 12:44:17 | sean-k-mooney | ok so i dont really see a way to fix this in code for exisitng release then | |
| 12:44:37 | sean-k-mooney | operators will just need to fix the RP inventories | |
| 12:44:49 | sean-k-mooney | e.g. set capasty to max int or something | |
| 12:44:59 | sean-k-mooney | the comptue node would fix it when it started back up | |
| 12:45:12 | gibi | basically the operator needs to resolve the overallocation | |
| 12:45:18 | sean-k-mooney | but while its down you can use osc to manually update it | |
| 12:45:26 | gibi | either by deleting allocations or by increasing inventory | |
| 12:45:35 | sean-k-mooney | gibi: right but if the host is down they cant really do deletes | |
| 12:45:49 | gibi | ture | |
| 12:45:50 | gibi | true | |
| 12:45:57 | gibi | the change the inventory via OSC | |
| 12:46:02 | gibi | then | |
| 12:46:06 | sean-k-mooney | yep | |
| 12:46:10 | gibi | that is the way | |
| 12:46:22 | gibi | and also investigate how you ended up in overallocation | |
| 12:46:28 | gibi | as placement should not allow that | |
| 12:47:48 | sean-k-mooney | it normally happens if you change things like cpu_dedicated_set or the amount of hugepages ectra | |
| 12:48:23 | sean-k-mooney | or actully more commanly the ram/disk/cpu allcoation ratios | |
| 12:49:21 | sean-k-mooney | im sure there are other ways too but i have most often seen it due to operators chanign config such that the current vms nolonger fit | |
| 12:50:25 | gibi | hm, maybe we should add a WARNING for the compute log / placement log if there is overallocation detected so the admin will detect the misconfiguration | |
| 12:50:47 | sean-k-mooney | to the perodic | |
| 12:51:02 | sean-k-mooney | update_avaialable_resouces when we recalulate the placment update | |
| 12:51:24 | sean-k-mooney | ya we could | |
| 12:52:06 | sean-k-mooney | im not sure how spamy that would be but it does indeicate the might need to heal allcoaiton or other wise investigate why | |
| 14:07:00 | gibi | sean-k-mooney: actually placement already has a warning | |
| 14:07:02 | gibi | sean-k-mooney: "WARNING placement.objects.resource_provider [None req-6f2253b9-a195-4bf9-8c7e-2a32271a8c0c admin admin] Resource provider 935b9ad6-d7d1-4b5a-bb49-022acbba7c72 is now over-capacity for VCPU" | |
| 14:07:23 | gibi | when I set the allocation ratio to lower to induce overallocation | |
| 14:15:27 | opendevreview | Balazs Gibizer proposed openstack/placement master: DNM: extra logs to troubleshoot overcapacity https://review.opendev.org/c/openstack/placement/+/808083 | |
| 15:33:10 | sean-k-mooney | ah nice | |
| 15:47:17 | sean-k-mooney | although if the compute agent is down you might not see that | |
| 17:03:11 | gibi | elodilles_pto: there is probably a stable only bug here https://bugs.launchpad.net/nova/+bug/1941819 but as it is probably only affect stein and older which are in EM I don't think I will spend time fixing it. Maybe the bug author try it. | |
| 18:40:37 | legochen | hey nova experts, one question - can someone point me the best practice about configuring nova-scheduler filter to distribute VMs to user-specified cabinets? | |
| 18:41:31 | dansmith | user-specified, meaning "at boot a user specifies where this should go" ? | |
| 18:41:35 | dansmith | or did you mean user-specific? | |
| 18:42:05 | legochen | user-specified, meaning "at boot a user specifies where this should go" ? <= yes | |
| 18:42:24 | dansmith | in general this is not a thing nova allows or intends to allow, with one exception: AZs | |
| 18:42:42 | dansmith | users can choose AZs, so if you want them to be able to choose, make AZs for them to specify | |
| 18:43:00 | dansmith | you can do an AZ per site, or per aisle, or per rack or something | |
| 18:43:58 | legochen | For example, I have multiple cabients in data center, users want to distribute their VMs to different cabinets equally in order to avoid SPOF of tor switch or power stuff. | |
| 18:44:15 | dansmith | that's what AZs are for | |
| 18:44:32 | legochen | hmm, per cabinet per AZ seems not that reasonable to me :( | |
| 18:49:23 | legochen | I was thinking to configure per aggregate per cabinet. And set property - cabient=A for aggregate A, cabinet=B for aggregate B…. then users can specify —hint cabinet=A while creating a VM. | |
| 18:49:50 | dansmith | that's what AZs are for | |
| 19:18:09 | opendevreview | xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126 | |
| #openstack-nova - 2021-09-10 | |||
| 02:12:18 | opendevreview | xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126 | |
| 02:41:15 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 06:08:09 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:00:49 | admin1 | seeking info on what happens during a migration .. say i have h1 ( hypervisor1 ) and h2 ( hypervisor2 ) .. when I want to migrate vm1 on h1 -> h2, what makes the ssh conection .. is it h1 nova -> h2 nova .. or h1 libvirt -> h2 libvirt or h1 root -> h2 root .. | |
| 07:24:31 | gibi | bauzas: o/ friendly reminder that we need to land melwitt's placemnt fix in Xena RC1 https://review.opendev.org/c/openstack/placement/+/807014 | |
| 07:25:08 | gibi | melwitt, dansmith: we are out of EU cores for now, so could you please look at a small doc patch before RC1 https://review.opendev.org/c/openstack/nova/+/807564 ? | |
| 07:25:38 | gibi | melwitt, dansmith: also the reno prelude is ready https://review.opendev.org/c/openstack/nova/+/807786 | |
| 07:56:14 | bauzas | gibi: yeah I know and I started to review it | |
| 07:56:26 | bauzas | but y'know how it's a bit difficult | |
| 07:56:42 | bauzas | + loving escalations during the last 2 days :( | |
| 07:57:17 | gibi | bauzas: thanks. feel free to ask questions about the patch I should be able to answer them | |
| 07:57:21 | gibi | ahh escalations. :/ | |
| 07:59:11 | bauzas | don't know what happended | |
| 07:59:31 | bauzas | maybe because of the moon or because of puppets | |
| 07:59:43 | bauzas | but looks like we have a lot of customers asking us to help them | |
| 08:00:01 | gibi | maybe they are back from summer vacation :) | |
| 08:01:16 | bauzas | they're not French :p | |
| 08:08:41 | gibi | then they are just nice and waited for the French to return from their summer vacation :P | |
| 08:28:07 | admin1 | one more question .. if cold migration works but live does not -- ceph backend .. where do i need to start looking .. the nova logs ( debug true ) does not provide any actionable insight .. | |
| 09:30:42 | gibi | bauzas: do you agree with my triage here https://bugs.launchpad.net/nova/+bug/1939856 that it is probably a new feature not a bugfix? | |
| 09:34:44 | bauzas | gibi: fwiw, I +wd https://review.opendev.org/c/openstack/nova/+/792357 | |
| 09:35:04 | gibi | bauzas: thanks | |
| 09:35:08 | bauzas | so we can start to use the new label | |
| 09:35:28 | bauzas | but then I'll provide a new change explaining what I'd like to have to helping contributors | |
| 09:35:40 | gibi | OK, I will check that | |