| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-09 | |||
| 12:42:21 | gibi | yepp | |
| 12:42:43 | sean-k-mooney | technically there are no filed change and we are just chanign form a 400 to 200 | |
| 12:42:58 | sean-k-mooney | but i assume that still need a microverion bump | |
| 12:43:02 | sean-k-mooney | so not backportable? | |
| 12:43:30 | gibi | as the 400 wasnt caused by a bug, the transformation that to 200 is a microversion bump | |
| 12:44:17 | sean-k-mooney | ok so i dont really see a way to fix this in code for exisitng release then | |
| 12:44:37 | sean-k-mooney | operators will just need to fix the RP inventories | |
| 12:44:49 | sean-k-mooney | e.g. set capasty to max int or something | |
| 12:44:59 | sean-k-mooney | the comptue node would fix it when it started back up | |
| 12:45:12 | gibi | basically the operator needs to resolve the overallocation | |
| 12:45:18 | sean-k-mooney | but while its down you can use osc to manually update it | |
| 12:45:26 | gibi | either by deleting allocations or by increasing inventory | |
| 12:45:35 | sean-k-mooney | gibi: right but if the host is down they cant really do deletes | |
| 12:45:49 | gibi | ture | |
| 12:45:50 | gibi | true | |
| 12:45:57 | gibi | the change the inventory via OSC | |
| 12:46:02 | gibi | then | |
| 12:46:06 | sean-k-mooney | yep | |
| 12:46:10 | gibi | that is the way | |
| 12:46:22 | gibi | and also investigate how you ended up in overallocation | |
| 12:46:28 | gibi | as placement should not allow that | |
| 12:47:48 | sean-k-mooney | it normally happens if you change things like cpu_dedicated_set or the amount of hugepages ectra | |
| 12:48:23 | sean-k-mooney | or actully more commanly the ram/disk/cpu allcoation ratios | |
| 12:49:21 | sean-k-mooney | im sure there are other ways too but i have most often seen it due to operators chanign config such that the current vms nolonger fit | |
| 12:50:25 | gibi | hm, maybe we should add a WARNING for the compute log / placement log if there is overallocation detected so the admin will detect the misconfiguration | |
| 12:50:47 | sean-k-mooney | to the perodic | |
| 12:51:02 | sean-k-mooney | update_avaialable_resouces when we recalulate the placment update | |
| 12:51:24 | sean-k-mooney | ya we could | |
| 12:52:06 | sean-k-mooney | im not sure how spamy that would be but it does indeicate the might need to heal allcoaiton or other wise investigate why | |
| 14:07:00 | gibi | sean-k-mooney: actually placement already has a warning | |
| 14:07:02 | gibi | sean-k-mooney: "WARNING placement.objects.resource_provider [None req-6f2253b9-a195-4bf9-8c7e-2a32271a8c0c admin admin] Resource provider 935b9ad6-d7d1-4b5a-bb49-022acbba7c72 is now over-capacity for VCPU" | |
| 14:07:23 | gibi | when I set the allocation ratio to lower to induce overallocation | |
| 14:15:27 | opendevreview | Balazs Gibizer proposed openstack/placement master: DNM: extra logs to troubleshoot overcapacity https://review.opendev.org/c/openstack/placement/+/808083 | |
| 15:33:10 | sean-k-mooney | ah nice | |
| 15:47:17 | sean-k-mooney | although if the compute agent is down you might not see that | |
| 17:03:11 | gibi | elodilles_pto: there is probably a stable only bug here https://bugs.launchpad.net/nova/+bug/1941819 but as it is probably only affect stein and older which are in EM I don't think I will spend time fixing it. Maybe the bug author try it. | |
| 18:40:37 | legochen | hey nova experts, one question - can someone point me the best practice about configuring nova-scheduler filter to distribute VMs to user-specified cabinets? | |
| 18:41:31 | dansmith | user-specified, meaning "at boot a user specifies where this should go" ? | |
| 18:41:35 | dansmith | or did you mean user-specific? | |
| 18:42:05 | legochen | user-specified, meaning "at boot a user specifies where this should go" ? <= yes | |
| 18:42:24 | dansmith | in general this is not a thing nova allows or intends to allow, with one exception: AZs | |
| 18:42:42 | dansmith | users can choose AZs, so if you want them to be able to choose, make AZs for them to specify | |
| 18:43:00 | dansmith | you can do an AZ per site, or per aisle, or per rack or something | |
| 18:43:58 | legochen | For example, I have multiple cabients in data center, users want to distribute their VMs to different cabinets equally in order to avoid SPOF of tor switch or power stuff. | |
| 18:44:15 | dansmith | that's what AZs are for | |
| 18:44:32 | legochen | hmm, per cabinet per AZ seems not that reasonable to me :( | |
| 18:49:23 | legochen | I was thinking to configure per aggregate per cabinet. And set property - cabient=A for aggregate A, cabinet=B for aggregate B…. then users can specify —hint cabinet=A while creating a VM. | |
| 18:49:50 | dansmith | that's what AZs are for | |
| 19:18:09 | opendevreview | xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126 | |
| #openstack-nova - 2021-09-10 | |||
| 02:12:18 | opendevreview | xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126 | |
| 02:41:15 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 06:08:09 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:00:49 | admin1 | seeking info on what happens during a migration .. say i have h1 ( hypervisor1 ) and h2 ( hypervisor2 ) .. when I want to migrate vm1 on h1 -> h2, what makes the ssh conection .. is it h1 nova -> h2 nova .. or h1 libvirt -> h2 libvirt or h1 root -> h2 root .. | |
| 07:24:31 | gibi | bauzas: o/ friendly reminder that we need to land melwitt's placemnt fix in Xena RC1 https://review.opendev.org/c/openstack/placement/+/807014 | |
| 07:25:08 | gibi | melwitt, dansmith: we are out of EU cores for now, so could you please look at a small doc patch before RC1 https://review.opendev.org/c/openstack/nova/+/807564 ? | |
| 07:25:38 | gibi | melwitt, dansmith: also the reno prelude is ready https://review.opendev.org/c/openstack/nova/+/807786 | |
| 07:56:14 | bauzas | gibi: yeah I know and I started to review it | |
| 07:56:26 | bauzas | but y'know how it's a bit difficult | |
| 07:56:42 | bauzas | + loving escalations during the last 2 days :( | |
| 07:57:17 | gibi | bauzas: thanks. feel free to ask questions about the patch I should be able to answer them | |
| 07:57:21 | gibi | ahh escalations. :/ | |
| 07:59:11 | bauzas | don't know what happended | |
| 07:59:31 | bauzas | maybe because of the moon or because of puppets | |
| 07:59:43 | bauzas | but looks like we have a lot of customers asking us to help them | |
| 08:00:01 | gibi | maybe they are back from summer vacation :) | |
| 08:01:16 | bauzas | they're not French :p | |
| 08:08:41 | gibi | then they are just nice and waited for the French to return from their summer vacation :P | |
| 08:28:07 | admin1 | one more question .. if cold migration works but live does not -- ceph backend .. where do i need to start looking .. the nova logs ( debug true ) does not provide any actionable insight .. | |
| 09:30:42 | gibi | bauzas: do you agree with my triage here https://bugs.launchpad.net/nova/+bug/1939856 that it is probably a new feature not a bugfix? | |
| 09:34:44 | bauzas | gibi: fwiw, I +wd https://review.opendev.org/c/openstack/nova/+/792357 | |
| 09:35:04 | gibi | bauzas: thanks | |
| 09:35:08 | bauzas | so we can start to use the new label | |
| 09:35:28 | bauzas | but then I'll provide a new change explaining what I'd like to have to helping contributors | |
| 09:35:40 | gibi | OK, I will check that | |
| 09:36:46 | bauzas | gibi: about the bug you asked me | |
| 09:37:06 | bauzas | it's honestly a bug but Wishlist looks good to me | |
| 09:37:40 | bauzas | as you said, given the instance was offloaded from the compute, it's no longer possible to ask nova to wait for deleting it | |
| 09:42:50 | gibi | I would be affraid to change the delete behavior of offloaded instances as the current behavior was there forever and it is not really leaking any resources (I checked ports and volumes are cleaned up) | |
| 09:43:01 | gibi | so probalby people out there are adapted to this | |
| 09:51:19 | bauzas | gibi: we can fix this by documenting :) | |
| 09:51:57 | gibi | I can accept a documentation patch for it sure | |
| 10:13:05 | kevko | Hi folks, in victoria I am hitting this -> /var/log/kolla/nova/nova-compute.log:2021-09-10 09:49:09.815 8 ERROR nova.virt.libvirt.driver [req-8082a663-ca30-4845-9e93-2429c500ba76 31031550a7c94928abbd9dfa8634fba1 7b2ac086026741aa82c6c9c96ded42ee - default default] [instance: cf3ae6f9-a971-4123-ae73-9b597364985a] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter | |
| 10:13:05 | kevko | type for 'vhost', expected: boolean | |
| 10:13:18 | kevko | is it bug ? | |
| 10:25:05 | sean-k-mooney | kevko: can you past the libvirt xml somewhere so we can see what was generated | |
| 10:25:19 | sean-k-mooney | also what libvirt verions are you useing | |
| 10:28:54 | kevko | sean-k-mooney: https://paste.opendev.org/show/809223/ | |
| 10:28:57 | kevko | debian version | |
| 10:28:59 | kevko | buster one | |
| 10:32:07 | sean-k-mooney | that meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L225 | |
| 10:32:40 | sean-k-mooney | can you provide teh xml that was generated so we can see where the type element was added | |
| 10:32:53 | sean-k-mooney | i assume this was an interface attach? | |
| 10:33:09 | kevko | sean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ? | |
| 10:33:16 | kevko | sean-k-mooney: yeah, interface attach | |
| 10:33:33 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | |
| 10:33:41 | sean-k-mooney | yes if you trun on debug you see a about a 8 line segment of xml printed for attach | |
| 10:34:10 | sean-k-mooney | we print the full xml when startign the vm and just the bit we are appending for attach | |
| 10:34:55 | kevko | ok, give me 5 minutes cca | |
| 10:37:46 | kevko | sean-k-mooney: https://paste.opendev.org/show/809224/ | |
| 10:39:15 | kevko | sean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/ | |