Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
13:48:34 tssurya which is why its not adding the column to the cell level dbs
13:49:36 shilpasd i have checked all that and in conf cell1 is configured as DB con
13:50:04 shilpasd and checked cell1 for cross_cell_move column but its not there
13:50:40 shilpasd finally i am doing stack with reclone YES, lets see further, will keep you posted
13:53:57 mloza hello, I have 13 computes and 12 of them are working fine. One of the problematic compute node's nova-compute service goes up and down in `openstack compute service list`. From the logs, I got tons of "Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed" and RMQ broken pipes. Redeploying the compute didn't fix the issue neither rebooting
13:53:59 mloza completely the controller nodes.
13:55:35 ohwhyosa mloza, have you checked placement logs?
13:55:48 tssurya mloza: looks like a message queue issue due to which the compute is not ablt to communicate to the controller, its not a nova-compute service issue I guess
13:55:56 tssurya able*
13:56:20 ohwhyosa I had a similar problem and it turns out the host-name was duplicated (because of a previous installation)
13:57:43 ohwhyosa try OS_TOKEN=$(openstack token issue -c id -f value)
13:58:14 ohwhyosa and then curl -s -H "X-Auth-Token: ${OS_TOKEN}" http://10.0.0.11:8780/resource_providers?name=${hostname}
13:58:41 ohwhyosa assumming that 10.0.0.11:8780 is your placement endpoint
13:59:00 ohwhyosa which you could check with openstack endpoint list I believe
13:59:06 mloza tssurya: When I redeployed the problematic compute, I change the hostname and IP address but still the issue persist. nova-compute keeps flapping
14:09:20 shilpasd tssurya: mriedem: stack solve the issue
14:09:45 mloza ohwhyosa: Just full of RMQ broken pipes and heartbeat missed. All logs are thrown in elasticsearch
14:13:26 ohwhyosa Did you check the api, mloza ?
14:17:58 mloza ohwhyosa: No such error in nova-api
14:19:48 ohwhyosa nope, the placement api, did you check the curl command result?
14:20:37 mloza ohwhyosa: let me check
14:20:56 jroll random question: has anyone discussed doing passthrough of a host TPM device to the guest? I see this BP from 2014 https://blueprints.launchpad.net/nova/+spec/add-libvirt-tpm . I guess what I'm wondering is if that's something that would be accepted, and if so, if anyone has ideas on the best route to implement (as it isn't just standard pci passthrough).
14:21:20 jroll context: we have users that have a requirement for a hardware tpm (still don't understand why vtpm isn't okay, but it is what it is)
14:27:59 gibi stephenfin: I'm +2 on https://review.opendev.org/#/c/609460
14:28:11 stephenfin gibi++
14:28:24 stephenfin Now to find another unwilling victim^H^H^H helpful soul :)
14:28:51 mloza ohwhyosa: https://pastebin.com/raw/bA3LScMr
14:29:24 mriedem jroll: there is an approved spec for tpm
14:29:43 mriedem https://specs.openstack.org/openstack/nova-specs/specs/train/approved/add-emulated-virtual-tpm.html
14:29:51 mriedem the code is lagging and was deferred from stein
14:29:53 mriedem windriver owns it
14:30:22 mriedem jroll: oh this is different,
14:30:25 jroll mriedem: passthrough, not virt... yeah
14:30:26 mriedem you want passthrough
14:31:05 mriedem i'd ask cfriesen about hw passthrough tpm if/when he's around
14:31:10 mloza ohwhyosa: I did redeploy twice. The first one, we deleted the old compute in `openstack resource provider ..` and the second time, we change the hostname and IP address but still RMQ issues are there.
14:33:20 mloza ohwhyosa: The RMQ issue is only a specific node and other computes are fine.
14:34:25 ohwhyosa hmmm so the old provider doesn't appear on the query, right?
14:34:39 ohwhyosa did you search by the current hostname or the old one?
14:38:01 kashyap jroll: Heya, just noticed your random question, yeah, was curious why vTPM (which is implemented in libvirt/QEMU) is not okay. Curious to hear when you know
14:38:26 jroll kashyap: probably FUD. idk. I'll let you know if I figure it out
14:38:58 mriedem jroll: let me guess, they are baremetal people that don't trust anything virtual, so even using VMs is a problem for them and they want hw passthrough wherever and whenever possible b/c they don't trust virtualization.
14:39:23 jroll mriedem: it's almost like you know where I work
14:39:40 mriedem heh
14:40:12 kashyap mriedem: Heh, that guess is perfectly reasonable
14:41:08 stephenfin dansmith: Could you take a look at https://review.opendev.org/#/c/609460 too? Someone has finally stumbled upon it
14:42:39 mloza ohwhyosa: yes, the old provider isn't in the list
14:42:44 mloza current one
14:42:59 mloza I checked the old and it wasn't there anymore
14:43:40 dansmith stephenfin: I'll look when there's a CI run from this year
14:43:57 stephenfin that'll happen automatically when it goes through the gate
14:44:39 dansmith yeah, but I'm not going to spend time reviewing it if it's like completely broken, you know?
14:45:31 sean-k-mooney dansmith: ha that could take a while to happen
14:45:39 openstackgerrit Stephen Finucane proposed openstack/nova master: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/609460
14:46:10 stephenfin Fair, but it's not. Just saying :)
14:46:49 sean-k-mooney stephenfin: wait you tested something locally :P
14:47:33 mloza ohwhyosa: btw, this is a kolla-ansible deployment of stable/rocky branch
15:01:14 belmoreira mriedem are you available? Hope you're good.
15:01:54 belmoreira what's the best way to fix wrong allocations? (requires_allocation_refresh = False)
15:02:46 mriedem belmoreira: coincidentally bauzas and i are talking about busted allocations in -placement
15:02:52 mriedem but should probably be talking about that here since it's a nova problem
15:03:03 bauzas actually yeah
15:03:34 mriedem i'm not sure what requires_allocation_refresh is
15:03:49 mriedem resource_provider_association_refresh ?
15:04:00 mriedem resource_provider_association_refresh doesn't touch allocations anyway
15:05:03 belmoreira this is only enable for ironic (the default is false). I can confirm but I'm assuming it always updates the allocations
15:05:09 mriedem belmoreira: well one part is fixing the many bugs that cause incorrect allocatoins, e.g. https://review.opendev.org/#/c/654067/
15:05:40 mriedem belmoreira: oh, requires_allocation_refresh in the ironic driver, that was removed awhile ago
15:05:52 belmoreira mriedem I agree. But after hitting the bugs operators need something to fix the busted allocations
15:06:27 mriedem yeah i know
15:06:39 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:06:39 mriedem i think https://bugs.launchpad.net/nova/+bug/1793569 is where we've been collecting the various notes on what some operators are doing for scripts
15:07:18 mriedem belmoreira: one option is if you know that some instance has the wrong allocations, you can delete the allocations in placement and then run the nova-manage placement heal_allocations command and it should fix the allocations
15:07:38 mriedem but that won't do anything about allocations held by migration records from some failed migration
15:08:06 ohwhyosa mloza, I recommend you checkout also with the people at #openstack-kolla
15:08:57 ohwhyosa I can't help you more, I'm kinda new myself (without the kinda) but since I had a similar problem recently and was helped by sean-k-mooney , I thought I'd pay the favor forward in case it would help
15:14:21 bauzas mriedem: you need to stop your APIs before cleaning up allocations, right?
15:15:10 mriedem umm
15:15:19 belmoreira mriedem in that case the operator needs to identify somehow the wrong allocations. I was thinking in having a "nova-manage" for that and fix or a periodic task
15:15:21 mriedem i guess it depends on what you mean by cleaning up
15:15:36 mriedem belmoreira: see comment 5 that i just posted https://bugs.launchpad.net/nova/+bug/1793569
15:15:37 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:15:39 bauzas mriedem: what you just said 'delete allocations and run nova-manage'
15:16:04 mriedem bauzas: heal_allocations won't do anything to an instance whose task_state is not None,
15:16:05 bauzas if you delete allocations, you could race if you leave users to be able to create new instances ;)
15:16:05 belmoreira mriedem we had that periodic task in the past, if I remember correctly
15:16:22 mriedem bauzas: creating new servers and their allocations isn't the problem you're trying to solve
15:16:39 bauzas I'm misunderstanding what you propose then
15:16:49 mriedem belmoreira: once everything is upgraded to pike, the resource tracker would stop updating allocations for non-ironic nodes
15:17:57 mriedem bauzas: if you're trying to heal allocations for a single instance, you could lock it, delete its allocatoins in placement, and then run heal_allocations and then unlock it
15:18:25 bauzas hah ok, that I understand
15:18:30 mriedem but as i've said in that bug above, and in irc many a time, heal_allocations nor the scripts in that bug from larsks and mnaser deal with stale allocations held by migration consumers
15:18:53 bauzas I thought you were saying about scrubbing the whole allocations table and just do the nova-manage placement heal stuff to recreate all the records
15:19:11 mriedem bauzas: i probably wouldn't do that on a production cloud with a lot of instances
15:19:17 bauzas of course
15:19:27 mriedem unless you use the --dry-run option which was also recently added and not backported
15:19:40 mriedem heal_allocations also doesn't deal with nested allocations (yet)
15:19:49 belmoreira mriedem right. Why it shouldn't continue to happen for libvirt if we continue to require it for ironic? (at least optional) We will have always bugs and this "sync" will be always required
15:20:02 mriedem so our tooling for operators is definitely lagging the complicated features we're shoving in
15:20:32 mriedem belmoreira: first, we no longer have that flag in the ironic driver as i said - it was removed some time ago, either rocky or stein

Earlier   Later