Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
13:30:40 mriedem shilpasd: you know that https://review.opendev.org/#/c/614012/ just merged yesterday, so you must be doing CD somewhere?
13:31:43 shilpasd mriedem: yes i know that, after pulling master today started getting this issue
13:36:19 tssurya shilpasd: did you run it on both cell0 and cell1 ?
13:39:07 shilpasd tssurya: after syncing DB, checked the table schema in that database for both cell0 and cell1, but it does not have the cross_cell_move column
13:40:03 sean-k-mooney alex_xu: erric woudl we be able to take a look at the CAT spec https://review.opendev.org/#/c/662264/3 johnthetubaguy i dont know if cache allocation and memory bandwith limits are somethign your interested in too for hpc but input is welcome if you are interested.
13:40:13 sean-k-mooney efried: ^
13:41:43 efried sean-k-mooney: ack
13:43:13 sean-k-mooney ill be working with lyarwood on the implementation he will likely take lead but im hopeing to get a yes or no on the general direction of the spec in the next week or two.
13:43:22 tssurya shilpasd: well the nova-manage db sync comand by default takes the db_connection from the nova.conf config file.
13:44:06 tssurya for cell1 syncing
13:44:06 tssurya hopefully you are running it on level1 and not level0
13:44:33 stephenfin gibi: Could you take a look at this long standing bugfix? https://review.opendev.org/#/c/609460/
13:44:39 mriedem shilpasd: confirm that the config file you're using when running nova-manage db sync has the [database]/connection pointed at the correct cell db
13:44:51 stephenfin (I don't know anyone else that knows that piece of the code in detail)
13:45:18 gibi stephenfin: I've put it in my queue
13:45:23 stephenfin Thanks :)
13:45:24 tssurya shilpasd: yea check the database connection in the config you are running it against, I am guessing its not pointing at the right db
13:47:24 shilpasd tssurya: i don't think so, since nova operations were working before pulling master
13:48:11 tssurya shilpasd: what I mean is that nova-manage db sync is not being run against the cell config file
13:48:20 tssurya perhaps
13:48:34 tssurya which is why its not adding the column to the cell level dbs
13:49:36 shilpasd i have checked all that and in conf cell1 is configured as DB con
13:50:04 shilpasd and checked cell1 for cross_cell_move column but its not there
13:50:40 shilpasd finally i am doing stack with reclone YES, lets see further, will keep you posted
13:53:57 mloza hello, I have 13 computes and 12 of them are working fine. One of the problematic compute node's nova-compute service goes up and down in `openstack compute service list`. From the logs, I got tons of "Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed" and RMQ broken pipes. Redeploying the compute didn't fix the issue neither rebooting
13:53:59 mloza completely the controller nodes.
13:55:35 ohwhyosa mloza, have you checked placement logs?
13:55:48 tssurya mloza: looks like a message queue issue due to which the compute is not ablt to communicate to the controller, its not a nova-compute service issue I guess
13:55:56 tssurya able*
13:56:20 ohwhyosa I had a similar problem and it turns out the host-name was duplicated (because of a previous installation)
13:57:43 ohwhyosa try OS_TOKEN=$(openstack token issue -c id -f value)
13:58:14 ohwhyosa and then curl -s -H "X-Auth-Token: ${OS_TOKEN}" http://10.0.0.11:8780/resource_providers?name=${hostname}
13:58:41 ohwhyosa assumming that 10.0.0.11:8780 is your placement endpoint
13:59:00 ohwhyosa which you could check with openstack endpoint list I believe
13:59:06 mloza tssurya: When I redeployed the problematic compute, I change the hostname and IP address but still the issue persist. nova-compute keeps flapping
14:09:20 shilpasd tssurya: mriedem: stack solve the issue
14:09:45 mloza ohwhyosa: Just full of RMQ broken pipes and heartbeat missed. All logs are thrown in elasticsearch
14:13:26 ohwhyosa Did you check the api, mloza ?
14:17:58 mloza ohwhyosa: No such error in nova-api
14:19:48 ohwhyosa nope, the placement api, did you check the curl command result?
14:20:37 mloza ohwhyosa: let me check
14:20:56 jroll random question: has anyone discussed doing passthrough of a host TPM device to the guest? I see this BP from 2014 https://blueprints.launchpad.net/nova/+spec/add-libvirt-tpm . I guess what I'm wondering is if that's something that would be accepted, and if so, if anyone has ideas on the best route to implement (as it isn't just standard pci passthrough).
14:21:20 jroll context: we have users that have a requirement for a hardware tpm (still don't understand why vtpm isn't okay, but it is what it is)
14:27:59 gibi stephenfin: I'm +2 on https://review.opendev.org/#/c/609460
14:28:11 stephenfin gibi++
14:28:24 stephenfin Now to find another unwilling victim^H^H^H helpful soul :)
14:28:51 mloza ohwhyosa: https://pastebin.com/raw/bA3LScMr
14:29:24 mriedem jroll: there is an approved spec for tpm
14:29:43 mriedem https://specs.openstack.org/openstack/nova-specs/specs/train/approved/add-emulated-virtual-tpm.html
14:29:51 mriedem the code is lagging and was deferred from stein
14:29:53 mriedem windriver owns it
14:30:22 mriedem jroll: oh this is different,
14:30:25 jroll mriedem: passthrough, not virt... yeah
14:30:26 mriedem you want passthrough
14:31:05 mriedem i'd ask cfriesen about hw passthrough tpm if/when he's around
14:31:10 mloza ohwhyosa: I did redeploy twice. The first one, we deleted the old compute in `openstack resource provider ..` and the second time, we change the hostname and IP address but still RMQ issues are there.
14:33:20 mloza ohwhyosa: The RMQ issue is only a specific node and other computes are fine.
14:34:25 ohwhyosa hmmm so the old provider doesn't appear on the query, right?
14:34:39 ohwhyosa did you search by the current hostname or the old one?
14:38:01 kashyap jroll: Heya, just noticed your random question, yeah, was curious why vTPM (which is implemented in libvirt/QEMU) is not okay. Curious to hear when you know
14:38:26 jroll kashyap: probably FUD. idk. I'll let you know if I figure it out
14:38:58 mriedem jroll: let me guess, they are baremetal people that don't trust anything virtual, so even using VMs is a problem for them and they want hw passthrough wherever and whenever possible b/c they don't trust virtualization.
14:39:23 jroll mriedem: it's almost like you know where I work
14:39:40 mriedem heh
14:40:12 kashyap mriedem: Heh, that guess is perfectly reasonable
14:41:08 stephenfin dansmith: Could you take a look at https://review.opendev.org/#/c/609460 too? Someone has finally stumbled upon it
14:42:39 mloza ohwhyosa: yes, the old provider isn't in the list
14:42:44 mloza current one
14:42:59 mloza I checked the old and it wasn't there anymore
14:43:40 dansmith stephenfin: I'll look when there's a CI run from this year
14:43:57 stephenfin that'll happen automatically when it goes through the gate
14:44:39 dansmith yeah, but I'm not going to spend time reviewing it if it's like completely broken, you know?
14:45:31 sean-k-mooney dansmith: ha that could take a while to happen
14:45:39 openstackgerrit Stephen Finucane proposed openstack/nova master: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/609460
14:46:10 stephenfin Fair, but it's not. Just saying :)
14:46:49 sean-k-mooney stephenfin: wait you tested something locally :P
14:47:33 mloza ohwhyosa: btw, this is a kolla-ansible deployment of stable/rocky branch
15:01:14 belmoreira mriedem are you available? Hope you're good.
15:01:54 belmoreira what's the best way to fix wrong allocations? (requires_allocation_refresh = False)
15:02:46 mriedem belmoreira: coincidentally bauzas and i are talking about busted allocations in -placement
15:02:52 mriedem but should probably be talking about that here since it's a nova problem
15:03:03 bauzas actually yeah
15:03:34 mriedem i'm not sure what requires_allocation_refresh is
15:03:49 mriedem resource_provider_association_refresh ?
15:04:00 mriedem resource_provider_association_refresh doesn't touch allocations anyway
15:05:03 belmoreira this is only enable for ironic (the default is false). I can confirm but I'm assuming it always updates the allocations
15:05:09 mriedem belmoreira: well one part is fixing the many bugs that cause incorrect allocatoins, e.g. https://review.opendev.org/#/c/654067/
15:05:40 mriedem belmoreira: oh, requires_allocation_refresh in the ironic driver, that was removed awhile ago
15:05:52 belmoreira mriedem I agree. But after hitting the bugs operators need something to fix the busted allocations
15:06:27 mriedem yeah i know
15:06:39 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:06:39 mriedem i think https://bugs.launchpad.net/nova/+bug/1793569 is where we've been collecting the various notes on what some operators are doing for scripts
15:07:18 mriedem belmoreira: one option is if you know that some instance has the wrong allocations, you can delete the allocations in placement and then run the nova-manage placement heal_allocations command and it should fix the allocations
15:07:38 mriedem but that won't do anything about allocations held by migration records from some failed migration
15:08:06 ohwhyosa mloza, I recommend you checkout also with the people at #openstack-kolla
15:08:57 ohwhyosa I can't help you more, I'm kinda new myself (without the kinda) but since I had a similar problem recently and was helped by sean-k-mooney , I thought I'd pay the favor forward in case it would help
15:14:21 bauzas mriedem: you need to stop your APIs before cleaning up allocations, right?
15:15:10 mriedem umm
15:15:19 belmoreira mriedem in that case the operator needs to identify somehow the wrong allocations. I was thinking in having a "nova-manage" for that and fix or a periodic task
15:15:21 mriedem i guess it depends on what you mean by cleaning up

Earlier   Later