Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
09:58:14 openstackgerrit Surya Seetharaman proposed openstack/nova master: Grab fresh info from the driver during nova start/stop actions https://review.opendev.org/665975
10:21:07 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Use consistent URL regex substitution https://review.opendev.org/665949
11:18:07 ohwhyosa hey there! I uploaded an iso ubuntu-server-18.04 (not live) and when connecting via console to the instance, it does detect it (it lets me get to the keymap selection part) but then it complains no cd-rom image is mounted, any idea why might that be?
11:21:19 sean-k-mooney not really while you can use iso its a little complicated in that you generally have to also a data volume to install to
11:22:24 sean-k-mooney when you select the iso as a disk image it gets attached as the root disk. im not sure if it is attahced as a cdrom or not
11:23:08 ohwhyosa thanks sean-k-mooney! the recommendation would be using other image formats if available then, right?
11:23:44 sean-k-mooney well normally you would locally install it in a vm and then upload that to the cloud or use a prebuild image
11:25:12 sean-k-mooney you can enable a boot menu too. you can convert the iso into a data volume in cinder and then boot an instance with a blank volume and install too but in generall you are better off preparing the image first rather then trying to do it in openstack
11:25:54 ohwhyosa nova.console.websocketproxy keeps on saying broken pipe, by the way, tried the openstack-ansible multinode and the aio (which should set up all the networking correctly by itself)
11:26:16 ohwhyosa It is only the console service that broken pipes, just saying in case there might be an issue behind it
11:26:24 ohwhyosa I used both latest and stein
11:26:28 ohwhyosa stable/stein
11:26:55 sean-k-mooney im not sure that sound more like a spice server or websockify issue rather then nova
11:27:15 sean-k-mooney ohwhyosa: if you have not seen https://github.com/openstack/diskimage-builder you should check it out
11:28:04 sean-k-mooney there is also virt-builder but tool like those are how people usually prepare image for openstack
11:29:21 ohwhyosa sean-k-mooney, thanks a ton! (and it happened both with noVNC and spice, but it is indeed happening with websockify)
13:01:45 efried shilpasd: You've hit this: https://review.opendev.org/#/c/614012/
13:01:58 efried shilpasd: sorry, you already said that.
13:02:18 efried are your conductor & computes running at the same level?
13:10:16 shilpasd efried: yes
13:10:50 efried shilpasd: In any case this sounds like a bug (it should behave better than that even if versions are mismatched) but something mriedem will have to look at once he gets here.
13:12:42 efried there he is :)
13:12:49 shilpasd efried: ok
13:12:51 shilpasd yes
13:14:22 shilpasd mriedem: i am getting http://paste.openstack.org/show/753170/ while service restart on master, can you help here to solve
13:14:25 alex_xu mriedem: we replied your comment on vpmem. also explored the bdm, let us know your thought
13:15:13 alex_xu mriedem: sean-k-mooney also in case you don't know, we already submit the vpmem code to gerrit https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virtual-persistent-memory, hope that is helpful on review spec
13:15:40 mriedem shilpasd: looks like you upgraded code without syncing the database
13:16:17 shilpasd mriedem: nova-manage db sync, is run still the issue
13:16:37 shilpasd any other command to be run?
13:18:45 sean-k-mooney alex_xu: sorry i have kept defering re reviewing https://review.opendev.org/#/c/601596/ ill grab a cup of coffee and go review before my next meeting
13:19:17 sean-k-mooney and cool im not sure i will have time to reivew the code today but ill skim over it
13:19:17 alex_xu sean-k-mooney: no problem, thanks
13:19:38 alex_xu \o/
13:24:13 mriedem shilpasd: if you check the table schema in that database does it have the cross_cell_move column?
13:28:26 shilpasd mriedem: there is no cross_cell_move column
13:28:34 mriedem then you didn't run the db sync properly
13:28:59 mriedem are you sure you're syncing your nova (cell1) database?
13:29:15 shilpasd mriedem: i am having multi node setup, and i have migrations record in it, is it cauing an issue?
13:29:31 shilpasd should i clen records and run db sync?
13:29:33 mriedem existing migration records should not be a problem
13:30:34 shilpasd ok
13:30:40 mriedem shilpasd: you know that https://review.opendev.org/#/c/614012/ just merged yesterday, so you must be doing CD somewhere?
13:31:43 shilpasd mriedem: yes i know that, after pulling master today started getting this issue
13:36:19 tssurya shilpasd: did you run it on both cell0 and cell1 ?
13:39:07 shilpasd tssurya: after syncing DB, checked the table schema in that database for both cell0 and cell1, but it does not have the cross_cell_move column
13:40:03 sean-k-mooney alex_xu: erric woudl we be able to take a look at the CAT spec https://review.opendev.org/#/c/662264/3 johnthetubaguy i dont know if cache allocation and memory bandwith limits are somethign your interested in too for hpc but input is welcome if you are interested.
13:40:13 sean-k-mooney efried: ^
13:41:43 efried sean-k-mooney: ack
13:43:13 sean-k-mooney ill be working with lyarwood on the implementation he will likely take lead but im hopeing to get a yes or no on the general direction of the spec in the next week or two.
13:43:22 tssurya shilpasd: well the nova-manage db sync comand by default takes the db_connection from the nova.conf config file.
13:44:06 tssurya for cell1 syncing
13:44:06 tssurya hopefully you are running it on level1 and not level0
13:44:33 stephenfin gibi: Could you take a look at this long standing bugfix? https://review.opendev.org/#/c/609460/
13:44:39 mriedem shilpasd: confirm that the config file you're using when running nova-manage db sync has the [database]/connection pointed at the correct cell db
13:44:51 stephenfin (I don't know anyone else that knows that piece of the code in detail)
13:45:18 gibi stephenfin: I've put it in my queue
13:45:23 stephenfin Thanks :)
13:45:24 tssurya shilpasd: yea check the database connection in the config you are running it against, I am guessing its not pointing at the right db
13:47:24 shilpasd tssurya: i don't think so, since nova operations were working before pulling master
13:48:11 tssurya shilpasd: what I mean is that nova-manage db sync is not being run against the cell config file
13:48:20 tssurya perhaps
13:48:34 tssurya which is why its not adding the column to the cell level dbs
13:49:36 shilpasd i have checked all that and in conf cell1 is configured as DB con
13:50:04 shilpasd and checked cell1 for cross_cell_move column but its not there
13:50:40 shilpasd finally i am doing stack with reclone YES, lets see further, will keep you posted
13:53:57 mloza hello, I have 13 computes and 12 of them are working fine. One of the problematic compute node's nova-compute service goes up and down in `openstack compute service list`. From the logs, I got tons of "Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed" and RMQ broken pipes. Redeploying the compute didn't fix the issue neither rebooting
13:53:59 mloza completely the controller nodes.
13:55:35 ohwhyosa mloza, have you checked placement logs?
13:55:48 tssurya mloza: looks like a message queue issue due to which the compute is not ablt to communicate to the controller, its not a nova-compute service issue I guess
13:55:56 tssurya able*
13:56:20 ohwhyosa I had a similar problem and it turns out the host-name was duplicated (because of a previous installation)
13:57:43 ohwhyosa try OS_TOKEN=$(openstack token issue -c id -f value)
13:58:14 ohwhyosa and then curl -s -H "X-Auth-Token: ${OS_TOKEN}" http://10.0.0.11:8780/resource_providers?name=${hostname}
13:58:41 ohwhyosa assumming that 10.0.0.11:8780 is your placement endpoint
13:59:00 ohwhyosa which you could check with openstack endpoint list I believe
13:59:06 mloza tssurya: When I redeployed the problematic compute, I change the hostname and IP address but still the issue persist. nova-compute keeps flapping
14:09:20 shilpasd tssurya: mriedem: stack solve the issue
14:09:45 mloza ohwhyosa: Just full of RMQ broken pipes and heartbeat missed. All logs are thrown in elasticsearch
14:13:26 ohwhyosa Did you check the api, mloza ?
14:17:58 mloza ohwhyosa: No such error in nova-api
14:19:48 ohwhyosa nope, the placement api, did you check the curl command result?
14:20:37 mloza ohwhyosa: let me check
14:20:56 jroll random question: has anyone discussed doing passthrough of a host TPM device to the guest? I see this BP from 2014 https://blueprints.launchpad.net/nova/+spec/add-libvirt-tpm . I guess what I'm wondering is if that's something that would be accepted, and if so, if anyone has ideas on the best route to implement (as it isn't just standard pci passthrough).
14:21:20 jroll context: we have users that have a requirement for a hardware tpm (still don't understand why vtpm isn't okay, but it is what it is)
14:27:59 gibi stephenfin: I'm +2 on https://review.opendev.org/#/c/609460
14:28:11 stephenfin gibi++
14:28:24 stephenfin Now to find another unwilling victim^H^H^H helpful soul :)
14:28:51 mloza ohwhyosa: https://pastebin.com/raw/bA3LScMr
14:29:24 mriedem jroll: there is an approved spec for tpm
14:29:43 mriedem https://specs.openstack.org/openstack/nova-specs/specs/train/approved/add-emulated-virtual-tpm.html
14:29:51 mriedem the code is lagging and was deferred from stein
14:29:53 mriedem windriver owns it
14:30:22 mriedem jroll: oh this is different,
14:30:25 jroll mriedem: passthrough, not virt... yeah
14:30:26 mriedem you want passthrough
14:31:05 mriedem i'd ask cfriesen about hw passthrough tpm if/when he's around
14:31:10 mloza ohwhyosa: I did redeploy twice. The first one, we deleted the old compute in `openstack resource provider ..` and the second time, we change the hostname and IP address but still RMQ issues are there.
14:33:20 mloza ohwhyosa: The RMQ issue is only a specific node and other computes are fine.
14:34:25 ohwhyosa hmmm so the old provider doesn't appear on the query, right?

Earlier   Later