| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-19 | |||
| 13:10:16 | shilpasd | efried: yes | |
| 13:10:50 | efried | shilpasd: In any case this sounds like a bug (it should behave better than that even if versions are mismatched) but something mriedem will have to look at once he gets here. | |
| 13:12:42 | efried | there he is :) | |
| 13:12:49 | shilpasd | efried: ok | |
| 13:12:51 | shilpasd | yes | |
| 13:14:22 | shilpasd | mriedem: i am getting http://paste.openstack.org/show/753170/ while service restart on master, can you help here to solve | |
| 13:14:25 | alex_xu | mriedem: we replied your comment on vpmem. also explored the bdm, let us know your thought | |
| 13:15:13 | alex_xu | mriedem: sean-k-mooney also in case you don't know, we already submit the vpmem code to gerrit https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virtual-persistent-memory, hope that is helpful on review spec | |
| 13:15:40 | mriedem | shilpasd: looks like you upgraded code without syncing the database | |
| 13:16:17 | shilpasd | mriedem: nova-manage db sync, is run still the issue | |
| 13:16:37 | shilpasd | any other command to be run? | |
| 13:18:45 | sean-k-mooney | alex_xu: sorry i have kept defering re reviewing https://review.opendev.org/#/c/601596/ ill grab a cup of coffee and go review before my next meeting | |
| 13:19:17 | sean-k-mooney | and cool im not sure i will have time to reivew the code today but ill skim over it | |
| 13:19:17 | alex_xu | sean-k-mooney: no problem, thanks | |
| 13:19:38 | alex_xu | \o/ | |
| 13:24:13 | mriedem | shilpasd: if you check the table schema in that database does it have the cross_cell_move column? | |
| 13:28:26 | shilpasd | mriedem: there is no cross_cell_move column | |
| 13:28:34 | mriedem | then you didn't run the db sync properly | |
| 13:28:59 | mriedem | are you sure you're syncing your nova (cell1) database? | |
| 13:29:15 | shilpasd | mriedem: i am having multi node setup, and i have migrations record in it, is it cauing an issue? | |
| 13:29:31 | shilpasd | should i clen records and run db sync? | |
| 13:29:33 | mriedem | existing migration records should not be a problem | |
| 13:30:34 | shilpasd | ok | |
| 13:30:40 | mriedem | shilpasd: you know that https://review.opendev.org/#/c/614012/ just merged yesterday, so you must be doing CD somewhere? | |
| 13:31:43 | shilpasd | mriedem: yes i know that, after pulling master today started getting this issue | |
| 13:36:19 | tssurya | shilpasd: did you run it on both cell0 and cell1 ? | |
| 13:39:07 | shilpasd | tssurya: after syncing DB, checked the table schema in that database for both cell0 and cell1, but it does not have the cross_cell_move column | |
| 13:40:03 | sean-k-mooney | alex_xu: erric woudl we be able to take a look at the CAT spec https://review.opendev.org/#/c/662264/3 johnthetubaguy i dont know if cache allocation and memory bandwith limits are somethign your interested in too for hpc but input is welcome if you are interested. | |
| 13:40:13 | sean-k-mooney | efried: ^ | |
| 13:41:43 | efried | sean-k-mooney: ack | |
| 13:43:13 | sean-k-mooney | ill be working with lyarwood on the implementation he will likely take lead but im hopeing to get a yes or no on the general direction of the spec in the next week or two. | |
| 13:43:22 | tssurya | shilpasd: well the nova-manage db sync comand by default takes the db_connection from the nova.conf config file. | |
| 13:44:06 | tssurya | for cell1 syncing | |
| 13:44:06 | tssurya | hopefully you are running it on level1 and not level0 | |
| 13:44:33 | stephenfin | gibi: Could you take a look at this long standing bugfix? https://review.opendev.org/#/c/609460/ | |
| 13:44:39 | mriedem | shilpasd: confirm that the config file you're using when running nova-manage db sync has the [database]/connection pointed at the correct cell db | |
| 13:44:51 | stephenfin | (I don't know anyone else that knows that piece of the code in detail) | |
| 13:45:18 | gibi | stephenfin: I've put it in my queue | |
| 13:45:23 | stephenfin | Thanks :) | |
| 13:45:24 | tssurya | shilpasd: yea check the database connection in the config you are running it against, I am guessing its not pointing at the right db | |
| 13:47:24 | shilpasd | tssurya: i don't think so, since nova operations were working before pulling master | |
| 13:48:11 | tssurya | shilpasd: what I mean is that nova-manage db sync is not being run against the cell config file | |
| 13:48:20 | tssurya | perhaps | |
| 13:48:34 | tssurya | which is why its not adding the column to the cell level dbs | |
| 13:49:36 | shilpasd | i have checked all that and in conf cell1 is configured as DB con | |
| 13:50:04 | shilpasd | and checked cell1 for cross_cell_move column but its not there | |
| 13:50:40 | shilpasd | finally i am doing stack with reclone YES, lets see further, will keep you posted | |
| 13:53:57 | mloza | hello, I have 13 computes and 12 of them are working fine. One of the problematic compute node's nova-compute service goes up and down in `openstack compute service list`. From the logs, I got tons of "Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed" and RMQ broken pipes. Redeploying the compute didn't fix the issue neither rebooting | |
| 13:53:59 | mloza | completely the controller nodes. | |
| 13:55:35 | ohwhyosa | mloza, have you checked placement logs? | |
| 13:55:48 | tssurya | mloza: looks like a message queue issue due to which the compute is not ablt to communicate to the controller, its not a nova-compute service issue I guess | |
| 13:55:56 | tssurya | able* | |
| 13:56:20 | ohwhyosa | I had a similar problem and it turns out the host-name was duplicated (because of a previous installation) | |
| 13:57:43 | ohwhyosa | try OS_TOKEN=$(openstack token issue -c id -f value) | |
| 13:58:14 | ohwhyosa | and then curl -s -H "X-Auth-Token: ${OS_TOKEN}" http://10.0.0.11:8780/resource_providers?name=${hostname} | |
| 13:58:41 | ohwhyosa | assumming that 10.0.0.11:8780 is your placement endpoint | |
| 13:59:00 | ohwhyosa | which you could check with openstack endpoint list I believe | |
| 13:59:06 | mloza | tssurya: When I redeployed the problematic compute, I change the hostname and IP address but still the issue persist. nova-compute keeps flapping | |
| 14:09:20 | shilpasd | tssurya: mriedem: stack solve the issue | |
| 14:09:45 | mloza | ohwhyosa: Just full of RMQ broken pipes and heartbeat missed. All logs are thrown in elasticsearch | |
| 14:13:26 | ohwhyosa | Did you check the api, mloza ? | |
| 14:17:58 | mloza | ohwhyosa: No such error in nova-api | |
| 14:19:48 | ohwhyosa | nope, the placement api, did you check the curl command result? | |
| 14:20:37 | mloza | ohwhyosa: let me check | |
| 14:20:56 | jroll | random question: has anyone discussed doing passthrough of a host TPM device to the guest? I see this BP from 2014 https://blueprints.launchpad.net/nova/+spec/add-libvirt-tpm . I guess what I'm wondering is if that's something that would be accepted, and if so, if anyone has ideas on the best route to implement (as it isn't just standard pci passthrough). | |
| 14:21:20 | jroll | context: we have users that have a requirement for a hardware tpm (still don't understand why vtpm isn't okay, but it is what it is) | |
| 14:27:59 | gibi | stephenfin: I'm +2 on https://review.opendev.org/#/c/609460 | |
| 14:28:11 | stephenfin | gibi++ | |
| 14:28:24 | stephenfin | Now to find another unwilling victim^H^H^H helpful soul :) | |
| 14:28:51 | mloza | ohwhyosa: https://pastebin.com/raw/bA3LScMr | |
| 14:29:24 | mriedem | jroll: there is an approved spec for tpm | |
| 14:29:43 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/train/approved/add-emulated-virtual-tpm.html | |
| 14:29:51 | mriedem | the code is lagging and was deferred from stein | |
| 14:29:53 | mriedem | windriver owns it | |
| 14:30:22 | mriedem | jroll: oh this is different, | |
| 14:30:25 | jroll | mriedem: passthrough, not virt... yeah | |
| 14:30:26 | mriedem | you want passthrough | |
| 14:31:05 | mriedem | i'd ask cfriesen about hw passthrough tpm if/when he's around | |
| 14:31:10 | mloza | ohwhyosa: I did redeploy twice. The first one, we deleted the old compute in `openstack resource provider ..` and the second time, we change the hostname and IP address but still RMQ issues are there. | |
| 14:33:20 | mloza | ohwhyosa: The RMQ issue is only a specific node and other computes are fine. | |
| 14:34:25 | ohwhyosa | hmmm so the old provider doesn't appear on the query, right? | |
| 14:34:39 | ohwhyosa | did you search by the current hostname or the old one? | |
| 14:38:01 | kashyap | jroll: Heya, just noticed your random question, yeah, was curious why vTPM (which is implemented in libvirt/QEMU) is not okay. Curious to hear when you know | |
| 14:38:26 | jroll | kashyap: probably FUD. idk. I'll let you know if I figure it out | |
| 14:38:58 | mriedem | jroll: let me guess, they are baremetal people that don't trust anything virtual, so even using VMs is a problem for them and they want hw passthrough wherever and whenever possible b/c they don't trust virtualization. | |
| 14:39:23 | jroll | mriedem: it's almost like you know where I work | |
| 14:39:40 | mriedem | heh | |
| 14:40:12 | kashyap | mriedem: Heh, that guess is perfectly reasonable | |
| 14:41:08 | stephenfin | dansmith: Could you take a look at https://review.opendev.org/#/c/609460 too? Someone has finally stumbled upon it | |
| 14:42:39 | mloza | ohwhyosa: yes, the old provider isn't in the list | |
| 14:42:44 | mloza | current one | |
| 14:42:59 | mloza | I checked the old and it wasn't there anymore | |
| 14:43:40 | dansmith | stephenfin: I'll look when there's a CI run from this year | |
| 14:43:57 | stephenfin | that'll happen automatically when it goes through the gate | |
| 14:44:39 | dansmith | yeah, but I'm not going to spend time reviewing it if it's like completely broken, you know? | |
| 14:45:31 | sean-k-mooney | dansmith: ha that could take a while to happen | |
| 14:45:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/609460 | |
| 14:46:10 | stephenfin | Fair, but it's not. Just saying :) | |
| 14:46:49 | sean-k-mooney | stephenfin: wait you tested something locally :P | |
| 14:47:33 | mloza | ohwhyosa: btw, this is a kolla-ansible deployment of stable/rocky branch | |