| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-19 | |||
| 07:31:38 | gmann | thanks for reporting it. | |
| 08:04:36 | openstackgerrit | Josephine Seifert proposed openstack/nova-specs master: Spec for the Nova part of Image Encryption https://review.opendev.org/608696 | |
| 08:15:39 | openstackgerrit | Josephine Seifert proposed openstack/nova-specs master: Spec for the Nova part of Image Encryption https://review.opendev.org/608696 | |
| 08:24:17 | openstackgerrit | jiasirui proposed openstack/nova-specs master: fix the spelling mistakes https://review.opendev.org/666244 | |
| 08:28:05 | shilpasd | <efried> i am facing issue in n-cpu start, due to nova/db/sqlalchemy/migrate_repo/versions/397_migrations_cross_cell_move.py | |
| 08:28:20 | shilpasd | efried: this got commited at https://review.opendev.org/#/c/614012/ | |
| 08:28:36 | shilpasd | error is:Error starting thread.: RemoteError: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'migrations.cross_cell_move' in 'field list'") | |
| 08:28:53 | shilpasd | i have run db manage command, but no success | |
| 08:28:59 | shilpasd | any solution? | |
| 08:30:36 | shilpasd | efried: mriedem: here is the detailed error log, http://paste.openstack.org/show/753170/ | |
| 08:44:08 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245 | |
| 09:38:20 | openstackgerrit | Merged openstack/nova master: Fix wrong assert methods https://review.opendev.org/665897 | |
| 09:50:45 | openstackgerrit | zhaixiaojun proposed openstack/python-novaclient master: Modify the url of upper_constraints_file https://review.opendev.org/665934 | |
| 09:57:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: xvp: Start using consoleauth tokens https://review.opendev.org/652967 | |
| 09:57:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: xvp: Remove use of '_LI' marker https://review.opendev.org/665425 | |
| 09:57:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-status: Remove consoleauth workaround check https://review.opendev.org/652968 | |
| 09:57:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove nova-consoleauth https://review.opendev.org/652969 | |
| 09:57:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove ConsoleAuthToken.to_dict https://review.opendev.org/652970 | |
| 09:57:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rework nova console diagram https://review.opendev.org/660147 | |
| 09:57:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Integrate 'sphinx.ext.imgconverter' https://review.opendev.org/665693 | |
| 09:58:14 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Grab fresh info from the driver during nova start/stop actions https://review.opendev.org/665975 | |
| 10:21:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Use consistent URL regex substitution https://review.opendev.org/665949 | |
| 11:18:07 | ohwhyosa | hey there! I uploaded an iso ubuntu-server-18.04 (not live) and when connecting via console to the instance, it does detect it (it lets me get to the keymap selection part) but then it complains no cd-rom image is mounted, any idea why might that be? | |
| 11:21:19 | sean-k-mooney | not really while you can use iso its a little complicated in that you generally have to also a data volume to install to | |
| 11:22:24 | sean-k-mooney | when you select the iso as a disk image it gets attached as the root disk. im not sure if it is attahced as a cdrom or not | |
| 11:23:08 | ohwhyosa | thanks sean-k-mooney! the recommendation would be using other image formats if available then, right? | |
| 11:23:44 | sean-k-mooney | well normally you would locally install it in a vm and then upload that to the cloud or use a prebuild image | |
| 11:25:12 | sean-k-mooney | you can enable a boot menu too. you can convert the iso into a data volume in cinder and then boot an instance with a blank volume and install too but in generall you are better off preparing the image first rather then trying to do it in openstack | |
| 11:25:54 | ohwhyosa | nova.console.websocketproxy keeps on saying broken pipe, by the way, tried the openstack-ansible multinode and the aio (which should set up all the networking correctly by itself) | |
| 11:26:16 | ohwhyosa | It is only the console service that broken pipes, just saying in case there might be an issue behind it | |
| 11:26:24 | ohwhyosa | I used both latest and stein | |
| 11:26:28 | ohwhyosa | stable/stein | |
| 11:26:55 | sean-k-mooney | im not sure that sound more like a spice server or websockify issue rather then nova | |
| 11:27:15 | sean-k-mooney | ohwhyosa: if you have not seen https://github.com/openstack/diskimage-builder you should check it out | |
| 11:28:04 | sean-k-mooney | there is also virt-builder but tool like those are how people usually prepare image for openstack | |
| 11:29:21 | ohwhyosa | sean-k-mooney, thanks a ton! (and it happened both with noVNC and spice, but it is indeed happening with websockify) | |
| 13:01:45 | efried | shilpasd: You've hit this: https://review.opendev.org/#/c/614012/ | |
| 13:01:58 | efried | shilpasd: sorry, you already said that. | |
| 13:02:18 | efried | are your conductor & computes running at the same level? | |
| 13:10:16 | shilpasd | efried: yes | |
| 13:10:50 | efried | shilpasd: In any case this sounds like a bug (it should behave better than that even if versions are mismatched) but something mriedem will have to look at once he gets here. | |
| 13:12:42 | efried | there he is :) | |
| 13:12:49 | shilpasd | efried: ok | |
| 13:12:51 | shilpasd | yes | |
| 13:14:22 | shilpasd | mriedem: i am getting http://paste.openstack.org/show/753170/ while service restart on master, can you help here to solve | |
| 13:14:25 | alex_xu | mriedem: we replied your comment on vpmem. also explored the bdm, let us know your thought | |
| 13:15:13 | alex_xu | mriedem: sean-k-mooney also in case you don't know, we already submit the vpmem code to gerrit https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virtual-persistent-memory, hope that is helpful on review spec | |
| 13:15:40 | mriedem | shilpasd: looks like you upgraded code without syncing the database | |
| 13:16:17 | shilpasd | mriedem: nova-manage db sync, is run still the issue | |
| 13:16:37 | shilpasd | any other command to be run? | |
| 13:18:45 | sean-k-mooney | alex_xu: sorry i have kept defering re reviewing https://review.opendev.org/#/c/601596/ ill grab a cup of coffee and go review before my next meeting | |
| 13:19:17 | alex_xu | sean-k-mooney: no problem, thanks | |
| 13:19:17 | sean-k-mooney | and cool im not sure i will have time to reivew the code today but ill skim over it | |
| 13:19:38 | alex_xu | \o/ | |
| 13:24:13 | mriedem | shilpasd: if you check the table schema in that database does it have the cross_cell_move column? | |
| 13:28:26 | shilpasd | mriedem: there is no cross_cell_move column | |
| 13:28:34 | mriedem | then you didn't run the db sync properly | |
| 13:28:59 | mriedem | are you sure you're syncing your nova (cell1) database? | |
| 13:29:15 | shilpasd | mriedem: i am having multi node setup, and i have migrations record in it, is it cauing an issue? | |
| 13:29:31 | shilpasd | should i clen records and run db sync? | |
| 13:29:33 | mriedem | existing migration records should not be a problem | |
| 13:30:34 | shilpasd | ok | |
| 13:30:40 | mriedem | shilpasd: you know that https://review.opendev.org/#/c/614012/ just merged yesterday, so you must be doing CD somewhere? | |
| 13:31:43 | shilpasd | mriedem: yes i know that, after pulling master today started getting this issue | |
| 13:36:19 | tssurya | shilpasd: did you run it on both cell0 and cell1 ? | |
| 13:39:07 | shilpasd | tssurya: after syncing DB, checked the table schema in that database for both cell0 and cell1, but it does not have the cross_cell_move column | |
| 13:40:03 | sean-k-mooney | alex_xu: erric woudl we be able to take a look at the CAT spec https://review.opendev.org/#/c/662264/3 johnthetubaguy i dont know if cache allocation and memory bandwith limits are somethign your interested in too for hpc but input is welcome if you are interested. | |
| 13:40:13 | sean-k-mooney | efried: ^ | |
| 13:41:43 | efried | sean-k-mooney: ack | |
| 13:43:13 | sean-k-mooney | ill be working with lyarwood on the implementation he will likely take lead but im hopeing to get a yes or no on the general direction of the spec in the next week or two. | |
| 13:43:22 | tssurya | shilpasd: well the nova-manage db sync comand by default takes the db_connection from the nova.conf config file. | |
| 13:44:06 | tssurya | hopefully you are running it on level1 and not level0 | |
| 13:44:06 | tssurya | for cell1 syncing | |
| 13:44:33 | stephenfin | gibi: Could you take a look at this long standing bugfix? https://review.opendev.org/#/c/609460/ | |
| 13:44:39 | mriedem | shilpasd: confirm that the config file you're using when running nova-manage db sync has the [database]/connection pointed at the correct cell db | |
| 13:44:51 | stephenfin | (I don't know anyone else that knows that piece of the code in detail) | |
| 13:45:18 | gibi | stephenfin: I've put it in my queue | |
| 13:45:23 | stephenfin | Thanks :) | |
| 13:45:24 | tssurya | shilpasd: yea check the database connection in the config you are running it against, I am guessing its not pointing at the right db | |
| 13:47:24 | shilpasd | tssurya: i don't think so, since nova operations were working before pulling master | |
| 13:48:11 | tssurya | shilpasd: what I mean is that nova-manage db sync is not being run against the cell config file | |
| 13:48:20 | tssurya | perhaps | |
| 13:48:34 | tssurya | which is why its not adding the column to the cell level dbs | |
| 13:49:36 | shilpasd | i have checked all that and in conf cell1 is configured as DB con | |
| 13:50:04 | shilpasd | and checked cell1 for cross_cell_move column but its not there | |
| 13:50:40 | shilpasd | finally i am doing stack with reclone YES, lets see further, will keep you posted | |
| 13:53:57 | mloza | hello, I have 13 computes and 12 of them are working fine. One of the problematic compute node's nova-compute service goes up and down in `openstack compute service list`. From the logs, I got tons of "Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed" and RMQ broken pipes. Redeploying the compute didn't fix the issue neither rebooting | |
| 13:53:59 | mloza | completely the controller nodes. | |
| 13:55:35 | ohwhyosa | mloza, have you checked placement logs? | |
| 13:55:48 | tssurya | mloza: looks like a message queue issue due to which the compute is not ablt to communicate to the controller, its not a nova-compute service issue I guess | |
| 13:55:56 | tssurya | able* | |
| 13:56:20 | ohwhyosa | I had a similar problem and it turns out the host-name was duplicated (because of a previous installation) | |
| 13:57:43 | ohwhyosa | try OS_TOKEN=$(openstack token issue -c id -f value) | |
| 13:58:14 | ohwhyosa | and then curl -s -H "X-Auth-Token: ${OS_TOKEN}" http://10.0.0.11:8780/resource_providers?name=${hostname} | |
| 13:58:41 | ohwhyosa | assumming that 10.0.0.11:8780 is your placement endpoint | |
| 13:59:00 | ohwhyosa | which you could check with openstack endpoint list I believe | |
| 13:59:06 | mloza | tssurya: When I redeployed the problematic compute, I change the hostname and IP address but still the issue persist. nova-compute keeps flapping | |
| 14:09:20 | shilpasd | tssurya: mriedem: stack solve the issue | |
| 14:09:45 | mloza | ohwhyosa: Just full of RMQ broken pipes and heartbeat missed. All logs are thrown in elasticsearch | |
| 14:13:26 | ohwhyosa | Did you check the api, mloza ? | |