| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-13 | |||
| 15:38:29 | gmann | yeah | |
| 15:39:19 | admin0 | sean-k-mooney, i did a grep mysql /etc/nova.conf .. then installed mysql-client .. and then when i do mysql -h (hostname as in the conf) -u username -p ( copy paste of the password) .. it works just fine i can login and select the database | |
| 15:39:51 | gmann | artom: sean-k-mooney do you have link where test_server_sequence_suspend_resume failing ? | |
| 15:40:29 | admin0 | i did a create table and drop table ( the same test table) to validate that the login is rw | |
| 15:41:17 | sean-k-mooney | admin0: odd im not certin what the issue is off the top of my head | |
| 15:41:51 | admin0 | is there a way to get more verbose/debug steps in nova-manage db sync ? | |
| 15:44:25 | admin0 | nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0 worked fine .. only in db sync its giving issues | |
| 15:48:46 | artom | gmann, https://zuul.opendev.org/t/openstack/build/0c87381c639e4d629fa2bd48ea2da4f2 | |
| 15:49:56 | gmann | artom: thanks, i can try to build logic of creating unique network in scenario test too but may be during weekend or tomorrow. | |
| 15:50:25 | gmann | failing scenario tests on this should be less-frequent than API tests or rare | |
| 15:55:43 | lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | |
| 15:55:48 | lyarwood | ^ can anyone take this? | |
| 15:56:43 | lyarwood | or we skip this week | |
| 15:58:31 | gmann | lyarwood: sorry, i will be in internal meeting at the same time, even would not be able to join too | |
| 15:59:55 | lyarwood | or mum is home, I can run through the agenda quickly | |
| 16:00:51 | lyarwood | #openstack-meeting-3 for a very quick meeting if people have anything | |
| 16:10:57 | sean-k-mooney | very efficent of you | |
| 16:11:00 | sean-k-mooney | lyarwood: ^ | |
| 16:11:08 | sean-k-mooney | you should run all meetings :) | |
| 16:11:38 | lyarwood | haha I didn't prep and just rushed through everything, 10/10. | |
| 16:11:59 | lyarwood | I'll sort the agenda out ahead of time next week while gibi_pto_24th is still out | |
| 16:12:12 | lyarwood | apologies for the mess today | |
| 16:14:05 | tosky | I just wanted thanks for the Zuul v3 effort and please keep going but you were faster :'( | |
| 16:21:12 | lyarwood | tosky: hehe sorry and thanks, will keep pushing until M3, have the evacuate stuff worked out locally, just need to test and push. | |
| 17:17:33 | sangeet | I need help. I am doing "nova-manege db sync" and it faile with error "ERROR: Could not access cell0.". "nova-manage cell_v2 map_cell0" "nova-manage api_db sync" were run successfully before. I have ssl enabled in my deployment. The issue seems that _url_cfg.connection in _TransactionFactory object is somehow not corect for nova_cell0. Hence the DB is rejection the connection. Please see http://paste.openstack.org/show/796820/. | |
| 17:17:33 | sangeet | But connection in [cell0_database] in nova.conf is set correctly. Where is _TransactionFactory object reading the connection value form? | |
| 17:19:00 | sean-k-mooney | sangeet: your the second person to ask abou this in 2 hours :) | |
| 17:19:16 | sangeet | Interesting | |
| 17:20:04 | sangeet | sean-k-mooney was it answered? Sorry I can not scroll up for previous chats. | |
| 17:20:29 | sean-k-mooney | am no | |
| 17:20:57 | sean-k-mooney | same bhavior "nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0" bot worked by sync failed | |
| 17:21:24 | sangeet | :-( | |
| 17:21:57 | sean-k-mooney | i think if you want to run sync on cell0 | |
| 17:22:13 | sean-k-mooney | you need to create a second config adn set teh database connection to point to cell0 | |
| 17:22:43 | sean-k-mooney | at least i think you had to do that in the past e.g. nova-manage did not read cell0_database | |
| 17:22:49 | sean-k-mooney | but i could be miss remembering | |
| 17:23:03 | sangeet | That is waht I have set .. please look at http://paste.openstack.org/show/796820/. | |
| 17:23:49 | sean-k-mooney | let me compare to my devstack | |
| 17:24:46 | sean-k-mooney | so have | |
| 17:24:48 | sean-k-mooney | [database] | |
| 17:24:51 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:24:53 | sean-k-mooney | [api_database] | |
| 17:24:55 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_api?charset=utf8 | |
| 17:25:10 | sean-k-mooney | so my db is point to cell0 in the conotler config at least | |
| 17:26:04 | sean-k-mooney | my cell1 contoller then point to cell1 | |
| 17:26:08 | sean-k-mooney | [database] | |
| 17:26:11 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell1?charset=utf8 | |
| 17:26:12 | sangeet | sean-k-mooney - Please not .. if I do a new openstack deployment with ssl it is working fine. Its only when I am doing an update from no ssl to ssl, I see the issue. Somehow url_cfg.connection in _TransactionFactory object is taking old connection | |
| 17:27:03 | sangeet | for nova_cell0 only | |
| 17:27:20 | sean-k-mooney | nova_cell0 is special | |
| 17:27:28 | sean-k-mooney | its not a real cell | |
| 17:27:38 | sean-k-mooney | it has a special meanin in the code | |
| 17:28:01 | sean-k-mooney | i dont know what [cell0_database] is | |
| 17:28:07 | sangeet | nova.conf has correct values as well ... so just can't figure out how the connection is still old link | |
| 17:29:02 | sean-k-mooney | i have never seen the "[cell0_database]" config section before | |
| 17:29:20 | sean-k-mooney | and its not in our config docs | |
| 17:33:29 | sean-k-mooney | sangeet: did you deploy with openstack helm? | |
| 17:33:46 | sangeet | yes | |
| 17:33:53 | sean-k-mooney | then its a helm bug i think | |
| 17:34:02 | sean-k-mooney | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L1724 | |
| 17:34:11 | sean-k-mooney | it is the only thing that creats that config section | |
| 17:34:28 | sean-k-mooney | i cant see anything in any other tool that does and i dont see any code that uses it in nova | |
| 17:34:35 | sean-k-mooney | there are also not docs related to it | |
| 17:34:48 | sean-k-mooney | based on http://codesearch.openstack.org/?q=cell0_database&i=nope&files=&repos= | |
| 17:36:38 | sangeet | Its ok ... we can ignore [cell0_database]. Even [database] has correct value of connection. | |
| 17:37:24 | sean-k-mooney | it does but cell0_database might be breaking nova manage somehow | |
| 17:45:39 | sean-k-mooney | sangeet: can you try adding --local_cell | |
| 17:47:59 | sean-k-mooney | i think i know why its failing | |
| 17:48:32 | sean-k-mooney | as i said that config option is not valid. no code in nova exits to use it | |
| 17:48:49 | sean-k-mooney | wehn you dont pass --loca_cell we lookup the cell0 from the db | |
| 17:48:51 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L188-L189 | |
| 17:49:24 | sean-k-mooney | so unless you have updated the cell0 database connection string in the cell maping table it will use the old value | |
| 17:50:08 | sean-k-mooney | sangeet: if you pass --local_cell it will skip that lookup although it wont update the schem of cell0 | |
| 17:50:28 | sean-k-mooney | you will have to do that manually | |
| 17:50:40 | sangeet | It is automated but let me try manually | |
| 17:50:55 | sean-k-mooney | but ya in anycae openstack helm has a bug | |
| 17:50:57 | sangeet | you are saying run nova-manabe db sync --local_cell? | |
| 17:51:03 | sean-k-mooney | yes | |
| 17:51:11 | sangeet | Let me try that real quick | |
| 17:52:04 | sean-k-mooney | openstack helm is not generting correct local.confs and as a result the automated db sync will fail if you are going form non ssl to ssl | |
| 17:52:26 | sean-k-mooney | since it will use the non ssl connection string from the db | |
| 17:52:36 | sangeet | awesome ... that worked. Can you please explain what --local_cell opetion does | |
| 17:53:04 | sean-k-mooney | it tells nova manage not to lookup the db connection string form nova_api.cell_mappings db table | |
| 17:53:52 | sangeet | Interestinfg | |
| 17:54:03 | sean-k-mooney | you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell | |
| 17:54:23 | sean-k-mooney | sangeet: alternitvily you can just update teh connection string in the cell mappign table in the nova_api db | |
| 17:54:48 | sean-k-mooney | in my devstack deployment its just mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:54:58 | sean-k-mooney | but you would need to add the ssl parmater to your version | |
| 17:55:50 | sangeet | Let me see what I have in the my database | |
| 17:56:25 | sean-k-mooney | mysql -e "select database_connection from nova_api.cell_mappings where name='cell0';" | |
| 17:56:42 | sean-k-mooney | by the way you should not really paste the result publicy | |
| 17:56:52 | sean-k-mooney | since it has the password in it... | |
| 17:57:20 | sean-k-mooney | anyway it does not have the ssl cert info right | |
| 17:57:29 | sangeet | You are correct .. it is without ssl | |
| 17:57:45 | sean-k-mooney | yep so you could just update it and then it would work | |
| 17:59:30 | sangeet | going back to useing --local-cell, do you see any issue with it? Also when you say "you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell" | |
| 17:59:31 | sean-k-mooney | by should work i mean nova manage shoudl work without --local-cell and as a result the automated update should work too | |
| 17:59:39 | sangeet | does it mean I have to run on each node | |
| 17:59:59 | sean-k-mooney | no so each cell has its own db right | |
| 18:00:07 | sangeet | yes | |