| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-13 | |||
| 16:11:08 | sean-k-mooney | you should run all meetings :) | |
| 16:11:38 | lyarwood | haha I didn't prep and just rushed through everything, 10/10. | |
| 16:11:59 | lyarwood | I'll sort the agenda out ahead of time next week while gibi_pto_24th is still out | |
| 16:12:12 | lyarwood | apologies for the mess today | |
| 16:14:05 | tosky | I just wanted thanks for the Zuul v3 effort and please keep going but you were faster :'( | |
| 16:21:12 | lyarwood | tosky: hehe sorry and thanks, will keep pushing until M3, have the evacuate stuff worked out locally, just need to test and push. | |
| 17:17:33 | sangeet | I need help. I am doing "nova-manege db sync" and it faile with error "ERROR: Could not access cell0.". "nova-manage cell_v2 map_cell0" "nova-manage api_db sync" were run successfully before. I have ssl enabled in my deployment. The issue seems that _url_cfg.connection in _TransactionFactory object is somehow not corect for nova_cell0. Hence the DB is rejection the connection. Please see http://paste.openstack.org/show/796820/. | |
| 17:17:33 | sangeet | But connection in [cell0_database] in nova.conf is set correctly. Where is _TransactionFactory object reading the connection value form? | |
| 17:19:00 | sean-k-mooney | sangeet: your the second person to ask abou this in 2 hours :) | |
| 17:19:16 | sangeet | Interesting | |
| 17:20:04 | sangeet | sean-k-mooney was it answered? Sorry I can not scroll up for previous chats. | |
| 17:20:29 | sean-k-mooney | am no | |
| 17:20:57 | sean-k-mooney | same bhavior "nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0" bot worked by sync failed | |
| 17:21:24 | sangeet | :-( | |
| 17:21:57 | sean-k-mooney | i think if you want to run sync on cell0 | |
| 17:22:13 | sean-k-mooney | you need to create a second config adn set teh database connection to point to cell0 | |
| 17:22:43 | sean-k-mooney | at least i think you had to do that in the past e.g. nova-manage did not read cell0_database | |
| 17:22:49 | sean-k-mooney | but i could be miss remembering | |
| 17:23:03 | sangeet | That is waht I have set .. please look at http://paste.openstack.org/show/796820/. | |
| 17:23:49 | sean-k-mooney | let me compare to my devstack | |
| 17:24:46 | sean-k-mooney | so have | |
| 17:24:48 | sean-k-mooney | [database] | |
| 17:24:51 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:24:53 | sean-k-mooney | [api_database] | |
| 17:24:55 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_api?charset=utf8 | |
| 17:25:10 | sean-k-mooney | so my db is point to cell0 in the conotler config at least | |
| 17:26:04 | sean-k-mooney | my cell1 contoller then point to cell1 | |
| 17:26:08 | sean-k-mooney | [database] | |
| 17:26:11 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell1?charset=utf8 | |
| 17:26:12 | sangeet | sean-k-mooney - Please not .. if I do a new openstack deployment with ssl it is working fine. Its only when I am doing an update from no ssl to ssl, I see the issue. Somehow url_cfg.connection in _TransactionFactory object is taking old connection | |
| 17:27:03 | sangeet | for nova_cell0 only | |
| 17:27:20 | sean-k-mooney | nova_cell0 is special | |
| 17:27:28 | sean-k-mooney | its not a real cell | |
| 17:27:38 | sean-k-mooney | it has a special meanin in the code | |
| 17:28:01 | sean-k-mooney | i dont know what [cell0_database] is | |
| 17:28:07 | sangeet | nova.conf has correct values as well ... so just can't figure out how the connection is still old link | |
| 17:29:02 | sean-k-mooney | i have never seen the "[cell0_database]" config section before | |
| 17:29:20 | sean-k-mooney | and its not in our config docs | |
| 17:33:29 | sean-k-mooney | sangeet: did you deploy with openstack helm? | |
| 17:33:46 | sangeet | yes | |
| 17:33:53 | sean-k-mooney | then its a helm bug i think | |
| 17:34:02 | sean-k-mooney | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L1724 | |
| 17:34:11 | sean-k-mooney | it is the only thing that creats that config section | |
| 17:34:28 | sean-k-mooney | i cant see anything in any other tool that does and i dont see any code that uses it in nova | |
| 17:34:35 | sean-k-mooney | there are also not docs related to it | |
| 17:34:48 | sean-k-mooney | based on http://codesearch.openstack.org/?q=cell0_database&i=nope&files=&repos= | |
| 17:36:38 | sangeet | Its ok ... we can ignore [cell0_database]. Even [database] has correct value of connection. | |
| 17:37:24 | sean-k-mooney | it does but cell0_database might be breaking nova manage somehow | |
| 17:45:39 | sean-k-mooney | sangeet: can you try adding --local_cell | |
| 17:47:59 | sean-k-mooney | i think i know why its failing | |
| 17:48:32 | sean-k-mooney | as i said that config option is not valid. no code in nova exits to use it | |
| 17:48:49 | sean-k-mooney | wehn you dont pass --loca_cell we lookup the cell0 from the db | |
| 17:48:51 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L188-L189 | |
| 17:49:24 | sean-k-mooney | so unless you have updated the cell0 database connection string in the cell maping table it will use the old value | |
| 17:50:08 | sean-k-mooney | sangeet: if you pass --local_cell it will skip that lookup although it wont update the schem of cell0 | |
| 17:50:28 | sean-k-mooney | you will have to do that manually | |
| 17:50:40 | sangeet | It is automated but let me try manually | |
| 17:50:55 | sean-k-mooney | but ya in anycae openstack helm has a bug | |
| 17:50:57 | sangeet | you are saying run nova-manabe db sync --local_cell? | |
| 17:51:03 | sean-k-mooney | yes | |
| 17:51:11 | sangeet | Let me try that real quick | |
| 17:52:04 | sean-k-mooney | openstack helm is not generting correct local.confs and as a result the automated db sync will fail if you are going form non ssl to ssl | |
| 17:52:26 | sean-k-mooney | since it will use the non ssl connection string from the db | |
| 17:52:36 | sangeet | awesome ... that worked. Can you please explain what --local_cell opetion does | |
| 17:53:04 | sean-k-mooney | it tells nova manage not to lookup the db connection string form nova_api.cell_mappings db table | |
| 17:53:52 | sangeet | Interestinfg | |
| 17:54:03 | sean-k-mooney | you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell | |
| 17:54:23 | sean-k-mooney | sangeet: alternitvily you can just update teh connection string in the cell mappign table in the nova_api db | |
| 17:54:48 | sean-k-mooney | in my devstack deployment its just mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:54:58 | sean-k-mooney | but you would need to add the ssl parmater to your version | |
| 17:55:50 | sangeet | Let me see what I have in the my database | |
| 17:56:25 | sean-k-mooney | mysql -e "select database_connection from nova_api.cell_mappings where name='cell0';" | |
| 17:56:42 | sean-k-mooney | by the way you should not really paste the result publicy | |
| 17:56:52 | sean-k-mooney | since it has the password in it... | |
| 17:57:20 | sean-k-mooney | anyway it does not have the ssl cert info right | |
| 17:57:29 | sangeet | You are correct .. it is without ssl | |
| 17:57:45 | sean-k-mooney | yep so you could just update it and then it would work | |
| 17:59:30 | sangeet | going back to useing --local-cell, do you see any issue with it? Also when you say "you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell" | |
| 17:59:31 | sean-k-mooney | by should work i mean nova manage shoudl work without --local-cell and as a result the automated update should work too | |
| 17:59:39 | sangeet | does it mean I have to run on each node | |
| 17:59:59 | sean-k-mooney | no so each cell has its own db right | |
| 18:00:07 | sangeet | yes | |
| 18:00:16 | sean-k-mooney | so each cell db needs to be updated | |
| 18:00:27 | sean-k-mooney | wehn you use --local-cell it only update well the locall cell db | |
| 18:00:39 | sean-k-mooney | not all deployment have db access between cells | |
| 18:00:41 | sangeet | will "nova-manage db sync --local-cell" not update all cells? | |
| 18:01:02 | sean-k-mooney | correct it only updates the cell pointed to in the database section | |
| 18:01:21 | sangeet | but "nova-manage db sync " does all? | |
| 18:01:31 | sean-k-mooney | actully no | |
| 18:01:38 | sean-k-mooney | it does the local cell and cell0 | |
| 18:02:04 | sean-k-mooney | you need to run nova-manage in each cell anyway | |
| 18:02:14 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L185-L186 | |
| 18:03:05 | sangeet | So it mean I run the command with our without --local-cell, it will update same cells? | |
| 18:04:18 | sean-k-mooney | no the default behavor is to update the cell in [database]/connection and cell0 | |
| 18:04:50 | sean-k-mooney | so if you run the command on the host with the conducto for cell1 it will update cell1 and cell0 | |
| 18:05:12 | sean-k-mooney | ran on the host with the conductor for cell2 it will update cell2 and cell0 | |
| 18:05:29 | sean-k-mooney | if you add --local-cell it will skip updating cell0 | |
| 18:05:51 | sean-k-mooney | unless [database]/connection point to cell0 | |
| 18:06:53 | sean-k-mooney | basically the nova.conf used by the nova api should have [database]/connection pointing at cell0, each conductors nova.conf should have [database]/connection pointing at the cell db | |
| 18:07:21 | sean-k-mooney | and the compute services should not have the database details defiend at all | |