| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-13 | |||
| 17:24:48 | sean-k-mooney | [database] | |
| 17:24:51 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:24:53 | sean-k-mooney | [api_database] | |
| 17:24:55 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_api?charset=utf8 | |
| 17:25:10 | sean-k-mooney | so my db is point to cell0 in the conotler config at least | |
| 17:26:04 | sean-k-mooney | my cell1 contoller then point to cell1 | |
| 17:26:08 | sean-k-mooney | [database] | |
| 17:26:11 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell1?charset=utf8 | |
| 17:26:12 | sangeet | sean-k-mooney - Please not .. if I do a new openstack deployment with ssl it is working fine. Its only when I am doing an update from no ssl to ssl, I see the issue. Somehow url_cfg.connection in _TransactionFactory object is taking old connection | |
| 17:27:03 | sangeet | for nova_cell0 only | |
| 17:27:20 | sean-k-mooney | nova_cell0 is special | |
| 17:27:28 | sean-k-mooney | its not a real cell | |
| 17:27:38 | sean-k-mooney | it has a special meanin in the code | |
| 17:28:01 | sean-k-mooney | i dont know what [cell0_database] is | |
| 17:28:07 | sangeet | nova.conf has correct values as well ... so just can't figure out how the connection is still old link | |
| 17:29:02 | sean-k-mooney | i have never seen the "[cell0_database]" config section before | |
| 17:29:20 | sean-k-mooney | and its not in our config docs | |
| 17:33:29 | sean-k-mooney | sangeet: did you deploy with openstack helm? | |
| 17:33:46 | sangeet | yes | |
| 17:33:53 | sean-k-mooney | then its a helm bug i think | |
| 17:34:02 | sean-k-mooney | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L1724 | |
| 17:34:11 | sean-k-mooney | it is the only thing that creats that config section | |
| 17:34:28 | sean-k-mooney | i cant see anything in any other tool that does and i dont see any code that uses it in nova | |
| 17:34:35 | sean-k-mooney | there are also not docs related to it | |
| 17:34:48 | sean-k-mooney | based on http://codesearch.openstack.org/?q=cell0_database&i=nope&files=&repos= | |
| 17:36:38 | sangeet | Its ok ... we can ignore [cell0_database]. Even [database] has correct value of connection. | |
| 17:37:24 | sean-k-mooney | it does but cell0_database might be breaking nova manage somehow | |
| 17:45:39 | sean-k-mooney | sangeet: can you try adding --local_cell | |
| 17:47:59 | sean-k-mooney | i think i know why its failing | |
| 17:48:32 | sean-k-mooney | as i said that config option is not valid. no code in nova exits to use it | |
| 17:48:49 | sean-k-mooney | wehn you dont pass --loca_cell we lookup the cell0 from the db | |
| 17:48:51 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L188-L189 | |
| 17:49:24 | sean-k-mooney | so unless you have updated the cell0 database connection string in the cell maping table it will use the old value | |
| 17:50:08 | sean-k-mooney | sangeet: if you pass --local_cell it will skip that lookup although it wont update the schem of cell0 | |
| 17:50:28 | sean-k-mooney | you will have to do that manually | |
| 17:50:40 | sangeet | It is automated but let me try manually | |
| 17:50:55 | sean-k-mooney | but ya in anycae openstack helm has a bug | |
| 17:50:57 | sangeet | you are saying run nova-manabe db sync --local_cell? | |
| 17:51:03 | sean-k-mooney | yes | |
| 17:51:11 | sangeet | Let me try that real quick | |
| 17:52:04 | sean-k-mooney | openstack helm is not generting correct local.confs and as a result the automated db sync will fail if you are going form non ssl to ssl | |
| 17:52:26 | sean-k-mooney | since it will use the non ssl connection string from the db | |
| 17:52:36 | sangeet | awesome ... that worked. Can you please explain what --local_cell opetion does | |
| 17:53:04 | sean-k-mooney | it tells nova manage not to lookup the db connection string form nova_api.cell_mappings db table | |
| 17:53:52 | sangeet | Interestinfg | |
| 17:54:03 | sean-k-mooney | you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell | |
| 17:54:23 | sean-k-mooney | sangeet: alternitvily you can just update teh connection string in the cell mappign table in the nova_api db | |
| 17:54:48 | sean-k-mooney | in my devstack deployment its just mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:54:58 | sean-k-mooney | but you would need to add the ssl parmater to your version | |
| 17:55:50 | sangeet | Let me see what I have in the my database | |
| 17:56:25 | sean-k-mooney | mysql -e "select database_connection from nova_api.cell_mappings where name='cell0';" | |
| 17:56:42 | sean-k-mooney | by the way you should not really paste the result publicy | |
| 17:56:52 | sean-k-mooney | since it has the password in it... | |
| 17:57:20 | sean-k-mooney | anyway it does not have the ssl cert info right | |
| 17:57:29 | sangeet | You are correct .. it is without ssl | |
| 17:57:45 | sean-k-mooney | yep so you could just update it and then it would work | |
| 17:59:30 | sangeet | going back to useing --local-cell, do you see any issue with it? Also when you say "you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell" | |
| 17:59:31 | sean-k-mooney | by should work i mean nova manage shoudl work without --local-cell and as a result the automated update should work too | |
| 17:59:39 | sangeet | does it mean I have to run on each node | |
| 17:59:59 | sean-k-mooney | no so each cell has its own db right | |
| 18:00:07 | sangeet | yes | |
| 18:00:16 | sean-k-mooney | so each cell db needs to be updated | |
| 18:00:27 | sean-k-mooney | wehn you use --local-cell it only update well the locall cell db | |
| 18:00:39 | sean-k-mooney | not all deployment have db access between cells | |
| 18:00:41 | sangeet | will "nova-manage db sync --local-cell" not update all cells? | |
| 18:01:02 | sean-k-mooney | correct it only updates the cell pointed to in the database section | |
| 18:01:21 | sangeet | but "nova-manage db sync " does all? | |
| 18:01:31 | sean-k-mooney | actully no | |
| 18:01:38 | sean-k-mooney | it does the local cell and cell0 | |
| 18:02:04 | sean-k-mooney | you need to run nova-manage in each cell anyway | |
| 18:02:14 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L185-L186 | |
| 18:03:05 | sangeet | So it mean I run the command with our without --local-cell, it will update same cells? | |
| 18:04:18 | sean-k-mooney | no the default behavor is to update the cell in [database]/connection and cell0 | |
| 18:04:50 | sean-k-mooney | so if you run the command on the host with the conducto for cell1 it will update cell1 and cell0 | |
| 18:05:12 | sean-k-mooney | ran on the host with the conductor for cell2 it will update cell2 and cell0 | |
| 18:05:29 | sean-k-mooney | if you add --local-cell it will skip updating cell0 | |
| 18:05:51 | sean-k-mooney | unless [database]/connection point to cell0 | |
| 18:06:53 | sean-k-mooney | basically the nova.conf used by the nova api should have [database]/connection pointing at cell0, each conductors nova.conf should have [database]/connection pointing at the cell db | |
| 18:07:21 | sean-k-mooney | and the compute services should not have the database details defiend at all | |
| 18:08:06 | sean-k-mooney | openstack helm shoudl then be automatinn runing the db sync once for each cell db using the correct cofigs | |
| 18:08:55 | sean-k-mooney | and the simplet way to do that is to delegate runing nova-magange db sync to the first conductor host in each cell | |
| 18:09:13 | sean-k-mooney | you can safly run it on each conductor host | |
| 18:09:28 | sean-k-mooney | as it is not destructive its just wasteful to do it | |
| 18:09:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/746210 | |
| 18:12:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/746210 | |
| 19:21:31 | sangeet | sean-k-mooney .. thanks a lot for your help | |
| 19:39:24 | sean-k-mooney | sangeet: no worries hopefully if admin0 comes back and sees this it might help them too | |
| 20:56:01 | mnaser | does nova run ceph jobs on every commit? | |
| 21:03:08 | dansmith | yes | |
| 21:29:50 | eandersson | btw I have a PR open for cloud-init for the OpenStack Datasource to increase the retry limit when identifying the metadata service. The current defaults only allows a single failure before giving up. https://github.com/canonical/cloud-init/pull/501 | |
| 21:30:12 | eandersson | If anyone has any thoughts feel free to drop a comment. | |
| 21:45:41 | openstackgerrit | Merged openstack/nova master: Fix FIP policy for admin_or_owner https://review.opendev.org/741811 | |
| 21:47:59 | prometheanfire | can we get more cores on https://review.opendev.org/744992 (oneliner, with comment) | |
| 21:58:58 | melwitt | prometheanfire: done | |
| 22:01:13 | prometheanfire | thanks | |
| 22:21:51 | openstackgerrit | Merged openstack/nova master: Add new default roles in multinic policies https://review.opendev.org/742330 | |
| 22:21:57 | openstackgerrit | Merged openstack/nova master: Correct the check_str and pass actual target in FIP pools policy https://review.opendev.org/741797 | |
| 22:22:05 | openstackgerrit | Merged openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | |
| #openstack-nova - 2020-08-14 | |||
| 02:21:53 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of FIP policies https://review.opendev.org/741812 | |
| 02:22:09 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in FIP policy https://review.opendev.org/741813 | |