| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-13 | |||
| 15:16:36 | gmann | by default, tempest tests does not pass the dedicated network while creating server so we go test by test and request for unique network to be created for test | |
| 15:16:39 | gmann | artom: yeah | |
| 15:17:29 | gmann | artom: https://review.opendev.org/#/c/721767/ | |
| 15:17:42 | gmann | which test failing? | |
| 15:18:29 | artom | gmann, looks like we're hitting it in tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume | |
| 15:18:59 | artom | And something else, but I can't figure out the test from the kibana info | |
| 15:19:52 | gmann | artom: ah that is scenario test which always has the unique network, let me confirm | |
| 15:20:09 | sean-k-mooney | there are a number of test failing | |
| 15:22:01 | sean-k-mooney | tempest-AttachInterfacesTestJSON tempest-test_creds-1679784040 | |
| 15:22:36 | gmann | for API tests, root cause is same and we can fix it in Tempest side | |
| 15:23:23 | sean-k-mooney | actully i need to filter out passing jobs | |
| 15:23:35 | sean-k-mooney | apparently we have some tests that try to create that error | |
| 15:24:12 | artom | sean-k-mooney, ah, it would explain why https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1a8/744280/6/check/nova-next/1a8a26e/controller/logs/screen-n-api.txt is from a passing job | |
| 15:24:30 | artom | I guess start by fixing test_server_sequence_suspend_resume then? | |
| 15:25:13 | sean-k-mooney | oh so 212 hits instead of ~500 | |
| 15:26:23 | sean-k-mooney | and some of those have failure that are not related to this | |
| 15:26:45 | sean-k-mooney | this will be annoying to fine a query for | |
| 15:26:54 | gmann | test_server_sequence_suspend_resume does not explicit create the network and test try to get one from neutron - https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L257 | |
| 15:27:58 | gmann | artom: the fix i mentioned will not help for test_server_sequence_suspend_resume , we need to do either create the network explicitly or add some common logic in scenario manager also like we do in API test | |
| 15:28:34 | lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | |
| 15:29:45 | gmann | this query -http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22HTTP%20exception%20thrown:%20Multiple%20possible%20networks%20found,%20use%20a%20Network%20ID%20to%20be%20more%20specific.%5C%22%20AND%20NOT%20message:%5C%22AttachInterfacesTestJSON%5C%22%20AND%20tags:%5C%22screen-n-api.txt%5C%22&from=7d | |
| 15:32:59 | admin0 | when i run nova-manage --debug db sync .. it says ERROR: Could not access cell0. .. i check the username and password and the user (nova_api) can acess nova_cell0 database .. what could be wrong ? | |
| 15:34:17 | admin0 | or does nova_api need access to nova database also ? | |
| 15:34:47 | sean-k-mooney | this is the query i came up with http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%5C%22%20AND%20NOT%20message:%5C%22AttachInterfacesTestJSON%5C%22%20AND%20NOT%20message:%5C%22tempest-test_creds%5C%22%20%20AND%20NOT%20build_status:%5C%22SUCCESS%5C%22&from=30d | |
| 15:36:37 | sean-k-mooney | admin0: nova manage gets the db creds from the nova.conf | |
| 15:36:54 | admin0 | i have validated it and it is correct | |
| 15:37:01 | admin0 | is there a way to debug what actually it is using | |
| 15:37:02 | gmann | yeah, AttachInterfacesTestJSON is most in failing | |
| 15:37:11 | admin0 | --debug does not seem to produce that much of a log | |
| 15:37:49 | sean-k-mooney | gmann: right but none of those builds failed | |
| 15:38:07 | lyarwood | AttributeError: module 'libvirt' has no attribute 'VIR_ERR_DEVICE_MISSING' - whops, my bad. | |
| 15:38:13 | sean-k-mooney | well wehn the failed that was nto the issue | |
| 15:38:29 | gmann | yeah | |
| 15:39:19 | admin0 | sean-k-mooney, i did a grep mysql /etc/nova.conf .. then installed mysql-client .. and then when i do mysql -h (hostname as in the conf) -u username -p ( copy paste of the password) .. it works just fine i can login and select the database | |
| 15:39:51 | gmann | artom: sean-k-mooney do you have link where test_server_sequence_suspend_resume failing ? | |
| 15:40:29 | admin0 | i did a create table and drop table ( the same test table) to validate that the login is rw | |
| 15:41:17 | sean-k-mooney | admin0: odd im not certin what the issue is off the top of my head | |
| 15:41:51 | admin0 | is there a way to get more verbose/debug steps in nova-manage db sync ? | |
| 15:44:25 | admin0 | nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0 worked fine .. only in db sync its giving issues | |
| 15:48:46 | artom | gmann, https://zuul.opendev.org/t/openstack/build/0c87381c639e4d629fa2bd48ea2da4f2 | |
| 15:49:56 | gmann | artom: thanks, i can try to build logic of creating unique network in scenario test too but may be during weekend or tomorrow. | |
| 15:50:25 | gmann | failing scenario tests on this should be less-frequent than API tests or rare | |
| 15:55:43 | lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | |
| 15:55:48 | lyarwood | ^ can anyone take this? | |
| 15:56:43 | lyarwood | or we skip this week | |
| 15:58:31 | gmann | lyarwood: sorry, i will be in internal meeting at the same time, even would not be able to join too | |
| 15:59:55 | lyarwood | or mum is home, I can run through the agenda quickly | |
| 16:00:51 | lyarwood | #openstack-meeting-3 for a very quick meeting if people have anything | |
| 16:10:57 | sean-k-mooney | very efficent of you | |
| 16:11:00 | sean-k-mooney | lyarwood: ^ | |
| 16:11:08 | sean-k-mooney | you should run all meetings :) | |
| 16:11:38 | lyarwood | haha I didn't prep and just rushed through everything, 10/10. | |
| 16:11:59 | lyarwood | I'll sort the agenda out ahead of time next week while gibi_pto_24th is still out | |
| 16:12:12 | lyarwood | apologies for the mess today | |
| 16:14:05 | tosky | I just wanted thanks for the Zuul v3 effort and please keep going but you were faster :'( | |
| 16:21:12 | lyarwood | tosky: hehe sorry and thanks, will keep pushing until M3, have the evacuate stuff worked out locally, just need to test and push. | |
| 17:17:33 | sangeet | I need help. I am doing "nova-manege db sync" and it faile with error "ERROR: Could not access cell0.". "nova-manage cell_v2 map_cell0" "nova-manage api_db sync" were run successfully before. I have ssl enabled in my deployment. The issue seems that _url_cfg.connection in _TransactionFactory object is somehow not corect for nova_cell0. Hence the DB is rejection the connection. Please see http://paste.openstack.org/show/796820/. | |
| 17:17:33 | sangeet | But connection in [cell0_database] in nova.conf is set correctly. Where is _TransactionFactory object reading the connection value form? | |
| 17:19:00 | sean-k-mooney | sangeet: your the second person to ask abou this in 2 hours :) | |
| 17:19:16 | sangeet | Interesting | |
| 17:20:04 | sangeet | sean-k-mooney was it answered? Sorry I can not scroll up for previous chats. | |
| 17:20:29 | sean-k-mooney | am no | |
| 17:20:57 | sean-k-mooney | same bhavior "nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0" bot worked by sync failed | |
| 17:21:24 | sangeet | :-( | |
| 17:21:57 | sean-k-mooney | i think if you want to run sync on cell0 | |
| 17:22:13 | sean-k-mooney | you need to create a second config adn set teh database connection to point to cell0 | |
| 17:22:43 | sean-k-mooney | at least i think you had to do that in the past e.g. nova-manage did not read cell0_database | |
| 17:22:49 | sean-k-mooney | but i could be miss remembering | |
| 17:23:03 | sangeet | That is waht I have set .. please look at http://paste.openstack.org/show/796820/. | |
| 17:23:49 | sean-k-mooney | let me compare to my devstack | |
| 17:24:46 | sean-k-mooney | so have | |
| 17:24:48 | sean-k-mooney | [database] | |
| 17:24:51 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | |
| 17:24:53 | sean-k-mooney | [api_database] | |
| 17:24:55 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_api?charset=utf8 | |
| 17:25:10 | sean-k-mooney | so my db is point to cell0 in the conotler config at least | |
| 17:26:04 | sean-k-mooney | my cell1 contoller then point to cell1 | |
| 17:26:08 | sean-k-mooney | [database] | |
| 17:26:11 | sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell1?charset=utf8 | |
| 17:26:12 | sangeet | sean-k-mooney - Please not .. if I do a new openstack deployment with ssl it is working fine. Its only when I am doing an update from no ssl to ssl, I see the issue. Somehow url_cfg.connection in _TransactionFactory object is taking old connection | |
| 17:27:03 | sangeet | for nova_cell0 only | |
| 17:27:20 | sean-k-mooney | nova_cell0 is special | |
| 17:27:28 | sean-k-mooney | its not a real cell | |
| 17:27:38 | sean-k-mooney | it has a special meanin in the code | |
| 17:28:01 | sean-k-mooney | i dont know what [cell0_database] is | |
| 17:28:07 | sangeet | nova.conf has correct values as well ... so just can't figure out how the connection is still old link | |
| 17:29:02 | sean-k-mooney | i have never seen the "[cell0_database]" config section before | |
| 17:29:20 | sean-k-mooney | and its not in our config docs | |
| 17:33:29 | sean-k-mooney | sangeet: did you deploy with openstack helm? | |
| 17:33:46 | sangeet | yes | |
| 17:33:53 | sean-k-mooney | then its a helm bug i think | |
| 17:34:02 | sean-k-mooney | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L1724 | |
| 17:34:11 | sean-k-mooney | it is the only thing that creats that config section | |
| 17:34:28 | sean-k-mooney | i cant see anything in any other tool that does and i dont see any code that uses it in nova | |
| 17:34:35 | sean-k-mooney | there are also not docs related to it | |
| 17:34:48 | sean-k-mooney | based on http://codesearch.openstack.org/?q=cell0_database&i=nope&files=&repos= | |
| 17:36:38 | sangeet | Its ok ... we can ignore [cell0_database]. Even [database] has correct value of connection. | |
| 17:37:24 | sean-k-mooney | it does but cell0_database might be breaking nova manage somehow | |
| 17:45:39 | sean-k-mooney | sangeet: can you try adding --local_cell | |
| 17:47:59 | sean-k-mooney | i think i know why its failing | |