| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-13 | |||
| 04:53:53 | openstackgerrit | Harshavardhan Metla proposed openstack/nova master: Raising a failure in case of 403 https://review.opendev.org/735068 | |
| 08:07:11 | openstackgerrit | Alexandre Arents proposed openstack/nova master: Rebase qcow2 images when unshelving an instance https://review.opendev.org/696084 | |
| 08:27:20 | openstackgerrit | Jie Li proposed openstack/nova master: fix revert resize failed https://review.opendev.org/746020 | |
| 12:35:06 | openstackgerrit | Merged openstack/nova master: [Trivial] Remove wrong format_message() conversion https://review.opendev.org/744280 | |
| 12:53:58 | openstackgerrit | Mark Goddard proposed openstack/nova master: docs: fix aggregate weight multiplier property names https://review.opendev.org/746128 | |
| 14:28:25 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in baremetal nodes policies https://review.opendev.org/741716 | |
| 14:28:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in baremetal nodes policy https://review.opendev.org/741717 | |
| 14:31:30 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566 | |
| 14:31:52 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568 | |
| 14:34:32 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add scope and new default roles in extensions policies https://review.opendev.org/743046 | |
| 14:36:30 | artom | Is logstash... working? | |
| 14:36:42 | artom | I'm trying to understand if "Multiple possible networks found, use a Network ID to be more specific" in the nova-next job is a recurring thing | |
| 14:55:44 | sean-k-mooney | artom: its a few days delayed | |
| 14:56:02 | sean-k-mooney | i think 3-5 days | |
| 14:57:19 | sean-k-mooney | artom: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%5C%22 | |
| 14:57:26 | sean-k-mooney | set it to 30days and you get hits | |
| 14:58:08 | artom | Ah yes, indeed. I feel like we've hit this before, for a different job | |
| 14:58:12 | artom | gmann ^^ sound familiar? | |
| 14:58:17 | sean-k-mooney | we have hit this many times | |
| 14:58:21 | sean-k-mooney | its a tempest bug | |
| 14:58:41 | sean-k-mooney | the tests are not wrtten correctly | |
| 14:59:07 | artom | I kinda figured that already :P | |
| 14:59:28 | artom | I was looking for a previous fix that I could plagiarize to avoid having to think about this ;) | |
| 14:59:38 | artom | Or to straight up "delegate" to someone else :P | |
| 15:01:11 | sean-k-mooney | the first failure was in this requiremnt update to neutron libs https://review.opendev.org/#/c/745237 | |
| 15:02:00 | sean-k-mooney | that may or may not be related to the cause | |
| 15:12:33 | gmann | artom: sean-k-mooney yeah, we fixed this issue while we see error. | |
| 15:15:50 | artom | gmann, do you remember what the fix was? Do we need to redo it somewhere else? | |
| 15:16:36 | gmann | by default, tempest tests does not pass the dedicated network while creating server so we go test by test and request for unique network to be created for test | |
| 15:16:39 | gmann | artom: yeah | |
| 15:17:29 | gmann | artom: https://review.opendev.org/#/c/721767/ | |
| 15:17:42 | gmann | which test failing? | |
| 15:18:29 | artom | gmann, looks like we're hitting it in tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume | |
| 15:18:59 | artom | And something else, but I can't figure out the test from the kibana info | |
| 15:19:52 | gmann | artom: ah that is scenario test which always has the unique network, let me confirm | |
| 15:20:09 | sean-k-mooney | there are a number of test failing | |
| 15:22:01 | sean-k-mooney | tempest-AttachInterfacesTestJSON tempest-test_creds-1679784040 | |
| 15:22:36 | gmann | for API tests, root cause is same and we can fix it in Tempest side | |
| 15:23:23 | sean-k-mooney | actully i need to filter out passing jobs | |
| 15:23:35 | sean-k-mooney | apparently we have some tests that try to create that error | |
| 15:24:12 | artom | sean-k-mooney, ah, it would explain why https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1a8/744280/6/check/nova-next/1a8a26e/controller/logs/screen-n-api.txt is from a passing job | |
| 15:24:30 | artom | I guess start by fixing test_server_sequence_suspend_resume then? | |
| 15:25:13 | sean-k-mooney | oh so 212 hits instead of ~500 | |
| 15:26:23 | sean-k-mooney | and some of those have failure that are not related to this | |
| 15:26:45 | sean-k-mooney | this will be annoying to fine a query for | |
| 15:26:54 | gmann | test_server_sequence_suspend_resume does not explicit create the network and test try to get one from neutron - https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L257 | |
| 15:27:58 | gmann | artom: the fix i mentioned will not help for test_server_sequence_suspend_resume , we need to do either create the network explicitly or add some common logic in scenario manager also like we do in API test | |
| 15:28:34 | lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | |
| 15:29:45 | gmann | this query -http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22HTTP%20exception%20thrown:%20Multiple%20possible%20networks%20found,%20use%20a%20Network%20ID%20to%20be%20more%20specific.%5C%22%20AND%20NOT%20message:%5C%22AttachInterfacesTestJSON%5C%22%20AND%20tags:%5C%22screen-n-api.txt%5C%22&from=7d | |
| 15:32:59 | admin0 | when i run nova-manage --debug db sync .. it says ERROR: Could not access cell0. .. i check the username and password and the user (nova_api) can acess nova_cell0 database .. what could be wrong ? | |
| 15:34:17 | admin0 | or does nova_api need access to nova database also ? | |
| 15:34:47 | sean-k-mooney | this is the query i came up with http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%5C%22%20AND%20NOT%20message:%5C%22AttachInterfacesTestJSON%5C%22%20AND%20NOT%20message:%5C%22tempest-test_creds%5C%22%20%20AND%20NOT%20build_status:%5C%22SUCCESS%5C%22&from=30d | |
| 15:36:37 | sean-k-mooney | admin0: nova manage gets the db creds from the nova.conf | |
| 15:36:54 | admin0 | i have validated it and it is correct | |
| 15:37:01 | admin0 | is there a way to debug what actually it is using | |
| 15:37:02 | gmann | yeah, AttachInterfacesTestJSON is most in failing | |
| 15:37:11 | admin0 | --debug does not seem to produce that much of a log | |
| 15:37:49 | sean-k-mooney | gmann: right but none of those builds failed | |
| 15:38:07 | lyarwood | AttributeError: module 'libvirt' has no attribute 'VIR_ERR_DEVICE_MISSING' - whops, my bad. | |
| 15:38:13 | sean-k-mooney | well wehn the failed that was nto the issue | |
| 15:38:29 | gmann | yeah | |
| 15:39:19 | admin0 | sean-k-mooney, i did a grep mysql /etc/nova.conf .. then installed mysql-client .. and then when i do mysql -h (hostname as in the conf) -u username -p ( copy paste of the password) .. it works just fine i can login and select the database | |
| 15:39:51 | gmann | artom: sean-k-mooney do you have link where test_server_sequence_suspend_resume failing ? | |
| 15:40:29 | admin0 | i did a create table and drop table ( the same test table) to validate that the login is rw | |
| 15:41:17 | sean-k-mooney | admin0: odd im not certin what the issue is off the top of my head | |
| 15:41:51 | admin0 | is there a way to get more verbose/debug steps in nova-manage db sync ? | |
| 15:44:25 | admin0 | nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0 worked fine .. only in db sync its giving issues | |
| 15:48:46 | artom | gmann, https://zuul.opendev.org/t/openstack/build/0c87381c639e4d629fa2bd48ea2da4f2 | |
| 15:49:56 | gmann | artom: thanks, i can try to build logic of creating unique network in scenario test too but may be during weekend or tomorrow. | |
| 15:50:25 | gmann | failing scenario tests on this should be less-frequent than API tests or rare | |
| 15:55:43 | lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | |
| 15:55:48 | lyarwood | ^ can anyone take this? | |
| 15:56:43 | lyarwood | or we skip this week | |
| 15:58:31 | gmann | lyarwood: sorry, i will be in internal meeting at the same time, even would not be able to join too | |
| 15:59:55 | lyarwood | or mum is home, I can run through the agenda quickly | |
| 16:00:51 | lyarwood | #openstack-meeting-3 for a very quick meeting if people have anything | |
| 16:10:57 | sean-k-mooney | very efficent of you | |
| 16:11:00 | sean-k-mooney | lyarwood: ^ | |
| 16:11:08 | sean-k-mooney | you should run all meetings :) | |
| 16:11:38 | lyarwood | haha I didn't prep and just rushed through everything, 10/10. | |
| 16:11:59 | lyarwood | I'll sort the agenda out ahead of time next week while gibi_pto_24th is still out | |
| 16:12:12 | lyarwood | apologies for the mess today | |
| 16:14:05 | tosky | I just wanted thanks for the Zuul v3 effort and please keep going but you were faster :'( | |
| 16:21:12 | lyarwood | tosky: hehe sorry and thanks, will keep pushing until M3, have the evacuate stuff worked out locally, just need to test and push. | |
| 17:17:33 | sangeet | I need help. I am doing "nova-manege db sync" and it faile with error "ERROR: Could not access cell0.". "nova-manage cell_v2 map_cell0" "nova-manage api_db sync" were run successfully before. I have ssl enabled in my deployment. The issue seems that _url_cfg.connection in _TransactionFactory object is somehow not corect for nova_cell0. Hence the DB is rejection the connection. Please see http://paste.openstack.org/show/796820/. | |
| 17:17:33 | sangeet | But connection in [cell0_database] in nova.conf is set correctly. Where is _TransactionFactory object reading the connection value form? | |
| 17:19:00 | sean-k-mooney | sangeet: your the second person to ask abou this in 2 hours :) | |
| 17:19:16 | sangeet | Interesting | |
| 17:20:04 | sangeet | sean-k-mooney was it answered? Sorry I can not scroll up for previous chats. | |
| 17:20:29 | sean-k-mooney | am no | |
| 17:20:57 | sean-k-mooney | same bhavior "nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0" bot worked by sync failed | |
| 17:21:24 | sangeet | :-( | |
| 17:21:57 | sean-k-mooney | i think if you want to run sync on cell0 | |
| 17:22:13 | sean-k-mooney | you need to create a second config adn set teh database connection to point to cell0 | |
| 17:22:43 | sean-k-mooney | at least i think you had to do that in the past e.g. nova-manage did not read cell0_database | |
| 17:22:49 | sean-k-mooney | but i could be miss remembering | |
| 17:23:03 | sangeet | That is waht I have set .. please look at http://paste.openstack.org/show/796820/. | |
| 17:23:49 | sean-k-mooney | let me compare to my devstack | |
| 17:24:46 | sean-k-mooney | so have | |
| 17:24:48 | sean-k-mooney | [database] | |