| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-04 | |||
| 15:33:37 | mriedem | we can see things really start to go south about the time that we hit the cell timeout failures | |
| 15:33:38 | mriedem | https://zuul.opendev.org/t/openstack/build/4085120e390f4f1e971c6ff61304a596/log/logs/mysql/error.txt.gz#213 | |
| 15:33:54 | dansmith | mriedem: yeah that seems weird | |
| 15:34:02 | efried | gibi: That would be fine. If you prefer I can un-implement the train bp and you can reuse it. I don't really care too much. But I think the old process hounds have a preference for creating a new bp (-ussuri suffix). | |
| 15:34:03 | dansmith | mriedem: are we actually upgrading mysql packages? | |
| 15:34:08 | mriedem | so i guess 2 starts would mean...old devstack and new devstack | |
| 15:34:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Correctly mock out security groups in NeutronFixture https://review.opendev.org/685927 | |
| 15:34:15 | stephenfin | mriedem: ^ | |
| 15:34:25 | stephenfin | think that's what you were looking for | |
| 15:36:18 | gibi | efried: OK, I will go with a new one. thanks | |
| 15:37:50 | mriedem | dansmith: we're not upgrading packages, same version for all 3 starts: | |
| 15:37:51 | mriedem | mysqld (mysqld 5.7.27-0ubuntu0.18.04.1) starting as process | |
| 15:38:06 | dansmith | mriedem: so why are we restarting it at all? | |
| 15:38:41 | mriedem | for one old and new devstack will just blindly try to restart the service i think | |
| 15:38:47 | mriedem | i need to dig into the grenade log | |
| 15:38:49 | dansmith | odd | |
| 15:39:15 | sean-k-mooney | dansmith: we also could be tweaking the mysql config between release | |
| 15:39:25 | dansmith | mriedem: unrelated, remind me.. are we not supposed to 404 for things like specifying a missing image id in my request? | |
| 15:39:35 | sean-k-mooney | although i dont think we actully do set much in grenade | |
| 15:39:36 | dansmith | sean-k-mooney: possible, but not sure what that would be | |
| 15:39:51 | dansmith | sean-k-mooney: but maybe some implicit restart in case it has changed or something | |
| 15:39:51 | mriedem | dansmith: 400 | |
| 15:39:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Correctly mock out security groups in NeutronFixture https://review.opendev.org/685927 | |
| 15:39:55 | dansmith | mriedem: okay | |
| 15:40:02 | mriedem | dansmith: 404 is if the resouce in the path is missing i think | |
| 15:40:06 | mriedem | 400 if a thing in the body is | |
| 15:40:12 | dansmith | mriedem: okay | |
| 15:40:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_server_tags' functional tests https://review.opendev.org/684335 | |
| 15:40:32 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_server_tags' functional tests https://review.opendev.org/684335 | |
| 15:41:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_rescue' functional tests https://review.opendev.org/684339 | |
| 15:42:19 | mriedem | stephenfin: +2 | |
| 15:42:24 | mriedem | gibi: i'm +2 on https://review.opendev.org/#/c/685927/ if you want to re-+2 | |
| 15:42:34 | stephenfin | ta | |
| 15:42:37 | gibi | mriedem: going... | |
| 15:42:47 | dansmith | mriedem: sean-k-mooney: in grenade's resource phase, mysql has to be up because we validate some things without the api, but we might then stop it between the upgrade steps I guess, but wouldn't expect to restart it more than that one time | |
| 15:42:59 | stephenfin | mriedem: Fixed you commit message nit on https://review.opendev.org/#/c/684335/ if you want to blast that through too | |
| 15:44:02 | mriedem | dansmith: yeah it's the more than once i'm confused about, so grenade log digging i go | |
| 15:44:19 | dansmith | yeah | |
| 15:44:23 | sean-k-mooney | dansmith: i think devstack would restart it twice becasue it will try to install mysql for both new and old and then explcitly restart it | |
| 15:44:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_servers' functional tests https://review.opendev.org/684336 | |
| 15:44:38 | mriedem | blasted | |
| 15:44:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hosts' functional tests https://review.opendev.org/684337 | |
| 15:44:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_rescue' functional tests https://review.opendev.org/684339 | |
| 15:44:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hypervisors' functional tests https://review.opendev.org/684340 | |
| 15:45:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_attach_interfaces' functional tests https://review.opendev.org/684341 | |
| 15:45:07 | dansmith | sean-k-mooney: well, that's why I was saying I wonder if we're upgrading packages, which would be one of those restarts.. but yeah, maybe it's all expected, I dunno | |
| 15:45:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_simple_tenant_usage' functional tests https://review.opendev.org/684342 | |
| 15:46:02 | sean-k-mooney | well devstack will do an apt-get update on each run and i think that will result in 2 mysql restart on on the old code and one on the new | |
| 15:46:20 | dansmith | sean-k-mooney: only if we actually changed the mysql package right? | |
| 15:46:24 | sean-k-mooney | i dont know of a reason to require a third restart but im looking | |
| 15:46:29 | dansmith | sean-k-mooney: if we're going from xenial to xenial it won't, I wouldn't think | |
| 15:46:34 | sean-k-mooney | dansmith: im not sure if it was conditional | |
| 15:47:45 | mriedem | i doubt the devstack code is conditional | |
| 15:47:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Neutron fixture cleanup https://review.opendev.org/684343 | |
| 15:47:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345 | |
| 15:47:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344 | |
| 15:47:52 | mriedem | devstack is pretty biased to just assume fresh install | |
| 15:48:11 | mriedem | so maybe a newer mysqld package triggers a restart on any update | |
| 15:48:11 | dansmith | mriedem: but we're not changing OS releases or pinning OS packages right? so the apt upgrade between grenade phases shouldn't change the packages for that right? | |
| 15:48:24 | mriedem | the version isn't changing | |
| 15:48:25 | sean-k-mooney | ya its not https://github.com/openstack/devstack/blob/master/lib/databases/mysql#L129 | |
| 15:48:26 | dansmith | a newer mysql package will definitely do a restart | |
| 15:48:42 | dansmith | I just expect no package change | |
| 15:48:54 | mriedem | devstack is just outright restarting as sean pointed out | |
| 15:49:17 | dansmith | oh I see | |
| 15:49:20 | dansmith | so that's one | |
| 15:49:20 | sean-k-mooney | to be fair it does also regenrate teh mysql config before it does that | |
| 15:49:38 | sean-k-mooney | so it need to do the restart to make sure that is applied | |
| 15:49:45 | dansmith | yeah | |
| 15:49:59 | dansmith | okay so you found 33% of them :D | |
| 15:50:30 | sean-k-mooney | well taht shoudl accound for 2 restart right | |
| 15:50:34 | sean-k-mooney | the old run and new run | |
| 15:51:06 | dansmith | so, package install is one start, config refresh 1, config refresh 2... is that your three starts mriedem ? | |
| 15:51:37 | mriedem | that's probably it | |
| 15:52:26 | dansmith | I don't remember the details, but weren't you saying we do something in the new phase that requires the db and then later the db seems to be gone? | |
| 15:52:35 | dansmith | so maybe it's something more obscure than just an ill-timed restart? | |
| 15:53:19 | mriedem | post-restart of the scheduler and before that timeout, the verify_instance command runs successfully which connects to the cell1 database | |
| 15:53:23 | mriedem | that ^? | |
| 15:53:40 | dansmith | yeah and then fails during actual schedule? | |
| 15:54:15 | mriedem | yup | |
| 15:54:22 | mriedem | but from the mysqld logs shit just goes bad fast it seems | |
| 15:54:39 | mriedem | https://zuul.opendev.org/t/openstack/build/4085120e390f4f1e971c6ff61304a596/log/logs/mysql/error.txt.gz#213 | |
| 15:55:01 | mriedem | 2019-10-03T21:49:22.135122Z 0 [Note] Event Scheduler: Loaded 0 events 2019-10-03T21:49:22.135429Z 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.7.27-0ubuntu0.18.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) 2019-10-03T21:49:37.141359Z 4 [Note] Aborted connection 4 to db: 'keystone' user: 'root' host: 'localhost' (Got an error reading communication packets) | |
| 15:55:10 | mriedem | i guess within 15 seconds | |
| 15:55:47 | mriedem | checking the grenade log to see what happens in that window | |
| 15:55:59 | mriedem | yeah so final restart here: | |
| 15:56:00 | mriedem | 2019-10-03 21:49:20.113 | + functions-common:restart_service:2202 : sudo /bin/systemctl restart mysql | |
| 15:56:52 | dansmith | why wouldn't that restart be before we start any of the openstack services? | |
| 15:57:07 | sean-k-mooney | it is | |
| 15:57:10 | sean-k-mooney | its in stack.sh | |
| 15:57:29 | sean-k-mooney | https://github.com/openstack/devstack/blob/729f8b8ca62d0e6cba339be82ff371fa5abee62c/stack.sh#L1063 | |
| 15:57:31 | mriedem | then we do dstat, mem tracker, etcd stuff and start setting up the keystone db | |
| 15:57:35 | sean-k-mooney | right before we start the services | |
| 15:57:50 | mriedem | yeah new side is recreating the dbs and running the sync | |
| 15:58:06 | mriedem | umm, | |
| 15:58:13 | mriedem | doesn't keystone do upgrades using db triggers? | |
| 15:58:16 | mriedem | lbragstad: ^ | |
| 15:58:23 | dansmith | I think it can do either | |
| 15:58:30 | sean-k-mooney | mriedem: keystone sir unning under uwsgi right | |
| 15:58:39 | mriedem | why would that matter? | |