| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-04 | |||
| 15:44:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hosts' functional tests https://review.opendev.org/684337 | |
| 15:44:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_rescue' functional tests https://review.opendev.org/684339 | |
| 15:44:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hypervisors' functional tests https://review.opendev.org/684340 | |
| 15:45:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_attach_interfaces' functional tests https://review.opendev.org/684341 | |
| 15:45:07 | dansmith | sean-k-mooney: well, that's why I was saying I wonder if we're upgrading packages, which would be one of those restarts.. but yeah, maybe it's all expected, I dunno | |
| 15:45:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_simple_tenant_usage' functional tests https://review.opendev.org/684342 | |
| 15:46:02 | sean-k-mooney | well devstack will do an apt-get update on each run and i think that will result in 2 mysql restart on on the old code and one on the new | |
| 15:46:20 | dansmith | sean-k-mooney: only if we actually changed the mysql package right? | |
| 15:46:24 | sean-k-mooney | i dont know of a reason to require a third restart but im looking | |
| 15:46:29 | dansmith | sean-k-mooney: if we're going from xenial to xenial it won't, I wouldn't think | |
| 15:46:34 | sean-k-mooney | dansmith: im not sure if it was conditional | |
| 15:47:45 | mriedem | i doubt the devstack code is conditional | |
| 15:47:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Neutron fixture cleanup https://review.opendev.org/684343 | |
| 15:47:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345 | |
| 15:47:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344 | |
| 15:47:52 | mriedem | devstack is pretty biased to just assume fresh install | |
| 15:48:11 | mriedem | so maybe a newer mysqld package triggers a restart on any update | |
| 15:48:11 | dansmith | mriedem: but we're not changing OS releases or pinning OS packages right? so the apt upgrade between grenade phases shouldn't change the packages for that right? | |
| 15:48:24 | mriedem | the version isn't changing | |
| 15:48:25 | sean-k-mooney | ya its not https://github.com/openstack/devstack/blob/master/lib/databases/mysql#L129 | |
| 15:48:26 | dansmith | a newer mysql package will definitely do a restart | |
| 15:48:42 | dansmith | I just expect no package change | |
| 15:48:54 | mriedem | devstack is just outright restarting as sean pointed out | |
| 15:49:17 | dansmith | oh I see | |
| 15:49:20 | dansmith | so that's one | |
| 15:49:20 | sean-k-mooney | to be fair it does also regenrate teh mysql config before it does that | |
| 15:49:38 | sean-k-mooney | so it need to do the restart to make sure that is applied | |
| 15:49:45 | dansmith | yeah | |
| 15:49:59 | dansmith | okay so you found 33% of them :D | |
| 15:50:30 | sean-k-mooney | well taht shoudl accound for 2 restart right | |
| 15:50:34 | sean-k-mooney | the old run and new run | |
| 15:51:06 | dansmith | so, package install is one start, config refresh 1, config refresh 2... is that your three starts mriedem ? | |
| 15:51:37 | mriedem | that's probably it | |
| 15:52:26 | dansmith | I don't remember the details, but weren't you saying we do something in the new phase that requires the db and then later the db seems to be gone? | |
| 15:52:35 | dansmith | so maybe it's something more obscure than just an ill-timed restart? | |
| 15:53:19 | mriedem | post-restart of the scheduler and before that timeout, the verify_instance command runs successfully which connects to the cell1 database | |
| 15:53:23 | mriedem | that ^? | |
| 15:53:40 | dansmith | yeah and then fails during actual schedule? | |
| 15:54:15 | mriedem | yup | |
| 15:54:22 | mriedem | but from the mysqld logs shit just goes bad fast it seems | |
| 15:54:39 | mriedem | https://zuul.opendev.org/t/openstack/build/4085120e390f4f1e971c6ff61304a596/log/logs/mysql/error.txt.gz#213 | |
| 15:55:01 | mriedem | 2019-10-03T21:49:22.135122Z 0 [Note] Event Scheduler: Loaded 0 events 2019-10-03T21:49:22.135429Z 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.7.27-0ubuntu0.18.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) 2019-10-03T21:49:37.141359Z 4 [Note] Aborted connection 4 to db: 'keystone' user: 'root' host: 'localhost' (Got an error reading communication packets) | |
| 15:55:10 | mriedem | i guess within 15 seconds | |
| 15:55:47 | mriedem | checking the grenade log to see what happens in that window | |
| 15:55:59 | mriedem | yeah so final restart here: | |
| 15:56:00 | mriedem | 2019-10-03 21:49:20.113 | + functions-common:restart_service:2202 : sudo /bin/systemctl restart mysql | |
| 15:56:52 | dansmith | why wouldn't that restart be before we start any of the openstack services? | |
| 15:57:07 | sean-k-mooney | it is | |
| 15:57:10 | sean-k-mooney | its in stack.sh | |
| 15:57:29 | sean-k-mooney | https://github.com/openstack/devstack/blob/729f8b8ca62d0e6cba339be82ff371fa5abee62c/stack.sh#L1063 | |
| 15:57:31 | mriedem | then we do dstat, mem tracker, etcd stuff and start setting up the keystone db | |
| 15:57:35 | sean-k-mooney | right before we start the services | |
| 15:57:50 | mriedem | yeah new side is recreating the dbs and running the sync | |
| 15:58:06 | mriedem | umm, | |
| 15:58:13 | mriedem | doesn't keystone do upgrades using db triggers? | |
| 15:58:16 | mriedem | lbragstad: ^ | |
| 15:58:23 | dansmith | I think it can do either | |
| 15:58:30 | sean-k-mooney | mriedem: keystone sir unning under uwsgi right | |
| 15:58:39 | mriedem | why would that matter? | |
| 15:58:46 | lbragstad | mriedem it depends | |
| 15:58:56 | sean-k-mooney | im wondering when we restart appache | |
| 15:58:56 | mriedem | my point is, is there a keystone db migration late in train that f's up the db | |
| 15:59:03 | mriedem | or f's up mysql | |
| 15:59:20 | sean-k-mooney | could it be starting keysone service before we reconfig the db | |
| 15:59:59 | mriedem | the keystone db sync on the new side happens here https://zuul.opendev.org/t/openstack/build/4085120e390f4f1e971c6ff61304a596/log/logs/grenade.sh.txt.gz#12915 | |
| 16:01:50 | mriedem | actually that looks like the old side db sync, so i'm confused | |
| 16:01:56 | mriedem | Loading script /opt/stack/old/keystone/keystone/common/sql/contract_repo/versions/025_placeholder.py | |
| 16:03:19 | sean-k-mooney | so in a greade run the wsgi sripts and pacakages would be installed already for keystoen so wehn we do https://github.com/openstack/devstack/blob/729f8b8ca62d0e6cba339be82ff371fa5abee62c/stack.sh#L892 which restart apache https://github.com/openstack/devstack/blob/94ca9f6756e7b677b1ee3fd2e32b555447e950dd/lib/apache#L79 | |
| 16:03:33 | sean-k-mooney | it could start running keystone | |
| 16:03:53 | sean-k-mooney | that would happen before the db migration are run on the new run | |
| 16:04:01 | mriedem | 2019-10-03 22:17:55.659 | + functions-common:_run_under_systemd:1564 : sudo systemctl start devstack@n-sch.service | |
| 16:04:01 | mriedem | this is when n-sch is restarted on the new side | |
| 16:04:13 | mriedem | so those times don't line up with mysqld error logs puking | |
| 16:07:18 | mriedem | after that we verify the instance created on the old side | |
| 16:07:19 | mriedem | 2019-10-03 22:20:00.703 | + /opt/stack/new/grenade/projects/60_nova/resources.sh:verify:175 : nova-manage cell_v2 verify_instance --uuid cb3d18d0-3266-43b8-9288-50151fe3b68b | |
| 16:08:55 | fungi | is there any formal specification for network_data.json? so far i've only found an example at https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata | |
| 16:09:44 | mriedem | fungi: not formal no, i think there is an abandoned spec somewhere that people reference... | |
| 16:10:08 | fungi | ahh, okay. is it treated as a stable api at least? | |
| 16:10:35 | fungi | came up in working on improving ipv6 support in glean on rh-derived distros | |
| 16:11:36 | mriedem | this is what i could find from the original spec https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact | |
| 16:12:24 | mriedem | and https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/multiple-fixed-ips-network-information.html#rest-api-impact but the multiple fixed ips thing never landed... | |
| 16:12:36 | mriedem | fungi: it's not microversioned, | |
| 16:12:40 | mriedem | it's supposed to be stable, | |
| 16:12:53 | mriedem | but i wouldn't be surprised if things slip through the nested dicts that get in htere | |
| 16:13:15 | fungi | no worries, that's probably good enough, thanks!!! | |
| 16:13:31 | mriedem | i remember this coming up in that latter spec because it says, "The old ip_address and netmask fields found in network_data.json need to be preserved for backward compatible reasons. This means those fields will continue to contain information of the first IP address." | |
| 16:13:48 | mriedem | so it's like old school backward compat pre-microversion where we add keys but don't remove old ones | |
| 16:13:53 | fungi | neat, i'd call that fairly stable | |
| 16:14:01 | mriedem | the metadata api also rarely changes | |
| 16:14:07 | fungi | i figured | |
| 16:14:10 | dansmith | yeah that's what we did for the multiple dns or multple subnets or whatever change we made | |
| 16:14:18 | fungi | thanks again, this is helpful | |
| 16:20:48 | sean-k-mooney | do the peole that maintain cloud-init have a specfication for what whoudl be returned | |
| 16:21:14 | sean-k-mooney | i know initlaly we based our metadata api on the ec2 version but we have diveraged along time ago | |
| 16:25:41 | sean-k-mooney | stephenfin: i added some info on what case will not work with sev currently https://review.opendev.org/#/c/686414/1/releasenotes/notes/bug-1845986-95cbede0a296b088.yaml | |
| 16:31:48 | sean-k-mooney | aspiers: are you working on a new version of the sev fix by the way? | |
| 16:50:30 | stephenfin | efried: Comment left on https://review.opendev.org/#/c/677443/ | |
| 16:50:32 | stephenfin | *reply | |
| 16:54:05 | efried | stephenfin: but I mean, did something in this patch expose the need? | |
| 16:54:25 | stephenfin | efried: Yeah, the option directives are indexed | |