Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-04
15:49:20 sean-k-mooney to be fair it does also regenrate teh mysql config before it does that
15:49:38 sean-k-mooney so it need to do the restart to make sure that is applied
15:49:45 dansmith yeah
15:49:59 dansmith okay so you found 33% of them :D
15:50:30 sean-k-mooney well taht shoudl accound for 2 restart right
15:50:34 sean-k-mooney the old run and new run
15:51:06 dansmith so, package install is one start, config refresh 1, config refresh 2... is that your three starts mriedem ?
15:51:37 mriedem that's probably it
15:52:26 dansmith I don't remember the details, but weren't you saying we do something in the new phase that requires the db and then later the db seems to be gone?
15:52:35 dansmith so maybe it's something more obscure than just an ill-timed restart?
15:53:19 mriedem post-restart of the scheduler and before that timeout, the verify_instance command runs successfully which connects to the cell1 database
15:53:23 mriedem that ^?
15:53:40 dansmith yeah and then fails during actual schedule?
15:54:15 mriedem yup
15:54:22 mriedem but from the mysqld logs shit just goes bad fast it seems
15:54:39 mriedem https://zuul.opendev.org/t/openstack/build/4085120e390f4f1e971c6ff61304a596/log/logs/mysql/error.txt.gz#213
15:55:01 mriedem 2019-10-03T21:49:22.135122Z 0 [Note] Event Scheduler: Loaded 0 events 2019-10-03T21:49:22.135429Z 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.7.27-0ubuntu0.18.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) 2019-10-03T21:49:37.141359Z 4 [Note] Aborted connection 4 to db: 'keystone' user: 'root' host: 'localhost' (Got an error reading communication packets)
15:55:10 mriedem i guess within 15 seconds
15:55:47 mriedem checking the grenade log to see what happens in that window
15:55:59 mriedem yeah so final restart here:
15:56:00 mriedem 2019-10-03 21:49:20.113 | + functions-common:restart_service:2202 : sudo /bin/systemctl restart mysql
15:56:52 dansmith why wouldn't that restart be before we start any of the openstack services?
15:57:07 sean-k-mooney it is
15:57:10 sean-k-mooney its in stack.sh
15:57:29 sean-k-mooney https://github.com/openstack/devstack/blob/729f8b8ca62d0e6cba339be82ff371fa5abee62c/stack.sh#L1063
15:57:31 mriedem then we do dstat, mem tracker, etcd stuff and start setting up the keystone db
15:57:35 sean-k-mooney right before we start the services
15:57:50 mriedem yeah new side is recreating the dbs and running the sync
15:58:06 mriedem umm,
15:58:13 mriedem doesn't keystone do upgrades using db triggers?
15:58:16 mriedem lbragstad: ^
15:58:23 dansmith I think it can do either
15:58:30 sean-k-mooney mriedem: keystone sir unning under uwsgi right
15:58:39 mriedem why would that matter?
15:58:46 lbragstad mriedem it depends
15:58:56 sean-k-mooney im wondering when we restart appache
15:58:56 mriedem my point is, is there a keystone db migration late in train that f's up the db
15:59:03 mriedem or f's up mysql
15:59:20 sean-k-mooney could it be starting keysone service before we reconfig the db
15:59:59 mriedem the keystone db sync on the new side happens here https://zuul.opendev.org/t/openstack/build/4085120e390f4f1e971c6ff61304a596/log/logs/grenade.sh.txt.gz#12915
16:01:50 mriedem actually that looks like the old side db sync, so i'm confused
16:01:56 mriedem Loading script /opt/stack/old/keystone/keystone/common/sql/contract_repo/versions/025_placeholder.py
16:03:19 sean-k-mooney so in a greade run the wsgi sripts and pacakages would be installed already for keystoen so wehn we do https://github.com/openstack/devstack/blob/729f8b8ca62d0e6cba339be82ff371fa5abee62c/stack.sh#L892 which restart apache https://github.com/openstack/devstack/blob/94ca9f6756e7b677b1ee3fd2e32b555447e950dd/lib/apache#L79
16:03:33 sean-k-mooney it could start running keystone
16:03:53 sean-k-mooney that would happen before the db migration are run on the new run
16:04:01 mriedem 2019-10-03 22:17:55.659 | + functions-common:_run_under_systemd:1564 : sudo systemctl start devstack@n-sch.service
16:04:01 mriedem this is when n-sch is restarted on the new side
16:04:13 mriedem so those times don't line up with mysqld error logs puking
16:07:18 mriedem after that we verify the instance created on the old side
16:07:19 mriedem 2019-10-03 22:20:00.703 | + /opt/stack/new/grenade/projects/60_nova/resources.sh:verify:175 : nova-manage cell_v2 verify_instance --uuid cb3d18d0-3266-43b8-9288-50151fe3b68b
16:08:55 fungi is there any formal specification for network_data.json? so far i've only found an example at https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata
16:09:44 mriedem fungi: not formal no, i think there is an abandoned spec somewhere that people reference...
16:10:08 fungi ahh, okay. is it treated as a stable api at least?
16:10:35 fungi came up in working on improving ipv6 support in glean on rh-derived distros
16:11:36 mriedem this is what i could find from the original spec https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact
16:12:24 mriedem and https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/multiple-fixed-ips-network-information.html#rest-api-impact but the multiple fixed ips thing never landed...
16:12:36 mriedem fungi: it's not microversioned,
16:12:40 mriedem it's supposed to be stable,
16:12:53 mriedem but i wouldn't be surprised if things slip through the nested dicts that get in htere
16:13:15 fungi no worries, that's probably good enough, thanks!!!
16:13:31 mriedem i remember this coming up in that latter spec because it says, "The old ip_address and netmask fields found in network_data.json need to be preserved for backward compatible reasons. This means those fields will continue to contain information of the first IP address."
16:13:48 mriedem so it's like old school backward compat pre-microversion where we add keys but don't remove old ones
16:13:53 fungi neat, i'd call that fairly stable
16:14:01 mriedem the metadata api also rarely changes
16:14:07 fungi i figured
16:14:10 dansmith yeah that's what we did for the multiple dns or multple subnets or whatever change we made
16:14:18 fungi thanks again, this is helpful
16:20:48 sean-k-mooney do the peole that maintain cloud-init have a specfication for what whoudl be returned
16:21:14 sean-k-mooney i know initlaly we based our metadata api on the ec2 version but we have diveraged along time ago
16:25:41 sean-k-mooney stephenfin: i added some info on what case will not work with sev currently https://review.opendev.org/#/c/686414/1/releasenotes/notes/bug-1845986-95cbede0a296b088.yaml
16:31:48 sean-k-mooney aspiers: are you working on a new version of the sev fix by the way?
16:50:30 stephenfin efried: Comment left on https://review.opendev.org/#/c/677443/
16:50:32 stephenfin *reply
16:54:05 efried stephenfin: but I mean, did something in this patch expose the need?
16:54:25 stephenfin efried: Yeah, the option directives are indexed
16:54:36 efried or just because this is the first patch hitting docs builds since whatever
16:54:57 efried ah, thx
16:56:26 efried +A
17:01:44 stephenfin ]ta
17:24:58 openstackgerrit Merged openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807
17:25:07 openstackgerrit Balazs Gibizer proposed openstack/nova-specs master: Support move operations with qos ports - Ussuri https://review.opendev.org/686779
17:25:07 openstackgerrit Merged openstack/nova master: FUP to I30916d8d10d70ce25523fa4961007cedbdfe8ad7 https://review.opendev.org/676228
17:25:22 gibi efried, mriedem_lunch: ^^
17:26:13 gibi and now I run away to have weekend...
17:50:55 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Neutron fixture cleanup https://review.opendev.org/684343
17:50:56 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345
17:50:56 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344
17:50:57 openstackgerrit Stephen Finucane proposed openstack/nova master: fixtures: Handle iterable params for 'NeutronFixture.list_*' https://review.opendev.org/686796
17:50:57 openstackgerrit Stephen Finucane proposed openstack/nova master: fixtures: Beef up NeutronFixture https://review.opendev.org/686795
17:50:58 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Change name of network provided by NeutronFixture https://review.opendev.org/686798
17:50:58 openstackgerrit Stephen Finucane proposed openstack/nova master: fixtures: Store 'device_id' when creating port in NeutronFixture https://review.opendev.org/686797
17:50:59 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Make it obvious where we're getting our names from https://review.opendev.org/686800
17:50:59 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Stop mocking the instance network cache https://review.opendev.org/686799
17:51:00 openstackgerrit Stephen Finucane proposed openstack/nova master: fixtures: Add support for security groups https://review.opendev.org/686802
17:51:00 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Make even more nova-net stuff optional https://review.opendev.org/686801
17:58:23 mriedem ociuhandu: looks like hyperv cloudbase ci is having problems
17:58:29 mriedem http://cloudbase-ci.com/nova/656656/30/devstack-build.log.gz
17:58:40 mriedem fatal: [10.222.0.14]: FAILED! => { "msg": "The conditional check 'devstack_clone_result.rc != 0' failed. The error was: error while evaluating conditional (devstack_clone_result.rc != 0): 'dict object' has no attribute 'rc'
17:59:36 ociuhandu mriedem: thank you, I'll ping the team asap and ask them to check and fix it
17:59:43 mriedem thanks

Earlier   Later