Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-31
15:35:51 dansmith okay always thought it was stack or nova or something
15:35:55 mriedem https://docs.openstack.org/nova/train/configuration/config.html#database.connection_recycle_time defaults to 1 hour, i wonder if we should drop that to like 10 minutes or something - not sure it would help anything
15:37:43 dansmith ah, like max_requests
15:37:53 dansmith the failure happens really early in the run though right?
15:38:06 mriedem post-upgrade?
15:38:36 dansmith yeah
15:38:42 dansmith early after starting services Imean
15:39:05 dansmith so if you set recycle to 1s maybe it would be doing lots of reconnecting and we could see if that matters, but it doesn't seem like a fix
15:39:08 dansmith connection+debug=100 maybe?
15:39:11 mriedem yeah i think so. last i checked the funny thing is we run verify_instance post-upgrade which is ok and then grenade tries to create a server or run tempest smoke and things fall over
15:39:59 dansmith yeah, but verify_instance wouldn't be the scheduler,
15:40:05 dansmith so if the scheduler is the only one affected somehow...
15:41:18 dansmith hmm
15:41:23 dansmith is scheduler getting restarted at all?
15:41:27 mriedem it's not only the scheduler, just mostly. some jobs hit it in the api too
15:41:37 mriedem yes scheduler gets restarted in the grenade run
15:41:53 dansmith I don't see it in the log
15:42:08 dansmith oh, I'm looking at a normal job
15:42:11 mriedem i have some notes here https://bugs.launchpad.net/nova/+bug/1844929/comments/3
15:42:11 openstack Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed]
15:42:33 dansmith I know, I read those previously, but I'm trying to avoid re-reading them to not get stuck down a given line of thinking
15:42:43 mriedem we restart the scheduler and it queries compute nodes and services from the cell1 db at that time ok
15:42:50 mriedem ack
15:43:18 dansmith I know we should,
15:43:38 dansmith I was just wondering if maybe we don't restart it actually and the first attempt fails
15:44:07 dansmith so,
15:44:08 dansmith https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/screen-n-sch.txt.gz
15:44:16 dansmith I see it started at 14:07:00 only
15:44:48 dansmith are we not keeping the log or something maybe?
15:46:00 mriedem it looks like that job failed smoke tests on the old side,
15:46:03 mriedem so never got to upgrade
15:46:11 mriedem https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/grenade.sh.summary.txt.gz
15:46:20 dansmith okay but the same error
15:46:36 mriedem oh that's interesting
15:51:33 dansmith the first instance of that happens well after the last "aborted connection" message in the mysql error log
15:51:54 dansmith which makes me think all those errors in the mysql log are from things we're doing during devstack setup
15:52:02 dansmith like one-shot client calls or something
15:53:09 dansmith and 18 minutes after the last "ready for connections" message from mysql
15:57:15 dansmith doesn't really look like any sort of unreasonable load or kernel events around that time
16:11:35 dansmith mriedem: https://review.opendev.org/#/c/692402/
16:12:14 dansmith mriedem: have any tricks to get a lot of grenade runs on that?
16:12:39 dansmith I know in the past we'd create like ten noop patches that depends-on something like that to get 10x the runs
16:12:47 dansmith seems irresponsible in today's climate :)
16:16:38 mriedem you could create a grenade patch that depends-on it
17:05:54 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP service: Do not perform version checks for deleted services https://review.opendev.org/692412
17:06:05 lyarwood melwitt: ^ when you're back, I'm likely missing something obvious here.
17:11:48 mriedem lyarwood: didn't mel already fix that bug in the Instance object when lazy-loading services?
17:12:25 mriedem https://review.opendev.org/#/c/582408/
17:14:19 lyarwood mriedem: that was the original bug when running through this upgrade scenario yeah but instead of ServiceNotFound we now see ServiceTooOld raised with that fix applied
17:25:06 mdbooth sean-k-mooney: https://review.opendev.org/#/c/687957/
17:29:08 sean-k-mooney mdbooth: thanks for the review. ill go though it tommorow im just about to go have dinner
17:29:28 mdbooth sean-k-mooney: ack
17:30:20 sean-k-mooney by the way i belive the way im comparing the numa toplogy constirant should compare things like pages size too
17:30:58 sean-k-mooney we dont actully want to try and fit the request to a host and compare that we just want to extrat the contaritns and assert they dont change
17:31:59 sean-k-mooney i can add some more test cases however to test different type of constraitn changes
17:32:24 sean-k-mooney to ensure that change hugpeage also will be detected ectra
17:32:52 sean-k-mooney i assume that is what you ment by poking at different levels
17:33:21 mdbooth sean-k-mooney: Yeah, I wasn't 100% on that, although I *did* go look up 1) the implementation of obj_to_primitive(), 2) the implementation of dict.__eq__ in python ;)
17:33:44 mdbooth sean-k-mooney: I think it's ok, I was just thinking for completeness
17:34:08 mdbooth But about the exception, I think it's important to positively assert that your last test case is being rejected for the reason you expect
17:34:24 sean-k-mooney ya i can do that
17:34:24 mdbooth Rather than for some other, coincidental reason
17:34:30 mdbooth (It is, btw, I checked that too)
17:34:40 sean-k-mooney the self.assertRaisss returns the exception raised
17:34:46 mdbooth It does
17:34:52 sean-k-mooney so i can save it to a var and then do more asserts
17:36:35 sean-k-mooney im pretty sure we can do that anyway
17:37:16 sean-k-mooney mdbooth: https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises
17:37:57 sean-k-mooney i know we can get teh exception via the context manager apparch but i think the non context manager form just returns the exception
17:38:06 mdbooth sean-k-mooney: You can do both
17:38:10 mdbooth You can do:
17:38:17 mdbooth ex = self.assertRaises(...)
17:38:22 sean-k-mooney yep
17:38:35 mdbooth Anyway, I don't care what colour the bike shed is ;)
17:38:38 sean-k-mooney so ill leave the exsting test tehn i can assert after on the body
17:39:09 sean-k-mooney well it was more that there is an easy way to be more specific for that last case
17:39:33 sean-k-mooney i can assert an excetion is raised and then assert what it was as you suggested
18:01:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Reset vm_state to original value if rebuild claim fails https://review.opendev.org/692185
18:24:15 melwitt mdbooth, lyarwood: thanx for the fast reviews on the get_hash_str patch!
18:54:46 efried mriedem: flavors were previously associated with instances just by ID, right?
18:55:11 efried But that was problematic because when you changed a flavor, you could no longer tell what was used to create the instance...
18:55:23 efried so we started copying the flavor content into the instance
18:55:30 efried and now we... no longer even store the original flavor ID?
18:59:52 mriedem we saved off the flavor information for a long time...not sure when that started, but it used to be in system_metadata rather than instance_extra
19:00:08 melwitt I think we still do store it, as instance_type column in the instances table
19:00:17 mriedem yes, we do
19:00:37 mriedem the embedded instance flavor was migrated from system_metadata to instance_extra back in kilo - dan did that work
19:01:03 mriedem instances.instance_type_id is the flavor.id
19:01:09 mriedem it's set on create and resize
19:01:27 efried is it used for anything, or just there kind of as a historical artifact?
19:02:01 mriedem quick grok shows the simple tenant usage api uses it
19:02:04 mriedem but probably shouldn't
19:02:30 efried okay.
19:02:31 mriedem oh it's only used there if the instance.flavor doesn't load
19:02:34 mriedem that's super old compat code though
19:02:51 mriedem it's used for filtering servers in GET /servers
19:03:07 mriedem and it's in the Migration object
19:03:13 mriedem old/new_instance_type_id
19:03:34 mriedem i think it's mostly vestigial though
19:03:36 efried And all of this (copying the flavor rather than just referring to it) is so things like rebuild behave sanely when the flavor is changed out of band.

Earlier   Later