Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-16
20:09:30 sean-k-mooney so the main issue is its not realible and we keep getting bug downstream that we cant fix
20:09:39 sean-k-mooney so we just want to stop supproting it
20:10:19 mriedem efried: is this a recent failure? b/c i'm not sure what is global about this one and it's been around awhile (the test that is)
20:10:34 efried mriedem: from what I could tell, it started hitting on 9/10
20:10:53 efried mriedem: I'm looking at the delta in test_aggregates.py for I9ab9d7d65378be564b3731b5227ede8cece71bef
20:11:06 efried https://review.opendev.org/#/c/671075/21/nova/tests/functional/test_aggregates.py
20:13:32 mriedem hmm, yeah probably related to that series
20:13:58 artom sean-k-mooney, https://review.opendev.org/#/c/682435/3
20:14:04 efried never mind, that one only merged yesterday.
20:14:10 artom Which mean that upstream whitebox has now "caught up" and can be used in tests
20:14:25 efried and its predecessor the day before
20:14:28 efried so that's not the culprit.
20:14:31 mriedem efried: so from the error it looks like the actual failure is not really the issue,
20:14:36 mriedem it's that we failed to put the instance on the host
20:14:38 mriedem b/c of the az filter
20:14:39 efried yup
20:14:47 artom Next step: update https://review.opendev.org/#/c/656890/ to move whitebox under openstack-qa/
20:14:49 sean-k-mooney artom: ok you didnt need to merge them for me to test with them
20:14:53 sean-k-mooney but ok
20:14:59 mriedem the boot server function is just waiting for the server to exit BUILD status, which it does when it changes to ERROR status
20:23:44 mriedem efried: so if i had to guess,
20:24:24 mriedem changing the base class on the test to ProviderUsageBaseTestCase but not cleaning up a bunch of the duplicate setup means we have multiple fixtures and multiple copies of the same conductor/api/scheduler services, and the api syncs informatoin to the scheduler about aggregates, which is probably not multi-scheduler aware in the functional tests,
20:24:40 mriedem so it's probably intermittent b/c sometimes we hit hte scheduler process that knows about the az metadata and sometimes we don't
20:25:40 efried mm. so why hitting since the 10th
20:25:41 mriedem i can put up a patch to cleanup that duplicate setup
20:25:50 mriedem efried: the first change to hit it in logstash was the one you pointed at
20:25:57 mriedem 671075
20:26:08 mriedem and then it was just a recheck grind
20:26:17 efried yeah...
20:26:27 mriedem see how many times someone named Eric Fried blindly rechecked it :)
20:26:44 mriedem i'll put up a patch
20:26:52 efried If only I had full debuggability from my phone.
20:27:39 mriedem or make shilpa responsible for investigating the failures and rechecking them
20:27:45 mriedem but whatever, done is done
20:30:28 efried mriedem: looks to me like everything (except the super()) through the self.computes = {} can be removed, yah?
20:30:51 mriedem yup, just did it, running the tests
20:31:11 efried k, just prepping for review :P
20:31:25 efried +2 "that's how I would have done it"
20:37:16 efried tests pass locally for me
20:37:34 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475
20:38:11 efried mriedem: two more lines
20:38:52 mriedem ack
20:39:42 efried mriedem: why only partial-bug?
20:40:02 mriedem b/c i don't know if it's the root fix
20:40:12 mriedem hence the "at least rule it out" comment
20:40:28 mriedem if we see the hits drop off in e-r by next week we can close the bug
20:40:45 efried this way requires us to remember to go back at the bug :)
20:40:48 efried but okay
20:40:58 mriedem i will when i cleanup old e-r queries
20:41:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475
20:42:57 efried thanks for the quick fix mriedem <high five>
20:43:08 mriedem <down low>
20:45:02 mriedem i wonder if the fake messaging driver doesn't do rpc fanout cast properly, if we should have some sort of fixture that blows up if you try to start more than one non-nova-compute service in a functional test
20:45:25 mriedem since that's twice in 2 weeks that we've had some issue from functional test inheritance doubling up on fixtures and stuff
20:46:02 sean-k-mooney are you suggestion on;ly allowing 1 comptue service in the fucntional tests
20:46:21 efried no
20:46:23 sean-k-mooney or havign a fixture to detect when you have more then one but have not set things up correctly
20:46:35 efried he's suggesting allowing only one *non* nova-compute service.
20:46:52 efried we clearly need multiple compute services for many tests.
20:47:01 efried but we should never need more than one superconductor
20:47:01 sean-k-mooney oh so only one conductor or scheduler
20:47:04 efried right
20:47:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove SchedulerReportClient from AggregateRequestFiltersTest https://review.opendev.org/682480
20:47:38 mriedem correct, one controller service
20:47:43 mriedem one of each
20:50:53 mriedem i think multiple apis and conductors are probably ok since those are stateless
20:51:02 mriedem but the scheduler has some stateful crap in the HostManager
20:51:06 mriedem like aggregate info
20:52:12 mriedem but yeah the test was definitely starting 2 schedulers
20:52:12 mriedem b'2019-09-16 15:10:19,491 INFO [nova.service] Starting scheduler node (version 19.1.0)'
20:52:17 mriedem b'2019-09-16 15:10:20,156 INFO [nova.service] Starting scheduler node (version 19.1.0)'
21:01:13 efried mriedem: what was the other bug with the duplicated services?
21:01:26 mriedem duplicated placement fixture
21:01:27 mriedem sec
21:01:41 mriedem https://github.com/openstack/nova/commit/5e1b096894f6de4cfbca254cf74dcfcf56358ea5#diff-5befe429f14247314e6ca487aa4e13bd
21:02:07 efried thank you sir
21:02:13 efried I'm posting a poison patch now
21:02:23 mriedem heh i was just writing my commit message ofr one
21:02:25 mriedem *for
21:02:30 efried oh, no bug for I057a07c8d0b880c8d09fc2e618ce1f7fc885beda ?
21:02:39 mriedem guess not
21:03:20 efried hm, if that one was placement fixture, my fix isn't going to hit it.
21:03:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Do not allow mutiple nova-scheduler workers https://review.opendev.org/682485
21:04:06 mriedem efried: ^ is what i was thinking of
21:04:28 efried whoah
21:04:46 efried Mine is way dumber, but more complete, and faster.
21:05:43 openstackgerrit Eric Fried proposed openstack/nova master: Only allow one non-compute service in tests https://review.opendev.org/682486
21:05:46 efried mriedem: ^
21:09:21 mriedem you're going to at least have failures on tests that intentionally start multiple api fixtures with different projects and roles (admin vs non-admin, project1 and projec2 for filtering, etc)
21:10:35 sean-k-mooney if we have multi cell tests then those would also fail correct
21:11:13 efried yeah, I guess I thought the cell conductors would have different names, but that doesn't make sense.
21:11:14 sean-k-mooney so we need a way to allow it
21:11:22 mriedem the api/conductor/scheduler services don't get registered in the cell dbs
21:11:24 mriedem so i don't think that matters here
21:12:03 mriedem but i'm also not sure how sophisticated our fixture stuff is to know where to create those services (they should go into cell0 but might not)
21:12:27 mriedem the CellDatabase fixture has a default of cell1 so the stuff probably just gets created there
21:12:40 mriedem sean-k-mooney: also btw i added https://review.opendev.org/#/c/670585/2 to https://review.opendev.org/#/c/670585/2 https://etherpad.openstack.org/p/nova-train-release-todo
21:12:43 mriedem so we don't forget
21:13:36 sean-k-mooney ok cool thanks. it would not be the end of the world if it sliped but im hopeing that we can drop supprot for this downstream before our next LTS
21:14:02 sean-k-mooney so in 24-18 months when people start using it we dont have to support it

Earlier   Later