| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 14:52:04 | sean-k-mooney | one factor for multinode jobs is it only starts stacking the second node after the first completes | |
| 14:52:38 | efried | I'm sure I'm thinking of it way too simplistically, but what I was thinking was just, like, if tempest-slow-py3 currently runs 10 tests, make a tempest-slow-py3-1 that runs the first 5 and a tempest-slow-py3-2 that runs the second 5. | |
| 14:53:01 | efried | i.e. one additional job, but since it runs in parallel, reduces the overall run time in half. | |
| 14:53:05 | mriedem | efried: that would be a pain in the ass to manage from a tempest perspective i'd think | |
| 14:53:26 | mriedem | i.e. managing what gets run per job | |
| 14:53:29 | sean-k-mooney | efried: most of the time is spend on devstack | |
| 14:53:39 | sean-k-mooney | efried: so you would not reduce the time by half | |
| 14:53:46 | efried | mmph | |
| 14:53:51 | mriedem | also note that the scenario tests are run in serial | |
| 14:53:53 | sean-k-mooney | well that might not be quite true the slow test are slow | |
| 14:53:57 | mriedem | which i tried to do something about but failed | |
| 14:54:08 | sean-k-mooney | but devstack is still a signifcant portion | |
| 14:54:10 | efried | "tempest_concurrency":2 | |
| 14:54:12 | mriedem | https://review.opendev.org/#/c/650300/ | |
| 14:54:17 | mriedem | no, | |
| 14:54:30 | sean-k-mooney | efried: we need to run them serially | |
| 14:54:34 | mriedem | because you want the api tests running at default concurrency, which is i think nproc/2 | |
| 14:54:40 | sean-k-mooney | they can fail due to resouce constratins if we dont | |
| 14:54:41 | mriedem | sean-k-mooney: we don't *need* to | |
| 14:54:45 | efried | So the above line in the job def is... not doing anything? | |
| 14:55:04 | efried | http://zuul.openstack.org/job/tempest-slow | |
| 14:55:13 | sean-k-mooney | mriedem: we used to get intermitent failture when we didnt right? | |
| 14:55:40 | mriedem | oh geez i didn't realize tempest-slow ran *everything* with only 2 workers | |
| 14:55:53 | mriedem | sean-k-mooney: yes but that was more about ssh issues | |
| 14:55:59 | mriedem | which might have been resolved by now | |
| 14:56:10 | mriedem | because in the long ago, only the scenario tests ran ssh | |
| 14:56:20 | sean-k-mooney | mriedem: oh ok | |
| 14:56:23 | mriedem | but tempest-full has been running with ssh integration on for years now | |
| 14:56:41 | mriedem | efried: so yeah that is definitely one reason that tempest-slow is slower, it's constrained to 2 workers | |
| 14:57:03 | sean-k-mooney | the scenario tests still use ssh | |
| 14:57:16 | sean-k-mooney | but unfrotuetly some api tests also do... | |
| 14:57:16 | mriedem | sean-k-mooney: yes i know | |
| 14:57:33 | mriedem | sean-k-mooney: i'm pretty sure ^ is intentional by the QA team | |
| 14:58:22 | sean-k-mooney | doing it via the connectivyt check is fine. but the api test were ment to work with any hyperviors and should work with the fake driver | |
| 14:58:45 | sean-k-mooney | but i know that distinction has kind of faded over the years | |
| 15:00:24 | mriedem | nova-next runs with full concurrency, api and scenario, and we don't have issues from that | |
| 15:00:40 | mriedem | https://zuul.opendev.org/t/openstack/build/21573b9826664ec8a456f4e3007a91c4/log/job-output.txt#31633 | |
| 15:00:53 | sean-k-mooney | dont we run the scenario tests with serially as a seperate step | |
| 15:01:15 | sean-k-mooney | we do that in one of the jobs | |
| 15:01:15 | mriedem | Ran: 579 tests in 2771.6131 sec. | |
| 15:01:25 | mriedem | sean-k-mooney: that's what the full env does | |
| 15:01:29 | mriedem | tempest-integrated-compute | |
| 15:01:54 | mriedem | https://github.com/openstack/tempest/blob/master/tox.ini#L106 | |
| 15:02:09 | efried | mriedem: if we made nova-next multinode, we could drop tempest-slow-py3, which would reduce the number of CI nodes consumed by one... but would likely inflate nova-next to 2.5h or more, wouldn't it? | |
| 15:02:11 | sean-k-mooney | ok ya | |
| 15:02:20 | sean-k-mooney | that is why we dont have issue with concurance 4 | |
| 15:02:33 | mriedem | efried: nova-next would still be faster than tempest-slow i think b/c we'd be avoiding non-compute tests, | |
| 15:02:41 | mriedem | but it's hard to say without just proposing | |
| 15:02:57 | mriedem | efried: and we'd be running nova-next with 4 test workers rather than 2 | |
| 15:03:35 | sean-k-mooney | i did a multinoe tempest full fun for one of the cpu pinning jobs | |
| 15:03:42 | sean-k-mooney | it was just aboud 2 hours | |
| 15:04:01 | mriedem | tempest-integrated-compute SUCCESS in 1h 38m 29s | |
| 15:04:17 | mriedem | i'd say if we can stay within a reasonable comparison time-wise to tempest-integrated-compute it's a win | |
| 15:04:29 | mriedem | so make nova-next multinode and drop tempest-slow-py3 from nova runs | |
| 15:05:42 | sean-k-mooney | oh sorry tempest-full with concurrance:1 is just about 2 hours | |
| 15:06:04 | sean-k-mooney | so with concurrance:4 it should be closer to 1.5h as you suggested | |
| 15:08:41 | mriedem | stephenfin: was there any reason why you didn't remove this cellsv1 mention here? https://github.com/openstack/nova/blob/5a1c2d4ffa0815e874f373a87eb38b1833d03b24/nova/conductor/manager.py#L567 | |
| 15:09:50 | efried | sean-k-mooney: do you understand mriedem's suggestion enough to propose... whatever change(s) are necessary? | |
| 15:10:10 | stephenfin | mriedem: I started doing it but got stuck because of highly coupled tests https://review.opendev.org/#/c/651316/2/nova/conductor/manager.py I'd prefer to leave it until that patch is finished | |
| 15:10:27 | stephenfin | Or at least put in a TODO to remove all the cells v1 stuff | |
| 15:11:24 | mriedem | stephenfin: that patch is wrong anyway | |
| 15:11:32 | mriedem | you can't just drop compat code w/o a major rpc version bump | |
| 15:13:51 | stephenfin | Yeah, I don't get why we do that. I get that we can't change the signature but can't we start crapping out if a too-old client calls us? | |
| 15:14:15 | stephenfin | i.e. 'if request_spec if None: raise Exception('too old')' | |
| 15:14:18 | mriedem | i'll let dansmith answer that one | |
| 15:14:42 | cdent | if cdent: raise Exception('too old') | |
| 15:15:13 | mriedem | i ask b/c i want to drop ocata-era error handling in _populate_instance_mapping which would only be hit in the cells v1 case, which is no longer possible | |
| 15:15:35 | dansmith | stephenfin: mriedem: Of course we *can* but we're breaking the contract/rules of that 5.x means vs 6.x. So yeah, we could go all wild-west and just deprecate minor versions by raising random exceptions in the receiving code, | |
| 15:15:52 | dansmith | but then it becomes hard to reason about when we can remove things and what the impacts will be | |
| 15:16:15 | dansmith | the version numbers and rules are there to make it easier for the humans to know what falls into what bucket, IMHO | |
| 15:17:14 | mriedem | i know, i just pinged you since you're better with the words on this | |
| 15:17:21 | mriedem | "we can but we shouldn't" | |
| 15:17:23 | dansmith | okay | |
| 15:17:26 | mriedem | bad habits etc | |
| 15:17:52 | stephenfin | Would bumping the major version each release be too expensive? | |
| 15:18:01 | mriedem | conductor doesn't need to change that often | |
| 15:18:02 | dansmith | it used to be done almost every release | |
| 15:18:08 | dansmith | but we don't change that much anymore, | |
| 15:18:12 | dansmith | so yeah, I think it's too expensive | |
| 15:18:12 | mriedem | but we are definitely due for a conductor comptue task api 2.0 | |
| 15:18:20 | mriedem | b/c there is a lot of old shit in here | |
| 15:18:31 | stephenfin | maybe that's the solution so | |
| 15:18:37 | mriedem | once we go to 2.0 that _populate_instance_mapping just gets dropped | |
| 15:19:11 | stephenfin | I can't remove that stuff yet but I can go to 2.0 and then drop that stuff | |
| 15:19:23 | openstackgerrit | Merged openstack/os-resource-classes master: Update the constraints url https://review.opendev.org/683872 | |
| 15:19:38 | stephenfin | mriedem: While I have you - can we remove the os-networks entirely when we drop nova-net? | |
| 15:20:08 | mriedem | i had an etherpad with notes about the more complicated apis that involved networks b/c they aren't all nova-net only anymore | |
| 15:20:09 | stephenfin | It seems some of the APIs work with neutron but most don't, and I'm trying to decide if we should selectively 404 them or 404 everything | |
| 15:20:15 | mriedem | that's the issue | |
| 15:20:24 | mriedem | if there are apis that work with neutron we can't just 410 those | |
| 15:20:30 | stephenfin | 410, sorry | |
| 15:20:32 | stephenfin | yeah | |
| 15:22:03 | stephenfin | I was afraid you'd say that :( Time to rework again | |
| 15:22:09 | stephenfin | Any idea where that old etherpad is? | |
| 15:22:10 | mriedem | btw this is the last time we did an rpc api major version bump https://review.opendev.org/#/c/541005/ | |
| 15:22:31 | mriedem | https://etherpad.openstack.org/p/nova-network-removal-rocky | |
| 15:22:39 | stephenfin | I've a rough idea from converting all the API sample functional tests over but maybe I've missed some stuff | |
| 15:25:24 | stephenfin | gibi: Just in case, you haven't tried adding floating IP stuff to NeutronFixture or some subclass, have you? | |
| 15:25:52 | gibi | stephenfin: let me check some notification tests | |
| 15:26:27 | stephenfin | gibi: It would be local if so. We don't have it in tree | |