| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 14:43:30 | stephenfin | on a call but yeah | |
| 14:43:50 | kashyap | stephenfin: How much of this is possible _today_ in Nova? https://kashyapc.fedorapeople.org/NUMA-pinning.txt | |
| 14:43:58 | kashyap | stephenfin: Sure, respond when you can | |
| 14:44:45 | stephenfin | 1 and 2 | |
| 14:44:50 | kashyap | stephenfin: Context is a QEMU dev was asking about it. | |
| 14:44:59 | stephenfin | we don't provide a way to configure IOThreads | |
| 14:45:04 | kashyap | Yeah, I see from the code that we don't do anything for IOThreads | |
| 14:45:13 | efried | mriedem: what would it take to split tempest-slow-py3 into two parts? | |
| 14:45:16 | stephenfin | and I don't know what a PXB device is so let's assume not for that too | |
| 14:46:51 | kashyap | stephenfin: For that, `grep` for 'pci-expander-bus' here: https://libvirt.org/formatdomain.html | |
| 14:46:56 | mriedem | efried: why? | |
| 14:47:21 | efried | mriedem: That would make a run take ~1.5h rather than ~2.5h | |
| 14:47:29 | efried | that last hour is always just waiting for tempest-slow-py3 to finish. | |
| 14:47:40 | openstackgerrit | Merged openstack/python-novaclient master: Update master for stable/train https://review.opendev.org/683627 | |
| 14:47:42 | kashyap | stephenfin: So the suggested IOThreads 'formula' for management tools is, _assuming_ we know the required info here: | |
| 14:47:50 | mriedem | efried: what do you mean the last hour is waiting for the job to finish? | |
| 14:47:54 | mriedem | for tempest to finish? | |
| 14:47:56 | mriedem | or to collect logs? | |
| 14:48:06 | kashyap | stephenfin: num_iothreads = min(num_devices, num_vcpus, num_host_cpus) | |
| 14:48:18 | kashyap | Thanks for the answer so far | |
| 14:49:24 | efried | mriedem: I mean that for a given patch, all the other jobs take <=1.5h to complete, but tempest-slow-py3 takes ~2.5h. Trying to reduce the wait time for a single patch to make it through CI. | |
| 14:50:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check instance mapping during scheduling https://review.opendev.org/683730 | |
| 14:50:58 | mriedem | efried: well, there are a few options. one is to go the way of tempest-integrated-compute and make a compute-specific job that only runs compute tests we care about (sort of like nova-next) but only runs slow tests (which tempest-integrated-compute does not), | |
| 14:51:12 | mriedem | the other major difference is tempest-slow* is a multinode job where nova-next and tempest-integrated-compute are single node | |
| 14:51:26 | mriedem | the multinode + slow combo is important to run some tests that we don't otherwise run | |
| 14:51:45 | mriedem | one idea is just drop tempest-slow-py3 from nova's job list and make nova-next multi-node | |
| 14:51:56 | mriedem | nova-next runs compute api and scenario tests, including slow | |
| 14:52:04 | sean-k-mooney | one factor for multinode jobs is it only starts stacking the second node after the first completes | |
| 14:52:38 | efried | I'm sure I'm thinking of it way too simplistically, but what I was thinking was just, like, if tempest-slow-py3 currently runs 10 tests, make a tempest-slow-py3-1 that runs the first 5 and a tempest-slow-py3-2 that runs the second 5. | |
| 14:53:01 | efried | i.e. one additional job, but since it runs in parallel, reduces the overall run time in half. | |
| 14:53:05 | mriedem | efried: that would be a pain in the ass to manage from a tempest perspective i'd think | |
| 14:53:26 | mriedem | i.e. managing what gets run per job | |
| 14:53:29 | sean-k-mooney | efried: most of the time is spend on devstack | |
| 14:53:39 | sean-k-mooney | efried: so you would not reduce the time by half | |
| 14:53:46 | efried | mmph | |
| 14:53:51 | mriedem | also note that the scenario tests are run in serial | |
| 14:53:53 | sean-k-mooney | well that might not be quite true the slow test are slow | |
| 14:53:57 | mriedem | which i tried to do something about but failed | |
| 14:54:08 | sean-k-mooney | but devstack is still a signifcant portion | |
| 14:54:10 | efried | "tempest_concurrency":2 | |
| 14:54:12 | mriedem | https://review.opendev.org/#/c/650300/ | |
| 14:54:17 | mriedem | no, | |
| 14:54:30 | sean-k-mooney | efried: we need to run them serially | |
| 14:54:34 | mriedem | because you want the api tests running at default concurrency, which is i think nproc/2 | |
| 14:54:40 | sean-k-mooney | they can fail due to resouce constratins if we dont | |
| 14:54:41 | mriedem | sean-k-mooney: we don't *need* to | |
| 14:54:45 | efried | So the above line in the job def is... not doing anything? | |
| 14:55:04 | efried | http://zuul.openstack.org/job/tempest-slow | |
| 14:55:13 | sean-k-mooney | mriedem: we used to get intermitent failture when we didnt right? | |
| 14:55:40 | mriedem | oh geez i didn't realize tempest-slow ran *everything* with only 2 workers | |
| 14:55:53 | mriedem | sean-k-mooney: yes but that was more about ssh issues | |
| 14:55:59 | mriedem | which might have been resolved by now | |
| 14:56:10 | mriedem | because in the long ago, only the scenario tests ran ssh | |
| 14:56:20 | sean-k-mooney | mriedem: oh ok | |
| 14:56:23 | mriedem | but tempest-full has been running with ssh integration on for years now | |
| 14:56:41 | mriedem | efried: so yeah that is definitely one reason that tempest-slow is slower, it's constrained to 2 workers | |
| 14:57:03 | sean-k-mooney | the scenario tests still use ssh | |
| 14:57:16 | sean-k-mooney | but unfrotuetly some api tests also do... | |
| 14:57:16 | mriedem | sean-k-mooney: yes i know | |
| 14:57:33 | mriedem | sean-k-mooney: i'm pretty sure ^ is intentional by the QA team | |
| 14:58:22 | sean-k-mooney | doing it via the connectivyt check is fine. but the api test were ment to work with any hyperviors and should work with the fake driver | |
| 14:58:45 | sean-k-mooney | but i know that distinction has kind of faded over the years | |
| 15:00:24 | mriedem | nova-next runs with full concurrency, api and scenario, and we don't have issues from that | |
| 15:00:40 | mriedem | https://zuul.opendev.org/t/openstack/build/21573b9826664ec8a456f4e3007a91c4/log/job-output.txt#31633 | |
| 15:00:53 | sean-k-mooney | dont we run the scenario tests with serially as a seperate step | |
| 15:01:15 | sean-k-mooney | we do that in one of the jobs | |
| 15:01:15 | mriedem | Ran: 579 tests in 2771.6131 sec. | |
| 15:01:25 | mriedem | sean-k-mooney: that's what the full env does | |
| 15:01:29 | mriedem | tempest-integrated-compute | |
| 15:01:54 | mriedem | https://github.com/openstack/tempest/blob/master/tox.ini#L106 | |
| 15:02:09 | efried | mriedem: if we made nova-next multinode, we could drop tempest-slow-py3, which would reduce the number of CI nodes consumed by one... but would likely inflate nova-next to 2.5h or more, wouldn't it? | |
| 15:02:11 | sean-k-mooney | ok ya | |
| 15:02:20 | sean-k-mooney | that is why we dont have issue with concurance 4 | |
| 15:02:33 | mriedem | efried: nova-next would still be faster than tempest-slow i think b/c we'd be avoiding non-compute tests, | |
| 15:02:41 | mriedem | but it's hard to say without just proposing | |
| 15:02:57 | mriedem | efried: and we'd be running nova-next with 4 test workers rather than 2 | |
| 15:03:35 | sean-k-mooney | i did a multinoe tempest full fun for one of the cpu pinning jobs | |
| 15:03:42 | sean-k-mooney | it was just aboud 2 hours | |
| 15:04:01 | mriedem | tempest-integrated-compute SUCCESS in 1h 38m 29s | |
| 15:04:17 | mriedem | i'd say if we can stay within a reasonable comparison time-wise to tempest-integrated-compute it's a win | |
| 15:04:29 | mriedem | so make nova-next multinode and drop tempest-slow-py3 from nova runs | |
| 15:05:42 | sean-k-mooney | oh sorry tempest-full with concurrance:1 is just about 2 hours | |
| 15:06:04 | sean-k-mooney | so with concurrance:4 it should be closer to 1.5h as you suggested | |
| 15:08:41 | mriedem | stephenfin: was there any reason why you didn't remove this cellsv1 mention here? https://github.com/openstack/nova/blob/5a1c2d4ffa0815e874f373a87eb38b1833d03b24/nova/conductor/manager.py#L567 | |
| 15:09:50 | efried | sean-k-mooney: do you understand mriedem's suggestion enough to propose... whatever change(s) are necessary? | |
| 15:10:10 | stephenfin | mriedem: I started doing it but got stuck because of highly coupled tests https://review.opendev.org/#/c/651316/2/nova/conductor/manager.py I'd prefer to leave it until that patch is finished | |
| 15:10:27 | stephenfin | Or at least put in a TODO to remove all the cells v1 stuff | |
| 15:11:24 | mriedem | stephenfin: that patch is wrong anyway | |
| 15:11:32 | mriedem | you can't just drop compat code w/o a major rpc version bump | |
| 15:13:51 | stephenfin | Yeah, I don't get why we do that. I get that we can't change the signature but can't we start crapping out if a too-old client calls us? | |
| 15:14:15 | stephenfin | i.e. 'if request_spec if None: raise Exception('too old')' | |
| 15:14:18 | mriedem | i'll let dansmith answer that one | |
| 15:14:42 | cdent | if cdent: raise Exception('too old') | |
| 15:15:13 | mriedem | i ask b/c i want to drop ocata-era error handling in _populate_instance_mapping which would only be hit in the cells v1 case, which is no longer possible | |
| 15:15:35 | dansmith | stephenfin: mriedem: Of course we *can* but we're breaking the contract/rules of that 5.x means vs 6.x. So yeah, we could go all wild-west and just deprecate minor versions by raising random exceptions in the receiving code, | |
| 15:15:52 | dansmith | but then it becomes hard to reason about when we can remove things and what the impacts will be | |
| 15:16:15 | dansmith | the version numbers and rules are there to make it easier for the humans to know what falls into what bucket, IMHO | |
| 15:17:14 | mriedem | i know, i just pinged you since you're better with the words on this | |
| 15:17:21 | mriedem | "we can but we shouldn't" | |
| 15:17:23 | dansmith | okay | |