| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 13:48:20 | mriedem | gibi: technically no because it was a regression in stein | |
| 13:48:32 | mriedem | gibi: but it would be good to get it fixed in train regardless | |
| 13:49:23 | mriedem | gibi: oh i guess that's a side effect of the other bug right? | |
| 13:50:21 | sean-k-mooney | stephenfin: so the fact we cant confiugre different compute nodes with different config has come up afew times recently | |
| 13:50:42 | sean-k-mooney | we could remove that test but should we file a bug to add support for that in the future | |
| 13:50:51 | gibi | mriedem: your migrate bug is visible if the rpc is pinned to 5.0 | |
| 13:51:01 | gibi | mriedem: the qos migrate bug is visible even if the rpc is pinned to 5.1 | |
| 13:51:20 | sean-k-mooney | stephenfin: i think there has to be a way to use mocking to make the CONF non global | |
| 13:51:59 | sean-k-mooney | and in effect then allow us to have different confics in different agents althoug set_flags likely wont work in that context | |
| 13:52:33 | mriedem | gibi: right so my bug, introduced in stein, is a problem if computes are pinned to rocky (5.0) and yours if computes are pinned to stein (5.1) | |
| 13:52:47 | mriedem | gibi: then yeah it's probably worth tagging for rc1 | |
| 13:53:03 | gibi | mriedem: OK, then we are on the same page | |
| 13:55:01 | gibi | mriedem: I will respin the fix based on your rpc api method suggestion. I haven't seen the live migration way yet | |
| 13:56:34 | mriedem | it's just a simple abstraction so the caller doesn't need to know the router.client or version internals | |
| 13:57:21 | gibi | stephenfin: theortically you can go and mock things via self.compute['test_compute1'].manager.driver and that will be service and driver selective | |
| 14:07:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | |
| 14:25:24 | mriedem | oh god schedule_and_build_instances is too big, have to write 100 LOC test case just to test one new line of code | |
| 14:43:05 | kashyap | stephenfin: Hey, about? | |
| 14:43:21 | kashyap | stephenfin: When you are -- | |
| 14:43:30 | stephenfin | on a call but yeah | |
| 14:43:50 | kashyap | stephenfin: How much of this is possible _today_ in Nova? https://kashyapc.fedorapeople.org/NUMA-pinning.txt | |
| 14:43:58 | kashyap | stephenfin: Sure, respond when you can | |
| 14:44:45 | stephenfin | 1 and 2 | |
| 14:44:50 | kashyap | stephenfin: Context is a QEMU dev was asking about it. | |
| 14:44:59 | stephenfin | we don't provide a way to configure IOThreads | |
| 14:45:04 | kashyap | Yeah, I see from the code that we don't do anything for IOThreads | |
| 14:45:13 | efried | mriedem: what would it take to split tempest-slow-py3 into two parts? | |
| 14:45:16 | stephenfin | and I don't know what a PXB device is so let's assume not for that too | |
| 14:46:51 | kashyap | stephenfin: For that, `grep` for 'pci-expander-bus' here: https://libvirt.org/formatdomain.html | |
| 14:46:56 | mriedem | efried: why? | |
| 14:47:21 | efried | mriedem: That would make a run take ~1.5h rather than ~2.5h | |
| 14:47:29 | efried | that last hour is always just waiting for tempest-slow-py3 to finish. | |
| 14:47:40 | openstackgerrit | Merged openstack/python-novaclient master: Update master for stable/train https://review.opendev.org/683627 | |
| 14:47:42 | kashyap | stephenfin: So the suggested IOThreads 'formula' for management tools is, _assuming_ we know the required info here: | |
| 14:47:50 | mriedem | efried: what do you mean the last hour is waiting for the job to finish? | |
| 14:47:54 | mriedem | for tempest to finish? | |
| 14:47:56 | mriedem | or to collect logs? | |
| 14:48:06 | kashyap | stephenfin: num_iothreads = min(num_devices, num_vcpus, num_host_cpus) | |
| 14:48:18 | kashyap | Thanks for the answer so far | |
| 14:49:24 | efried | mriedem: I mean that for a given patch, all the other jobs take <=1.5h to complete, but tempest-slow-py3 takes ~2.5h. Trying to reduce the wait time for a single patch to make it through CI. | |
| 14:50:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check instance mapping during scheduling https://review.opendev.org/683730 | |
| 14:50:58 | mriedem | efried: well, there are a few options. one is to go the way of tempest-integrated-compute and make a compute-specific job that only runs compute tests we care about (sort of like nova-next) but only runs slow tests (which tempest-integrated-compute does not), | |
| 14:51:12 | mriedem | the other major difference is tempest-slow* is a multinode job where nova-next and tempest-integrated-compute are single node | |
| 14:51:26 | mriedem | the multinode + slow combo is important to run some tests that we don't otherwise run | |
| 14:51:45 | mriedem | one idea is just drop tempest-slow-py3 from nova's job list and make nova-next multi-node | |
| 14:51:56 | mriedem | nova-next runs compute api and scenario tests, including slow | |
| 14:52:04 | sean-k-mooney | one factor for multinode jobs is it only starts stacking the second node after the first completes | |
| 14:52:38 | efried | I'm sure I'm thinking of it way too simplistically, but what I was thinking was just, like, if tempest-slow-py3 currently runs 10 tests, make a tempest-slow-py3-1 that runs the first 5 and a tempest-slow-py3-2 that runs the second 5. | |
| 14:53:01 | efried | i.e. one additional job, but since it runs in parallel, reduces the overall run time in half. | |
| 14:53:05 | mriedem | efried: that would be a pain in the ass to manage from a tempest perspective i'd think | |
| 14:53:26 | mriedem | i.e. managing what gets run per job | |
| 14:53:29 | sean-k-mooney | efried: most of the time is spend on devstack | |
| 14:53:39 | sean-k-mooney | efried: so you would not reduce the time by half | |
| 14:53:46 | efried | mmph | |
| 14:53:51 | mriedem | also note that the scenario tests are run in serial | |
| 14:53:53 | sean-k-mooney | well that might not be quite true the slow test are slow | |
| 14:53:57 | mriedem | which i tried to do something about but failed | |
| 14:54:08 | sean-k-mooney | but devstack is still a signifcant portion | |
| 14:54:10 | efried | "tempest_concurrency":2 | |
| 14:54:12 | mriedem | https://review.opendev.org/#/c/650300/ | |
| 14:54:17 | mriedem | no, | |
| 14:54:30 | sean-k-mooney | efried: we need to run them serially | |
| 14:54:34 | mriedem | because you want the api tests running at default concurrency, which is i think nproc/2 | |
| 14:54:40 | sean-k-mooney | they can fail due to resouce constratins if we dont | |
| 14:54:41 | mriedem | sean-k-mooney: we don't *need* to | |
| 14:54:45 | efried | So the above line in the job def is... not doing anything? | |
| 14:55:04 | efried | http://zuul.openstack.org/job/tempest-slow | |
| 14:55:13 | sean-k-mooney | mriedem: we used to get intermitent failture when we didnt right? | |
| 14:55:40 | mriedem | oh geez i didn't realize tempest-slow ran *everything* with only 2 workers | |
| 14:55:53 | mriedem | sean-k-mooney: yes but that was more about ssh issues | |
| 14:55:59 | mriedem | which might have been resolved by now | |
| 14:56:10 | mriedem | because in the long ago, only the scenario tests ran ssh | |
| 14:56:20 | sean-k-mooney | mriedem: oh ok | |
| 14:56:23 | mriedem | but tempest-full has been running with ssh integration on for years now | |
| 14:56:41 | mriedem | efried: so yeah that is definitely one reason that tempest-slow is slower, it's constrained to 2 workers | |
| 14:57:03 | sean-k-mooney | the scenario tests still use ssh | |
| 14:57:16 | sean-k-mooney | but unfrotuetly some api tests also do... | |
| 14:57:16 | mriedem | sean-k-mooney: yes i know | |
| 14:57:33 | mriedem | sean-k-mooney: i'm pretty sure ^ is intentional by the QA team | |
| 14:58:22 | sean-k-mooney | doing it via the connectivyt check is fine. but the api test were ment to work with any hyperviors and should work with the fake driver | |
| 14:58:45 | sean-k-mooney | but i know that distinction has kind of faded over the years | |
| 15:00:24 | mriedem | nova-next runs with full concurrency, api and scenario, and we don't have issues from that | |
| 15:00:40 | mriedem | https://zuul.opendev.org/t/openstack/build/21573b9826664ec8a456f4e3007a91c4/log/job-output.txt#31633 | |
| 15:00:53 | sean-k-mooney | dont we run the scenario tests with serially as a seperate step | |
| 15:01:15 | sean-k-mooney | we do that in one of the jobs | |
| 15:01:15 | mriedem | Ran: 579 tests in 2771.6131 sec. | |
| 15:01:25 | mriedem | sean-k-mooney: that's what the full env does | |
| 15:01:29 | mriedem | tempest-integrated-compute | |
| 15:01:54 | mriedem | https://github.com/openstack/tempest/blob/master/tox.ini#L106 | |
| 15:02:09 | efried | mriedem: if we made nova-next multinode, we could drop tempest-slow-py3, which would reduce the number of CI nodes consumed by one... but would likely inflate nova-next to 2.5h or more, wouldn't it? | |
| 15:02:11 | sean-k-mooney | ok ya | |
| 15:02:20 | sean-k-mooney | that is why we dont have issue with concurance 4 | |
| 15:02:33 | mriedem | efried: nova-next would still be faster than tempest-slow i think b/c we'd be avoiding non-compute tests, | |
| 15:02:41 | mriedem | but it's hard to say without just proposing | |
| 15:02:57 | mriedem | efried: and we'd be running nova-next with 4 test workers rather than 2 | |
| 15:03:35 | sean-k-mooney | i did a multinoe tempest full fun for one of the cpu pinning jobs | |
| 15:03:42 | sean-k-mooney | it was just aboud 2 hours | |
| 15:04:01 | mriedem | tempest-integrated-compute SUCCESS in 1h 38m 29s | |
| 15:04:17 | mriedem | i'd say if we can stay within a reasonable comparison time-wise to tempest-integrated-compute it's a win | |
| 15:04:29 | mriedem | so make nova-next multinode and drop tempest-slow-py3 from nova runs | |