| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-26 | |||
| 09:23:10 | bauzas | gibi: do you know if gmann did the tempest patch ? | |
| 09:23:17 | bauzas | I can check, I just didn't had the time yet | |
| 09:23:32 | gibi | bauzas: here is the tempest pin series https://review.opendev.org/q/topic:wallaby-pin-tempest it needs love | |
| 09:23:44 | bauzas | I can surely provide love | |
| 09:23:45 | gibi | the DNM test patches are failing | |
| 09:24:33 | bauzas | then I guess the love has to be on finding why the DNM patches are failing | |
| 09:24:35 | bauzas | lovely | |
| 09:24:51 | bauzas | that's just 12 hours I haven't looked at zuul files | |
| 09:48:18 | bauzas | mmm | |
| 09:48:40 | bauzas | gibi: does those skipttest exceptions in tempest look correct to you ? | |
| 09:48:42 | bauzas | 2023-01-26 01:06:13.455899 | controller | unittest2.case.SkipTest: Identity api v2 is not enabled | |
| 09:48:51 | bauzas | I have seen gmann rechecking on such errors | |
| 09:49:06 | bauzas | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0a4/871782/2/check/tempest-full-py3/0a42465/job-output.txt | |
| 09:49:17 | gibi | it does not look good, but maybe gmann rechecked on it as he changed one of the depends-ons | |
| 09:50:31 | bauzas | I just asked for a recheck | |
| 09:50:44 | bauzas | on the devstack patch | |
| 09:51:03 | bauzas | anyway, the changes themselves on both tempest and devstack seem logic to me | |
| 09:51:04 | gibi | we will see | |
| 09:51:12 | bauzas | so I guess this is just a transient issue | |
| 09:52:22 | bauzas | at least keystone was running | |
| 09:53:48 | sahid | o/ sean-k-mooney, bauzas I have noticed your new comments | |
| 09:53:51 | sahid | working on it ! | |
| 09:53:53 | sahid | thanks | |
| 09:54:33 | bauzas | sahid: thanks | |
| 09:54:47 | bauzas | sahid: ping me when you're done with those, and I'll rereview | |
| 09:55:13 | sean-k-mooney[m] | the rpc pin case is not something i orginally tought of but i dont think its a large change to just make sure we dont retrun a 500 from the api and add a test for that so hopefully it wont take too much to adress that | |
| 09:55:33 | bauzas | sahid: to clarify, sorry but you don't need to change the RPC client, just make sure that on the API you can verify it | |
| 09:56:26 | sean-k-mooney[m] | yep so add a functional test that pins to 6.0 | |
| 09:56:32 | sean-k-mooney[m] | use the new microversion | |
| 09:56:46 | sean-k-mooney[m] | and assert an excption is raised | |
| 09:56:53 | sean-k-mooney[m] | ideally it should be a 409 | |
| 09:57:06 | sean-k-mooney[m] | the same as when the compute is not upgraded | |
| 09:57:53 | sean-k-mooney[m] | you should not expose the crrent RPC pin in the excption | |
| 09:58:48 | sean-k-mooney[m] | just that the could does not meet the requirements for the new microversion and you should use the old one like the other exception | |
| 09:59:35 | sean-k-mooney[m] | bauzas actully is there any reason not to use the same excpetion here as in the api when the compute service is to low | |
| 10:00:04 | bauzas | good question | |
| 10:00:29 | bauzas | problem is, evacuation is defined by policy, right? | |
| 10:00:52 | sean-k-mooney[m] | i guess its and admin api and they might want to know but if they manually pinned. perhapse its diffent admins that do upgrade vs day to day | |
| 10:01:04 | sean-k-mooney[m] | well all apis are controlable by policy | |
| 10:01:06 | bauzas | so you can change the policy to have evacuation (without a host param) be supported for endusers | |
| 10:01:28 | bauzas | if so, we could return something like 'sorry, compute service is too low' | |
| 10:01:31 | sean-k-mooney[m] | you could | |
| 10:01:45 | bauzas | I don't know whether it would be a problem for our operators then | |
| 10:01:56 | kashyap | This timeout is blocking a couple of patches. /me is trying to find where exactly is the time_out -- https://zuul.opendev.org/t/openstack/build/453f991eeeb34498b132eb84de3301db/logs | |
| 10:02:43 | sean-k-mooney[m] | sound like just a slow node to be honest | |
| 10:03:05 | sean-k-mooney[m] | and that job might be hitting up agagisnt the timeout anyway but lests see what the normal runtime is | |
| 10:03:25 | kashyap | So only a full recheck is the only option? :( | |
| 10:03:33 | kashyap | (Already did it once) | |
| 10:03:37 | sean-k-mooney[m] | normally 90 mins or so | |
| 10:03:51 | kashyap | sean-k-mooney[m]: For the full recheck? | |
| 10:04:45 | sean-k-mooney[m] | that job normally takes 90 mins | |
| 10:04:46 | sean-k-mooney[m] | https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-compute&project=openstack/nova | |
| 10:05:26 | sean-k-mooney[m] | there have been 5 time outs in the last 300 runs of that job | |
| 10:10:07 | sean-k-mooney[m] | the 3 time outs are form 4 providers so i dont really see any corralation | |
| 10:10:33 | sean-k-mooney[m] | it would be good to see if there is anything odd in the devstack or tempet runs | |
| 10:10:49 | sean-k-mooney[m] | but something took more time then normal | |
| 10:11:31 | sean-k-mooney[m] | so yes a recheck is the way to proceed but it would be good to see if say a lot of swap was used or an image/pacakge download was slow | |
| 10:18:40 | sean-k-mooney[m] | looking at a passing run vs failing devstack too ~900 seconds vs ~1200 | |
| 10:19:29 | sean-k-mooney[m] | its seams to be pretty even across apt install pip install and osc | |
| 10:19:58 | sean-k-mooney[m] | so i think this is diskio or just general cpu/disk/net performace related | |
| 10:20:27 | sean-k-mooney[m] | if the devstack install is 33% slower the tempest execation will likely be similarly reduced in perfromace | |
| 10:20:49 | sean-k-mooney[m] | and a normal run is at 75% of the build limit so it cant really tollerate that much of a slow down | |
| 10:26:33 | sahid | bauzas, sean-k-mooney[m] humm i may missing somethinhg, this is not what you are looking for? https://review.opendev.org/c/openstack/nova/+/858384/34/nova/tests/unit/api/openstack/compute/test_evacuate.py#431 | |
| 10:27:16 | sahid | that is when using microversion 2.95 with a hosts that are not fully upgraded | |
| 10:27:22 | sean-k-mooney[m] | no | |
| 10:27:39 | sean-k-mooney[m] | that is pininng the min compute service version | |
| 10:27:57 | sean-k-mooney[m] | we were talking about the rpc version pin | |
| 10:28:47 | sean-k-mooney[m] | the ones you pin in https://docs.openstack.org/nova/latest/configuration/config.html#upgrade-levels | |
| 10:29:26 | sean-k-mooney[m] | so [upgrade_levels]/compute=6.0 | |
| 10:30:41 | bauzas | sahid: the missing unittest I was referring was to verify that you return an exception if you set the parameter and call a old compute | |
| 10:31:09 | bauzas | sahid: for the functest in the other change (the microversion one), yeah, what sean-k-mooney said | |
| 10:32:41 | sahid | bauzas: this part? https://review.opendev.org/c/openstack/nova/+/858383/25/nova/compute/manager.py#3831 | |
| 10:33:13 | bauzas | sahid: no sorry | |
| 10:33:15 | bauzas | sec | |
| 10:33:53 | bauzas | sahid: in https://review.opendev.org/c/openstack/nova/+/858383/25/nova/compute/rpcapi.py#1108 | |
| 10:33:59 | bauzas | sahid: you return an exception | |
| 10:34:10 | bauzas | and I'm eventually OK with it (sorry for the comments) | |
| 10:34:37 | bauzas | sahid: butn | |
| 10:36:12 | bauzas | sahid: I don't see any unittests for verifying the RPC call | |
| 10:36:41 | sahid | yes good point | |
| 10:37:20 | bauzas | and we also need to have unittests for the conductor RPC API, unfortunately :( | |
| 10:37:37 | bauzas | sahid: sec, will find you where we have unittests for both | |
| 10:42:52 | bauzas | sahid: one example of a simple unittest for testing it https://github.com/openstack/nova/blob/master/nova/tests/unit/conductor/test_conductor.py#L4734-L4752 | |
| 10:43:13 | bauzas | gosh, our tests are so horrible to read :( | |
| 10:43:32 | bauzas | https://github.com/openstack/nova/blob/master/nova/tests/unit/conductor/test_conductor.py#L4754-L4788 are also good examples | |
| 10:43:43 | bauzas | just add two checks there and I'm all good (for the conductor API) | |
| 10:43:56 | bauzas | (even if we apparently missed the other version checks...) | |
| 10:44:14 | sahid | working on it! thank for you help guys | |
| 10:45:32 | bauzas | sahid: for the compute RPC API, here is the existing test https://github.com/openstack/nova/blob/master/nova/tests/unit/compute/test_rpcapi.py#L883-L916 | |
| 10:45:40 | bauzas | HTH | |
| 10:45:58 | bauzas | sahid: np, you're next in my review queue once you're all set | |
| 10:51:41 | kashyap | sean-k-mooney[m]: Sorry for my delay, just reading back the scroll of your analysis | |
| 10:54:39 | sean-k-mooney | tl;dr the io on those nodes that timeed out looks like its about 33% slower then normal. we are normally at 75% of the timeout so we dont have the headroom in that case | |
| 11:09:18 | sean-k-mooney | gibi: bauzas can we land gmann's placement service role patch https://review.opendev.org/c/openstack/placement/+/865618 | |
| 12:41:38 | opendevreview | Merged openstack/nova master: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/861738 | |
| 13:15:20 | kashyap | sean-k-mooney: Thanks for the summary. It's nearly 2-ish hours, and still no sign of a vote - https://review.opendev.org/c/openstack/nova/+/870794 | |
| 13:16:39 | sean-k-mooney | its in gate | |
| 13:16:49 | sean-k-mooney | its been runing for 1hr 46mins | |
| 13:17:05 | sean-k-mooney | and tempest-integrated-compute passed already | |
| 13:17:15 | sean-k-mooney | you can watch it here if you like https://zuul.openstack.org/status#870794 | |