| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-03 | |||
| 07:46:33 | bauzas | huzzaaaaah \o/ https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-n-cpu.txt?severity=4#12685-12689 | |
| 07:46:44 | bauzas | NotImplementedError: direct_snapshot() is not implemented | |
| 07:47:12 | bauzas | looks like we regressed | |
| 07:51:29 | bauzas | hmmmm | |
| 07:51:53 | lyarwood | Morning | |
| 07:52:09 | lyarwood | yeah I've not had time to look at nova-lvm failures thanks to a few downstream fires | |
| 07:52:17 | lyarwood | they should be out now so I'll look this morning | |
| 07:52:20 | bauzas | ok, I have better understanding | |
| 07:52:22 | lyarwood | bauzas: that isn't the failure | |
| 07:52:25 | bauzas | correct | |
| 07:52:33 | bauzas | we catch the notimplemented exception | |
| 07:52:36 | bauzas | it's failing later | |
| 07:52:51 | bauzas | when calling glance I'd say | |
| 07:53:06 | bauzas | https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-n-cpu.txt?severity=4#12772 | |
| 07:53:08 | lyarwood | yeah the g-api 502 is something I think I've seen before when we run out of space during the upload | |
| 07:53:27 | bauzas | yup | |
| 07:53:57 | bauzas | we want to upload the snapshot and we fail, right? | |
| 07:54:22 | lyarwood | we are in the process of uploading the snapshot and something fails | |
| 07:54:29 | lyarwood | it might even be a timeout in httpd | |
| 07:54:39 | lyarwood | given the size of the images we are uploading for these tests | |
| 07:55:18 | lyarwood | brb | |
| 07:56:23 | bauzas | lyarwood: https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-g-api.txt?severity=3#3555 | |
| 07:56:37 | bauzas | this error matches the req-id of the n-cpu issue | |
| 08:11:54 | bauzas | well, can't figure out more why the temporary image fails | |
| 08:12:09 | bauzas | abhishekk: around ? | |
| 08:12:43 | bauzas | abhishekk: we have a nova job failure (nova-lvm) which fails constantly because of glance API returning 502 https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-g-api.txt?severity=3#3555 | |
| 08:12:55 | bauzas | on an image upload | |
| 08:34:47 | abhishekk | bauzas, looking | |
| 08:35:02 | bauzas | abhishekk: thanks, wrote an email about this | |
| 08:35:10 | bauzas | (to summarize) | |
| 08:35:17 | abhishekk | yes, just went through it | |
| 08:36:35 | abhishekk | bauzas, are you sure failure is because of this warning ? | |
| 08:37:01 | bauzas | abhishekk: I'm nearly sure the test is failing because glance is returning a 502 | |
| 08:37:25 | abhishekk | ok, will have a look and get back to you | |
| 08:38:05 | bauzas | abhishekk: but I relate the glance g-api warning with the same req-id of the failing glance image upload | |
| 08:38:25 | bauzas | correlation isn't consequence, I know, but that's a starting point | |
| 08:38:33 | abhishekk | ack | |
| 08:39:17 | lyarwood | bauzas: the test is 100% failing because of the 502 it just isn't clear why g-api or httpd is returning it | |
| 08:39:27 | lyarwood | sorry got my head stuck downstream again | |
| 08:39:40 | bauzas | yup, that's what I said to abhishekk | |
| 08:39:53 | bauzas | we have an image upload failure | |
| 08:40:00 | bauzas | but we don't know why | |
| 08:40:57 | bauzas | lyarwood: no worries about downstream rush, I feel your pain, maybe just one thought : as this is failing, should we make the job non-voting until we figure out the cause ? | |
| 08:41:16 | lyarwood | Yup happy to if you can raise a bug | |
| 08:41:29 | bauzas | good call, I forgot this :facepalm: | |
| 08:43:06 | abhishekk | I can see HTTP 413 in g-api log | |
| 08:46:20 | bauzas | lyarwood: abhishekk: gibi: https://bugs.launchpad.net/nova/+bug/1938765 | |
| 08:48:38 | abhishekk | ack | |
| 08:53:34 | bauzas | I need to disappear for 10 mins | |
| 09:00:37 | abhishekk | I think this is causing problem, DEBUG oslo.limit.limit [None req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-ListImageFiltersTestJSON-1499681503 tempest-ListImageFiltersTestJSON-1499681503-project] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1024 and delta 0] {{(pid=108370) enforce_limits /usr/local/lib/python3.8/dist-packages/oslo_limit/limit.py:268}} | |
| 09:00:37 | abhishekk | Aug 01 03:51:33.042224 ubuntu-focal-ovh-bhs1-0025715861 devstack@g-api.service[108370]: [pid: 108370|app: 0|req: 188/333] 127.0.0.1 () {46 vars in 1190 bytes} [Sun Aug 1 03:51:32 2021] PUT /v2/images/4d204807-d5e5-4736-821f-fae99a0c066e/file => generated 448 bytes in 743 msecs (HTTP/1.1 413) 4 headers in 187 bytes (1 switches on core 0) | |
| 09:01:23 | lyarwood | sounds about right, these images are going to be pretty big | |
| 09:02:23 | lyarwood | https://opendev.org/openstack/devstack/commit/5a642450d6ac94ff1ea2bea3e7ce3887ca79dcc4 | |
| 09:02:27 | lyarwood | there we go | |
| 09:03:49 | abhishekk | I think this will fix the problem | |
| 09:11:40 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm https://review.opendev.org/c/openstack/nova/+/803322 | |
| 09:11:41 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM - Trigger nova-lvm to test I824655387a10ac9b813c1b7b2399e25ed95f7fc3 https://review.opendev.org/c/openstack/nova/+/803323 | |
| 09:11:43 | lyarwood | bauzas: / abhishekk ; ^ there we go | |
| 09:12:11 | abhishekk | lyarwood, ack | |
| 09:16:53 | bauzas | lyarwood: and we return a 502 if we go over the limit ? :O | |
| 09:18:23 | lyarwood | bauzas: 413 but somehow that's translated into a 503 by the time n-cpu sees it | |
| 09:18:38 | lyarwood | bauzas: that could be a separate bug report against devstack tbh | |
| 09:18:48 | bauzas | yup, agreed | |
| 09:18:57 | lyarwood | 502 even sorry | |
| 09:19:00 | bauzas | either way, excellent catch | |
| 09:19:14 | bauzas | I didn't know about glance limiting now the uploads | |
| 09:19:15 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests https://review.opendev.org/c/openstack/nova/+/803226 | |
| 09:34:33 | abhishekk | I see 6 413 in glance logs, that sums up total failure cases in the job | |
| 09:59:39 | abhishekk | job is passing now | |
| 10:04:56 | lyarwood | abhishekk: awesome thanks for your help with that :) | |
| 10:34:59 | gibi | lyarwood, bauzas thanks for fixing that up. | |
| 10:56:21 | lyarwood | grenade jobs blocked by https://bugs.launchpad.net/grenade/+bug/1938763 FWIW | |
| 10:58:01 | gibi | lyarwood: thanks I saw the error in the job run but I did not know that it is filed as a bug. I only see one failure in grenade by this so far | |
| 10:59:23 | lyarwood | np https://zuul.opendev.org/t/openstack/builds?job_name=grenade&job_name=nova-grenade-multinode&job_name=grenade-multinode&branch=master&result=failure shows it a little better | |
| 11:00:42 | gibi | lyarwood: yepp that see a lot better that it is hit grenade not just nova grenade | |
| 11:00:47 | gibi | thanks | |
| 11:01:09 | gibi | (i filtered the query for the nova grenade job only) | |
| 11:06:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed https://review.opendev.org/c/openstack/nova/+/803233 | |
| 11:06:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 12:08:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support move ops with extended resource request https://review.opendev.org/c/openstack/nova/+/800087 | |
| 12:08:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] refactor interface attach with qos https://review.opendev.org/c/openstack/nova/+/800088 | |
| 12:08:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support interaface attach / detach with new resource request format https://review.opendev.org/c/openstack/nova/+/800089 | |
| 12:11:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place https://review.opendev.org/c/openstack/nova/+/793621 | |
| 12:12:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP support extended res req in heal port allocation https://review.opendev.org/c/openstack/nova/+/802060 | |
| 12:21:59 | bauzas | gibi: stephenfin: sean-k-mooney: others, I'm eventually done with the generic-mdevs series | |
| 12:22:38 | bauzas | gibi: stephenfin: sean-k-mooney: given I'll be on PTO for 3 weeks after Friday, if you want to review the series, I'd love to see your comments before I'm done | |
| 12:22:53 | bauzas | https://review.opendev.org/q/topic:%22bp%252Fgeneric-mdevs%22+(status:open%20OR%20status:merged) | |
| 12:23:35 | sean-k-mooney | bauzas: almost finished getting through the 450+ emails i got when i was off for 2 days | |
| 12:23:49 | sean-k-mooney | but sure ill try an take a look today | |
| 12:28:34 | opendevreview | Sylvain Bauza proposed openstack/nova master: Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 12:38:55 | gibi | bauzas: ack, I will try to do an initial read today | |
| 13:20:13 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests https://review.opendev.org/c/openstack/nova/+/803226 | |
| 14:10:29 | opendevreview | Elod Illes proposed openstack/nova stable/train: DNM: devstack-gate test https://review.opendev.org/c/openstack/nova/+/803351 | |
| 14:14:34 | gibi | bauzas: finished reading your mdev series, left comments in the reviews | |
| 14:25:36 | bauzas | melwitt: reviewed https://review.opendev.org/c/openstack/placement/+/669170 and had a concern about the default name which is lowercase | |
| 14:25:49 | bauzas | gibi: thanks, appreciated, looking | |
| 14:25:56 | bauzas | (good timing fwiw) | |
| 14:26:47 | bauzas | gibi: oh the renaming thing | |
| 14:26:52 | bauzas | gibi: I should have been explicit | |
| 14:27:19 | bauzas | given I'll on PTO in 4 days, I wanted to only modify the needed names | |