| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-03 | |||
| 08:37:25 | abhishekk | ok, will have a look and get back to you | |
| 08:38:05 | bauzas | abhishekk: but I relate the glance g-api warning with the same req-id of the failing glance image upload | |
| 08:38:25 | bauzas | correlation isn't consequence, I know, but that's a starting point | |
| 08:38:33 | abhishekk | ack | |
| 08:39:17 | lyarwood | bauzas: the test is 100% failing because of the 502 it just isn't clear why g-api or httpd is returning it | |
| 08:39:27 | lyarwood | sorry got my head stuck downstream again | |
| 08:39:40 | bauzas | yup, that's what I said to abhishekk | |
| 08:39:53 | bauzas | we have an image upload failure | |
| 08:40:00 | bauzas | but we don't know why | |
| 08:40:57 | bauzas | lyarwood: no worries about downstream rush, I feel your pain, maybe just one thought : as this is failing, should we make the job non-voting until we figure out the cause ? | |
| 08:41:16 | lyarwood | Yup happy to if you can raise a bug | |
| 08:41:29 | bauzas | good call, I forgot this :facepalm: | |
| 08:43:06 | abhishekk | I can see HTTP 413 in g-api log | |
| 08:46:20 | bauzas | lyarwood: abhishekk: gibi: https://bugs.launchpad.net/nova/+bug/1938765 | |
| 08:48:38 | abhishekk | ack | |
| 08:53:34 | bauzas | I need to disappear for 10 mins | |
| 09:00:37 | abhishekk | I think this is causing problem, DEBUG oslo.limit.limit [None req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-ListImageFiltersTestJSON-1499681503 tempest-ListImageFiltersTestJSON-1499681503-project] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1024 and delta 0] {{(pid=108370) enforce_limits /usr/local/lib/python3.8/dist-packages/oslo_limit/limit.py:268}} | |
| 09:00:37 | abhishekk | Aug 01 03:51:33.042224 ubuntu-focal-ovh-bhs1-0025715861 devstack@g-api.service[108370]: [pid: 108370|app: 0|req: 188/333] 127.0.0.1 () {46 vars in 1190 bytes} [Sun Aug 1 03:51:32 2021] PUT /v2/images/4d204807-d5e5-4736-821f-fae99a0c066e/file => generated 448 bytes in 743 msecs (HTTP/1.1 413) 4 headers in 187 bytes (1 switches on core 0) | |
| 09:01:23 | lyarwood | sounds about right, these images are going to be pretty big | |
| 09:02:23 | lyarwood | https://opendev.org/openstack/devstack/commit/5a642450d6ac94ff1ea2bea3e7ce3887ca79dcc4 | |
| 09:02:27 | lyarwood | there we go | |
| 09:03:49 | abhishekk | I think this will fix the problem | |
| 09:11:40 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm https://review.opendev.org/c/openstack/nova/+/803322 | |
| 09:11:41 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM - Trigger nova-lvm to test I824655387a10ac9b813c1b7b2399e25ed95f7fc3 https://review.opendev.org/c/openstack/nova/+/803323 | |
| 09:11:43 | lyarwood | bauzas: / abhishekk ; ^ there we go | |
| 09:12:11 | abhishekk | lyarwood, ack | |
| 09:16:53 | bauzas | lyarwood: and we return a 502 if we go over the limit ? :O | |
| 09:18:23 | lyarwood | bauzas: 413 but somehow that's translated into a 503 by the time n-cpu sees it | |
| 09:18:38 | lyarwood | bauzas: that could be a separate bug report against devstack tbh | |
| 09:18:48 | bauzas | yup, agreed | |
| 09:18:57 | lyarwood | 502 even sorry | |
| 09:19:00 | bauzas | either way, excellent catch | |
| 09:19:14 | bauzas | I didn't know about glance limiting now the uploads | |
| 09:19:15 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests https://review.opendev.org/c/openstack/nova/+/803226 | |
| 09:34:33 | abhishekk | I see 6 413 in glance logs, that sums up total failure cases in the job | |
| 09:59:39 | abhishekk | job is passing now | |
| 10:04:56 | lyarwood | abhishekk: awesome thanks for your help with that :) | |
| 10:34:59 | gibi | lyarwood, bauzas thanks for fixing that up. | |
| 10:56:21 | lyarwood | grenade jobs blocked by https://bugs.launchpad.net/grenade/+bug/1938763 FWIW | |
| 10:58:01 | gibi | lyarwood: thanks I saw the error in the job run but I did not know that it is filed as a bug. I only see one failure in grenade by this so far | |
| 10:59:23 | lyarwood | np https://zuul.opendev.org/t/openstack/builds?job_name=grenade&job_name=nova-grenade-multinode&job_name=grenade-multinode&branch=master&result=failure shows it a little better | |
| 11:00:42 | gibi | lyarwood: yepp that see a lot better that it is hit grenade not just nova grenade | |
| 11:00:47 | gibi | thanks | |
| 11:01:09 | gibi | (i filtered the query for the nova grenade job only) | |
| 11:06:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed https://review.opendev.org/c/openstack/nova/+/803233 | |
| 11:06:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 12:08:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support move ops with extended resource request https://review.opendev.org/c/openstack/nova/+/800087 | |
| 12:08:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] refactor interface attach with qos https://review.opendev.org/c/openstack/nova/+/800088 | |
| 12:08:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support interaface attach / detach with new resource request format https://review.opendev.org/c/openstack/nova/+/800089 | |
| 12:11:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place https://review.opendev.org/c/openstack/nova/+/793621 | |
| 12:12:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP support extended res req in heal port allocation https://review.opendev.org/c/openstack/nova/+/802060 | |
| 12:21:59 | bauzas | gibi: stephenfin: sean-k-mooney: others, I'm eventually done with the generic-mdevs series | |
| 12:22:38 | bauzas | gibi: stephenfin: sean-k-mooney: given I'll be on PTO for 3 weeks after Friday, if you want to review the series, I'd love to see your comments before I'm done | |
| 12:22:53 | bauzas | https://review.opendev.org/q/topic:%22bp%252Fgeneric-mdevs%22+(status:open%20OR%20status:merged) | |
| 12:23:35 | sean-k-mooney | bauzas: almost finished getting through the 450+ emails i got when i was off for 2 days | |
| 12:23:49 | sean-k-mooney | but sure ill try an take a look today | |
| 12:28:34 | opendevreview | Sylvain Bauza proposed openstack/nova master: Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 12:38:55 | gibi | bauzas: ack, I will try to do an initial read today | |
| 13:20:13 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests https://review.opendev.org/c/openstack/nova/+/803226 | |
| 14:10:29 | opendevreview | Elod Illes proposed openstack/nova stable/train: DNM: devstack-gate test https://review.opendev.org/c/openstack/nova/+/803351 | |
| 14:14:34 | gibi | bauzas: finished reading your mdev series, left comments in the reviews | |
| 14:25:36 | bauzas | melwitt: reviewed https://review.opendev.org/c/openstack/placement/+/669170 and had a concern about the default name which is lowercase | |
| 14:25:49 | bauzas | gibi: thanks, appreciated, looking | |
| 14:25:56 | bauzas | (good timing fwiw) | |
| 14:26:47 | bauzas | gibi: oh the renaming thing | |
| 14:26:52 | bauzas | gibi: I should have been explicit | |
| 14:27:19 | bauzas | given I'll on PTO in 4 days, I wanted to only modify the needed names | |
| 14:27:45 | bauzas | and for the internal methods names, I'll be adding a follow-up after the last change | |
| 14:32:28 | gibi | bauzas: followup for the internal renaming works for me | |
| 14:33:00 | bauzas | looking at your other comments | |
| 14:33:14 | gibi | bauzas: still the externally visible things should be renamed along the way. E.g. the exception texts | |
| 14:34:18 | gibi | InvalidLibvirtGPUConfig | |
| 14:34:32 | gibi | I guess the messag from that ends up in the logs | |
| 14:35:22 | bauzas | you're maybe right | |
| 14:35:48 | bauzas | gibi: I don't think it would be an upgrade modification if we provide different logs, right? | |
| 14:36:05 | gibi | not an upgrade impact no | |
| 14:48:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support move ops with extended resource request https://review.opendev.org/c/openstack/nova/+/800087 | |
| 14:48:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] refactor interface attach with qos https://review.opendev.org/c/openstack/nova/+/800088 | |
| 14:48:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support interaface attach / detach with new resource request format https://review.opendev.org/c/openstack/nova/+/800089 | |
| 14:49:44 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place https://review.opendev.org/c/openstack/nova/+/793621 | |
| 14:49:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP support extended res req in heal port allocation https://review.opendev.org/c/openstack/nova/+/802060 | |
| 15:33:06 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Fix incorrect reservation IDs in unit tests https://review.opendev.org/c/openstack/nova/+/803363 | |
| 15:50:15 | gibi | nova meeting starts in 10 minutes here in the channell | |
| 15:50:34 | sean-k-mooney | gibi: were you not ment to be on PTO this week? | |
| 15:50:38 | sean-k-mooney | or is that next week | |
| 15:50:55 | sean-k-mooney | oh it was last week | |
| 15:56:09 | gibi | yepp that was last week | |
| 15:56:27 | gibi | and that will be next week as well (which I should mention on the meeting too) | |
| 15:59:30 | bauzas | me too | |
| 15:59:40 | bauzas | for the next 3 weeks | |
| 16:00:19 | gibi | #startmeeting nova | |
| 16:00:19 | opendevmeet | Meeting started Tue Aug 3 16:00:19 2021 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:19 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:19 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:26 | gibi | o/ | |
| 16:00:27 | dansmith | o/ | |
| 16:00:30 | bauzas | humpf, I was about to upload :) | |
| 16:00:37 | elodilles | o/ | |
| 16:00:39 | bauzas | \o | |
| 16:01:39 | gibi | #topic Bugs (stuck/critical) | |