| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-02 | |||
| 16:00:24 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests https://review.opendev.org/c/openstack/nova/+/803226 | |
| 16:42:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide the mdev class for every PCI device https://review.opendev.org/c/openstack/nova/+/802918 | |
| 16:42:51 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM (yet) : Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 16:42:51 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed https://review.opendev.org/c/openstack/nova/+/803233 | |
| 17:15:10 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed https://review.opendev.org/c/openstack/nova/+/803233 | |
| 17:15:10 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 22:19:12 | opendevreview | Samuel proposed openstack/nova-specs master: Migrate Instance Between Projects https://review.opendev.org/c/openstack/nova-specs/+/802034 | |
| #openstack-nova - 2021-08-03 | |||
| 01:03:41 | songwenping | bauzas, sean-k-mooney[m]: thanks for last week's response. | |
| 07:29:25 | bauzas | gibi: morning | |
| 07:29:35 | bauzas | looks like the nova-lvm job is on the weeds | |
| 07:32:15 | gibi | bauzas: o/ I have to be off for 2 hours | |
| 07:33:12 | gibi | it seems it is in the weeds for daays | |
| 07:33:12 | bauzas | no worries, I'll try to look at the problme | |
| 07:33:15 | bauzas | yup | |
| 07:33:28 | bauzas | a snapshot not found or something like this | |
| 07:33:52 | gibi | last success was 2021-07-16T08:36:39 | |
| 07:34:08 | bauzas | yep http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22nova-lvm%5C%22%20build_status%3A%5C%22FAILURE%5C%22 | |
| 07:34:16 | bauzas | (if logstash is still working correctly) | |
| 07:35:25 | gibi | but then how we managed to merge things?! | |
| 07:35:43 | gibi | was this job turned off? | |
| 07:36:43 | gibi | or is it selectively runnig? | |
| 07:37:36 | gibi | it doesn't have a too special irrelevant-files in .zuul.yaml | |
| 07:37:40 | gibi | so I don't get it | |
| 07:37:48 | gibi | I have a bunch of patches that nova-lvm isn't run on | |
| 07:38:00 | bauzas | excellent question | |
| 07:38:11 | gibi | like https://review.opendev.org/c/openstack/nova/+/800086/13 | |
| 07:38:23 | gibi | pingig lyarwood :) | |
| 07:38:27 | bauzas | zuul is pretty https://zuul.openstack.org/builds?job_name=nova-lvm | |
| 07:38:44 | gibi | and I'm dropping off now. be back later | |
| 07:38:44 | bauzas | none of the job builds worked since 2 weeks | |
| 07:38:51 | bauzas | gibi: no worries and thanks | |
| 07:40:05 | yonglihe | HI, everyone | |
| 07:42:55 | bauzas | ok, the nova-lvm job is triggered on libvirt changes hence the non-failure on other changes and why we merged other stuff | |
| 07:43:01 | bauzas | https://zuul.openstack.org/job/nova-lvm | |
| 07:43:39 | bauzas | now, looks like all runs fail with the same tempest tests with the same exception being a snapshot not found | |
| 07:46:33 | bauzas | huzzaaaaah \o/ https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-n-cpu.txt?severity=4#12685-12689 | |
| 07:46:44 | bauzas | NotImplementedError: direct_snapshot() is not implemented | |
| 07:47:12 | bauzas | looks like we regressed | |
| 07:51:29 | bauzas | hmmmm | |
| 07:51:53 | lyarwood | Morning | |
| 07:52:09 | lyarwood | yeah I've not had time to look at nova-lvm failures thanks to a few downstream fires | |
| 07:52:17 | lyarwood | they should be out now so I'll look this morning | |
| 07:52:20 | bauzas | ok, I have better understanding | |
| 07:52:22 | lyarwood | bauzas: that isn't the failure | |
| 07:52:25 | bauzas | correct | |
| 07:52:33 | bauzas | we catch the notimplemented exception | |
| 07:52:36 | bauzas | it's failing later | |
| 07:52:51 | bauzas | when calling glance I'd say | |
| 07:53:06 | bauzas | https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-n-cpu.txt?severity=4#12772 | |
| 07:53:08 | lyarwood | yeah the g-api 502 is something I think I've seen before when we run out of space during the upload | |
| 07:53:27 | bauzas | yup | |
| 07:53:57 | bauzas | we want to upload the snapshot and we fail, right? | |
| 07:54:22 | lyarwood | we are in the process of uploading the snapshot and something fails | |
| 07:54:29 | lyarwood | it might even be a timeout in httpd | |
| 07:54:39 | lyarwood | given the size of the images we are uploading for these tests | |
| 07:55:18 | lyarwood | brb | |
| 07:56:23 | bauzas | lyarwood: https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-g-api.txt?severity=3#3555 | |
| 07:56:37 | bauzas | this error matches the req-id of the n-cpu issue | |
| 08:11:54 | bauzas | well, can't figure out more why the temporary image fails | |
| 08:12:09 | bauzas | abhishekk: around ? | |
| 08:12:43 | bauzas | abhishekk: we have a nova job failure (nova-lvm) which fails constantly because of glance API returning 502 https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-g-api.txt?severity=3#3555 | |
| 08:12:55 | bauzas | on an image upload | |
| 08:34:47 | abhishekk | bauzas, looking | |
| 08:35:02 | bauzas | abhishekk: thanks, wrote an email about this | |
| 08:35:10 | bauzas | (to summarize) | |
| 08:35:17 | abhishekk | yes, just went through it | |
| 08:36:35 | abhishekk | bauzas, are you sure failure is because of this warning ? | |
| 08:37:01 | bauzas | abhishekk: I'm nearly sure the test is failing because glance is returning a 502 | |
| 08:37:25 | abhishekk | ok, will have a look and get back to you | |
| 08:38:05 | bauzas | abhishekk: but I relate the glance g-api warning with the same req-id of the failing glance image upload | |
| 08:38:25 | bauzas | correlation isn't consequence, I know, but that's a starting point | |
| 08:38:33 | abhishekk | ack | |
| 08:39:17 | lyarwood | bauzas: the test is 100% failing because of the 502 it just isn't clear why g-api or httpd is returning it | |
| 08:39:27 | lyarwood | sorry got my head stuck downstream again | |
| 08:39:40 | bauzas | yup, that's what I said to abhishekk | |
| 08:39:53 | bauzas | we have an image upload failure | |
| 08:40:00 | bauzas | but we don't know why | |
| 08:40:57 | bauzas | lyarwood: no worries about downstream rush, I feel your pain, maybe just one thought : as this is failing, should we make the job non-voting until we figure out the cause ? | |
| 08:41:16 | lyarwood | Yup happy to if you can raise a bug | |
| 08:41:29 | bauzas | good call, I forgot this :facepalm: | |
| 08:43:06 | abhishekk | I can see HTTP 413 in g-api log | |
| 08:46:20 | bauzas | lyarwood: abhishekk: gibi: https://bugs.launchpad.net/nova/+bug/1938765 | |
| 08:48:38 | abhishekk | ack | |
| 08:53:34 | bauzas | I need to disappear for 10 mins | |
| 09:00:37 | abhishekk | I think this is causing problem, DEBUG oslo.limit.limit [None req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-ListImageFiltersTestJSON-1499681503 tempest-ListImageFiltersTestJSON-1499681503-project] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1024 and delta 0] {{(pid=108370) enforce_limits /usr/local/lib/python3.8/dist-packages/oslo_limit/limit.py:268}} | |
| 09:00:37 | abhishekk | Aug 01 03:51:33.042224 ubuntu-focal-ovh-bhs1-0025715861 devstack@g-api.service[108370]: [pid: 108370|app: 0|req: 188/333] 127.0.0.1 () {46 vars in 1190 bytes} [Sun Aug 1 03:51:32 2021] PUT /v2/images/4d204807-d5e5-4736-821f-fae99a0c066e/file => generated 448 bytes in 743 msecs (HTTP/1.1 413) 4 headers in 187 bytes (1 switches on core 0) | |
| 09:01:23 | lyarwood | sounds about right, these images are going to be pretty big | |
| 09:02:23 | lyarwood | https://opendev.org/openstack/devstack/commit/5a642450d6ac94ff1ea2bea3e7ce3887ca79dcc4 | |
| 09:02:27 | lyarwood | there we go | |
| 09:03:49 | abhishekk | I think this will fix the problem | |
| 09:11:40 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm https://review.opendev.org/c/openstack/nova/+/803322 | |
| 09:11:41 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM - Trigger nova-lvm to test I824655387a10ac9b813c1b7b2399e25ed95f7fc3 https://review.opendev.org/c/openstack/nova/+/803323 | |
| 09:11:43 | lyarwood | bauzas: / abhishekk ; ^ there we go | |
| 09:12:11 | abhishekk | lyarwood, ack | |
| 09:16:53 | bauzas | lyarwood: and we return a 502 if we go over the limit ? :O | |
| 09:18:23 | lyarwood | bauzas: 413 but somehow that's translated into a 503 by the time n-cpu sees it | |
| 09:18:38 | lyarwood | bauzas: that could be a separate bug report against devstack tbh | |
| 09:18:48 | bauzas | yup, agreed | |
| 09:18:57 | lyarwood | 502 even sorry | |
| 09:19:00 | bauzas | either way, excellent catch | |