| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-01 | |||
| 16:46:19 | elodilles | (and the devstack patch merged successfully on 27th March, so it either not fully blocking or it was fixed on stable/2025.1 somehow) | |
| 16:46:30 | gmann | seems like glance fail to import image? | |
| 16:46:32 | gmann | #link https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#12807 | |
| 16:47:33 | sean-k-mooney | 953cc449-8f49-4d56-ab3e-49b54f49f937 failed to import image 833ccb36-83b8-41a7-b932-86884e4cdfc0 to the filesystem.: NoneType: None | |
| 16:47:47 | dansmith | is this a multinode setup? | |
| 16:47:53 | dansmith | with glance-remote configured? | |
| 16:48:13 | dansmith | that sort of looks like it can't do import because it can't look up the configured self-reference url of a node that did the stage, maybe? | |
| 16:48:31 | sean-k-mooney | its multinode and multi store but im note sure if that also means multiple glance apis | |
| 16:48:50 | sean-k-mooney | oh no | |
| 16:48:53 | sean-k-mooney | its single node | |
| 16:48:57 | sean-k-mooney | but multiple backends | |
| 16:49:03 | dansmith | yeah, single node and no g-api-r service | |
| 16:50:05 | dansmith | that must be the import-from-http test and not volume upload then | |
| 16:50:44 | sean-k-mooney | its empest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_upload[id-d8f1ca95-3d5b-44a3-b8ca-909691c9532d,image] | |
| 16:50:52 | sean-k-mooney | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/testr_results.html | |
| 16:51:57 | sean-k-mooney | its a post to POST https://10.0.18.121/volume/v3/volumes/5726aafd-b7f7-4dbd-be64-265076f0efbb/action so asking cidner to upload the voluem to glance | |
| 16:51:59 | sean-k-mooney | Body: {"os-volume_upload_image": {"image_name": "tempest-VolumesActionsTest-Image-2109450379", "disk_format": "raw"}} | |
| 16:52:12 | dansmith | I also don't expect volume upload to use import | |
| 16:53:03 | dansmith | sean-k-mooney: I meant the failed task that gmann linked to | |
| 16:53:30 | dansmith | let's not debug it here | |
| 16:53:30 | sean-k-mooney | oh ok | |
| 16:53:31 | gibi | Uggla: I think we can close the meeting and let the folks continue troubleshooting | |
| 16:53:41 | sean-k-mooney | ya | |
| 16:53:42 | gmann | I think there are some image delete request in log | |
| 16:53:47 | gmann | #link https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19062 | |
| 16:53:51 | gmann | yeah let's debug later | |
| 16:54:08 | Uggla | yep can we looks at that after the meeting ? | |
| 16:54:24 | sean-k-mooney | sure | |
| 16:54:25 | Uggla | I skip the vmwareapi 3rd-party CI efforts Highlights as fwiesel is ooo this week. | |
| 16:54:37 | Uggla | Latest topic | |
| 16:54:43 | Uggla | #topic Open discussion | |
| 16:55:00 | Uggla | If there is not, I'll close the meeting. | |
| 16:55:21 | Uggla | anything more to discuss ? | |
| 16:55:51 | Uggla | 3... | |
| 16:56:13 | Uggla | 2... | |
| 16:56:38 | Uggla | 1... | |
| 16:56:51 | bauzas | . | |
| 16:57:14 | Uggla | thanks all | |
| 16:57:16 | Uggla | #endmeeting | |
| 16:57:16 | opendevmeet | Meeting ended Tue Apr 1 16:57:16 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:57:16 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.html | |
| 16:57:16 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.txt | |
| 16:57:16 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.log.html | |
| 16:57:21 | elodilles | thanks Uggla o/ | |
| 16:57:30 | bauzas | thanks Uggla | |
| 16:57:34 | masahito | thank you | |
| 16:57:35 | gibi | thanks | |
| 16:58:16 | Uggla | I thought @dansmith would like to discuss brainfuck formating. :) | |
| 16:58:16 | sean-k-mooney | dansmith: gmann: i have a thoey but im not sure if it makes sense. looking at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/glance/glance-api_conf.txt the default backend is file(cheap), this is a volume upload test and the job has ceph. so i would assome that | |
| 16:58:18 | sean-k-mooney | cinder is using ceph. could the issue be that cidner is expectign to upload to the robust(ceph) store but we are trying to upload to the default file store | |
| 16:58:49 | sean-k-mooney | its called black | |
| 16:59:03 | sean-k-mooney | with its one line per function argument | |
| 17:00:17 | sean-k-mooney | dansmith: what im wondering is did cinder create a volume snapshot on ceph and then try and use the interoperable imporat flow to import from the ceph store to file store or somethign odd like that | |
| 17:00:34 | dansmith | you can't use import like that | |
| 17:00:38 | dansmith | so I don't think so | |
| 17:00:56 | dansmith | you're thinking of adding the location directly | |
| 17:01:21 | sean-k-mooney | perhaps | |
| 17:01:23 | sean-k-mooney | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/cinder/cinder_conf.txt | |
| 17:01:28 | sean-k-mooney | cidner is using ceph | |
| 17:01:48 | sean-k-mooney | im just wondering if cidner ig getting confused about how to upload the image | |
| 17:02:23 | dansmith | maybe best to ask a cinder person to look | |
| 17:03:06 | sean-k-mooney | ya im going to very quieckly try and find the chagne id in there logs | |
| 17:03:15 | sean-k-mooney | *request id | |
| 17:04:58 | elodilles | gmann sean-k-mooney : it looks like this patch landed around the time when things started to break: https://review.opendev.org/c/openstack/tempest/+/938592 | |
| 17:05:24 | gmann | elodilles: yeah, I was about to debug that and someone rechecked and it merged and then I forgot | |
| 17:05:53 | gmann | I am checking some tempest logs if something test is making request which is racing GET and DELETE | |
| 17:05:59 | gmann | I will do after TC meeting | |
| 17:06:04 | sean-k-mooney | there is a trace back related to that i think in cidner | |
| 17:06:12 | sean-k-mooney | it thinks it uploaded the image DEBUG cinder.volume.manager [None req-ca38b299-90c1-489f-93c7-3fd8b32b5e69 tempest-VolumesActionsTest-2112541612 None] Uploaded volume to glance image-id: 439cbb04-d5ef-480f-89c9-84bf2d4c749b | |
| 17:06:19 | elodilles | thanks in advance gmann o/ | |
| 17:06:47 | gmann | because this is image id whihc is 404 in failing test and there is successful delete request https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19069 | |
| 17:07:03 | gmann | and after this DELETE request there are GET and 404 | |
| 17:07:30 | opendevreview | Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149 | |
| 17:07:30 | opendevreview | Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816 | |
| 17:07:31 | opendevreview | Dan Smith proposed openstack/nova master: Add one-time-use devices docs and reno https://review.opendev.org/c/openstack/nova/+/944262 | |
| 17:07:31 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Functional tests for one-time-use devices https://review.opendev.org/c/openstack/nova/+/946065 | |
| 17:07:43 | gmann | sean-k-mooney: I am searhing via image id from the tempest test failing traceback: | |
| 17:07:43 | gmann | Details: {'message': 'No image found with ID 3eb80d77-c55b-40d1-bf5d-1e22607350e8<br /><br />\n\n\n', 'code': '404 Not Found', 'title': 'Not Found'} | |
| 17:07:52 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-c-vol.txt#4972-5037 | |
| 17:08:02 | dansmith | gibi: that ^ tests the full workflow in a single test.. there's a lot of setup so I'm not sure if it's really useful to split that into multiple cases | |
| 17:08:40 | dansmith | gibi: also not sure its feasible to test anything other than the happy path there (the cache issue is covered in unit tests) | |
| 17:09:09 | sean-k-mooney | gmann: do you know where that 3eb80d77-c55b-40d1-bf5d-1e22607350e8 is coming form | |
| 17:09:37 | sean-k-mooney | we ca see in the start fo the test logs that the id of the create image is 439cbb04-d5ef-480f-89c9-84bf2d4c749b | |
| 17:10:02 | dansmith | I assume testing things like startup failures due to improper config of a VF or something are not really needed to be covered in functional since they're covered in unit | |
| 17:10:17 | sean-k-mooney | gmann: oh this is doing multiple operations in on etest | |
| 17:10:58 | gmann | sean-k-mooney: its is from cinder upload_volume https://github.com/openstack/tempest/blob/80c0477f78c71a2bd2e1a324c41cd2f50329b200/tempest/api/volume/test_volumes_actions.py#L123 | |
| 17:12:08 | sean-k-mooney | so what happing is the raw iamge is uploading fine | |
| 17:12:13 | sean-k-mooney | and the qcow is failing | |
| 17:13:00 | sean-k-mooney | gmann: the qcow fails because of https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-c-vol.txt#4972-5037 | |
| 17:13:41 | sean-k-mooney | its failing in the format inspector code | |
| 17:14:08 | dansmith | on the cinder side | |
| 17:14:12 | sean-k-mooney | yep | |
| 17:14:27 | dansmith | and that's their oooold integrated format inspector code | |
| 17:14:32 | gmann | but in that case, I expect upload_volume to fail instead of giving image_id | |
| 17:14:47 | sean-k-mooney | so its a ceph backed cinder volume. and we are asking cidner ot upload the volume as an image in qcow2 format | |
| 17:14:58 | dansmith | and apparently because filename is none | |
| 17:15:23 | sean-k-mooney | gmann: i have not looked at the cidner code but i belive the worklaod is create teh glance image then upload the data | |
| 17:15:37 | sean-k-mooney | so they are probaly failing in between those two steps | |
| 17:17:07 | sean-k-mooney | gmann: form the looks of that trace the ceph cinder volume driver does not support converting the format when uploading a volume to an image | |
| 17:17:27 | sean-k-mooney | gmann: so in the near term we proably should tweak that job to only use raw | |