| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-01 | |||
| 16:36:44 | Uggla | I close this topic by saying we will not have this meeting next week due to the vPTG. | |
| 16:36:56 | Uggla | #topic Stable Branches | |
| 16:36:57 | masahito | i'm not sure i mentioned it before. if my topic is selected for the ptg topics, i prefer 14utc or later. | |
| 16:37:23 | Uggla | masahito, yes, this is somewhere in my mind. | |
| 16:37:30 | masahito | thanks. | |
| 16:37:35 | Uggla | #info stable/2024.* gates broken with nova-ceph-multistore job failure (test case test_volume_upload fails - No image found with ID...) | |
| 16:37:41 | Uggla | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:38:08 | Uggla | elodilles, the floor is yours | |
| 16:38:16 | elodilles | yeah, i couldn't identify the root cause yet for that job failure yet ^^^ :( | |
| 16:38:42 | elodilles | so if anyone has any idea, it is appreciated o:) | |
| 16:39:24 | elodilles | stable/2023.2 gate is fine, and afaik stable/2025.1, too, but not 100% on that | |
| 16:39:57 | elodilles | and that is all i can say now | |
| 16:41:14 | sean-k-mooney | the iso patch has been rechecked again... | |
| 16:41:22 | sean-k-mooney | still hiting that volume bug | |
| 16:41:37 | sean-k-mooney | i havent looked at it yet either ill admit | |
| 16:42:10 | sean-k-mooney | i assume cinder of glance shoudl be seeing the same failure? | |
| 16:42:48 | gmann | I think I saw it on stable/2025.1 also (have not checked if it is same or different?) https://review.opendev.org/c/openstack/devstack/+/945239/comments/231d655b_52020b94 | |
| 16:43:46 | sean-k-mooney | that still the nova job | |
| 16:44:00 | sean-k-mooney | i was hopign that was a non nova one showing the same issue | |
| 16:44:30 | elodilles | gmann: looks like same | |
| 16:44:53 | Uggla | looks the same yes. | |
| 16:45:09 | sean-k-mooney | dansmith: do you know off the top of your head if there is anything special about the nova-ceph-multistore job that woudl cause no image found on volume upload? | |
| 16:45:30 | dansmith | volume upload to glance? | |
| 16:45:49 | sean-k-mooney | presumable yes | |
| 16:46:09 | dansmith | not really, that seems sort of impossible, | |
| 16:46:16 | dansmith | since you create first and then upload | |
| 16:46:19 | elodilles | (and the devstack patch merged successfully on 27th March, so it either not fully blocking or it was fixed on stable/2025.1 somehow) | |
| 16:46:30 | gmann | seems like glance fail to import image? | |
| 16:46:32 | gmann | #link https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#12807 | |
| 16:47:33 | sean-k-mooney | 953cc449-8f49-4d56-ab3e-49b54f49f937 failed to import image 833ccb36-83b8-41a7-b932-86884e4cdfc0 to the filesystem.: NoneType: None | |
| 16:47:47 | dansmith | is this a multinode setup? | |
| 16:47:53 | dansmith | with glance-remote configured? | |
| 16:48:13 | dansmith | that sort of looks like it can't do import because it can't look up the configured self-reference url of a node that did the stage, maybe? | |
| 16:48:31 | sean-k-mooney | its multinode and multi store but im note sure if that also means multiple glance apis | |
| 16:48:50 | sean-k-mooney | oh no | |
| 16:48:53 | sean-k-mooney | its single node | |
| 16:48:57 | sean-k-mooney | but multiple backends | |
| 16:49:03 | dansmith | yeah, single node and no g-api-r service | |
| 16:50:05 | dansmith | that must be the import-from-http test and not volume upload then | |
| 16:50:44 | sean-k-mooney | its empest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_upload[id-d8f1ca95-3d5b-44a3-b8ca-909691c9532d,image] | |
| 16:50:52 | sean-k-mooney | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/testr_results.html | |
| 16:51:57 | sean-k-mooney | its a post to POST https://10.0.18.121/volume/v3/volumes/5726aafd-b7f7-4dbd-be64-265076f0efbb/action so asking cidner to upload the voluem to glance | |
| 16:51:59 | sean-k-mooney | Body: {"os-volume_upload_image": {"image_name": "tempest-VolumesActionsTest-Image-2109450379", "disk_format": "raw"}} | |
| 16:52:12 | dansmith | I also don't expect volume upload to use import | |
| 16:53:03 | dansmith | sean-k-mooney: I meant the failed task that gmann linked to | |
| 16:53:30 | dansmith | let's not debug it here | |
| 16:53:30 | sean-k-mooney | oh ok | |
| 16:53:31 | gibi | Uggla: I think we can close the meeting and let the folks continue troubleshooting | |
| 16:53:41 | sean-k-mooney | ya | |
| 16:53:42 | gmann | I think there are some image delete request in log | |
| 16:53:47 | gmann | #link https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19062 | |
| 16:53:51 | gmann | yeah let's debug later | |
| 16:54:08 | Uggla | yep can we looks at that after the meeting ? | |
| 16:54:24 | sean-k-mooney | sure | |
| 16:54:25 | Uggla | I skip the vmwareapi 3rd-party CI efforts Highlights as fwiesel is ooo this week. | |
| 16:54:37 | Uggla | Latest topic | |
| 16:54:43 | Uggla | #topic Open discussion | |
| 16:55:00 | Uggla | If there is not, I'll close the meeting. | |
| 16:55:21 | Uggla | anything more to discuss ? | |
| 16:55:51 | Uggla | 3... | |
| 16:56:13 | Uggla | 2... | |
| 16:56:38 | Uggla | 1... | |
| 16:56:51 | bauzas | . | |
| 16:57:14 | Uggla | thanks all | |
| 16:57:16 | Uggla | #endmeeting | |
| 16:57:16 | opendevmeet | Meeting ended Tue Apr 1 16:57:16 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:57:16 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.html | |
| 16:57:16 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.txt | |
| 16:57:16 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.log.html | |
| 16:57:21 | elodilles | thanks Uggla o/ | |
| 16:57:30 | bauzas | thanks Uggla | |
| 16:57:34 | masahito | thank you | |
| 16:57:35 | gibi | thanks | |
| 16:58:16 | Uggla | I thought @dansmith would like to discuss brainfuck formating. :) | |
| 16:58:16 | sean-k-mooney | dansmith: gmann: i have a thoey but im not sure if it makes sense. looking at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/glance/glance-api_conf.txt the default backend is file(cheap), this is a volume upload test and the job has ceph. so i would assome that | |
| 16:58:18 | sean-k-mooney | cinder is using ceph. could the issue be that cidner is expectign to upload to the robust(ceph) store but we are trying to upload to the default file store | |
| 16:58:49 | sean-k-mooney | its called black | |
| 16:59:03 | sean-k-mooney | with its one line per function argument | |
| 17:00:17 | sean-k-mooney | dansmith: what im wondering is did cinder create a volume snapshot on ceph and then try and use the interoperable imporat flow to import from the ceph store to file store or somethign odd like that | |
| 17:00:34 | dansmith | you can't use import like that | |
| 17:00:38 | dansmith | so I don't think so | |
| 17:00:56 | dansmith | you're thinking of adding the location directly | |
| 17:01:21 | sean-k-mooney | perhaps | |
| 17:01:23 | sean-k-mooney | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/cinder/cinder_conf.txt | |
| 17:01:28 | sean-k-mooney | cidner is using ceph | |
| 17:01:48 | sean-k-mooney | im just wondering if cidner ig getting confused about how to upload the image | |
| 17:02:23 | dansmith | maybe best to ask a cinder person to look | |
| 17:03:06 | sean-k-mooney | ya im going to very quieckly try and find the chagne id in there logs | |
| 17:03:15 | sean-k-mooney | *request id | |
| 17:04:58 | elodilles | gmann sean-k-mooney : it looks like this patch landed around the time when things started to break: https://review.opendev.org/c/openstack/tempest/+/938592 | |
| 17:05:24 | gmann | elodilles: yeah, I was about to debug that and someone rechecked and it merged and then I forgot | |
| 17:05:53 | gmann | I am checking some tempest logs if something test is making request which is racing GET and DELETE | |
| 17:05:59 | gmann | I will do after TC meeting | |
| 17:06:04 | sean-k-mooney | there is a trace back related to that i think in cidner | |
| 17:06:12 | sean-k-mooney | it thinks it uploaded the image DEBUG cinder.volume.manager [None req-ca38b299-90c1-489f-93c7-3fd8b32b5e69 tempest-VolumesActionsTest-2112541612 None] Uploaded volume to glance image-id: 439cbb04-d5ef-480f-89c9-84bf2d4c749b | |
| 17:06:19 | elodilles | thanks in advance gmann o/ | |
| 17:06:47 | gmann | because this is image id whihc is 404 in failing test and there is successful delete request https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19069 | |
| 17:07:03 | gmann | and after this DELETE request there are GET and 404 | |
| 17:07:30 | opendevreview | Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149 | |
| 17:07:30 | opendevreview | Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816 | |