Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-04-01
16:42:48 gmann I think I saw it on stable/2025.1 also (have not checked if it is same or different?) https://review.opendev.org/c/openstack/devstack/+/945239/comments/231d655b_52020b94
16:43:46 sean-k-mooney that still the nova job
16:44:00 sean-k-mooney i was hopign that was a non nova one showing the same issue
16:44:30 elodilles gmann: looks like same
16:44:53 Uggla looks the same yes.
16:45:09 sean-k-mooney dansmith: do you know off the top of your head if there is anything special about the nova-ceph-multistore job that woudl cause no image found on volume upload?
16:45:30 dansmith volume upload to glance?
16:45:49 sean-k-mooney presumable yes
16:46:09 dansmith not really, that seems sort of impossible,
16:46:16 dansmith since you create first and then upload
16:46:19 elodilles (and the devstack patch merged successfully on 27th March, so it either not fully blocking or it was fixed on stable/2025.1 somehow)
16:46:30 gmann seems like glance fail to import image?
16:46:32 gmann #link https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#12807
16:47:33 sean-k-mooney 953cc449-8f49-4d56-ab3e-49b54f49f937 failed to import image 833ccb36-83b8-41a7-b932-86884e4cdfc0 to the filesystem.: NoneType: None
16:47:47 dansmith is this a multinode setup?
16:47:53 dansmith with glance-remote configured?
16:48:13 dansmith that sort of looks like it can't do import because it can't look up the configured self-reference url of a node that did the stage, maybe?
16:48:31 sean-k-mooney its multinode and multi store but im note sure if that also means multiple glance apis
16:48:50 sean-k-mooney oh no
16:48:53 sean-k-mooney its single node
16:48:57 sean-k-mooney but multiple backends
16:49:03 dansmith yeah, single node and no g-api-r service
16:50:05 dansmith that must be the import-from-http test and not volume upload then
16:50:44 sean-k-mooney its empest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_upload[id-d8f1ca95-3d5b-44a3-b8ca-909691c9532d,image]
16:50:52 sean-k-mooney https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/testr_results.html
16:51:57 sean-k-mooney its a post to POST https://10.0.18.121/volume/v3/volumes/5726aafd-b7f7-4dbd-be64-265076f0efbb/action so asking cidner to upload the voluem to glance
16:51:59 sean-k-mooney Body: {"os-volume_upload_image": {"image_name": "tempest-VolumesActionsTest-Image-2109450379", "disk_format": "raw"}}
16:52:12 dansmith I also don't expect volume upload to use import
16:53:03 dansmith sean-k-mooney: I meant the failed task that gmann linked to
16:53:30 sean-k-mooney oh ok
16:53:30 dansmith let's not debug it here
16:53:31 gibi Uggla: I think we can close the meeting and let the folks continue troubleshooting
16:53:41 sean-k-mooney ya
16:53:42 gmann I think there are some image delete request in log
16:53:47 gmann #link https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19062
16:53:51 gmann yeah let's debug later
16:54:08 Uggla yep can we looks at that after the meeting ?
16:54:24 sean-k-mooney sure
16:54:25 Uggla I skip the vmwareapi 3rd-party CI efforts Highlights as fwiesel is ooo this week.
16:54:37 Uggla Latest topic
16:54:43 Uggla #topic Open discussion
16:55:00 Uggla If there is not, I'll close the meeting.
16:55:21 Uggla anything more to discuss ?
16:55:51 Uggla 3...
16:56:13 Uggla 2...
16:56:38 Uggla 1...
16:56:51 bauzas .
16:57:14 Uggla thanks all
16:57:16 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.log.html
16:57:16 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.txt
16:57:16 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-04-01-16.01.html
16:57:16 opendevmeet Meeting ended Tue Apr 1 16:57:16 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:57:16 Uggla #endmeeting
16:57:21 elodilles thanks Uggla o/
16:57:30 bauzas thanks Uggla
16:57:34 masahito thank you
16:57:35 gibi thanks
16:58:16 sean-k-mooney dansmith: gmann: i have a thoey but im not sure if it makes sense. looking at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/glance/glance-api_conf.txt the default backend is file(cheap), this is a volume upload test and the job has ceph. so i would assome that
16:58:16 Uggla I thought @dansmith would like to discuss brainfuck formating. :)
16:58:18 sean-k-mooney cinder is using ceph. could the issue be that cidner is expectign to upload to the robust(ceph) store but we are trying to upload to the default file store
16:58:49 sean-k-mooney its called black
16:59:03 sean-k-mooney with its one line per function argument
17:00:17 sean-k-mooney dansmith: what im wondering is did cinder create a volume snapshot on ceph and then try and use the interoperable imporat flow to import from the ceph store to file store or somethign odd like that
17:00:34 dansmith you can't use import like that
17:00:38 dansmith so I don't think so
17:00:56 dansmith you're thinking of adding the location directly
17:01:21 sean-k-mooney perhaps
17:01:23 sean-k-mooney https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/cinder/cinder_conf.txt
17:01:28 sean-k-mooney cidner is using ceph
17:01:48 sean-k-mooney im just wondering if cidner ig getting confused about how to upload the image
17:02:23 dansmith maybe best to ask a cinder person to look
17:03:06 sean-k-mooney ya im going to very quieckly try and find the chagne id in there logs
17:03:15 sean-k-mooney *request id
17:04:58 elodilles gmann sean-k-mooney : it looks like this patch landed around the time when things started to break: https://review.opendev.org/c/openstack/tempest/+/938592
17:05:24 gmann elodilles: yeah, I was about to debug that and someone rechecked and it merged and then I forgot
17:05:53 gmann I am checking some tempest logs if something test is making request which is racing GET and DELETE
17:05:59 gmann I will do after TC meeting
17:06:04 sean-k-mooney there is a trace back related to that i think in cidner
17:06:12 sean-k-mooney it thinks it uploaded the image DEBUG cinder.volume.manager [None req-ca38b299-90c1-489f-93c7-3fd8b32b5e69 tempest-VolumesActionsTest-2112541612 None] Uploaded volume to glance image-id: 439cbb04-d5ef-480f-89c9-84bf2d4c749b
17:06:19 elodilles thanks in advance gmann o/
17:06:47 gmann because this is image id whihc is 404 in failing test and there is successful delete request https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19069
17:07:03 gmann and after this DELETE request there are GET and 404
17:07:30 opendevreview Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816
17:07:30 opendevreview Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149
17:07:31 opendevreview Dan Smith proposed openstack/nova master: WIP: Functional tests for one-time-use devices https://review.opendev.org/c/openstack/nova/+/946065
17:07:31 opendevreview Dan Smith proposed openstack/nova master: Add one-time-use devices docs and reno https://review.opendev.org/c/openstack/nova/+/944262
17:07:43 gmann Details: {'message': 'No image found with ID 3eb80d77-c55b-40d1-bf5d-1e22607350e8<br /><br />\n\n\n', 'code': '404 Not Found', 'title': 'Not Found'}
17:07:43 gmann sean-k-mooney: I am searhing via image id from the tempest test failing traceback:
17:07:52 sean-k-mooney https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-c-vol.txt#4972-5037
17:08:02 dansmith gibi: that ^ tests the full workflow in a single test.. there's a lot of setup so I'm not sure if it's really useful to split that into multiple cases
17:08:40 dansmith gibi: also not sure its feasible to test anything other than the happy path there (the cache issue is covered in unit tests)
17:09:09 sean-k-mooney gmann: do you know where that 3eb80d77-c55b-40d1-bf5d-1e22607350e8 is coming form
17:09:37 sean-k-mooney we ca see in the start fo the test logs that the id of the create image is 439cbb04-d5ef-480f-89c9-84bf2d4c749b
17:10:02 dansmith I assume testing things like startup failures due to improper config of a VF or something are not really needed to be covered in functional since they're covered in unit
17:10:17 sean-k-mooney gmann: oh this is doing multiple operations in on etest
17:10:58 gmann sean-k-mooney: its is from cinder upload_volume https://github.com/openstack/tempest/blob/80c0477f78c71a2bd2e1a324c41cd2f50329b200/tempest/api/volume/test_volumes_actions.py#L123
17:12:08 sean-k-mooney so what happing is the raw iamge is uploading fine
17:12:13 sean-k-mooney and the qcow is failing
17:13:00 sean-k-mooney gmann: the qcow fails because of https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-c-vol.txt#4972-5037
17:13:41 sean-k-mooney its failing in the format inspector code

Earlier   Later