Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-04-01
16:57:34 masahito thank you
16:57:35 gibi thanks
16:58:16 sean-k-mooney dansmith: gmann: i have a thoey but im not sure if it makes sense. looking at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/glance/glance-api_conf.txt the default backend is file(cheap), this is a volume upload test and the job has ceph. so i would assome that
16:58:16 Uggla I thought @dansmith would like to discuss brainfuck formating. :)
16:58:18 sean-k-mooney cinder is using ceph. could the issue be that cidner is expectign to upload to the robust(ceph) store but we are trying to upload to the default file store
16:58:49 sean-k-mooney its called black
16:59:03 sean-k-mooney with its one line per function argument
17:00:17 sean-k-mooney dansmith: what im wondering is did cinder create a volume snapshot on ceph and then try and use the interoperable imporat flow to import from the ceph store to file store or somethign odd like that
17:00:34 dansmith you can't use import like that
17:00:38 dansmith so I don't think so
17:00:56 dansmith you're thinking of adding the location directly
17:01:21 sean-k-mooney perhaps
17:01:23 sean-k-mooney https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54c/openstack/54c93553fc6c4797b25077fb29a89338/controller/logs/etc/cinder/cinder_conf.txt
17:01:28 sean-k-mooney cidner is using ceph
17:01:48 sean-k-mooney im just wondering if cidner ig getting confused about how to upload the image
17:02:23 dansmith maybe best to ask a cinder person to look
17:03:06 sean-k-mooney ya im going to very quieckly try and find the chagne id in there logs
17:03:15 sean-k-mooney *request id
17:04:58 elodilles gmann sean-k-mooney : it looks like this patch landed around the time when things started to break: https://review.opendev.org/c/openstack/tempest/+/938592
17:05:24 gmann elodilles: yeah, I was about to debug that and someone rechecked and it merged and then I forgot
17:05:53 gmann I am checking some tempest logs if something test is making request which is racing GET and DELETE
17:05:59 gmann I will do after TC meeting
17:06:04 sean-k-mooney there is a trace back related to that i think in cidner
17:06:12 sean-k-mooney it thinks it uploaded the image DEBUG cinder.volume.manager [None req-ca38b299-90c1-489f-93c7-3fd8b32b5e69 tempest-VolumesActionsTest-2112541612 None] Uploaded volume to glance image-id: 439cbb04-d5ef-480f-89c9-84bf2d4c749b
17:06:19 elodilles thanks in advance gmann o/
17:06:47 gmann because this is image id whihc is 404 in failing test and there is successful delete request https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-g-api.txt#19069
17:07:03 gmann and after this DELETE request there are GET and 404
17:07:30 opendevreview Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816
17:07:30 opendevreview Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149
17:07:31 opendevreview Dan Smith proposed openstack/nova master: WIP: Functional tests for one-time-use devices https://review.opendev.org/c/openstack/nova/+/946065
17:07:31 opendevreview Dan Smith proposed openstack/nova master: Add one-time-use devices docs and reno https://review.opendev.org/c/openstack/nova/+/944262
17:07:43 gmann Details: {'message': 'No image found with ID 3eb80d77-c55b-40d1-bf5d-1e22607350e8<br /><br />\n\n\n', 'code': '404 Not Found', 'title': 'Not Found'}
17:07:43 gmann sean-k-mooney: I am searhing via image id from the tempest test failing traceback:
17:07:52 sean-k-mooney https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-c-vol.txt#4972-5037
17:08:02 dansmith gibi: that ^ tests the full workflow in a single test.. there's a lot of setup so I'm not sure if it's really useful to split that into multiple cases
17:08:40 dansmith gibi: also not sure its feasible to test anything other than the happy path there (the cache issue is covered in unit tests)
17:09:09 sean-k-mooney gmann: do you know where that 3eb80d77-c55b-40d1-bf5d-1e22607350e8 is coming form
17:09:37 sean-k-mooney we ca see in the start fo the test logs that the id of the create image is 439cbb04-d5ef-480f-89c9-84bf2d4c749b
17:10:02 dansmith I assume testing things like startup failures due to improper config of a VF or something are not really needed to be covered in functional since they're covered in unit
17:10:17 sean-k-mooney gmann: oh this is doing multiple operations in on etest
17:10:58 gmann sean-k-mooney: its is from cinder upload_volume https://github.com/openstack/tempest/blob/80c0477f78c71a2bd2e1a324c41cd2f50329b200/tempest/api/volume/test_volumes_actions.py#L123
17:12:08 sean-k-mooney so what happing is the raw iamge is uploading fine
17:12:13 sean-k-mooney and the qcow is failing
17:13:00 sean-k-mooney gmann: the qcow fails because of https://zuul.opendev.org/t/openstack/build/54c93553fc6c4797b25077fb29a89338/log/controller/logs/screen-c-vol.txt#4972-5037
17:13:41 sean-k-mooney its failing in the format inspector code
17:14:08 dansmith on the cinder side
17:14:12 sean-k-mooney yep
17:14:27 dansmith and that's their oooold integrated format inspector code
17:14:32 gmann but in that case, I expect upload_volume to fail instead of giving image_id
17:14:47 sean-k-mooney so its a ceph backed cinder volume. and we are asking cidner ot upload the volume as an image in qcow2 format
17:14:58 dansmith and apparently because filename is none
17:15:23 sean-k-mooney gmann: i have not looked at the cidner code but i belive the worklaod is create teh glance image then upload the data
17:15:37 sean-k-mooney so they are probaly failing in between those two steps
17:17:07 sean-k-mooney gmann: form the looks of that trace the ceph cinder volume driver does not support converting the format when uploading a volume to an image
17:17:27 sean-k-mooney gmann: so in the near term we proably should tweak that job to only use raw
17:17:47 sean-k-mooney and if cidner ever support that with ceph backed volumes we can turn it back on
17:19:50 sean-k-mooney looking at the trace they are either expectign a host monted volume or a qemu rbd path to be passed as the path to qemu image. but its obviously not passing either given its None
17:23:08 sean-k-mooney dansmith: gmann i think the short term fix would be to add disk_format=raw here https://github.com/openstack/nova/blob/master/.zuul.yaml#L712
17:23:22 sean-k-mooney based on the default change in https://review.opendev.org/c/openstack/tempest/+/938592/4/tempest/config.py
17:25:05 sean-k-mooney qcow makes sensce fo lvm josb perhaps since the volujme will be a local block device and they can call qemu image on it
17:25:17 dansmith why not have the cinder team confirm and do that if so?
17:25:22 dansmith isn't this on a stable job?
17:25:37 sean-k-mooney its likely on master too
17:25:44 sean-k-mooney but yes
17:26:02 dansmith how would that not be 100% fail?
17:26:26 gmann but this is not 100% failure in stable and master pass correctly ?
17:26:27 dansmith my point is, let's not just go papering over some failure with a devstack change as this seems like it should be a cinder bug no?
17:26:47 sean-k-mooney right i think it is a ciner bug
17:26:48 gmann dansmith: yeah, that what I was wondering, its not 100%
17:27:06 dansmith right, so I say kick it over to them
17:27:08 sean-k-mooney so i dont see it as papering over
17:27:16 sean-k-mooney we need to report it to them as a gate blocker
17:27:40 sean-k-mooney and then eitehr revert the tempst defualt change until its fixed or just skip it in our job until fixed
17:27:56 sean-k-mooney but i agree the first step is get cinder involved
17:27:56 dansmith if glance is not using ceph as well, then asking them to upload a qcow2 image is totally legit
17:28:09 sean-k-mooney glance is usign file and ceph
17:28:25 sean-k-mooney so yes uploading qcow is ligit in the job
17:28:32 sean-k-mooney given file is the default
17:28:49 sean-k-mooney even if it was not the default it would be ligit but slow
17:28:56 dansmith yeah
17:29:11 sean-k-mooney we also have force image convertion to raw in this job
17:29:30 sean-k-mooney which could be a diffent issue
17:29:50 sean-k-mooney its not failing in galnce however so its not the current issue
17:32:10 supamatt :wave: hi folks this change broke security groups https://review.opendev.org/c/openstack/nova/+/811521 , if any security groups with the same name exist in the project (whether the VM is using that sec group or not). Results in VMs not being able to be built with an error saying "Multiple security groups found matching . Use an ID to be more specific"
17:33:08 sean-k-mooney if you relying on passign the name instead of the uuid
17:33:54 sean-k-mooney it woudl also only fail if that conflicting named security group was shared with you
17:34:37 sean-k-mooney supamatt: im not really sure how to adress the conflicting requirements
17:34:53 sean-k-mooney prefering the local one seams incorrect
17:35:11 sean-k-mooney if there is a name conflict it should be raised as an error by nova
17:36:05 masahito gmann: gibi: hi please review this api bug fix https://review.opendev.org/c/openstack/nova/+/939658 The v2.100 follows the same fix, so it might be ok for W+1.
17:36:48 supamatt sean-k-mooney: that's the thing, I am passing the uuid to nova. But nova is errors if any security has the same name in a project, regardless if I use that sec group to provision a vm.
17:37:30 sean-k-mooney supamatt: on ok
17:37:34 sean-k-mooney *oh
17:37:41 sean-k-mooney that simpler to fix
17:37:58 sean-k-mooney supamatt: have you filed a bug for this yet
17:38:17 supamatt I was checking lp now to see if there was an existing one
17:38:32 supamatt but it seems I am the first to see this, so I will file one
17:39:08 sean-k-mooney if you pass a name then the current behavior is arguabel correct. we could prefer the one from the current project but that might eb surprisign, if you pass a uuid we shoudl use the one you said even if there are duplicate names so that a ligit bug
17:40:38 sean-k-mooney supamatt: so taht a new feature in epoxy which release tomorrow
17:41:10 sean-k-mooney we wont have a fix for it in the release but we likely will do another release after the ptg

Earlier   Later