| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-29 | |||
| 14:51:27 | kevko | I can see that it's trying to process 0 bytes image from glance ... | |
| 14:51:36 | kevko | what i am doing bad ? | |
| 14:51:46 | kevko | some ceph settings ? copy on write ? what :( ? | |
| 14:51:53 | kevko | can anybody help ? thank you very much | |
| 14:54:15 | sean-k-mooney | kevko: this is a boot form volume instance correct | |
| 14:56:11 | sean-k-mooney | the boot form volume image rebuild was mainly implemeted for rebuilding form a normal image not a snapshot | |
| 14:56:20 | sean-k-mooney | not both shoudl be supprotable | |
| 14:56:57 | sean-k-mooney | but im not sure how well tested this is and if it works today. | |
| 14:58:38 | sean-k-mooney | im not sure if https://review.opendev.org/c/openstack/nova/+/930754 would adress this or not | |
| 14:58:59 | kevko | sean-k-mooney: this is my script to test | |
| 14:59:01 | kevko | https://paste.openstack.org/show/bpEXTVEN6E6Jc0ZTnXta/ | |
| 14:59:59 | sean-k-mooney | ok so you takign the server create form precreated volume flow | |
| 15:00:13 | sean-k-mooney | which is one of the two boot form volume workflows | |
| 15:00:48 | sean-k-mooney | then your creating a normla snapshot and then rebuildign to that | |
| 15:00:58 | kevko | sean-k-mooney: yeah, that should work no ? | |
| 15:01:31 | kevko | it's kinda a new feature to rebuild with cinder backed volume ...from 2.93 microversion | |
| 15:01:47 | kevko | I will be very grateful if you help me :) | |
| 15:02:18 | sean-k-mooney | so the reason im hesitating is i dont know if that supprot the special optimiation that we do when cinder and glance share the same storage backend | |
| 15:04:01 | sean-k-mooney | cidner and glance need to be configured in a spefic way in general to allwo them to do thing snaphosts when creatign new voluems form an image/snapshot for example | |
| 15:04:41 | sean-k-mooney | the main testing around https://specs.openstack.org/openstack/nova-specs/specs/zed/implemented/volume-backed-server-rebuild.html i belive was done with out ceph | |
| 15:05:00 | sean-k-mooney | so glance on file/swirft and cidner on lvm | |
| 15:05:16 | kevko | sean-k-mooney: i was able to catch incoming files with rsync in infinity loop | |
| 15:05:18 | kevko | sean-k-mooney: https://ibb.co/KK2KzfR | |
| 15:05:53 | kevko | sean-k-mooney: so i was wondering if there is some bug maybe ? because there is a 0 bytes image and also bigger one (cirros) | |
| 15:06:57 | sean-k-mooney | so what the cinder backend shoudl be doing in this case is the volume snapshot shoudl be creating a snapshot in ceph | |
| 15:07:01 | kevko | sean-k-mooney: https://ibb.co/F4NJDhB | |
| 15:07:14 | sean-k-mooney | and hten the rebuidl on the cider driver side should be just askign ceph to restore it | |
| 15:07:35 | sean-k-mooney | there should be no data trasfer in either code path in this case | |
| 15:07:59 | kevko | https://ibb.co/48gFG7f << attempt to import 0 bytes image into volumes pool | |
| 15:08:31 | sean-k-mooney | yep which impleis that you have an issue in how you have configred cinder an glance to tlak to each other | |
| 15:08:57 | sean-k-mooney | you are using the same ceph cluster for glance and cinder correct? | |
| 15:09:24 | kevko | sean-k-mooney: sure, I'm not an expert ... but at least I'm trying to follow the threads and I'm getting further and further ... and it's strange that internal Ceph isn't used ... and that the data is being copied... That also seemed strange to me. | |
| 15:09:34 | kevko | sean-k-mooney: yes, same cluster ... | |
| 15:10:09 | sean-k-mooney | https://docs.openstack.org/cinder/latest/admin/volume-backed-image.html | |
| 15:10:10 | kevko | sean-k-mooney: what do you think how i configured glance<->cinder to talk each other ? | |
| 15:10:21 | sean-k-mooney | have you confiugred glance ot supprot volume backed images | |
| 15:10:53 | sean-k-mooney | kevko: this is really not a nova issue by the way the error is on the cinder side | |
| 15:12:24 | sean-k-mooney | i belvie you need to configure that for https://specs.openstack.org/openstack/cinder-specs/specs/liberty/clone-image-in-glance-cinder-backend.html to work | |
| 15:13:11 | kevko | sean-k-mooney: this is my conf https://paste.openstack.org/show/bc0ZzJIdENRaizbJR1vG/ | |
| 15:14:06 | sean-k-mooney | part of me hopes SECRET_OMMITED is the acutal password and your are just trolling peopel :P | |
| 15:14:47 | kevko | :D | |
| 15:14:56 | kevko | that's not my pass :P | |
| 15:15:00 | kevko | hahaaha | |
| 15:15:06 | sean-k-mooney | it would make me sad if it was becasue now you woudl have to change it | |
| 15:15:25 | kevko | sean-k-mooney: i know what i am doing :) | |
| 15:15:57 | kevko | only this snapshots i am debuggign 2 days and dont' know what else i can try | |
| 15:15:58 | sean-k-mooney | so im not an expert ofn this but looking at your config, glance is not configured to show "show_multiple_locations" | |
| 15:16:31 | sean-k-mooney | which mean that cidner cant detect that its the same ceph cluster | |
| 15:16:42 | sean-k-mooney | and it can jsut ask ceph to do the restore | |
| 15:16:54 | sean-k-mooney | the issue with show_multiple_locations is that it has security implciations | |
| 15:17:35 | sean-k-mooney | it makes the ceph storage url "rbd://..." visable n the responce to a normal user | |
| 15:18:06 | kevko | sean-k-mooney: oh god, please say to me that i just made a mistake and kolla is somehow merging those two files in bad way :D | |
| 15:18:09 | kevko | sean-k-mooney: https://paste.openstack.org/show/b6A2uTpzPSWzs0EOl5iE/ | |
| 15:18:26 | kevko | sean-k-mooney: yeah i know what is the option ..i had it ... | |
| 15:18:43 | kevko | sean-k-mooney: but add raw formats to another file ..so wondering if it is not owervwiting | |
| 15:19:13 | sean-k-mooney | i have a backup of my kolla config somewher let me see where i set that | |
| 15:19:36 | sean-k-mooney | my most recent install currently does not have cinder for reasons | |
| 15:23:33 | kevko | let me reconfigure and test | |
| 15:23:39 | sean-k-mooney | oh the host with that backup is currently booted inot windows | |
| 15:23:45 | sean-k-mooney | i think /etc/kolla/config/glance-api.conf is correct | |
| 15:24:05 | kevko | sean-k-mooney: yeah, but you can also store into /etc/kolla/config/glance.conf | |
| 15:24:12 | kevko | then it's applied to all glance configs | |
| 15:24:31 | sean-k-mooney | right i was going to check the precedence | |
| 15:24:34 | kevko | and i don't have cache ..so ..it doesn't matter where it is .. | |
| 15:24:40 | kevko | yes | |
| 15:25:38 | kevko | this is btw disabled in kolla by default ..because then the locations are visible for users ..which is not good | |
| 15:25:43 | kevko | i need to implement the fix | |
| 15:25:55 | sean-k-mooney | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/tasks/config.yml#L61-L76 | |
| 15:26:16 | sean-k-mooney | its looking for "{{ node_custom_config }}/glance/glance-api.conf" | |
| 15:26:28 | sean-k-mooney | not /etc/kolla/config/glance-api.conf | |
| 15:26:36 | kevko | yeah i know :) | |
| 15:26:52 | kevko | that's the reason why I ve said oh god :D | |
| 15:26:56 | kevko | i've spotted it | |
| 15:26:59 | sean-k-mooney | ah | |
| 15:27:00 | kevko | my test is running now | |
| 15:27:11 | sean-k-mooney | i have done that before... | |
| 15:27:20 | sean-k-mooney | forgot to create the service folder | |
| 15:28:28 | sean-k-mooney | one of the things i like most about kolla is its config system | |
| 15:28:47 | sean-k-mooney | but i woudl be lying if i said i never put the file in the worng place | |
| 15:29:36 | kevko | sean-k-mooney: yeah i like it also, but i've also find some bugs in past ... especially with merge ini files | |
| 15:32:54 | sean-k-mooney | the ini merging was there before i got invovled but the yaml merging is my fault :) although i did take insperation form the ini version | |
| 15:44:03 | kevko | sean-k-mooney: error :) | |
| 15:44:46 | kevko | sean-k-mooney: same error | |
| 15:51:33 | opendevreview | Ildiko Vancsa proposed openstack/nova master: Update contributor process documentation https://review.opendev.org/c/openstack/nova/+/936759 | |
| 15:55:27 | ildikov | sean-k-mooney: bauzas: I created the patch to update the contributor docs ^^ | |
| 15:55:28 | sean-k-mooney | kevko: with a newly created vm | |
| 15:56:11 | sean-k-mooney | ildikov: yep i just saw eamil notificaton come in, thansk for takign the time to start on that | |
| 15:56:25 | sean-k-mooney | ill be finishing soso but ill try and take a look before i leave for today | |
| 15:56:28 | ildikov | sean-k-mooney: bauzas: I deleted the parts we said are outdated, but I don't know how people are supposed to find the tracking etherpad and interact with it, so I wasn't able to write that part myself | |
| 15:56:54 | ildikov | sean-k-mooney: no rush, thank you for adding it to your todo list | |
| 15:57:21 | sean-k-mooney | ildikov: the delete look correct but ya we shoudl add a short paragah on how to use the etherpad | |
| 15:58:18 | ildikov | sean-k-mooney: great! And yes, I'm hoping to rely on you and bauzas to type up that paragraph. | |
| 15:58:25 | sean-k-mooney | its basiclly add you reivw by addign a new bullet under the correct section with a link to the bug/bluepirnt in lanchpad then under that add either a link to the gerrit topic or each patch | |
| 15:59:05 | sean-k-mooney | you can also addd a short note like needs second +2 or simlar | |
| 15:59:54 | sean-k-mooney | we dont actully have a copypaste template to use but we proably can add one | |
| 16:00:56 | sean-k-mooney | the last thing i woudl suggest peopel do is ping use on irc (via matrix bride if that is easier) to let us know but lests see what bauzas things | |
| 16:00:57 | ildikov | 'copypaste template' to easily add a new entry you mean? | |
| 16:00:59 | sean-k-mooney | *thinks | |
| 16:01:09 | sean-k-mooney | ildikov: yep | |
| 16:01:28 | ildikov | ok, cool | |