| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-24 | |||
| 20:58:33 | sean-k-mooney | the only thin i was expecting to be differnt was i expected the snapshot to be raw but since we actully upload a qcow file and it is decalred as a qcow then that working as indended | |
| 21:29:21 | sean-k-mooney | dansmith: i got this when i treid to shelve using raw images https://paste.opendev.org/show/bxtLREMcntQ73IvUdBDq/ | |
| 21:29:52 | dansmith | hrm, I wonder if that's the same bug that was fooling the tests | |
| 21:30:09 | sean-k-mooney | so snapshot actully works | |
| 21:30:11 | dansmith | because it is looking for NotImplemented, which I thought was just the mock missing it | |
| 21:30:20 | sean-k-mooney | but snapshot when shelving does not | |
| 21:33:18 | sean-k-mooney | im just going to test snapshot on its own sepreatly again quickly | |
| 21:33:31 | sean-k-mooney | then ill call it aday an have a look again in the morning | |
| 21:33:56 | sean-k-mooney | yep upload fine in raw format as expected | |
| 21:37:32 | sean-k-mooney | weird | |
| 21:37:58 | sean-k-mooney | so if so if i do it again it worked | |
| 21:38:19 | sean-k-mooney | but the only thing i changed is i deleted the old snapshot form the sehelve and the one i did manually before tryign again | |
| 21:42:06 | sean-k-mooney | dansmith: ya so if i take a snapshot i cant shelve | |
| 21:42:13 | sean-k-mooney | if i delete the snapshot i can | |
| 21:44:35 | melwitt | sean-k-mooney: when you say using raw images do you mean you set images_type = raw or flat? | |
| 21:44:50 | sean-k-mooney | sorry good question | |
| 21:45:16 | sean-k-mooney | im using a cirros uec image | |
| 21:45:58 | sean-k-mooney | https://paste.opendev.org/show/b5LlnYZfb8Kq5g1RCiLz/ | |
| 21:46:39 | sean-k-mooney | and i have set the https://paste.opendev.org/show/824881/ | |
| 21:46:52 | sean-k-mooney | i have set the correct image properites for the ram disk | |
| 21:47:14 | melwitt | yeah I mean like a case where shelving the instance will result in a raw image being uploaded to glance (rather than qcow2) | |
| 21:47:16 | sean-k-mooney | so the kernel is uploaed as disk_format aki the ramdisk as ari and the root filesystem as ami | |
| 21:47:52 | sean-k-mooney | melwitt: so i have use_cow_iamge = false and force_raw_iamages=true | |
| 21:47:59 | sean-k-mooney | so it shoudl be uploaded as raw | |
| 21:48:25 | melwitt | I don't think so.. otherwise we wouldn't have hit the bug right? what's being uploaded is qcow2 | |
| 21:48:45 | melwitt | force raw images means force backing files to be raw format | |
| 21:49:04 | sean-k-mooney | what being uploaded matches the format of teh root disk with dan's patch | |
| 21:49:37 | sean-k-mooney | melwitt: yes but when you dont have use_cow_images=true it also froces the root disk to be raw | |
| 21:49:57 | melwitt | so if you boot an instance and it pulls a qcow2 image down from glance if force raw images is true it will convert the image to raw and use it as a backing file and then the instance disk will be qcow2 (assuming images_type = qcow2) | |
| 21:50:13 | sean-k-mooney | no | |
| 21:50:31 | sean-k-mooney | in that case if you hve set use_cow_iamge = false and force_raw_iamages=true | |
| 21:50:38 | sean-k-mooney | and boot form a qcow in glance | |
| 21:50:52 | sean-k-mooney | the vm should use a raw root disk after we convert it with qemu-img | |
| 21:51:04 | sean-k-mooney | i have qcows so ill test that now to confirm | |
| 21:51:05 | melwitt | I don't think so, given those are the defaults | |
| 21:51:19 | melwitt | you get a qcow2 image uploaded to glance when you snapshot, right? | |
| 21:51:37 | sean-k-mooney | we default to use_cow_iamge=true | |
| 21:51:39 | sean-k-mooney | not false | |
| 21:51:41 | melwitt | yes | |
| 21:52:19 | sean-k-mooney | melwitt: no as i said if i have "use_cow_iamge = false and force_raw_iamages=true" then the snapshot uploaded is raw | |
| 21:52:23 | melwitt | ok, sorry I think I read what you wrote as opposite | |
| 21:52:42 | sean-k-mooney | and if i have "use_cow_iamge = true and force_raw_iamages=true" the snapshot is qcow | |
| 21:52:46 | sean-k-mooney | with dans patch applied | |
| 21:52:56 | melwitt | ok, yes | |
| 21:53:00 | sean-k-mooney | in both cases the vm was created from an ami formated image in galnce | |
| 21:54:40 | melwitt | ok, cool. I had been thinking about the raw case so if that is working also then that answers what I was wondering | |
| 21:55:19 | sean-k-mooney | so i just booted form a qcow in glance with "use_cow_iamge = false and force_raw_iamages=true" and taht resulst in a raw root disk as expected | |
| 21:55:29 | sean-k-mooney | ill try snapshot and shelve quicklys | |
| 21:56:29 | sean-k-mooney | yep so snapshot creates a raw sanpshot | |
| 21:56:40 | melwitt | ok. yeah I was thinking about it because of that old code comment "glance forces ami disk format to be ami" I don't understand what it meant. so I wondered if a raw is uploaded and not labeled "ami" will glance somehow have a problem with it | |
| 21:57:09 | melwitt | like have a problem creating an instance from it | |
| 21:57:40 | sean-k-mooney | i think its just wrong | |
| 21:57:46 | sean-k-mooney | or at least it is today | |
| 21:58:17 | sean-k-mooney | i dont think glance has any idea what ami is | |
| 21:58:34 | sean-k-mooney | i can upload the 3 pars of the iamge as raw and it wont care | |
| 21:58:47 | melwitt | ok. mystery comment then I guess :) | |
| 21:59:51 | sean-k-mooney | so i get the same odd behavior when shelving a instance created form a glance qcow when i also have a snapshot | |
| 22:02:48 | sean-k-mooney | so it look like for soem reason the image backidn is tryign to use direct_snapshot | |
| 22:03:00 | sean-k-mooney | that fails and then we abort | |
| 22:03:02 | sean-k-mooney | https://paste.opendev.org/show/bKsmlsy1ClyGhEg1XTVp/ | |
| 22:03:57 | melwitt | it actually does that for everything, tries direct_snapshot first and then falls back on regular snapshot if not implemented | |
| 22:04:00 | sean-k-mooney | we are not on ceph so we do not expect driect_snapshot to work | |
| 22:04:12 | melwitt | it's weird, not sure why it's done that way | |
| 22:04:39 | sean-k-mooney | its because for ceph we can sometiems do it and othertiems not | |
| 22:04:51 | melwitt | https://github.com/openstack/nova/blob/df39222b106326a4c28dee26b7127a61174d6b51/nova/virt/libvirt/driver.py#L3205 | |
| 22:04:51 | sean-k-mooney | depending on if glance and nova are on the same cluster i think | |
| 22:05:14 | melwitt | ack | |
| 22:05:44 | sean-k-mooney | but the odd part is dan is not changin any of that in https://review.opendev.org/c/openstack/nova/+/924866 | |
| 22:07:01 | sean-k-mooney | it also does not make sense why this works if i dont alreayd have a snapshot and why creating a snapshot seperatly also works | |
| 22:07:08 | melwitt | huh you keep getting bad gateway from glance. are there any errors in the g-api log? | |
| 22:07:34 | sean-k-mooney | oh you think we are just logging ingocrrectly | |
| 22:07:41 | sean-k-mooney | ill check | |
| 22:08:37 | melwitt | I don't see how 502 bad gateway can be related to your test or the patch but I have seen weird things in upstream CI before | |
| 22:09:11 | sean-k-mooney | i might have hit an image size quota | |
| 22:09:18 | melwitt | ohhhh yeah that's what it is | |
| 22:09:27 | sean-k-mooney | DEBUG oslo.limit.limit [None req-635da87a-1d0b-45e9-b3d1-df7b8b2decf2 admin admin] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1849 and delta 0] | |
| 22:09:37 | melwitt | when you exceed quota, glance gives a 502, and it confused me so much | |
| 22:09:59 | sean-k-mooney | taht is really inccorect usage of http repsoce codes | |
| 22:10:22 | melwitt | and every time I guess I have to re figure it out to find that it's a quota limit issue | |
| 22:10:27 | melwitt | yeah, seriously | |
| 22:11:31 | sean-k-mooney | i have horizon im just setting them to -1 for now | |
| 22:12:24 | sean-k-mooney | lol of course the image qutas are not there | |
| 22:14:52 | melwitt | to do it manually it's openstack --os-cloud devstack-system-admin registered limit set --default-limit $limit --resource-name $name $registered_limit_id | |
| 22:15:34 | melwitt | to get the list openstack --os-cloud devstack registered limit list | |
| 22:15:40 | sean-k-mooney | isnt that for unified limits? | |
| 22:15:50 | sean-k-mooney | i guess glance uses that already | |
| 22:15:50 | melwitt | yeah, glance uses unified limits for its quotas | |
| 22:16:39 | melwitt | so I think -1 won't work as unlimited, but you can try if you want | |
| 22:16:51 | sean-k-mooney | it wont any more no | |
| 22:16:59 | sean-k-mooney | why is the default 1000 | |
| 22:17:01 | sean-k-mooney | which is 1G | |
| 22:17:19 | melwitt | qcow2 favoritism | |
| 22:17:28 | melwitt | (I don't actually know) | |
| 22:17:45 | sean-k-mooney | that only work for like cirros, alpine or tinycore | |
| 22:18:02 | sean-k-mooney | like most cloud images are under a gig but barely | |
| 22:18:02 | melwitt | yeah there's that too | |
| 22:20:57 | melwitt | you don't actually shouldn't pass --resource-name because it's by id and --resource-name is if you want to update the name. I made a mistake in the nova docs | |
| 22:21:11 | melwitt | *you actually shouldn't | |
| 22:21:43 | sean-k-mooney | ya i got a dup-licet key error | |
| 22:21:51 | sean-k-mooney | Conflict occurred attempting to store registered_limit - Duplicate entry. (HTTP 409) (Request-ID: req-f0f478fe-f682-4081-8837-bc89a09e79f2) | |