Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-24
18:58:02 sean-k-mooney dansmith: from the commit message i was expecting larger changes whic makes me sad as they are just subtler changes instead
19:00:20 sean-k-mooney i think i need to test this in devstack porperly beause im not sure i trust just unit test coverage in this case
19:05:25 dansmith the unit testing is garbage, as proved by the fact that they've been skipping this case on libvirt for ages
19:05:28 dansmith but yeah, please do
19:09:58 sean-k-mooney im basically concerend we might be really storign other things in the instnace_system_metadata
19:10:32 sean-k-mooney so i want ot boot an ami guest and look at the db
19:11:46 dansmith this change only really affects snapshot, fwiw
19:29:24 sean-k-mooney ya so really just need to test snapshot, rebuild form snapshot, boot form snapshot and shelve
20:28:43 sean-k-mooney had do fix a few things first but ok i have guest booted form the ami formated image
20:30:33 opendevreview Merged openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732
20:32:44 sean-k-mooney so with force_raw_images = True and use_cow_images = True
20:32:54 sean-k-mooney it create the snaphsot in qcow format
20:33:36 sean-k-mooney which is the format of the root disk of the runing vm but not the format of the orginal glance image or the backing file
20:33:46 sean-k-mooney im going to double check that now
20:34:02 sean-k-mooney and ill also save the image and check what it actully contins too
20:35:24 sean-k-mooney yes backing file is raw and root disk is a qcow and the snapshot was uploaded as a qcow
20:38:19 sean-k-mooney and it actully contaiens a qcow...
20:38:22 sean-k-mooney https://paste.opendev.org/show/bgiZgIIqkdSgDuotQFMO/
20:38:53 sean-k-mooney dansmith: right... im testing your patch right now and i should be expecting it to be fixed
20:40:00 sean-k-mooney so with force_raw_images = True and use_cow_images = True the new behvior is to snapshot in qcow2 format
20:40:42 sean-k-mooney we also flattend the image properly before uploading
20:41:45 sean-k-mooney ill quickly do shelve, boot form snapshot and rebuild then flip the config options to now use cow images
20:57:28 sean-k-mooney so shelve, rebuild (to ami, to qcow snapthost form ami, to iso) work, as does boot form snapshot
20:57:49 sean-k-mooney dansmith: so in the force_raw_images = True and use_cow_images = True that mostly tracks
20:58:33 sean-k-mooney the only thin i was expecting to be differnt was i expected the snapshot to be raw but since we actully upload a qcow file and it is decalred as a qcow then that working as indended
21:29:21 sean-k-mooney dansmith: i got this when i treid to shelve using raw images https://paste.opendev.org/show/bxtLREMcntQ73IvUdBDq/
21:29:52 dansmith hrm, I wonder if that's the same bug that was fooling the tests
21:30:09 sean-k-mooney so snapshot actully works
21:30:11 dansmith because it is looking for NotImplemented, which I thought was just the mock missing it
21:30:20 sean-k-mooney but snapshot when shelving does not
21:33:18 sean-k-mooney im just going to test snapshot on its own sepreatly again quickly
21:33:31 sean-k-mooney then ill call it aday an have a look again in the morning
21:33:56 sean-k-mooney yep upload fine in raw format as expected
21:37:32 sean-k-mooney weird
21:37:58 sean-k-mooney so if so if i do it again it worked
21:38:19 sean-k-mooney but the only thing i changed is i deleted the old snapshot form the sehelve and the one i did manually before tryign again
21:42:06 sean-k-mooney dansmith: ya so if i take a snapshot i cant shelve
21:42:13 sean-k-mooney if i delete the snapshot i can
21:44:35 melwitt sean-k-mooney: when you say using raw images do you mean you set images_type = raw or flat?
21:44:50 sean-k-mooney sorry good question
21:45:16 sean-k-mooney im using a cirros uec image
21:45:58 sean-k-mooney https://paste.opendev.org/show/b5LlnYZfb8Kq5g1RCiLz/
21:46:39 sean-k-mooney and i have set the https://paste.opendev.org/show/824881/
21:46:52 sean-k-mooney i have set the correct image properites for the ram disk
21:47:14 melwitt yeah I mean like a case where shelving the instance will result in a raw image being uploaded to glance (rather than qcow2)
21:47:16 sean-k-mooney so the kernel is uploaed as disk_format aki the ramdisk as ari and the root filesystem as ami
21:47:52 sean-k-mooney melwitt: so i have use_cow_iamge = false and force_raw_iamages=true
21:47:59 sean-k-mooney so it shoudl be uploaded as raw
21:48:25 melwitt I don't think so.. otherwise we wouldn't have hit the bug right? what's being uploaded is qcow2
21:48:45 melwitt force raw images means force backing files to be raw format
21:49:04 sean-k-mooney what being uploaded matches the format of teh root disk with dan's patch
21:49:37 sean-k-mooney melwitt: yes but when you dont have use_cow_images=true it also froces the root disk to be raw
21:49:57 melwitt so if you boot an instance and it pulls a qcow2 image down from glance if force raw images is true it will convert the image to raw and use it as a backing file and then the instance disk will be qcow2 (assuming images_type = qcow2)
21:50:13 sean-k-mooney no
21:50:31 sean-k-mooney in that case if you hve set use_cow_iamge = false and force_raw_iamages=true
21:50:38 sean-k-mooney and boot form a qcow in glance
21:50:52 sean-k-mooney the vm should use a raw root disk after we convert it with qemu-img
21:51:04 sean-k-mooney i have qcows so ill test that now to confirm
21:51:05 melwitt I don't think so, given those are the defaults
21:51:19 melwitt you get a qcow2 image uploaded to glance when you snapshot, right?
21:51:37 sean-k-mooney we default to use_cow_iamge=true
21:51:39 sean-k-mooney not false
21:51:41 melwitt yes
21:52:19 sean-k-mooney melwitt: no as i said if i have "use_cow_iamge = false and force_raw_iamages=true" then the snapshot uploaded is raw
21:52:23 melwitt ok, sorry I think I read what you wrote as opposite
21:52:42 sean-k-mooney and if i have "use_cow_iamge = true and force_raw_iamages=true" the snapshot is qcow
21:52:46 sean-k-mooney with dans patch applied
21:52:56 melwitt ok, yes
21:53:00 sean-k-mooney in both cases the vm was created from an ami formated image in galnce
21:54:40 melwitt ok, cool. I had been thinking about the raw case so if that is working also then that answers what I was wondering
21:55:19 sean-k-mooney so i just booted form a qcow in glance with "use_cow_iamge = false and force_raw_iamages=true" and taht resulst in a raw root disk as expected
21:55:29 sean-k-mooney ill try snapshot and shelve quicklys
21:56:29 sean-k-mooney yep so snapshot creates a raw sanpshot
21:56:40 melwitt ok. yeah I was thinking about it because of that old code comment "glance forces ami disk format to be ami" I don't understand what it meant. so I wondered if a raw is uploaded and not labeled "ami" will glance somehow have a problem with it
21:57:09 melwitt like have a problem creating an instance from it
21:57:40 sean-k-mooney i think its just wrong
21:57:46 sean-k-mooney or at least it is today
21:58:17 sean-k-mooney i dont think glance has any idea what ami is
21:58:34 sean-k-mooney i can upload the 3 pars of the iamge as raw and it wont care
21:58:47 melwitt ok. mystery comment then I guess :)
21:59:51 sean-k-mooney so i get the same odd behavior when shelving a instance created form a glance qcow when i also have a snapshot
22:02:48 sean-k-mooney so it look like for soem reason the image backidn is tryign to use direct_snapshot
22:03:00 sean-k-mooney that fails and then we abort
22:03:02 sean-k-mooney https://paste.opendev.org/show/bKsmlsy1ClyGhEg1XTVp/
22:03:57 melwitt it actually does that for everything, tries direct_snapshot first and then falls back on regular snapshot if not implemented
22:04:00 sean-k-mooney we are not on ceph so we do not expect driect_snapshot to work
22:04:12 melwitt it's weird, not sure why it's done that way
22:04:39 sean-k-mooney its because for ceph we can sometiems do it and othertiems not
22:04:51 melwitt https://github.com/openstack/nova/blob/df39222b106326a4c28dee26b7127a61174d6b51/nova/virt/libvirt/driver.py#L3205
22:04:51 sean-k-mooney depending on if glance and nova are on the same cluster i think
22:05:14 melwitt ack
22:05:44 sean-k-mooney but the odd part is dan is not changin any of that in https://review.opendev.org/c/openstack/nova/+/924866
22:07:01 sean-k-mooney it also does not make sense why this works if i dont alreayd have a snapshot and why creating a snapshot seperatly also works
22:07:08 melwitt huh you keep getting bad gateway from glance. are there any errors in the g-api log?
22:07:34 sean-k-mooney oh you think we are just logging ingocrrectly
22:07:41 sean-k-mooney ill check
22:08:37 melwitt I don't see how 502 bad gateway can be related to your test or the patch but I have seen weird things in upstream CI before
22:09:11 sean-k-mooney i might have hit an image size quota
22:09:18 melwitt ohhhh yeah that's what it is
22:09:27 sean-k-mooney DEBUG oslo.limit.limit [None req-635da87a-1d0b-45e9-b3d1-df7b8b2decf2 admin admin] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1849 and delta 0]

Earlier   Later