Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-24
14:58:45 dansmith or, cinder is setting the disk_format to ami, but uploading a qcow2 I guess
15:45:58 ykarel whoami-rajat, once around if you can check/confirm ^
16:02:37 opendevreview Dan Smith proposed openstack/nova master: DNM re-enable UEC images for testing https://review.opendev.org/c/openstack/nova/+/924865
16:02:37 opendevreview Dan Smith proposed openstack/nova master: WIP: Remove AMI snapshot format special case https://review.opendev.org/c/openstack/nova/+/924866
17:04:05 _colby sean-k-mooney: bauzas: thanks. We were able to get instances running with the vgpu and the A40. Its still creating all the resource providers for every virtual function (event the ones not allowed/configured in nova) unlike the centos 8 machines. So we might have some issues with vgpu recycling and removing mdevs. Luckily for this host its going to be a single project using it and the vgpu sizes will stay the same.
17:05:29 _colby We are using the latest Nvidia vgpu drivers
17:09:55 dansmith melwitt: sean-k-mooney: early signs show that patch combo repro's and fixes the problem
17:11:24 sean-k-mooney dansmith: just back. that is good to hear
17:13:14 dansmith I need to fix unit tests
17:13:49 melwitt sounds good but I think I missed, how does the snapshot metadata path get taken with rescue?
17:14:16 melwitt or maybe it's that rescue tempest tests make snapshots?
17:15:21 melwitt ok yeah I see that stable rescue test makes a snapshot to create a rescue image
17:22:04 sean-k-mooney dansmith: so am i correct in thinking that there may be instance that have had snapshot created in the past for ami guess that are not the wrong format in glance
17:22:32 dansmith I failed to parse that question
17:22:32 sean-k-mooney do you know if glance allow the format to be updated to reflect reality?
17:22:53 dansmith glance does not allow updating the disk_format after creation (except internally due to image conversion)
17:23:19 sean-k-mooney ack so if you booted a guest from a uec/ami image and took a snapshot
17:23:25 sean-k-mooney that was uploaded as ami
17:23:43 dansmith you're asking about existing snapshots that have an incorrect disk_format? that will be a problem for everyone, even not AMI.. like if you have always had isos or qcows registered as raw because nobody checked, those are all now broken
17:23:45 sean-k-mooney then we can nologner use that snapshot correct
17:23:58 sean-k-mooney right
17:23:59 dansmith yes
17:24:21 sean-k-mooney ok im wondering if we need to consider a tools or something to help with that case in the future
17:25:02 sean-k-mooney basically tring ot think of if there is anythign we can do for isntance that are shelved with the wrogn format for example
17:27:15 dansmith yeah, I mean hacking the glance DB is probably the only way, other than a glance-manage automation of that
17:27:29 dansmith I say we wait until someone demands it
17:29:41 sean-k-mooney ya i also have no idea how powsisbel it woudl even be if your using glance with cinder as a backend for example
17:45:29 dansmith zomg
17:45:46 dansmith one of those tests was silently failing because a mock didn't set disk_format
17:46:25 dansmith and now that I'm removing the line that was failing and being covered up. by the test, we're actually running a test that use ami and failing for other broken crap
17:47:26 dansmith because a missing o.vo object attribute raises NotImplementedError, which the test is ignoring because some drivers don't implement snapshot
17:47:29 dansmith *facepalm*
17:48:00 dansmith actually, this test wasn't running for libvirt *at all* for that reason, image type aside
18:05:05 opendevreview Dan Smith proposed openstack/nova master: Remove AMI snapshot format special case https://review.opendev.org/c/openstack/nova/+/924866
18:05:13 dansmith melwitt: sean-k-mooney: please review carefully ^
18:07:52 frickler bauzas: sean-k-mooney: dansmith: 2024.1 part of the cve fix seems ready now https://review.opendev.org/c/openstack/nova/+/924732
18:08:38 dansmith frickler: that had to wait for the de-AMI-ification yeah?
18:09:07 dansmith asking because of its relevance to the above patch, although I guess the ship has sailed now
18:10:42 frickler from what I saw, it was only waiting on https://review.opendev.org/c/openstack/nova/+/923831 in order to fix jobs, but I may have missed something
18:11:20 dansmith that's the de-AMI-ification of which I speak
18:11:39 frickler ah, that's done then, I'd say
18:55:52 sean-k-mooney frickler: we will likely keep merging the backport even though there is a regresion in snapshot functionaltiy but we may give it a few days before proposing new releases
18:56:14 sean-k-mooney to also include https://review.opendev.org/c/openstack/nova/+/924866
18:56:54 sean-k-mooney and its backports
18:58:02 sean-k-mooney dansmith: from the commit message i was expecting larger changes whic makes me sad as they are just subtler changes instead
19:00:20 sean-k-mooney i think i need to test this in devstack porperly beause im not sure i trust just unit test coverage in this case
19:05:25 dansmith the unit testing is garbage, as proved by the fact that they've been skipping this case on libvirt for ages
19:05:28 dansmith but yeah, please do
19:09:58 sean-k-mooney im basically concerend we might be really storign other things in the instnace_system_metadata
19:10:32 sean-k-mooney so i want ot boot an ami guest and look at the db
19:11:46 dansmith this change only really affects snapshot, fwiw
19:29:24 sean-k-mooney ya so really just need to test snapshot, rebuild form snapshot, boot form snapshot and shelve
20:28:43 sean-k-mooney had do fix a few things first but ok i have guest booted form the ami formated image
20:30:33 opendevreview Merged openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732
20:32:44 sean-k-mooney so with force_raw_images = True and use_cow_images = True
20:32:54 sean-k-mooney it create the snaphsot in qcow format
20:33:36 sean-k-mooney which is the format of the root disk of the runing vm but not the format of the orginal glance image or the backing file
20:33:46 sean-k-mooney im going to double check that now
20:34:02 sean-k-mooney and ill also save the image and check what it actully contins too
20:35:24 sean-k-mooney yes backing file is raw and root disk is a qcow and the snapshot was uploaded as a qcow
20:38:19 sean-k-mooney and it actully contaiens a qcow...
20:38:22 sean-k-mooney https://paste.opendev.org/show/bgiZgIIqkdSgDuotQFMO/
20:38:53 sean-k-mooney dansmith: right... im testing your patch right now and i should be expecting it to be fixed
20:40:00 sean-k-mooney so with force_raw_images = True and use_cow_images = True the new behvior is to snapshot in qcow2 format
20:40:42 sean-k-mooney we also flattend the image properly before uploading
20:41:45 sean-k-mooney ill quickly do shelve, boot form snapshot and rebuild then flip the config options to now use cow images
20:57:28 sean-k-mooney so shelve, rebuild (to ami, to qcow snapthost form ami, to iso) work, as does boot form snapshot
20:57:49 sean-k-mooney dansmith: so in the force_raw_images = True and use_cow_images = True that mostly tracks
20:58:33 sean-k-mooney the only thin i was expecting to be differnt was i expected the snapshot to be raw but since we actully upload a qcow file and it is decalred as a qcow then that working as indended
21:29:21 sean-k-mooney dansmith: i got this when i treid to shelve using raw images https://paste.opendev.org/show/bxtLREMcntQ73IvUdBDq/
21:29:52 dansmith hrm, I wonder if that's the same bug that was fooling the tests
21:30:09 sean-k-mooney so snapshot actully works
21:30:11 dansmith because it is looking for NotImplemented, which I thought was just the mock missing it
21:30:20 sean-k-mooney but snapshot when shelving does not
21:33:18 sean-k-mooney im just going to test snapshot on its own sepreatly again quickly
21:33:31 sean-k-mooney then ill call it aday an have a look again in the morning
21:33:56 sean-k-mooney yep upload fine in raw format as expected
21:37:32 sean-k-mooney weird
21:37:58 sean-k-mooney so if so if i do it again it worked
21:38:19 sean-k-mooney but the only thing i changed is i deleted the old snapshot form the sehelve and the one i did manually before tryign again
21:42:06 sean-k-mooney dansmith: ya so if i take a snapshot i cant shelve
21:42:13 sean-k-mooney if i delete the snapshot i can
21:44:35 melwitt sean-k-mooney: when you say using raw images do you mean you set images_type = raw or flat?
21:44:50 sean-k-mooney sorry good question
21:45:16 sean-k-mooney im using a cirros uec image
21:45:58 sean-k-mooney https://paste.opendev.org/show/b5LlnYZfb8Kq5g1RCiLz/
21:46:39 sean-k-mooney and i have set the https://paste.opendev.org/show/824881/
21:46:52 sean-k-mooney i have set the correct image properites for the ram disk
21:47:14 melwitt yeah I mean like a case where shelving the instance will result in a raw image being uploaded to glance (rather than qcow2)
21:47:16 sean-k-mooney so the kernel is uploaed as disk_format aki the ramdisk as ari and the root filesystem as ami
21:47:52 sean-k-mooney melwitt: so i have use_cow_iamge = false and force_raw_iamages=true
21:47:59 sean-k-mooney so it shoudl be uploaded as raw
21:48:25 melwitt I don't think so.. otherwise we wouldn't have hit the bug right? what's being uploaded is qcow2
21:48:45 melwitt force raw images means force backing files to be raw format
21:49:04 sean-k-mooney what being uploaded matches the format of teh root disk with dan's patch
21:49:37 sean-k-mooney melwitt: yes but when you dont have use_cow_images=true it also froces the root disk to be raw
21:49:57 melwitt so if you boot an instance and it pulls a qcow2 image down from glance if force raw images is true it will convert the image to raw and use it as a backing file and then the instance disk will be qcow2 (assuming images_type = qcow2)
21:50:13 sean-k-mooney no
21:50:31 sean-k-mooney in that case if you hve set use_cow_iamge = false and force_raw_iamages=true
21:50:38 sean-k-mooney and boot form a qcow in glance

Earlier   Later