| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-24 | |||
| 14:53:40 | ykarel | sean-k-mooney, wrt your alternate solution re. configure tempest for image create/upload | |
| 14:54:18 | ykarel | have you checked whoami-rajat comment regarding issue in format inspector? | |
| 14:54:45 | dansmith | ykarel: which issue? | |
| 14:54:46 | ykarel | as the images are created from snapshot, so tempest config doesn't seem involved here | |
| 14:54:56 | ykarel | dansmith, https://bugs.launchpad.net/nova/+bug/2073944 | |
| 14:56:08 | dansmith | ah | |
| 14:57:23 | dansmith | perhaps bfv instances and cinder is uploading always as raw even if it's qcow2? | |
| 14:58:45 | dansmith | or, cinder is setting the disk_format to ami, but uploading a qcow2 I guess | |
| 15:45:58 | ykarel | whoami-rajat, once around if you can check/confirm ^ | |
| 16:02:37 | opendevreview | Dan Smith proposed openstack/nova master: DNM re-enable UEC images for testing https://review.opendev.org/c/openstack/nova/+/924865 | |
| 16:02:37 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Remove AMI snapshot format special case https://review.opendev.org/c/openstack/nova/+/924866 | |
| 17:04:05 | _colby | sean-k-mooney: bauzas: thanks. We were able to get instances running with the vgpu and the A40. Its still creating all the resource providers for every virtual function (event the ones not allowed/configured in nova) unlike the centos 8 machines. So we might have some issues with vgpu recycling and removing mdevs. Luckily for this host its going to be a single project using it and the vgpu sizes will stay the same. | |
| 17:05:29 | _colby | We are using the latest Nvidia vgpu drivers | |
| 17:09:55 | dansmith | melwitt: sean-k-mooney: early signs show that patch combo repro's and fixes the problem | |
| 17:11:24 | sean-k-mooney | dansmith: just back. that is good to hear | |
| 17:13:14 | dansmith | I need to fix unit tests | |
| 17:13:49 | melwitt | sounds good but I think I missed, how does the snapshot metadata path get taken with rescue? | |
| 17:14:16 | melwitt | or maybe it's that rescue tempest tests make snapshots? | |
| 17:15:21 | melwitt | ok yeah I see that stable rescue test makes a snapshot to create a rescue image | |
| 17:22:04 | sean-k-mooney | dansmith: so am i correct in thinking that there may be instance that have had snapshot created in the past for ami guess that are not the wrong format in glance | |
| 17:22:32 | dansmith | I failed to parse that question | |
| 17:22:32 | sean-k-mooney | do you know if glance allow the format to be updated to reflect reality? | |
| 17:22:53 | dansmith | glance does not allow updating the disk_format after creation (except internally due to image conversion) | |
| 17:23:19 | sean-k-mooney | ack so if you booted a guest from a uec/ami image and took a snapshot | |
| 17:23:25 | sean-k-mooney | that was uploaded as ami | |
| 17:23:43 | dansmith | you're asking about existing snapshots that have an incorrect disk_format? that will be a problem for everyone, even not AMI.. like if you have always had isos or qcows registered as raw because nobody checked, those are all now broken | |
| 17:23:45 | sean-k-mooney | then we can nologner use that snapshot correct | |
| 17:23:58 | sean-k-mooney | right | |
| 17:23:59 | dansmith | yes | |
| 17:24:21 | sean-k-mooney | ok im wondering if we need to consider a tools or something to help with that case in the future | |
| 17:25:02 | sean-k-mooney | basically tring ot think of if there is anythign we can do for isntance that are shelved with the wrogn format for example | |
| 17:27:15 | dansmith | yeah, I mean hacking the glance DB is probably the only way, other than a glance-manage automation of that | |
| 17:27:29 | dansmith | I say we wait until someone demands it | |
| 17:29:41 | sean-k-mooney | ya i also have no idea how powsisbel it woudl even be if your using glance with cinder as a backend for example | |
| 17:45:29 | dansmith | zomg | |
| 17:45:46 | dansmith | one of those tests was silently failing because a mock didn't set disk_format | |
| 17:46:25 | dansmith | and now that I'm removing the line that was failing and being covered up. by the test, we're actually running a test that use ami and failing for other broken crap | |
| 17:47:26 | dansmith | because a missing o.vo object attribute raises NotImplementedError, which the test is ignoring because some drivers don't implement snapshot | |
| 17:47:29 | dansmith | *facepalm* | |
| 17:48:00 | dansmith | actually, this test wasn't running for libvirt *at all* for that reason, image type aside | |
| 18:05:05 | opendevreview | Dan Smith proposed openstack/nova master: Remove AMI snapshot format special case https://review.opendev.org/c/openstack/nova/+/924866 | |
| 18:05:13 | dansmith | melwitt: sean-k-mooney: please review carefully ^ | |
| 18:07:52 | frickler | bauzas: sean-k-mooney: dansmith: 2024.1 part of the cve fix seems ready now https://review.opendev.org/c/openstack/nova/+/924732 | |
| 18:08:38 | dansmith | frickler: that had to wait for the de-AMI-ification yeah? | |
| 18:09:07 | dansmith | asking because of its relevance to the above patch, although I guess the ship has sailed now | |
| 18:10:42 | frickler | from what I saw, it was only waiting on https://review.opendev.org/c/openstack/nova/+/923831 in order to fix jobs, but I may have missed something | |
| 18:11:20 | dansmith | that's the de-AMI-ification of which I speak | |
| 18:11:39 | frickler | ah, that's done then, I'd say | |
| 18:55:52 | sean-k-mooney | frickler: we will likely keep merging the backport even though there is a regresion in snapshot functionaltiy but we may give it a few days before proposing new releases | |
| 18:56:14 | sean-k-mooney | to also include https://review.opendev.org/c/openstack/nova/+/924866 | |
| 18:56:54 | sean-k-mooney | and its backports | |
| 18:58:02 | sean-k-mooney | dansmith: from the commit message i was expecting larger changes whic makes me sad as they are just subtler changes instead | |
| 19:00:20 | sean-k-mooney | i think i need to test this in devstack porperly beause im not sure i trust just unit test coverage in this case | |
| 19:05:25 | dansmith | the unit testing is garbage, as proved by the fact that they've been skipping this case on libvirt for ages | |
| 19:05:28 | dansmith | but yeah, please do | |
| 19:09:58 | sean-k-mooney | im basically concerend we might be really storign other things in the instnace_system_metadata | |
| 19:10:32 | sean-k-mooney | so i want ot boot an ami guest and look at the db | |
| 19:11:46 | dansmith | this change only really affects snapshot, fwiw | |
| 19:29:24 | sean-k-mooney | ya so really just need to test snapshot, rebuild form snapshot, boot form snapshot and shelve | |
| 20:28:43 | sean-k-mooney | had do fix a few things first but ok i have guest booted form the ami formated image | |
| 20:30:33 | opendevreview | Merged openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732 | |
| 20:32:44 | sean-k-mooney | so with force_raw_images = True and use_cow_images = True | |
| 20:32:54 | sean-k-mooney | it create the snaphsot in qcow format | |
| 20:33:36 | sean-k-mooney | which is the format of the root disk of the runing vm but not the format of the orginal glance image or the backing file | |
| 20:33:46 | sean-k-mooney | im going to double check that now | |
| 20:34:02 | sean-k-mooney | and ill also save the image and check what it actully contins too | |
| 20:35:24 | sean-k-mooney | yes backing file is raw and root disk is a qcow and the snapshot was uploaded as a qcow | |
| 20:38:19 | sean-k-mooney | and it actully contaiens a qcow... | |
| 20:38:22 | sean-k-mooney | https://paste.opendev.org/show/bgiZgIIqkdSgDuotQFMO/ | |
| 20:38:53 | sean-k-mooney | dansmith: right... im testing your patch right now and i should be expecting it to be fixed | |
| 20:40:00 | sean-k-mooney | so with force_raw_images = True and use_cow_images = True the new behvior is to snapshot in qcow2 format | |
| 20:40:42 | sean-k-mooney | we also flattend the image properly before uploading | |
| 20:41:45 | sean-k-mooney | ill quickly do shelve, boot form snapshot and rebuild then flip the config options to now use cow images | |
| 20:57:28 | sean-k-mooney | so shelve, rebuild (to ami, to qcow snapthost form ami, to iso) work, as does boot form snapshot | |
| 20:57:49 | sean-k-mooney | dansmith: so in the force_raw_images = True and use_cow_images = True that mostly tracks | |
| 20:58:33 | sean-k-mooney | the only thin i was expecting to be differnt was i expected the snapshot to be raw but since we actully upload a qcow file and it is decalred as a qcow then that working as indended | |
| 21:29:21 | sean-k-mooney | dansmith: i got this when i treid to shelve using raw images https://paste.opendev.org/show/bxtLREMcntQ73IvUdBDq/ | |
| 21:29:52 | dansmith | hrm, I wonder if that's the same bug that was fooling the tests | |
| 21:30:09 | sean-k-mooney | so snapshot actully works | |
| 21:30:11 | dansmith | because it is looking for NotImplemented, which I thought was just the mock missing it | |
| 21:30:20 | sean-k-mooney | but snapshot when shelving does not | |
| 21:33:18 | sean-k-mooney | im just going to test snapshot on its own sepreatly again quickly | |
| 21:33:31 | sean-k-mooney | then ill call it aday an have a look again in the morning | |
| 21:33:56 | sean-k-mooney | yep upload fine in raw format as expected | |
| 21:37:32 | sean-k-mooney | weird | |
| 21:37:58 | sean-k-mooney | so if so if i do it again it worked | |
| 21:38:19 | sean-k-mooney | but the only thing i changed is i deleted the old snapshot form the sehelve and the one i did manually before tryign again | |
| 21:42:06 | sean-k-mooney | dansmith: ya so if i take a snapshot i cant shelve | |
| 21:42:13 | sean-k-mooney | if i delete the snapshot i can | |
| 21:44:35 | melwitt | sean-k-mooney: when you say using raw images do you mean you set images_type = raw or flat? | |
| 21:44:50 | sean-k-mooney | sorry good question | |
| 21:45:16 | sean-k-mooney | im using a cirros uec image | |
| 21:45:58 | sean-k-mooney | https://paste.opendev.org/show/b5LlnYZfb8Kq5g1RCiLz/ | |
| 21:46:39 | sean-k-mooney | and i have set the https://paste.opendev.org/show/824881/ | |
| 21:46:52 | sean-k-mooney | i have set the correct image properites for the ram disk | |
| 21:47:14 | melwitt | yeah I mean like a case where shelving the instance will result in a raw image being uploaded to glance (rather than qcow2) | |
| 21:47:16 | sean-k-mooney | so the kernel is uploaed as disk_format aki the ramdisk as ari and the root filesystem as ami | |
| 21:47:52 | sean-k-mooney | melwitt: so i have use_cow_iamge = false and force_raw_iamages=true | |
| 21:47:59 | sean-k-mooney | so it shoudl be uploaded as raw | |
| 21:48:25 | melwitt | I don't think so.. otherwise we wouldn't have hit the bug right? what's being uploaded is qcow2 | |