| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-24 | |||
| 11:15:12 | sean-k-mooney | i just dont recall if we generate that upstream or not | |
| 11:15:47 | mikal | I'll just drop the SPICE concurrency patch for now. There is zero chance of me pursuing a libvirt patch right now. | |
| 12:46:39 | opendevreview | Merged openstack/nova stable/2024.1: Stop using split UEC image (mostly) https://review.opendev.org/c/openstack/nova/+/923831 | |
| 13:14:58 | frickler | sean-k-mooney: bauzas: grenade jobs are failing on the 2023.1 patch, but should grenade actually still run there when zed is eom? https://review.opendev.org/c/openstack/nova/+/924734 | |
| 13:16:40 | frickler | (seems more of the setuptools woe happening there) | |
| 13:16:51 | frickler | also https://zuul.opendev.org/t/openstack/build/c8fbe90f70974d2f918ee7fa6da54575 looks recheckable to me | |
| 13:19:44 | sean-k-mooney | frickler: we have debated that before. so its failign because of the setuptool issue right | |
| 13:21:14 | frickler | sean-k-mooney: the grenade failures that I looked at, yes. note the failure is on the controller, the later failure on compute1 is just secondary | |
| 13:21:46 | sean-k-mooney | i skimed them this morning and didnt see anytign really related to the patch | |
| 13:22:16 | sean-k-mooney | i didnt really look into test_get_list_deleted_instance_actions closely | |
| 13:23:24 | sean-k-mooney | Build of instance 38107228-1260-473d-b6a8-7f3148a135ce aborted: Failed to allocate the network(s), not rescheduling.: nova.exception.BuildAbortException: Build of instance 38107228-1260-473d-b6a8-7f3148a135ce aborted: Failed to allocate the network(s), not rescheduling. | |
| 13:23:36 | sean-k-mooney | ok so that is also unrelated | |
| 13:24:31 | bauzas | .... | |
| 13:31:32 | sean-k-mooney | frickler: so yes i think that will pass if we reject if the upper constraits bump for packaging has been merged | |
| 13:31:39 | sean-k-mooney | to fix the focal issues | |
| 13:38:19 | dansmith | ykarel: around? | |
| 13:39:06 | dansmith | oh I guess sean-k-mooney and ralonsoh have already discussed the image content failure | |
| 13:41:09 | ralonsoh | dansmith, most probably will change the neutron jobs to use qcow2 images, we are still testing | |
| 13:41:24 | dansmith | ralonsoh: excellent | |
| 14:15:55 | sean-k-mooney | dansmith: longterm we should fix tempets but that the short term solution. for our 2024.1 branch we have merge the revert back to qcow | |
| 14:16:11 | sean-k-mooney | and we had previously green result on 2023.2 | |
| 14:16:14 | dansmith | long-term we should drop ami/uec/whatever support :) | |
| 14:16:25 | sean-k-mooney | :) perhaps | |
| 14:16:49 | sean-k-mooney | alpine now provide cloud images by the way with cloud init in qcow format | |
| 14:17:16 | sean-k-mooney | so i might look at usign those instead of cirros again if we continue to see kernel panics | |
| 14:19:18 | sean-k-mooney | dansmith: a i saw you repsonded on https://bugs.launchpad.net/cinder/+bug/2073413 as well | |
| 14:19:44 | dansmith | yup | |
| 14:20:12 | sean-k-mooney | v2 qcow images might be safe if the lack of the feature mask means they cant have backing files | |
| 14:20:28 | sean-k-mooney | but if they can and we just didnt ahve a clean way to detech them before v3 | |
| 14:20:39 | sean-k-mooney | then im much less open to supprotign them going forward | |
| 14:22:23 | dansmith | yeah, I think they're probably safe here and we can exclude them, but I'm in no big rush to enable them.. that's pretty old at this point, none of the things I've tested use that old format (of course) and you can qemu-img convert your way to a v3 very easily | |
| 14:22:27 | sean-k-mooney | frickler: i can hack a poc of that i have a job that pull form my file share but i can likely just update the url | |
| 14:22:40 | dansmith | I'm working on a patch against oslo, but I don't think we need to apply it everywhere first | |
| 14:53:40 | ykarel | sean-k-mooney, wrt your alternate solution re. configure tempest for image create/upload | |
| 14:54:18 | ykarel | have you checked whoami-rajat comment regarding issue in format inspector? | |
| 14:54:45 | dansmith | ykarel: which issue? | |
| 14:54:46 | ykarel | as the images are created from snapshot, so tempest config doesn't seem involved here | |
| 14:54:56 | ykarel | dansmith, https://bugs.launchpad.net/nova/+bug/2073944 | |
| 14:56:08 | dansmith | ah | |
| 14:57:23 | dansmith | perhaps bfv instances and cinder is uploading always as raw even if it's qcow2? | |
| 14:58:45 | dansmith | or, cinder is setting the disk_format to ami, but uploading a qcow2 I guess | |
| 15:45:58 | ykarel | whoami-rajat, once around if you can check/confirm ^ | |
| 16:02:37 | opendevreview | Dan Smith proposed openstack/nova master: DNM re-enable UEC images for testing https://review.opendev.org/c/openstack/nova/+/924865 | |
| 16:02:37 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Remove AMI snapshot format special case https://review.opendev.org/c/openstack/nova/+/924866 | |
| 17:04:05 | _colby | sean-k-mooney: bauzas: thanks. We were able to get instances running with the vgpu and the A40. Its still creating all the resource providers for every virtual function (event the ones not allowed/configured in nova) unlike the centos 8 machines. So we might have some issues with vgpu recycling and removing mdevs. Luckily for this host its going to be a single project using it and the vgpu sizes will stay the same. | |
| 17:05:29 | _colby | We are using the latest Nvidia vgpu drivers | |
| 17:09:55 | dansmith | melwitt: sean-k-mooney: early signs show that patch combo repro's and fixes the problem | |
| 17:11:24 | sean-k-mooney | dansmith: just back. that is good to hear | |
| 17:13:14 | dansmith | I need to fix unit tests | |
| 17:13:49 | melwitt | sounds good but I think I missed, how does the snapshot metadata path get taken with rescue? | |
| 17:14:16 | melwitt | or maybe it's that rescue tempest tests make snapshots? | |
| 17:15:21 | melwitt | ok yeah I see that stable rescue test makes a snapshot to create a rescue image | |
| 17:22:04 | sean-k-mooney | dansmith: so am i correct in thinking that there may be instance that have had snapshot created in the past for ami guess that are not the wrong format in glance | |
| 17:22:32 | dansmith | I failed to parse that question | |
| 17:22:32 | sean-k-mooney | do you know if glance allow the format to be updated to reflect reality? | |
| 17:22:53 | dansmith | glance does not allow updating the disk_format after creation (except internally due to image conversion) | |
| 17:23:19 | sean-k-mooney | ack so if you booted a guest from a uec/ami image and took a snapshot | |
| 17:23:25 | sean-k-mooney | that was uploaded as ami | |
| 17:23:43 | dansmith | you're asking about existing snapshots that have an incorrect disk_format? that will be a problem for everyone, even not AMI.. like if you have always had isos or qcows registered as raw because nobody checked, those are all now broken | |
| 17:23:45 | sean-k-mooney | then we can nologner use that snapshot correct | |
| 17:23:58 | sean-k-mooney | right | |
| 17:23:59 | dansmith | yes | |
| 17:24:21 | sean-k-mooney | ok im wondering if we need to consider a tools or something to help with that case in the future | |
| 17:25:02 | sean-k-mooney | basically tring ot think of if there is anythign we can do for isntance that are shelved with the wrogn format for example | |
| 17:27:15 | dansmith | yeah, I mean hacking the glance DB is probably the only way, other than a glance-manage automation of that | |
| 17:27:29 | dansmith | I say we wait until someone demands it | |
| 17:29:41 | sean-k-mooney | ya i also have no idea how powsisbel it woudl even be if your using glance with cinder as a backend for example | |
| 17:45:29 | dansmith | zomg | |
| 17:45:46 | dansmith | one of those tests was silently failing because a mock didn't set disk_format | |
| 17:46:25 | dansmith | and now that I'm removing the line that was failing and being covered up. by the test, we're actually running a test that use ami and failing for other broken crap | |
| 17:47:26 | dansmith | because a missing o.vo object attribute raises NotImplementedError, which the test is ignoring because some drivers don't implement snapshot | |
| 17:47:29 | dansmith | *facepalm* | |
| 17:48:00 | dansmith | actually, this test wasn't running for libvirt *at all* for that reason, image type aside | |
| 18:05:05 | opendevreview | Dan Smith proposed openstack/nova master: Remove AMI snapshot format special case https://review.opendev.org/c/openstack/nova/+/924866 | |
| 18:05:13 | dansmith | melwitt: sean-k-mooney: please review carefully ^ | |
| 18:07:52 | frickler | bauzas: sean-k-mooney: dansmith: 2024.1 part of the cve fix seems ready now https://review.opendev.org/c/openstack/nova/+/924732 | |
| 18:08:38 | dansmith | frickler: that had to wait for the de-AMI-ification yeah? | |
| 18:09:07 | dansmith | asking because of its relevance to the above patch, although I guess the ship has sailed now | |
| 18:10:42 | frickler | from what I saw, it was only waiting on https://review.opendev.org/c/openstack/nova/+/923831 in order to fix jobs, but I may have missed something | |
| 18:11:20 | dansmith | that's the de-AMI-ification of which I speak | |
| 18:11:39 | frickler | ah, that's done then, I'd say | |
| 18:55:52 | sean-k-mooney | frickler: we will likely keep merging the backport even though there is a regresion in snapshot functionaltiy but we may give it a few days before proposing new releases | |
| 18:56:14 | sean-k-mooney | to also include https://review.opendev.org/c/openstack/nova/+/924866 | |
| 18:56:54 | sean-k-mooney | and its backports | |
| 18:58:02 | sean-k-mooney | dansmith: from the commit message i was expecting larger changes whic makes me sad as they are just subtler changes instead | |
| 19:00:20 | sean-k-mooney | i think i need to test this in devstack porperly beause im not sure i trust just unit test coverage in this case | |
| 19:05:25 | dansmith | the unit testing is garbage, as proved by the fact that they've been skipping this case on libvirt for ages | |
| 19:05:28 | dansmith | but yeah, please do | |
| 19:09:58 | sean-k-mooney | im basically concerend we might be really storign other things in the instnace_system_metadata | |
| 19:10:32 | sean-k-mooney | so i want ot boot an ami guest and look at the db | |
| 19:11:46 | dansmith | this change only really affects snapshot, fwiw | |
| 19:29:24 | sean-k-mooney | ya so really just need to test snapshot, rebuild form snapshot, boot form snapshot and shelve | |
| 20:28:43 | sean-k-mooney | had do fix a few things first but ok i have guest booted form the ami formated image | |
| 20:30:33 | opendevreview | Merged openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732 | |
| 20:32:44 | sean-k-mooney | so with force_raw_images = True and use_cow_images = True | |
| 20:32:54 | sean-k-mooney | it create the snaphsot in qcow format | |
| 20:33:36 | sean-k-mooney | which is the format of the root disk of the runing vm but not the format of the orginal glance image or the backing file | |
| 20:33:46 | sean-k-mooney | im going to double check that now | |
| 20:34:02 | sean-k-mooney | and ill also save the image and check what it actully contins too | |
| 20:35:24 | sean-k-mooney | yes backing file is raw and root disk is a qcow and the snapshot was uploaded as a qcow | |