| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-24 | |||
| 17:06:23 | dansmith | jssfr: ack :) | |
| #openstack-nova - 2025-09-25 | |||
| 11:13:03 | LarsErikP | Hi.. I'm experienceing the exact same scenario as this post from 2019, running Caracal: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5QSBZDKQLA7HJF3IYMLYOF7PACWX5QQE/ am I doing something wrong here? | |
| 11:14:21 | LarsErikP | volume-backed instance - create snapshot (either in horizon or with openstack server image create) - volume snapshots are created, but the image in glance is created with disk_format=qcow2 even though everything along the way here is in raw | |
| 11:14:59 | LarsErikP | I've tried to set snapshot_image_format=raw both on compute nodes and nova-api servers, but it does not make a difference | |
| 11:16:41 | LarsErikP | and I understand from the api-ref for this, that this scenario should indeed both create the volume snap, and an "empty" glance image with bdm properties pointing to the volume snap. But this fails with a http 415 and "Image disk_format qcow2 does not match stream data raw" | |
| 11:34:57 | LarsErikP | got rid of the http 415 when setting image_format/require_image_format_match to false in glance! | |
| 11:55:46 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 13:51:21 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 13:51:21 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 16:26:12 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 19:39:38 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 19:39:38 | opendevreview | melanie witt proposed openstack/nova master: FUP for vTPM live migration https://review.opendev.org/c/openstack/nova/+/962309 | |
| 20:02:36 | melwitt | LarsErikP: if I'm understanding you correctly, what you're seeing is the "empty" glance image that gets created when you snapshot a volume backed instance. nova calls cinder to snapshot the actual volume but will also call glance to create an empty/dummy image whose purpose is only to hold image metadata https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/compute/api.py#L3459-L3466 | |
| 20:03:23 | melwitt | bc this is not a "real" image the disk_format attribute is just thrown away and not provided to glance AFAICT. maybe glance defaults to qcow2, not sure | |
| 20:05:28 | melwitt | oh but you're saying the glance image create fails due to the disk_format being wrong/not specified. this sounds like a bug that is now uncovered since glance began validating image disk formats | |
| 20:08:30 | melwitt | I don't find an existing bug open about this so as far as I know this has not been reported yet | |
| 21:06:29 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 21:09:59 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| #openstack-nova - 2025-09-26 | |||
| 06:01:58 | LarsErikP | melwitt: correct, but it failed creation because of the error I described | |
| 06:03:10 | melwitt | LarsErikP: yeah, I think you have found a bug, unless this is some known issue that is documented somewhere. I did not find anything indicating that though | |
| 06:03:31 | LarsErikP | Which I suspect happened because of the "require_image_format_match". Glance somehow interprets the "dummy"/empty image as raw, when nova creates it as qcow2 | |
| 06:07:29 | LarsErikP | melwitt: I'll report a bug to glance then =) | |
| 06:07:32 | melwitt | LarsErikP: right. glance used to not validate that the formats match (it is a relatively recent development) so in the past it went through without an error but now that glance is validating by default, it reveals the mismatch. nova does not actually set anything in the disk_format field AFAICT, it leaves it empty and maybe glance defaults to qcow2 if empty (I could be wrong). and as you found you can work around it by disabling the | |
| 06:07:32 | melwitt | format validation. I'm not sure what/how we could fix this but it seems like a bug to me or at least if it's a known issue it should be documented somewhere | |
| 06:08:15 | LarsErikP | yeah. I'll file a bug, and describe it as good as I can. I guess that'll help troubleshooting, and pin-pointing what's happening | |
| 06:08:36 | melwitt | LarsErikP: it might be considered a nova bug. I'm not sure if there is a practical way nova could check the format and send the right one to glance. if we can do that then that would be the right fix I think | |
| 06:12:07 | LarsErikP | Good point. Either that, or there is something that can be fixed in the format validation itself. I don't know enough details about how this is done. It may be worth mentioning that I tried to set the "snapshot_image_format" option in nova.conf to raw in all relevant nova-servers as well. Did not change the behaviour | |
| 06:12:53 | LarsErikP | Which led me to belive that nova is just hardcoded to upload that "dummy" image as qcow2. I did not search in nova-code to confirm that though | |
| 06:13:54 | melwitt | yeah, same. this is why I think nova isn't sending any disk_format to glance https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/compute/api.py#L3465-L3466 | |
| 06:16:17 | LarsErikP | ah. good catch. It's definitely not hardcoding qcow2 there ;) | |
| 06:16:45 | LarsErikP | so.. do you think I should file the bug to nova or glance then? | |
| 06:17:28 | melwitt | I'm thinking nova. we'll discuss it first and if we think it should be glance we can change it to glance | |
| 06:18:05 | melwitt | I think nova is supposed to be sending the right format but it's not so clear when it's the volume-backed case where it involves the weird dummy image | |
| 06:18:49 | LarsErikP | right. Something to chew on for the developers then.. I'll finnish my morning coffee and write up a bug report =) | |
| 06:19:34 | melwitt | cool, thanks | |
| 06:58:59 | LarsErikP | melwitt: https://bugs.launchpad.net/nova/+bug/2125730 =) | |
| 08:18:03 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix debug PciPassthroughFilter https://review.opendev.org/c/openstack/nova/+/907976 | |
| 09:46:50 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 12:02:25 | gibi | dansmith: this is a small fix for the devstack fake-virt support due to stable-compute-uuid prevented the fake nova-compute restarts https://review.opendev.org/c/openstack/devstack/+/962300 | |
| 13:30:02 | dansmith | gibi: would you rather just make it like ironic and not have to have the same rules? | |
| 13:35:55 | gibi | dnaI tried that but | |
| 13:36:15 | gibi | dansmith: I tried that but our own functional testing depends on the fake driver to test the compute_id logic | |
| 13:38:39 | dansmith | ah okay | |
| #openstack-nova - 2025-09-27 | |||
| 03:06:23 | opendevreview | melanie witt proposed openstack/nova master: Opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 03:06:24 | opendevreview | melanie witt proposed openstack/nova master: FUP for vTPM live migration https://review.opendev.org/c/openstack/nova/+/962309 | |
| 03:06:24 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 16:22:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Leave chown operation of tpm data directory to libvirt https://review.opendev.org/c/openstack/nova/+/962446 | |
| 16:23:27 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| 16:43:29 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| 23:36:51 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| #openstack-nova - 2025-09-28 | |||
| 14:15:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 15:27:16 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| 15:29:44 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 16:13:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 17:06:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 18:38:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 19:28:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| #openstack-nova - 2025-09-29 | |||
| 08:20:20 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 08:46:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 09:43:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 10:00:07 | opendevreview | Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657 | |
| 10:11:49 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch nova-conductor to use global executor https://review.opendev.org/c/openstack/nova/+/962478 | |
| 11:12:10 | opendevreview | Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657 | |
| 11:28:28 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 12:14:16 | gibi | dansmith: fyi we can catch threading slowdowns now with rally https://review.opendev.org/c/openstack/nova/+/960130/30#message-1f12ccedc335aa0e59d469a478012a41d76fd67c | |
| 12:14:33 | gibi | (this was a simulated slowdown by injecting sleep to nova) | |
| 12:40:42 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch nova-conductor to use global executor https://review.opendev.org/c/openstack/nova/+/962478 | |
| 13:09:06 | opendevreview | Merged openstack/nova stable/2025.1: Regression test for ImagePropsWeigher due to missing context https://review.opendev.org/c/openstack/nova/+/961805 | |
| 13:58:58 | dansmith | gibi: nice.. hopefully that will be stable enough to be useful.. even within a single worker performance has varied a lot in the past (due to noisy neighbors) but even if we get a smoke signal that will be good | |
| 14:04:47 | gibi | if the performance of the worker fluctuates in a ~10 mintues window then yeah we will get false postive signals | |
| 14:05:20 | gibi | the window size depends one length of our rally scenarios | |
| 14:17:39 | sean-k-mooney-pto | while we could have noise neighber effects i think its a lease worth trying this for a while and see if it becomes a problem | |
| 14:18:15 | sean-k-mooney-pto | gibi: presumable the rally job wont be voting at least not in the short term | |
| 14:18:32 | gibi | won't be voting for a while | |
| 14:18:35 | sean-k-mooney-pto | so if it a perodic or its non voting | |
| 14:18:43 | sean-k-mooney-pto | i think this ok for now | |
| 14:19:19 | sean-k-mooney-pto | gibi: do you think it need to run in check longterm or would you jsut rung this in weekly perodic? | |
| 14:23:29 | gibi | right now it is a cheap job so I would run it non-voting. But if we add more scenarios (I planning to ) then it might be expensive enough to run it only periodically | |
| 14:23:55 | gibi | or if we move it to a label with many subnodes | |
| 14:24:11 | sean-k-mooney-pto | ack | |
| 14:24:47 | sean-k-mooney-pto | this is the kind fo job where it would be nice if there as a persitent mysql db where we coudl log some data across runs | |
| 14:25:12 | sean-k-mooney-pto | i.e. to track over all trends across time | |
| 14:26:24 | sean-k-mooney-pto | but ya i was thining the same start we non voting and hten move to perodic + experimental if its becomes expensive | |
| 14:27:12 | sean-k-mooney-pto | havign it in experminetal so you can spot check specific changes is ueful if we go the perodic route | |
| 14:29:45 | sean-k-mooney | gibi: the last day before i went on pto i spend a little time tryign to see if we could get any better infro form atop or the logs on overall load. i made some progress but the binary format and way that tool works is a littel painful | |
| 14:40:14 | gibi | sean-k-mooney: I actually noticed that when a service is stopped systemd already nicely logs the memory swap high watermark and the cpu time of the service | |
| 14:40:20 | gibi | so I went with that | |
| 14:40:32 | sean-k-mooney | yes i saw that also | |
| 14:40:52 | sean-k-mooney | however we only have the logs form the second run correct | |
| 14:41:02 | sean-k-mooney | unless thats also in the journal dump? | |
| 14:41:56 | gibi | we have journal logs from both runs, and I do an extra service restart now at the end to get the resource usage after the second run as well. Then I pull the data out from the journal into eg https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eb1/openstack/eb1c16166a9e42b5be0987f5d8ca0d01/results/resource-usage-devstack%40n-sch.service.txt | |
| 14:43:23 | sean-k-mooney | ah cool | |
| 14:46:00 | jssfr | dansmith (and maybe others following https://review.opendev.org/c/openstack/nova/+/955657 ): I'm going to be AFK starting october 1st until november 25th. Is there anything else I need to do before that to get it over the line? | |
| 14:47:31 | dansmith | jssfr: sorry, just been really busy with other stuff.. I think it mostly looks good to me now but I was just going to do some tests locally that I haven't had a chance to do yet | |
| 14:47:45 | dansmith | but others may feel more comfortable just looking at it | |
| 14:48:25 | jssfr | related question: is it possible to take over another person's gerrit patch? | |
| 14:48:34 | jssfr | I might find a substitute who can continue to look into it while I'm away. | |
| 14:48:56 | dansmith | yep, easy and common | |
| 14:50:05 | jssfr | is there anything that needs to be taken into account when doing that? or can another gerrit user just take my patch and run `git review`? | |
| 14:50:36 | opendevreview | Merged openstack/nova stable/2025.1: Add admin context when filling metadata on ImagePropsWeigher https://review.opendev.org/c/openstack/nova/+/961806 | |