| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-24 | |||
| 15:54:14 | melwitt | whitebox has the same libvirt and qemu version and I _think_ the same distro. I'm stumped so far | |
| 15:55:53 | melwitt | I have been looking for differences between the two because I don't get how it works there and not here. the cpu model looks to be the same. whitebox sets some extra cpu flags but afaict that is specifically for some cpu flags related tests in whitebox and nothing to do with them really being needed by the environment | |
| 15:56:58 | melwitt | in the googling I've done so far regarding the mutex assertion failure I have only found that it has something to do with cpu stuff. like model or arch or something like that? | |
| 15:57:52 | melwitt | ERROR:system/cpus.c is where it's raised from i.e. https://github.com/qemu/qemu/blob/master/system/cpus.c | |
| 15:58:24 | dansmith | what's the diff in the XMLs? | |
| 15:59:14 | melwitt | it looked the same at an eyeballing but let me actually diff them | |
| 16:00:12 | dansmith | yeah let's actually diff | |
| 16:00:32 | dansmith | if they're the same I can't really imagine how they wouldn't work the same unless it's distro/qemu | |
| 16:05:19 | melwitt | yeah. looking at this I'm reminded that there is virt_type=kvm in whitebox whereas upstream is virt_type=qemu. I know that the former means nested virt but from what I googled that should not affect whether the tpm would work | |
| 16:07:44 | clarkb | you can run virt_type=kvm upstream too. But half of our resources will fail (no nested virt) and you get occasional failures elsewhere due to bugs in nested virt. Still maybe worth testing if it helps narrow down the problem? | |
| 16:07:47 | dansmith | yeah but I think it changes a bunch of how cpu stuff works, so.. maybe that is the reason it's crashing there and there's some odd overlap | |
| 16:07:48 | melwitt | diff https://paste.openstack.org/show/829214 | |
| 16:08:01 | clarkb | (there are nested virt node labels in zuul to exclude the half of resources that can't do it) | |
| 16:09:03 | melwitt | thanks clarkb, I wondered about that. I have vague memory of nested virt being difficult or problematic upstream so it's good to know that it is indeed possible | |
| 16:09:19 | melwitt | I will try | |
| 16:10:17 | clarkb | ya I would say its still difficult in that we do still see occasional errors related to it that re basically impossible to debug | |
| 16:10:35 | clarkb | but for cases like this it may be helpful and we've got the ability ot at least try and do it without too much trouble | |
| 16:11:01 | melwitt | makes sense ++ | |
| 16:15:54 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 16:48:51 | jssfr | dansmith, I'm confused about one of your comments on https://review.opendev.org/c/openstack/nova/+/955657 , can you please check the unresolved one and help me find the issue? | |
| 17:03:03 | melwitt | dansmith, clarkb: it was indeed nested virt making the difference, works now with virt_type=kvm! | |
| 17:04:52 | dansmith | good.. I guess? :) | |
| 17:05:13 | opendevreview | Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657 | |
| 17:05:27 | jssfr | dansmith, ugh. sometimes I have tomatoes on the eyes (as we germans say). thanks, hopefully fixed. | |
| 17:06:23 | dansmith | jssfr: ack :) | |
| #openstack-nova - 2025-09-25 | |||
| 11:13:03 | LarsErikP | Hi.. I'm experienceing the exact same scenario as this post from 2019, running Caracal: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5QSBZDKQLA7HJF3IYMLYOF7PACWX5QQE/ am I doing something wrong here? | |
| 11:14:21 | LarsErikP | volume-backed instance - create snapshot (either in horizon or with openstack server image create) - volume snapshots are created, but the image in glance is created with disk_format=qcow2 even though everything along the way here is in raw | |
| 11:14:59 | LarsErikP | I've tried to set snapshot_image_format=raw both on compute nodes and nova-api servers, but it does not make a difference | |
| 11:16:41 | LarsErikP | and I understand from the api-ref for this, that this scenario should indeed both create the volume snap, and an "empty" glance image with bdm properties pointing to the volume snap. But this fails with a http 415 and "Image disk_format qcow2 does not match stream data raw" | |
| 11:34:57 | LarsErikP | got rid of the http 415 when setting image_format/require_image_format_match to false in glance! | |
| 11:55:46 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 13:51:21 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 13:51:21 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 16:26:12 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 19:39:38 | opendevreview | melanie witt proposed openstack/nova master: FUP for vTPM live migration https://review.opendev.org/c/openstack/nova/+/962309 | |
| 19:39:38 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 20:02:36 | melwitt | LarsErikP: if I'm understanding you correctly, what you're seeing is the "empty" glance image that gets created when you snapshot a volume backed instance. nova calls cinder to snapshot the actual volume but will also call glance to create an empty/dummy image whose purpose is only to hold image metadata https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/compute/api.py#L3459-L3466 | |
| 20:03:23 | melwitt | bc this is not a "real" image the disk_format attribute is just thrown away and not provided to glance AFAICT. maybe glance defaults to qcow2, not sure | |
| 20:05:28 | melwitt | oh but you're saying the glance image create fails due to the disk_format being wrong/not specified. this sounds like a bug that is now uncovered since glance began validating image disk formats | |
| 20:08:30 | melwitt | I don't find an existing bug open about this so as far as I know this has not been reported yet | |
| 21:06:29 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 21:09:59 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| #openstack-nova - 2025-09-26 | |||
| 06:01:58 | LarsErikP | melwitt: correct, but it failed creation because of the error I described | |
| 06:03:10 | melwitt | LarsErikP: yeah, I think you have found a bug, unless this is some known issue that is documented somewhere. I did not find anything indicating that though | |
| 06:03:31 | LarsErikP | Which I suspect happened because of the "require_image_format_match". Glance somehow interprets the "dummy"/empty image as raw, when nova creates it as qcow2 | |
| 06:07:29 | LarsErikP | melwitt: I'll report a bug to glance then =) | |
| 06:07:32 | melwitt | format validation. I'm not sure what/how we could fix this but it seems like a bug to me or at least if it's a known issue it should be documented somewhere | |
| 06:07:32 | melwitt | LarsErikP: right. glance used to not validate that the formats match (it is a relatively recent development) so in the past it went through without an error but now that glance is validating by default, it reveals the mismatch. nova does not actually set anything in the disk_format field AFAICT, it leaves it empty and maybe glance defaults to qcow2 if empty (I could be wrong). and as you found you can work around it by disabling the | |
| 06:08:15 | LarsErikP | yeah. I'll file a bug, and describe it as good as I can. I guess that'll help troubleshooting, and pin-pointing what's happening | |
| 06:08:36 | melwitt | LarsErikP: it might be considered a nova bug. I'm not sure if there is a practical way nova could check the format and send the right one to glance. if we can do that then that would be the right fix I think | |
| 06:12:07 | LarsErikP | Good point. Either that, or there is something that can be fixed in the format validation itself. I don't know enough details about how this is done. It may be worth mentioning that I tried to set the "snapshot_image_format" option in nova.conf to raw in all relevant nova-servers as well. Did not change the behaviour | |
| 06:12:53 | LarsErikP | Which led me to belive that nova is just hardcoded to upload that "dummy" image as qcow2. I did not search in nova-code to confirm that though | |
| 06:13:54 | melwitt | yeah, same. this is why I think nova isn't sending any disk_format to glance https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/compute/api.py#L3465-L3466 | |
| 06:16:17 | LarsErikP | ah. good catch. It's definitely not hardcoding qcow2 there ;) | |
| 06:16:45 | LarsErikP | so.. do you think I should file the bug to nova or glance then? | |
| 06:17:28 | melwitt | I'm thinking nova. we'll discuss it first and if we think it should be glance we can change it to glance | |
| 06:18:05 | melwitt | I think nova is supposed to be sending the right format but it's not so clear when it's the volume-backed case where it involves the weird dummy image | |
| 06:18:49 | LarsErikP | right. Something to chew on for the developers then.. I'll finnish my morning coffee and write up a bug report =) | |
| 06:19:34 | melwitt | cool, thanks | |
| 06:58:59 | LarsErikP | melwitt: https://bugs.launchpad.net/nova/+bug/2125730 =) | |
| 08:18:03 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix debug PciPassthroughFilter https://review.opendev.org/c/openstack/nova/+/907976 | |
| 09:46:50 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 12:02:25 | gibi | dansmith: this is a small fix for the devstack fake-virt support due to stable-compute-uuid prevented the fake nova-compute restarts https://review.opendev.org/c/openstack/devstack/+/962300 | |
| 13:30:02 | dansmith | gibi: would you rather just make it like ironic and not have to have the same rules? | |
| 13:35:55 | gibi | dnaI tried that but | |
| 13:36:15 | gibi | dansmith: I tried that but our own functional testing depends on the fake driver to test the compute_id logic | |
| 13:38:39 | dansmith | ah okay | |
| #openstack-nova - 2025-09-27 | |||
| 03:06:23 | opendevreview | melanie witt proposed openstack/nova master: Opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 03:06:24 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 03:06:24 | opendevreview | melanie witt proposed openstack/nova master: FUP for vTPM live migration https://review.opendev.org/c/openstack/nova/+/962309 | |
| 16:22:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Leave chown operation of tpm data directory to libvirt https://review.opendev.org/c/openstack/nova/+/962446 | |
| 16:23:27 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| 16:43:29 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| 23:36:51 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| #openstack-nova - 2025-09-28 | |||
| 14:15:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 15:27:16 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Drop redundant chown of tpm data directory https://review.opendev.org/c/openstack/nova/+/962446 | |
| 15:29:44 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 16:13:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 17:06:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 18:38:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 19:28:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| #openstack-nova - 2025-09-29 | |||
| 08:20:20 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 08:46:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 09:43:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 10:00:07 | opendevreview | Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657 | |
| 10:11:49 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch nova-conductor to use global executor https://review.opendev.org/c/openstack/nova/+/962478 | |
| 11:12:10 | opendevreview | Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657 | |
| 11:28:28 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 12:14:16 | gibi | dansmith: fyi we can catch threading slowdowns now with rally https://review.opendev.org/c/openstack/nova/+/960130/30#message-1f12ccedc335aa0e59d469a478012a41d76fd67c | |
| 12:14:33 | gibi | (this was a simulated slowdown by injecting sleep to nova) | |
| 12:40:42 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch nova-conductor to use global executor https://review.opendev.org/c/openstack/nova/+/962478 | |
| 13:09:06 | opendevreview | Merged openstack/nova stable/2025.1: Regression test for ImagePropsWeigher due to missing context https://review.opendev.org/c/openstack/nova/+/961805 | |
| 13:58:58 | dansmith | gibi: nice.. hopefully that will be stable enough to be useful.. even within a single worker performance has varied a lot in the past (due to noisy neighbors) but even if we get a smoke signal that will be good | |
| 14:04:47 | gibi | if the performance of the worker fluctuates in a ~10 mintues window then yeah we will get false postive signals | |
| 14:05:20 | gibi | the window size depends one length of our rally scenarios | |
| 14:17:39 | sean-k-mooney-pto | while we could have noise neighber effects i think its a lease worth trying this for a while and see if it becomes a problem | |
| 14:18:15 | sean-k-mooney-pto | gibi: presumable the rally job wont be voting at least not in the short term | |
| 14:18:32 | gibi | won't be voting for a while | |
| 14:18:35 | sean-k-mooney-pto | so if it a perodic or its non voting | |
| 14:18:43 | sean-k-mooney-pto | i think this ok for now | |