Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-24
14:12:14 bauzas gibi: can't join the eventlet meeting unfortunately :(
14:12:19 gibi bauzas: no worries
14:12:42 gibi the doc has some links and text about where we are if you later want to take a look
14:17:30 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190
14:17:30 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022
14:19:22 dansmith I've not made much progress on my thing in the last week due to other distractions
14:22:18 gibi same here. but I want to get back to the rally stuff this week finally
14:26:13 gibi btw, for me devstack in 24.04 in a VM generates strangely high io load recently. So high that it affects the normla use of the host. I'm not seem to able to pinpoint the source in the VM. I rolled back libvirt and qemu to pre sept 16 version in sid and that seems to help at least to get it back to unnoticed if I don't measure it. Maybe it is also related of my use of iommu and IGB for SRIOV...
14:28:03 gibi so that is anoher reason I did not progress with rally :)
14:42:23 tkajinam Can we merge https://review.opendev.org/c/openstack/osc-placement/+/935121 ? (post unmaintained transition change, kept for some time)
14:42:58 gibi tkajinam: done
14:43:11 tkajinam gibi, thx !
14:54:24 opendevreview Merged openstack/osc-placement master: reno: Update master for unmaintained/2023.1 https://review.opendev.org/c/openstack/osc-placement/+/935121
14:56:54 atmark hello, is nova host-evacuate-live concurrent or serial?
15:07:15 tkajinam atmark, API calls by cli would be done in serial but the operation is asynchronous so actual migration may be run concurrently.
15:07:48 tkajinam atmark, note that the max_concurrent_live_migrations option controls how a single nova-compute can run live migration concurrently so by default it processes multiple live migration requests one by one
15:07:52 tkajinam unless you tune it
15:18:05 opendevreview Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657
15:37:39 melwitt bauzas, dansmith: for your awareness, the spec re-proposal for vtpm live migration is available for review at https://review.opendev.org/c/openstack/nova-specs/+/961564 and the updated patch series is at https://review.opendev.org/q/topic:%22bp/vtpm-live-migration%22+is:open
15:37:55 bauzas ack, will try to review it shortly
15:38:17 melwitt ty
15:38:52 dansmith melwitt: I know, I've had the notification marked in my client to follow up, I just keep ... not
15:38:57 dansmith sorry
15:39:57 melwitt dansmith: sure np
15:44:58 melwitt it when they are back
15:44:58 melwitt for tempest testing I have https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/955969 proposed and I will be adding a test to it for the resize-opt-in. I have also been trying to get vtpm to work with regular tempest but have been unsuccessful. I have got all the setup in place but the guest crashes in qemu and I can't find why. they work fine in whitebox obvs but regular tempest no. I will probably ask sean-k-mooney about
15:47:57 dansmith by "regular tempest" do you mean in an upstream job?
15:50:09 melwitt yes
15:50:30 melwitt the unsuccess is here at the top of the series https://review.opendev.org/c/openstack/nova/+/957477
15:51:17 melwitt it gets tpm enabled in the capabilities and everything and the guest xml properly gets the tpm device in it but the guest just crashes when it tries to start, example: https://zuul.opendev.org/t/openstack/build/35be637426de4f7db36e56d76c554964/log/compute1/logs/libvirt/libvirt/qemu/instance-00000003_log.txt
15:52:08 melwitt the regular tempest patch is https://review.opendev.org/c/openstack/tempest/+/957475 and what I linked above is the nova patch that depends-on it to see if it could work
15:52:13 dansmith um, duh
15:52:14 dansmith reason=crashed
15:52:15 dansmith there
15:52:18 melwitt lol right
15:52:18 dansmith is your reason
15:52:53 dansmith it does say starting tpm emulator so it seems like it supports it.. are the whitebox and regular tests running different distros/qemu versions, etc?
15:53:26 melwitt I looked in the qemu source code out of curiosity and did not find anything helpful there. it's failing some mutex assertion but I dunno what that could mean as far as cause. and the libvirtd log isn't helping either as to showing why it happens
15:54:14 melwitt whitebox has the same libvirt and qemu version and I _think_ the same distro. I'm stumped so far
15:55:53 melwitt I have been looking for differences between the two because I don't get how it works there and not here. the cpu model looks to be the same. whitebox sets some extra cpu flags but afaict that is specifically for some cpu flags related tests in whitebox and nothing to do with them really being needed by the environment
15:56:58 melwitt in the googling I've done so far regarding the mutex assertion failure I have only found that it has something to do with cpu stuff. like model or arch or something like that?
15:57:52 melwitt ERROR:system/cpus.c is where it's raised from i.e. https://github.com/qemu/qemu/blob/master/system/cpus.c
15:58:24 dansmith what's the diff in the XMLs?
15:59:14 melwitt it looked the same at an eyeballing but let me actually diff them
16:00:12 dansmith yeah let's actually diff
16:00:32 dansmith if they're the same I can't really imagine how they wouldn't work the same unless it's distro/qemu
16:05:19 melwitt yeah. looking at this I'm reminded that there is virt_type=kvm in whitebox whereas upstream is virt_type=qemu. I know that the former means nested virt but from what I googled that should not affect whether the tpm would work
16:07:44 clarkb you can run virt_type=kvm upstream too. But half of our resources will fail (no nested virt) and you get occasional failures elsewhere due to bugs in nested virt. Still maybe worth testing if it helps narrow down the problem?
16:07:47 dansmith yeah but I think it changes a bunch of how cpu stuff works, so.. maybe that is the reason it's crashing there and there's some odd overlap
16:07:48 melwitt diff https://paste.openstack.org/show/829214
16:08:01 clarkb (there are nested virt node labels in zuul to exclude the half of resources that can't do it)
16:09:03 melwitt thanks clarkb, I wondered about that. I have vague memory of nested virt being difficult or problematic upstream so it's good to know that it is indeed possible
16:09:19 melwitt I will try
16:10:17 clarkb ya I would say its still difficult in that we do still see occasional errors related to it that re basically impossible to debug
16:10:35 clarkb but for cases like this it may be helpful and we've got the ability ot at least try and do it without too much trouble
16:11:01 melwitt makes sense ++
16:15:54 opendevreview melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477
16:48:51 jssfr dansmith, I'm confused about one of your comments on https://review.opendev.org/c/openstack/nova/+/955657 , can you please check the unresolved one and help me find the issue?
17:03:03 melwitt dansmith, clarkb: it was indeed nested virt making the difference, works now with virt_type=kvm!
17:04:52 dansmith good.. I guess? :)
17:05:13 opendevreview Jonas Schäfer proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657
17:05:27 jssfr dansmith, ugh. sometimes I have tomatoes on the eyes (as we germans say). thanks, hopefully fixed.
17:06:23 dansmith jssfr: ack :)
#openstack-nova - 2025-09-25
11:13:03 LarsErikP Hi.. I'm experienceing the exact same scenario as this post from 2019, running Caracal: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5QSBZDKQLA7HJF3IYMLYOF7PACWX5QQE/ am I doing something wrong here?
11:14:21 LarsErikP volume-backed instance - create snapshot (either in horizon or with openstack server image create) - volume snapshots are created, but the image in glance is created with disk_format=qcow2 even though everything along the way here is in raw
11:14:59 LarsErikP I've tried to set snapshot_image_format=raw both on compute nodes and nova-api servers, but it does not make a difference
11:16:41 LarsErikP and I understand from the api-ref for this, that this scenario should indeed both create the volume snap, and an "empty" glance image with bdm properties pointing to the volume snap. But this fails with a http 415 and "Image disk_format qcow2 does not match stream data raw"
11:34:57 LarsErikP got rid of the http 415 when setting image_format/require_image_format_match to false in glance!
11:55:46 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270
13:51:21 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270
13:51:21 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190
16:26:12 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270
19:39:38 opendevreview melanie witt proposed openstack/nova master: FUP for vTPM live migration https://review.opendev.org/c/openstack/nova/+/962309
19:39:38 opendevreview melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477
20:02:36 melwitt LarsErikP: if I'm understanding you correctly, what you're seeing is the "empty" glance image that gets created when you snapshot a volume backed instance. nova calls cinder to snapshot the actual volume but will also call glance to create an empty/dummy image whose purpose is only to hold image metadata https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/compute/api.py#L3459-L3466
20:03:23 melwitt bc this is not a "real" image the disk_format attribute is just thrown away and not provided to glance AFAICT. maybe glance defaults to qcow2, not sure
20:05:28 melwitt oh but you're saying the glance image create fails due to the disk_format being wrong/not specified. this sounds like a bug that is now uncovered since glance began validating image disk formats
20:08:30 melwitt I don't find an existing bug open about this so as far as I know this has not been reported yet
21:06:29 opendevreview melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477
21:09:59 opendevreview melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477
#openstack-nova - 2025-09-26
06:01:58 LarsErikP melwitt: correct, but it failed creation because of the error I described
06:03:10 melwitt LarsErikP: yeah, I think you have found a bug, unless this is some known issue that is documented somewhere. I did not find anything indicating that though
06:03:31 LarsErikP Which I suspect happened because of the "require_image_format_match". Glance somehow interprets the "dummy"/empty image as raw, when nova creates it as qcow2
06:07:29 LarsErikP melwitt: I'll report a bug to glance then =)
06:07:32 melwitt format validation. I'm not sure what/how we could fix this but it seems like a bug to me or at least if it's a known issue it should be documented somewhere
06:07:32 melwitt LarsErikP: right. glance used to not validate that the formats match (it is a relatively recent development) so in the past it went through without an error but now that glance is validating by default, it reveals the mismatch. nova does not actually set anything in the disk_format field AFAICT, it leaves it empty and maybe glance defaults to qcow2 if empty (I could be wrong). and as you found you can work around it by disabling the
06:08:15 LarsErikP yeah. I'll file a bug, and describe it as good as I can. I guess that'll help troubleshooting, and pin-pointing what's happening
06:08:36 melwitt LarsErikP: it might be considered a nova bug. I'm not sure if there is a practical way nova could check the format and send the right one to glance. if we can do that then that would be the right fix I think
06:12:07 LarsErikP Good point. Either that, or there is something that can be fixed in the format validation itself. I don't know enough details about how this is done. It may be worth mentioning that I tried to set the "snapshot_image_format" option in nova.conf to raw in all relevant nova-servers as well. Did not change the behaviour
06:12:53 LarsErikP Which led me to belive that nova is just hardcoded to upload that "dummy" image as qcow2. I did not search in nova-code to confirm that though
06:13:54 melwitt yeah, same. this is why I think nova isn't sending any disk_format to glance https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/compute/api.py#L3465-L3466
06:16:17 LarsErikP ah. good catch. It's definitely not hardcoding qcow2 there ;)
06:16:45 LarsErikP so.. do you think I should file the bug to nova or glance then?
06:17:28 melwitt I'm thinking nova. we'll discuss it first and if we think it should be glance we can change it to glance
06:18:05 melwitt I think nova is supposed to be sending the right format but it's not so clear when it's the volume-backed case where it involves the weird dummy image
06:18:49 LarsErikP right. Something to chew on for the developers then.. I'll finnish my morning coffee and write up a bug report =)
06:19:34 melwitt cool, thanks
06:58:59 LarsErikP melwitt: https://bugs.launchpad.net/nova/+bug/2125730 =)
08:18:03 opendevreview Arnaud Morin proposed openstack/nova master: Fix debug PciPassthroughFilter https://review.opendev.org/c/openstack/nova/+/907976
09:46:50 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270

Earlier   Later