| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-10-22 | |||
| 11:12:31 | zigo | (and that's what I'm currently trying to fix) | |
| 11:12:32 | sean-k-mooney | we are tryign to require "valid" boot image and currently that includes a valid GPT partion table | |
| 11:12:51 | sean-k-mooney | let me see if i can find the perio bug | |
| 11:13:00 | sean-k-mooney | i have forgotten much of the context ath this point | |
| 11:13:30 | zigo | Right. We aren't trying to prevent border-line wrongly not respecting the spec images. | |
| 11:13:49 | zigo | Just something that Qemu can boot safely. | |
| 11:14:38 | zigo | It'd be ok to be more respecting the spec if we were writing a spec validation tool for qemu, but that's not our goal ... | |
| 11:14:47 | sean-k-mooney | https://bugs.launchpad.net/oslo.utils/+bug/2091114 | |
| 11:15:03 | zigo | Thanks, reading. | |
| 14:36:01 | Uggla | Does someone knows about https://review.opendev.org/c/openstack/nova-specs/+/964444 ? I mean do we have the owners of this patch around ? | |
| 14:43:23 | sean-k-mooney | i just saw the propal come in yesterday | |
| 14:43:38 | sean-k-mooney | so i dont know anything more beyond what is in the review | |
| 15:30:47 | zigo | It is wrong that these magic byte checks are doing any security improvements. Plus I'm not sure where this image is comming from, I'm not sure it's from Debian, but it's been there in our older cloud that I'm trying to migrate. | |
| 15:30:47 | zigo | "Removing checks from a security layer because Debian's images have been wrong for a long time is not a good plan, IMHO." | |
| 15:30:47 | zigo | dansmith: I do not agree with your comment in the review. From your review: | |
| 15:31:17 | dansmith | zigo: understood :) | |
| 15:31:19 | zigo | IMO, we should re-open the topic during the vPTG. | |
| 15:31:49 | zigo | I do agree some kind of checking is good, and thanks for writing these. | |
| 15:32:34 | zigo | Though blocking existing (customer) VMs that's been there for so long isn't helpful. | |
| 15:32:56 | zigo | If I'm being ignored, I'll cary this as Debian specific patches ... :P | |
| 15:33:04 | dansmith | these things are 100% security related btw.. the configuration of the MBR is read and interpreted by the hypervisor, the emulated BIOS/firmware, etc.. so it's entirely possible there's an attack vector there | |
| 15:33:41 | zigo | At what stage the hypervisor reads it?!? | |
| 15:34:00 | zigo | Shouldn't this be done by the virtual BIOS when booting? | |
| 15:34:08 | zigo | (ie: not even in Qemu) | |
| 15:34:24 | dansmith | the hypervisor is running the virtual bios :) | |
| 15:34:40 | zigo | Well in qemu, no? | |
| 15:35:02 | dansmith | yes.. you know qemu is the thing that was hackable with the previous image attack right? | |
| 15:35:24 | zigo | If there was a way to escape when Qemu boots an HDD, that would be a serious security issue in Qemu. | |
| 15:35:43 | dansmith | escape is not the only attack mode, of course | |
| 15:35:45 | zigo | Well, the thing that was hackable, was the image conversion. | |
| 15:36:06 | dansmith | no it wasnt.. you might want to re-read the bug :) | |
| 15:36:11 | zigo | That's very different from executing sea-BIOS instructions. | |
| 15:36:15 | dansmith | it was literally qemu itself | |
| 15:37:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 15:37:34 | zigo | When reading the bootloader stuff?!? | |
| 15:38:42 | dansmith | when reading the image and executing the guest.. not the bootloader specifically, but again, we're talking about being defensive for the *next* thing here | |
| 15:39:33 | dansmith | anyway, re-open the discussion of supporting spec-violating images in the vPTG if you want, and carry debian patches to remove these matches-the-spec image security checks if you think that's the best plan | |
| 15:40:19 | zigo | IMO, here, we're talking about being deffensive against the sea-BIOS being executed and doing weird thing, which is *not* our job, and which is hopefully harmless. | |
| 15:40:52 | zigo | But fine ... :P | |
| 15:41:07 | dansmith | well, I thought that inspecting the details of even things like qcow2 data structures was completely not our job until qemu told us it was | |
| 15:41:13 | zigo | BTW, my first iteration got things wrong, what I added wasn't the boot signature I've found. | |
| 15:41:28 | zigo | I've found (0x00, 0x01, 0x00) | |
| 15:42:02 | zigo | dansmith: Right. It probably deserves a discussion. :) | |
| 15:42:28 | zigo | Thanks for this begining of a chat already. | |
| 15:43:27 | dansmith | personally, I think we've already had the discussion, and we've already had a prior art bug that shows another distro realized they were wrong and posted the easy "make the image match the spec" commands, which we could put in the docs | |
| 15:44:34 | zigo | I probably can also patch the MBR of these old VMs... | |
| 15:44:43 | zigo | I'll see if that path works and let you know. | |
| 15:46:40 | dansmith | maybe glance could have a feature in image conversion to fix these things specifically with parted.. that's something I'd be on board with and probably would address your migration issue | |
| 15:50:56 | sean-k-mooney | zigo: (0x00, 0x01, 0x00) is because of a bug in parted | |
| 15:51:09 | sean-k-mooney | acording to the the flatcar image fix pr anyway | |
| 15:51:22 | zigo | Ah... thanks a lot! :) | |
| 15:51:26 | zigo | I'll look it up. | |
| 15:51:56 | zigo | FYI, the image I'm talking about is probably running Debian Jessie ... :) | |
| 15:52:13 | sean-k-mooney | https://github.com/flatcar/scripts/pull/2668#issuecomment-2660859901 | |
| 15:53:54 | sean-k-mooney | i have no idea how the flatcar image buildign work by the way | |
| 15:54:46 | sean-k-mooney | they provided a workaroudn for old images https://bugs.launchpad.net/oslo.utils/+bug/2091114/comments/12 | |
| 15:56:39 | sean-k-mooney | by the way on the glance side ther eis https://github.com/openstack/glance/blob/master/glance/common/config.py#L123C1-L138 to allow folks to weaken the security | |
| 15:56:57 | sean-k-mooney | but that is not really a good longterm approch and im not conviced that shoudl be in oslo | |
| 15:57:17 | zigo | Surprisingly, Glance isn't complaining, only Nova, not sure why. | |
| 15:57:20 | dansmith | can't (and shouldn't) be in oslo | |
| 15:58:16 | kevko | hi, i am running nova upgrade check FROM caracal container when all others containers (kolla-ansible deployment) are still antelope ... this should pass , shouldn't ? as it is slurp ...but check failed ... is it ok ? https://paste.openstack.org/show/bXbzW9LwWQwQIEu8vEUV/ | |
| 15:58:17 | dansmith | glance hasn't had upload stream safety checks for very long, so depends on what you're upgrading to | |
| 15:58:19 | sean-k-mooney | dansmith: i was honesstly kind of surpised ot see that in glance if im honest i assuem this is planned to be deprecated and remvoed eventually | |
| 15:58:49 | kevko | point is that i will upgrade control plane to caracal ...and my 300 compute servers will be upgraded one by one during the weeks | |
| 15:58:54 | dansmith | sean-k-mooney: yes, I don't like it either | |
| 15:59:34 | sean-k-mooney | kevko: ya you shoudl be able to upgrade the contolers and do the compute later | |
| 16:00:25 | sean-k-mooney | kevko: that looks like you have not rant either the db migration or somethign similar to that | |
| 16:00:38 | sean-k-mooney | it could also be using the wrogn db | |
| 16:01:58 | sean-k-mooney | the compute_id was added by https://github.com/openstack/nova/commit/a47fdef1bfba8b1e2c3279f2c51eda3d8a985ec1 | |
| 16:02:07 | sean-k-mooney | in bobcat | |
| 16:02:35 | kevko | yeah, but nova-status upgrade check command sounds like "this script will tell you if you can upgrade , which of course including db migration " | |
| 16:03:34 | sean-k-mooney | well your using a nova-status form carical but have not applied a db migration form bobcat | |
| 16:03:36 | kevko | sean-k-mooney: as I said ... antelope is slurp ...so upgrade to caracal and caracal's nova-status upgrade check should work no ? | |
| 16:04:05 | sean-k-mooney | i would need to check our doc for if the status check shoudl be done after db sync or not | |
| 16:04:13 | kevko | sean-k-mooney: yeah, but why I should update db migration from bobcat when antelope -> caracal is supported slurp upgrade ? | |
| 16:04:45 | sean-k-mooney | its not that you do them form bobcat it that you woudl run all the caracal ones | |
| 16:04:49 | kevko | that's the point ... | |
| 16:06:37 | kevko | sean-k-mooney: In my understanding of the world, a DB migration is part of the upgrade... which means the upgrade check should take into account that I’m running it before the upgrade, right? At least that makes more sense to me… | |
| 16:06:40 | sean-k-mooney | kevko: in gerade we run nova stats after you have done the db upgrade | |
| 16:06:42 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L104 | |
| 16:07:11 | sean-k-mooney | keveko you shoudl run the antelop version before the dbsync and the caracal one after | |
| 16:07:28 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix: add a default dev_type in spec https://review.opendev.org/c/openstack/nova/+/964552 | |
| 16:08:38 | sean-k-mooney | kevko: https://docs.openstack.org/nova/latest/admin/upgrades.html#rolling-upgrade-process | |
| 16:08:44 | sean-k-mooney | that what we docuemtn as well by the way | |
| 16:09:14 | sean-k-mooney | the db sysnc syste the schema which will work with the old code | |
| 16:09:35 | sean-k-mooney | the upgade check expct the new db schema | |
| 16:12:15 | kevko | sean-k-mooney: hmm, thank you.... so upgrade: antelope container -> run upgrade check from antelope container -> exec db migrations to caracal -> replace antelope container to caracal -> run upgrade check again from caracal container | |
| 16:18:27 | sean-k-mooney | basiclly. the first execution fo the status check tells you if you have any online migration or other thing you need to do before upgrading | |
| 16:18:37 | sean-k-mooney | the second tell you the same after you have upgraded | |
| 16:24:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 18:54:14 | opendevreview | Merged openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | |
| 21:47:13 | atmark | Hello, I have instances that automatically got shutdown on few computes after restarting nova | |
| 21:47:20 | atmark | `During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.` | |
| 21:47:36 | atmark | Full logs https://paste.openstack.org/show/bZzH7XnMnZut2zZzTvuk/ | |
| 21:51:10 | atmark | OpenStack Version is Yoga | |
| 21:52:40 | atmark | Which table in the db do I need to look at to identify more VMs that has power state mismatch? | |
| #openstack-nova - 2025-10-23 | |||
| 08:15:43 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix: avoid comparing dev_type if absent https://review.opendev.org/c/openstack/nova/+/964552 | |
| 14:39:12 | Uggla | But it should not change too much now. Please ping me if you can see pitfalls. | |
| 14:39:12 | Uggla | Info: https://etherpad.opendev.org/p/nova-2026.1-ptg is almost done. I still need to schedule a couple of topics, especially one in Japan friendly timeframe. | |
| 14:52:56 | nicolairuckel | @Uggla: I think there was a misunderstanding. Friday is the only day that doesn't work for me. | |