| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-10-22 | |||
| 15:41:07 | dansmith | well, I thought that inspecting the details of even things like qcow2 data structures was completely not our job until qemu told us it was | |
| 15:41:13 | zigo | BTW, my first iteration got things wrong, what I added wasn't the boot signature I've found. | |
| 15:41:28 | zigo | I've found (0x00, 0x01, 0x00) | |
| 15:42:02 | zigo | dansmith: Right. It probably deserves a discussion. :) | |
| 15:42:28 | zigo | Thanks for this begining of a chat already. | |
| 15:43:27 | dansmith | personally, I think we've already had the discussion, and we've already had a prior art bug that shows another distro realized they were wrong and posted the easy "make the image match the spec" commands, which we could put in the docs | |
| 15:44:34 | zigo | I probably can also patch the MBR of these old VMs... | |
| 15:44:43 | zigo | I'll see if that path works and let you know. | |
| 15:46:40 | dansmith | maybe glance could have a feature in image conversion to fix these things specifically with parted.. that's something I'd be on board with and probably would address your migration issue | |
| 15:50:56 | sean-k-mooney | zigo: (0x00, 0x01, 0x00) is because of a bug in parted | |
| 15:51:09 | sean-k-mooney | acording to the the flatcar image fix pr anyway | |
| 15:51:22 | zigo | Ah... thanks a lot! :) | |
| 15:51:26 | zigo | I'll look it up. | |
| 15:51:56 | zigo | FYI, the image I'm talking about is probably running Debian Jessie ... :) | |
| 15:52:13 | sean-k-mooney | https://github.com/flatcar/scripts/pull/2668#issuecomment-2660859901 | |
| 15:53:54 | sean-k-mooney | i have no idea how the flatcar image buildign work by the way | |
| 15:54:46 | sean-k-mooney | they provided a workaroudn for old images https://bugs.launchpad.net/oslo.utils/+bug/2091114/comments/12 | |
| 15:56:39 | sean-k-mooney | by the way on the glance side ther eis https://github.com/openstack/glance/blob/master/glance/common/config.py#L123C1-L138 to allow folks to weaken the security | |
| 15:56:57 | sean-k-mooney | but that is not really a good longterm approch and im not conviced that shoudl be in oslo | |
| 15:57:17 | zigo | Surprisingly, Glance isn't complaining, only Nova, not sure why. | |
| 15:57:20 | dansmith | can't (and shouldn't) be in oslo | |
| 15:58:16 | kevko | hi, i am running nova upgrade check FROM caracal container when all others containers (kolla-ansible deployment) are still antelope ... this should pass , shouldn't ? as it is slurp ...but check failed ... is it ok ? https://paste.openstack.org/show/bXbzW9LwWQwQIEu8vEUV/ | |
| 15:58:17 | dansmith | glance hasn't had upload stream safety checks for very long, so depends on what you're upgrading to | |
| 15:58:19 | sean-k-mooney | dansmith: i was honesstly kind of surpised ot see that in glance if im honest i assuem this is planned to be deprecated and remvoed eventually | |
| 15:58:49 | kevko | point is that i will upgrade control plane to caracal ...and my 300 compute servers will be upgraded one by one during the weeks | |
| 15:58:54 | dansmith | sean-k-mooney: yes, I don't like it either | |
| 15:59:34 | sean-k-mooney | kevko: ya you shoudl be able to upgrade the contolers and do the compute later | |
| 16:00:25 | sean-k-mooney | kevko: that looks like you have not rant either the db migration or somethign similar to that | |
| 16:00:38 | sean-k-mooney | it could also be using the wrogn db | |
| 16:01:58 | sean-k-mooney | the compute_id was added by https://github.com/openstack/nova/commit/a47fdef1bfba8b1e2c3279f2c51eda3d8a985ec1 | |
| 16:02:07 | sean-k-mooney | in bobcat | |
| 16:02:35 | kevko | yeah, but nova-status upgrade check command sounds like "this script will tell you if you can upgrade , which of course including db migration " | |
| 16:03:34 | sean-k-mooney | well your using a nova-status form carical but have not applied a db migration form bobcat | |
| 16:03:36 | kevko | sean-k-mooney: as I said ... antelope is slurp ...so upgrade to caracal and caracal's nova-status upgrade check should work no ? | |
| 16:04:05 | sean-k-mooney | i would need to check our doc for if the status check shoudl be done after db sync or not | |
| 16:04:13 | kevko | sean-k-mooney: yeah, but why I should update db migration from bobcat when antelope -> caracal is supported slurp upgrade ? | |
| 16:04:45 | sean-k-mooney | its not that you do them form bobcat it that you woudl run all the caracal ones | |
| 16:04:49 | kevko | that's the point ... | |
| 16:06:37 | kevko | sean-k-mooney: In my understanding of the world, a DB migration is part of the upgrade... which means the upgrade check should take into account that I’m running it before the upgrade, right? At least that makes more sense to me… | |
| 16:06:40 | sean-k-mooney | kevko: in gerade we run nova stats after you have done the db upgrade | |
| 16:06:42 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L104 | |
| 16:07:11 | sean-k-mooney | keveko you shoudl run the antelop version before the dbsync and the caracal one after | |
| 16:07:28 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix: add a default dev_type in spec https://review.opendev.org/c/openstack/nova/+/964552 | |
| 16:08:38 | sean-k-mooney | kevko: https://docs.openstack.org/nova/latest/admin/upgrades.html#rolling-upgrade-process | |
| 16:08:44 | sean-k-mooney | that what we docuemtn as well by the way | |
| 16:09:14 | sean-k-mooney | the db sysnc syste the schema which will work with the old code | |
| 16:09:35 | sean-k-mooney | the upgade check expct the new db schema | |
| 16:12:15 | kevko | sean-k-mooney: hmm, thank you.... so upgrade: antelope container -> run upgrade check from antelope container -> exec db migrations to caracal -> replace antelope container to caracal -> run upgrade check again from caracal container | |
| 16:18:27 | sean-k-mooney | basiclly. the first execution fo the status check tells you if you have any online migration or other thing you need to do before upgrading | |
| 16:18:37 | sean-k-mooney | the second tell you the same after you have upgraded | |
| 16:24:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 18:54:14 | opendevreview | Merged openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | |
| 21:47:13 | atmark | Hello, I have instances that automatically got shutdown on few computes after restarting nova | |
| 21:47:20 | atmark | `During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.` | |
| 21:47:36 | atmark | Full logs https://paste.openstack.org/show/bZzH7XnMnZut2zZzTvuk/ | |
| 21:51:10 | atmark | OpenStack Version is Yoga | |
| 21:52:40 | atmark | Which table in the db do I need to look at to identify more VMs that has power state mismatch? | |
| #openstack-nova - 2025-10-23 | |||
| 08:15:43 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix: avoid comparing dev_type if absent https://review.opendev.org/c/openstack/nova/+/964552 | |
| 14:39:12 | Uggla | But it should not change too much now. Please ping me if you can see pitfalls. | |
| 14:39:12 | Uggla | Info: https://etherpad.opendev.org/p/nova-2026.1-ptg is almost done. I still need to schedule a couple of topics, especially one in Japan friendly timeframe. | |
| 14:52:56 | nicolairuckel | @Uggla: I think there was a misunderstanding. Friday is the only day that doesn't work for me. | |
| 14:53:59 | Uggla | nicolairuckel ok sorry. Though I'll try to move it to another day | |
| 14:54:29 | nicolairuckel | No problem, thank you. | |
| 23:03:46 | opendevreview | melanie witt proposed openstack/nova master: Add vtpm_secret_(uuid|value) to LibvirtLiveMigrateData https://review.opendev.org/c/openstack/nova/+/952628 | |
| 23:03:47 | opendevreview | melanie witt proposed openstack/nova master: TPM: bump service version and require it for live migration https://review.opendev.org/c/openstack/nova/+/962051 | |
| 23:03:47 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | |
| 23:03:48 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | |
| 23:03:48 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | |
| 23:03:49 | opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | |
| 23:03:49 | opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | |
| 23:03:50 | opendevreview | melanie witt proposed openstack/nova master: TPM: opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 23:03:51 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 23:03:51 | opendevreview | melanie witt proposed openstack/nova master: TPM: add documentation and reno for live migration https://review.opendev.org/c/openstack/nova/+/962889 | |
| #openstack-nova - 2025-10-24 | |||
| 00:27:31 | opendevreview | melanie witt proposed openstack/nova master: TPM: bump service version and require it for live migration https://review.opendev.org/c/openstack/nova/+/962051 | |
| 00:27:31 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | |
| 00:27:32 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | |
| 00:27:32 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | |
| 00:27:33 | opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | |
| 00:27:33 | opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | |
| 00:27:34 | opendevreview | melanie witt proposed openstack/nova master: TPM: opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 00:28:36 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 00:28:36 | opendevreview | melanie witt proposed openstack/nova master: TPM: add documentation and reno for live migration https://review.opendev.org/c/openstack/nova/+/962889 | |
| 07:57:31 | zigo | Has this been fixed since Xena? Or is it an unknown feature of the conductor that it may opens more than 65k file descriptors? | |
| 07:57:31 | zigo | 208570 | |
| 07:57:31 | zigo | # grep -i "Too many open files" /var/log/nova/nova-conductor.log | wc -l | |
| 08:09:44 | bauzas | zigo: AFAIK, nothing in Nova writes those logs | |
| 08:10:01 | bauzas | this is not a nova issue | |
| 08:10:01 | zigo | Maybe oslo.messaging does? It looks like it's from it. | |
| 08:11:21 | bauzas | https://codesearch.openstack.org/?q=too%20many%20open%20files&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 08:11:45 | bauzas | I don't see any repo code | |
| 08:12:08 | bauzas | that's just an OS issue | |
| 08:12:33 | bauzas | zigo: ulimit -n ? | |
| 08:12:49 | zigo | ERROR oslo.messaging._drivers.impl_rabbit [req-a93bf70e-c334-401b-920e-d5086c6c61d8 - - - - -] Connection failed: [Errno 24] Too many open files (retrying in 0 seconds): OSError: [Errno 24] Too many open files | |
| 08:12:49 | zigo | Full error is: | |
| 08:13:16 | zigo | It's set to 64k in my case, I can't believe oslo.messaging needs that many FDs. | |
| 08:13:44 | bauzas | voilà | |
| 08:13:53 | zigo | Oh ... https://bugs.launchpad.net/oslo.messaging/+bug/1949964 :) | |
| 08:13:57 | bauzas | so that's just rabbit that provides the OSError | |
| 08:14:31 | bauzas | hahaha, LOL | |
| 08:14:59 | zigo | Issue is in python3-amqp | |