| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-04 | |||
| 23:08:20 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| #openstack-nova - 2026-08-05 | |||
| 03:12:00 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| 03:12:30 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| 04:15:10 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 04:16:00 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 04:26:32 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 04:55:43 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| 04:56:57 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| 06:18:29 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 06:18:30 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 07:26:05 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 07:26:06 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 07:32:44 | gibi | gmaan: re threadpool, not yet, I will try to get to it today but there are higher prio things in front of it at the moment | |
| 07:40:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: check tempest for instability https://review.opendev.org/c/openstack/nova/+/999378 | |
| 07:43:05 | bauzas | gibi: may I gently ask you for a short +W ? https://review.opendev.org/c/openstack/nova/+/999607 | |
| 07:44:56 | gibi | bauzas: done. Have you checked if a failing post copy live migration in this case leaves the instance on the source node in ACTIVE state and we leak nothing on the dest node? | |
| 07:45:36 | bauzas | gibi: at least I saw that by the tempest whitebox test, the instance is back to the source | |
| 07:45:47 | bauzas | https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/939498 | |
| 07:46:24 | gibi | OK, thanks | |
| 07:46:55 | bauzas | but I'll verify whether the instance is back ACTIVE for sure | |
| 07:51:29 | opendevreview | Kamil Sambor proposed openstack/nova master: libvirt: retry get_guest() on InstanceNotFound during post-copy VIF reattach https://review.opendev.org/c/openstack/nova/+/997380 | |
| 08:07:35 | mattcrees | Good morning! Would anyone be free to give me another review of my bugfix? https://review.opendev.org/c/openstack/nova/+/982660 | |
| 09:07:30 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 09:07:30 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 09:33:40 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 09:33:40 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 09:34:09 | tkajinam | hmm I was able to fix unit tests but am struggling to fix functional tests. I don't know why that patching of update_provider_tree is ignored ... | |
| 09:40:47 | opendevreview | Merged openstack/nova master: doc: Document postcopy incompatibility with vGPU live migration https://review.opendev.org/c/openstack/nova/+/999607 | |
| 11:06:54 | opendevreview | Merged openstack/osc-placement master: tox: Fix outdated functional test environments https://review.opendev.org/c/openstack/osc-placement/+/998825 | |
| 11:08:17 | opendevreview | Elod Illes proposed openstack/nova stable/2025.1: [stable-only] Set skip-level grenade as voting https://review.opendev.org/c/openstack/nova/+/999742 | |
| 11:14:48 | opendevreview | Merged openstack/osc-placement master: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/998816 | |
| 11:16:38 | opendevreview | Merged openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495 | |
| 11:17:49 | opendevreview | Merged openstack/osc-placement master: Exclude tests directory from coverage calculation https://review.opendev.org/c/openstack/osc-placement/+/906665 | |
| 12:19:53 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2026.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999813 | |
| 12:21:48 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.2: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999814 | |
| 12:23:21 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999815 | |
| 12:42:10 | jangutter[m] | Hi folks, we ran into an issue in one of our OpenStack clusters where the autoincrement ID in the nova.instance_system_metadata table ran into the max for the column size https://opendev.org/openstack/nova/src/branch/master/nova/db/main/models.py#L993 | |
| 12:42:10 | jangutter[m] | For now, we’re promoting that column to BigInteger, but I was wondering if there’s a process for reclaiming old ID’s and resetting the autoincrement? Our combination of massive amounts of instance properties plus churn in instance launches triggered this, but I was wondering if there’s a periodic maintenance step we’re missing. | |
| 12:42:31 | sean-k-mooney | jangutter[m]: your not the first | |
| 12:43:02 | sean-k-mooney | so we didnt do that to avoid the upgrade impact but we have considered doing that in the past | |
| 12:43:12 | sean-k-mooney | everntully we will need to do the same upstream | |
| 12:43:28 | sean-k-mooney | what we did for our old internal cloud | |
| 12:43:30 | jangutter[m] | [sean-k-mooney](https://matrix.to/#/@sean-k-mooney:matrix.org) I was literally calculating how soon folks like CERN or telcos would run into this! | |
| 12:43:42 | sean-k-mooney | was manually reindex the records in teh table | |
| 12:44:02 | sean-k-mooney | ya so our internal cloud hit this about 2-3 years ago | |
| 12:44:08 | jangutter[m] | We took down our DB and re-imported (since we could take the downtime) | |
| 12:44:12 | sean-k-mooney | melwitt had the pleasure of helping them fix it | |
| 12:45:05 | sean-k-mooney | i think we did somethign simialr we shuffled the row ids back to the start of the reange fillign the gaps and then reset the increment point | |
| 12:45:06 | jangutter[m] | But yeah, migrating the primary index is not to be taken lightly. | |
| 12:45:48 | sean-k-mooney | the way we woudl do that upstream woudl be to add a new column and index then later hava a contract migration to drop the old integer column | |
| 12:46:21 | sean-k-mooney | so you woudl do the expand schema migration ot add the new colume then upgrade nova then we woudl drop the old primary index column | |
| 12:47:15 | sean-k-mooney | jangutter[m]: i dont know how shareable or robust the scripting we did internaly was but i could see adding a nova-manage command to do this if it was safe to do so | |
| 12:48:14 | jangutter[m] | You’d still run out of indexes, but hopefully other things would bother you first (like the rise of the Zombie people in 2045) | |
| 12:48:14 | sean-k-mooney | but ya the instance system metadata table is the first to hit it because there are many entreis per vm | |
| 12:49:46 | jangutter[m] | Thanks for confirming! | |
| 12:50:15 | sean-k-mooney | im just checking if i have any refence ot the downstream issue its been a while | |
| 12:50:47 | jangutter[m] | Yeah, strangely I didn’t find any launchpad issues | |
| 12:51:28 | sean-k-mooney | we discused it in the ptg a few years ago but i dont know if we created a bug | |
| 12:52:22 | sean-k-mooney | https://redhat.atlassian.net/browse/OSPRH-4216 | |
| 12:53:25 | sean-k-mooney | ok so that was orgially imported form bugzill 2 jira instances ago | |
| 12:54:47 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=2164247 | |
| 12:56:13 | sean-k-mooney | https://etherpad.opendev.org/p/nova-bobcat-ptg#L431 | |
| 12:57:03 | sean-k-mooney | jangutter[m]: so it looks like we agreed new PK will be big ints and to ask operators what there prefence woudl be for upgrades and if they woudl prefer reindexing | |
| 12:57:31 | sean-k-mooney | jangutter[m]: so if you have a prefence let us know :) | |
| 12:58:43 | jangutter[m] | We’re planning our Epoxy migrations now (and looks like most of them will be offline), so we might not be a good example. | |
| 13:00:08 | sean-k-mooney | i think most operator would just like sometihng that works :) | |
| 13:00:18 | sean-k-mooney | where they dont have to think to hard about it | |
| 13:01:46 | sean-k-mooney | ultimatly we need to change all our primary keys to bigints eventually | |
| 13:01:55 | jangutter[m] | Yeah, and the fixes are different whether you’re having an outage or not, of course. | |
| 13:04:48 | jangutter[m] | At least this isn’t as bad as the Y2038 problem. If you’ve been running a cluster long enough for this to affect you, chances are you know a bit about the internals of OpenStack. | |
| 13:08:11 | sean-k-mooney | ya the internal culsterwas running since like osp7 and upgraded along the way so 6-8 years of primarlly a dev/ci workload with many many short lived vms | |
| 13:09:22 | sean-k-mooney | anyeay if melwitt rememebr andy more or still has the script somewhere perhaps she can comment on how feasible it woudl be to have a nova-manage command to "heal" it when needed but that a workaround not a real fix | |
| 13:10:03 | sean-k-mooney | jangutter[m]: have ye fixed this for your current cloud | |
| 13:10:41 | sean-k-mooney | you mentioned you just promoted the column right | |
| 13:11:23 | sean-k-mooney | i dont know that we ever join on that primary key so for that specici table droping that column might also be an option | |
| 13:12:22 | sean-k-mooney | the unique constraitn is the instance id or uuid and the metadata key i think and we alwasy join on the instance id | |
| 13:14:32 | sean-k-mooney | ya its on the instanfce uuid we actuly do not have a proper uniwue constrating on that table... | |
| 13:14:35 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L987-L1004 | |
| 13:15:23 | sean-k-mooney | i wonder if that would be the better fix in this case add a uniqeconstrat on instnace_uuid + key colum and drop id entirly | |
| 13:16:06 | sean-k-mooney | the instance uuid is effectivly the real "primary key" based on our actul usage which is why its already got its own index | |
| 13:21:35 | sean-k-mooney | ah this has the soft delete mixin | |
| 13:22:00 | sean-k-mooney | so ya we cant actully change the colume name like that or delete the id colume and use a compistie primay key | |
| 13:22:15 | sean-k-mooney | so the only options are reindex or promot to BigInt | |
| 13:25:10 | sean-k-mooney | jangutter[m]: you will need to promote the deleted colume to the same type because of softdelete fyi | |
| 13:25:25 | sean-k-mooney | otherwase you wont be abel to delete if the value exceeds int max | |
| 13:38:07 | jangutter[m] | oh yes, we saw you need to do it in pairs. | |
| 13:40:42 | jangutter[m] | We managed to speedrun it in 2-3 years with a combination of ~50 properties per instance and a tight create/delete loop. | |
| 13:41:02 | sean-k-mooney | yep that will do it | |
| 13:41:28 | sean-k-mooney | we did the reindex by sufflleing all the rows down to avoid needing to mess with the schema | |
| 13:43:04 | sean-k-mooney | as noted we dont actully use the id values | |
| 13:43:12 | sean-k-mooney | so the hack is basiclly | |
| 13:43:15 | sean-k-mooney | SET @num := 0; | |
| 13:43:17 | sean-k-mooney | UPDATE instance_system_metadata SET id = @num := (@num+1); | |
| 13:43:19 | sean-k-mooney | ALTER TABLE instance_system_metadata AUTO_INCREMENT =1; | |
| 13:44:35 | sean-k-mooney | i say basiclly becase if the deleted colum is non 0 you have to set delete=id as well | |
| 13:44:49 | sean-k-mooney | to keep the soft delete state correct | |
| 13:45:02 | sean-k-mooney | im sure you can express that as a single query but i think we didi it in python | |
| 13:45:51 | jangutter[m] | That would be a handy hack to have in nova-manage. | |
| 13:46:28 | jangutter[m] | but you have to be sure the id isn’t referenced elsewhere. | |
| 13:51:52 | sean-k-mooney | ya and there are other tabels that we can or coudl do the same for | |