| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-05 | |||
| 11:08:17 | opendevreview | Elod Illes proposed openstack/nova stable/2025.1: [stable-only] Set skip-level grenade as voting https://review.opendev.org/c/openstack/nova/+/999742 | |
| 11:14:48 | opendevreview | Merged openstack/osc-placement master: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/998816 | |
| 11:16:38 | opendevreview | Merged openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495 | |
| 11:17:49 | opendevreview | Merged openstack/osc-placement master: Exclude tests directory from coverage calculation https://review.opendev.org/c/openstack/osc-placement/+/906665 | |
| 12:19:53 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2026.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999813 | |
| 12:21:48 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.2: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999814 | |
| 12:23:21 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999815 | |
| 12:42:10 | jangutter[m] | Hi folks, we ran into an issue in one of our OpenStack clusters where the autoincrement ID in the nova.instance_system_metadata table ran into the max for the column size https://opendev.org/openstack/nova/src/branch/master/nova/db/main/models.py#L993 | |
| 12:42:10 | jangutter[m] | For now, we’re promoting that column to BigInteger, but I was wondering if there’s a process for reclaiming old ID’s and resetting the autoincrement? Our combination of massive amounts of instance properties plus churn in instance launches triggered this, but I was wondering if there’s a periodic maintenance step we’re missing. | |
| 12:42:31 | sean-k-mooney | jangutter[m]: your not the first | |
| 12:43:02 | sean-k-mooney | so we didnt do that to avoid the upgrade impact but we have considered doing that in the past | |
| 12:43:12 | sean-k-mooney | everntully we will need to do the same upstream | |
| 12:43:28 | sean-k-mooney | what we did for our old internal cloud | |
| 12:43:30 | jangutter[m] | [sean-k-mooney](https://matrix.to/#/@sean-k-mooney:matrix.org) I was literally calculating how soon folks like CERN or telcos would run into this! | |
| 12:43:42 | sean-k-mooney | was manually reindex the records in teh table | |
| 12:44:02 | sean-k-mooney | ya so our internal cloud hit this about 2-3 years ago | |
| 12:44:08 | jangutter[m] | We took down our DB and re-imported (since we could take the downtime) | |
| 12:44:12 | sean-k-mooney | melwitt had the pleasure of helping them fix it | |
| 12:45:05 | sean-k-mooney | i think we did somethign simialr we shuffled the row ids back to the start of the reange fillign the gaps and then reset the increment point | |
| 12:45:06 | jangutter[m] | But yeah, migrating the primary index is not to be taken lightly. | |
| 12:45:48 | sean-k-mooney | the way we woudl do that upstream woudl be to add a new column and index then later hava a contract migration to drop the old integer column | |
| 12:46:21 | sean-k-mooney | so you woudl do the expand schema migration ot add the new colume then upgrade nova then we woudl drop the old primary index column | |
| 12:47:15 | sean-k-mooney | jangutter[m]: i dont know how shareable or robust the scripting we did internaly was but i could see adding a nova-manage command to do this if it was safe to do so | |
| 12:48:14 | jangutter[m] | You’d still run out of indexes, but hopefully other things would bother you first (like the rise of the Zombie people in 2045) | |
| 12:48:14 | sean-k-mooney | but ya the instance system metadata table is the first to hit it because there are many entreis per vm | |
| 12:49:46 | jangutter[m] | Thanks for confirming! | |
| 12:50:15 | sean-k-mooney | im just checking if i have any refence ot the downstream issue its been a while | |
| 12:50:47 | jangutter[m] | Yeah, strangely I didn’t find any launchpad issues | |
| 12:51:28 | sean-k-mooney | we discused it in the ptg a few years ago but i dont know if we created a bug | |
| 12:52:22 | sean-k-mooney | https://redhat.atlassian.net/browse/OSPRH-4216 | |
| 12:53:25 | sean-k-mooney | ok so that was orgially imported form bugzill 2 jira instances ago | |
| 12:54:47 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=2164247 | |
| 12:56:13 | sean-k-mooney | https://etherpad.opendev.org/p/nova-bobcat-ptg#L431 | |
| 12:57:03 | sean-k-mooney | jangutter[m]: so it looks like we agreed new PK will be big ints and to ask operators what there prefence woudl be for upgrades and if they woudl prefer reindexing | |
| 12:57:31 | sean-k-mooney | jangutter[m]: so if you have a prefence let us know :) | |
| 12:58:43 | jangutter[m] | We’re planning our Epoxy migrations now (and looks like most of them will be offline), so we might not be a good example. | |
| 13:00:08 | sean-k-mooney | i think most operator would just like sometihng that works :) | |
| 13:00:18 | sean-k-mooney | where they dont have to think to hard about it | |
| 13:01:46 | sean-k-mooney | ultimatly we need to change all our primary keys to bigints eventually | |
| 13:01:55 | jangutter[m] | Yeah, and the fixes are different whether you’re having an outage or not, of course. | |
| 13:04:48 | jangutter[m] | At least this isn’t as bad as the Y2038 problem. If you’ve been running a cluster long enough for this to affect you, chances are you know a bit about the internals of OpenStack. | |
| 13:08:11 | sean-k-mooney | ya the internal culsterwas running since like osp7 and upgraded along the way so 6-8 years of primarlly a dev/ci workload with many many short lived vms | |
| 13:09:22 | sean-k-mooney | anyeay if melwitt rememebr andy more or still has the script somewhere perhaps she can comment on how feasible it woudl be to have a nova-manage command to "heal" it when needed but that a workaround not a real fix | |
| 13:10:03 | sean-k-mooney | jangutter[m]: have ye fixed this for your current cloud | |
| 13:10:41 | sean-k-mooney | you mentioned you just promoted the column right | |
| 13:11:23 | sean-k-mooney | i dont know that we ever join on that primary key so for that specici table droping that column might also be an option | |
| 13:12:22 | sean-k-mooney | the unique constraitn is the instance id or uuid and the metadata key i think and we alwasy join on the instance id | |
| 13:14:32 | sean-k-mooney | ya its on the instanfce uuid we actuly do not have a proper uniwue constrating on that table... | |
| 13:14:35 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L987-L1004 | |
| 13:15:23 | sean-k-mooney | i wonder if that would be the better fix in this case add a uniqeconstrat on instnace_uuid + key colum and drop id entirly | |
| 13:16:06 | sean-k-mooney | the instance uuid is effectivly the real "primary key" based on our actul usage which is why its already got its own index | |
| 13:21:35 | sean-k-mooney | ah this has the soft delete mixin | |
| 13:22:00 | sean-k-mooney | so ya we cant actully change the colume name like that or delete the id colume and use a compistie primay key | |
| 13:22:15 | sean-k-mooney | so the only options are reindex or promot to BigInt | |
| 13:25:10 | sean-k-mooney | jangutter[m]: you will need to promote the deleted colume to the same type because of softdelete fyi | |
| 13:25:25 | sean-k-mooney | otherwase you wont be abel to delete if the value exceeds int max | |
| 13:38:07 | jangutter[m] | oh yes, we saw you need to do it in pairs. | |
| 13:40:42 | jangutter[m] | We managed to speedrun it in 2-3 years with a combination of ~50 properties per instance and a tight create/delete loop. | |
| 13:41:02 | sean-k-mooney | yep that will do it | |
| 13:41:28 | sean-k-mooney | we did the reindex by sufflleing all the rows down to avoid needing to mess with the schema | |
| 13:43:04 | sean-k-mooney | as noted we dont actully use the id values | |
| 13:43:12 | sean-k-mooney | so the hack is basiclly | |
| 13:43:15 | sean-k-mooney | SET @num := 0; | |
| 13:43:17 | sean-k-mooney | UPDATE instance_system_metadata SET id = @num := (@num+1); | |
| 13:43:19 | sean-k-mooney | ALTER TABLE instance_system_metadata AUTO_INCREMENT =1; | |
| 13:44:35 | sean-k-mooney | i say basiclly becase if the deleted colum is non 0 you have to set delete=id as well | |
| 13:44:49 | sean-k-mooney | to keep the soft delete state correct | |
| 13:45:02 | sean-k-mooney | im sure you can express that as a single query but i think we didi it in python | |
| 13:45:51 | jangutter[m] | That would be a handy hack to have in nova-manage. | |
| 13:46:28 | jangutter[m] | but you have to be sure the id isn’t referenced elsewhere. | |
| 13:51:52 | sean-k-mooney | ya and there are other tabels that we can or coudl do the same for | |
| 13:52:09 | sean-k-mooney | basiclly a `nova-manage db compact` command | |
| 13:52:52 | sean-k-mooney | i think to do that properly you need to do it with a full table lock and batch it to now block operations | |
| 13:53:19 | sean-k-mooney | or do it offline but but if you do it 1000 rows at a time ectra you could do it online | |
| 14:13:20 | MaxLamprecht[m] | Hey folks, I wanted to ask regarding some attachment fixes I proposed a while ago(we merged 1 out of 3 fixes a while ago). https://review.opendev.org/c/openstack/nova/+/974832 and https://review.opendev.org/c/openstack/nova/+/972045 are still open with valid reproducers/regression tests. Is there something I can do to get more attention on this topic? | |
| 14:24:30 | opendevreview | minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841 | |
| 14:39:26 | tkajinam | clif, hj could you try https://review.opendev.org/c/openstack/nova/+/999780 when you have time ? | |
| 14:39:41 | tkajinam | there are a few functional tests still failing but CI shows at least unit tests are passing | |
| 14:42:29 | clif | tkajinam: passes for me: py3: OK (113.29=setup[2.17]+cmd[99.77,7.69,3.66] seconds) | |
| 14:42:58 | tkajinam | clif, that's good. could you also run functional tests there, too ? | |
| 14:43:05 | clif | sure | |
| 14:43:06 | tkajinam | I expect two reshape tests may fail | |
| 14:49:07 | clif | tkajinam: I get two failures in functional: nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev and nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev_mixed | |
| 14:49:12 | tkajinam | yeah | |
| 14:49:40 | tkajinam | it's wired that update_available_resource is called there though it's mocked out | |
| 15:03:58 | opendevreview | Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup (B6) https://review.opendev.org/c/openstack/nova/+/980676 | |
| 15:06:14 | melwitt | MaxLamprecht[m]: channel ping is good, both have green CI, i will try to look soon | |
| 15:08:47 | Uggla | Reminder upstream bug triage in ~20mn. | |
| 15:12:24 | lajoskatona | sean-k-mooney: Hi, there is a patch ( https://review.opendev.org/c/openstack/nova/+/982085) for https://bugs.launchpad.net/nova/+bug/2051685 (After repeat of incomplete migration nova applies wrong (status=error) migration context in update_available_resource periodic job) | |
| 15:12:48 | lajoskatona | sean-k-mooney: and a reproducer functional test under the commit. As I remember we had a short chat about it, if you have time can you please check it? Bence / rubasov started to work on it but He was moved to another project so I am kind of responsible for it now :) | |
| 15:15:19 | opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | |
| 15:28:04 | opendevreview | minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841 | |
| 15:33:38 | Uggla | Link reminder: meet.google.com/zjr-rxus-hzj (upstream triage) | |
| 15:40:34 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 15:40:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 16:37:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add reproducer for bug/2162912 https://review.opendev.org/c/openstack/nova/+/999877 | |
| 17:06:49 | tkajinam | sean-k-mooney dansmith https://review.opendev.org/c/openstack/nova/+/999780 fixes the unit/func tests in amd machine | |
| 17:07:00 | tkajinam | verified in ci by https://review.opendev.org/c/openstack/nova/+/999729 | |
| 17:07:06 | tkajinam | just fyi | |
| 17:07:18 | tkajinam | (but appreciate your reviews, if you have time | |