Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-08-05
09:07:30 opendevreview Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729
09:07:30 opendevreview Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780
09:33:40 opendevreview Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729
09:33:40 opendevreview Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780
09:34:09 tkajinam hmm I was able to fix unit tests but am struggling to fix functional tests. I don't know why that patching of update_provider_tree is ignored ...
09:40:47 opendevreview Merged openstack/nova master: doc: Document postcopy incompatibility with vGPU live migration https://review.opendev.org/c/openstack/nova/+/999607
11:06:54 opendevreview Merged openstack/osc-placement master: tox: Fix outdated functional test environments https://review.opendev.org/c/openstack/osc-placement/+/998825
11:08:17 opendevreview Elod Illes proposed openstack/nova stable/2025.1: [stable-only] Set skip-level grenade as voting https://review.opendev.org/c/openstack/nova/+/999742
11:14:48 opendevreview Merged openstack/osc-placement master: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/998816
11:16:38 opendevreview Merged openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495
11:17:49 opendevreview Merged openstack/osc-placement master: Exclude tests directory from coverage calculation https://review.opendev.org/c/openstack/osc-placement/+/906665
12:19:53 opendevreview Elod Illes proposed openstack/osc-placement stable/2026.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999813
12:21:48 opendevreview Elod Illes proposed openstack/osc-placement stable/2025.2: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999814
12:23:21 opendevreview Elod Illes proposed openstack/osc-placement stable/2025.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999815
12:42:10 jangutter[m] For now, we’re promoting that column to BigInteger, but I was wondering if there’s a process for reclaiming old ID’s and resetting the autoincrement? Our combination of massive amounts of instance properties plus churn in instance launches triggered this, but I was wondering if there’s a periodic maintenance step we’re missing.
12:42:10 jangutter[m] Hi folks, we ran into an issue in one of our OpenStack clusters where the autoincrement ID in the nova.instance_system_metadata table ran into the max for the column size https://opendev.org/openstack/nova/src/branch/master/nova/db/main/models.py#L993
12:42:31 sean-k-mooney jangutter[m]: your not the first
12:43:02 sean-k-mooney so we didnt do that to avoid the upgrade impact but we have considered doing that in the past
12:43:12 sean-k-mooney everntully we will need to do the same upstream
12:43:28 sean-k-mooney what we did for our old internal cloud
12:43:30 jangutter[m] [sean-k-mooney](https://matrix.to/#/@sean-k-mooney:matrix.org) I was literally calculating how soon folks like CERN or telcos would run into this!
12:43:42 sean-k-mooney was manually reindex the records in teh table
12:44:02 sean-k-mooney ya so our internal cloud hit this about 2-3 years ago
12:44:08 jangutter[m] We took down our DB and re-imported (since we could take the downtime)
12:44:12 sean-k-mooney melwitt had the pleasure of helping them fix it
12:45:05 sean-k-mooney i think we did somethign simialr we shuffled the row ids back to the start of the reange fillign the gaps and then reset the increment point
12:45:06 jangutter[m] But yeah, migrating the primary index is not to be taken lightly.
12:45:48 sean-k-mooney the way we woudl do that upstream woudl be to add a new column and index then later hava a contract migration to drop the old integer column
12:46:21 sean-k-mooney so you woudl do the expand schema migration ot add the new colume then upgrade nova then we woudl drop the old primary index column
12:47:15 sean-k-mooney jangutter[m]: i dont know how shareable or robust the scripting we did internaly was but i could see adding a nova-manage command to do this if it was safe to do so
12:48:14 sean-k-mooney but ya the instance system metadata table is the first to hit it because there are many entreis per vm
12:48:14 jangutter[m] You’d still run out of indexes, but hopefully other things would bother you first (like the rise of the Zombie people in 2045)
12:49:46 jangutter[m] Thanks for confirming!
12:50:15 sean-k-mooney im just checking if i have any refence ot the downstream issue its been a while
12:50:47 jangutter[m] Yeah, strangely I didn’t find any launchpad issues
12:51:28 sean-k-mooney we discused it in the ptg a few years ago but i dont know if we created a bug
12:52:22 sean-k-mooney https://redhat.atlassian.net/browse/OSPRH-4216
12:53:25 sean-k-mooney ok so that was orgially imported form bugzill 2 jira instances ago
12:54:47 sean-k-mooney https://bugzilla.redhat.com/show_bug.cgi?id=2164247
12:56:13 sean-k-mooney https://etherpad.opendev.org/p/nova-bobcat-ptg#L431
12:57:03 sean-k-mooney jangutter[m]: so it looks like we agreed new PK will be big ints and to ask operators what there prefence woudl be for upgrades and if they woudl prefer reindexing
12:57:31 sean-k-mooney jangutter[m]: so if you have a prefence let us know :)
12:58:43 jangutter[m] We’re planning our Epoxy migrations now (and looks like most of them will be offline), so we might not be a good example.
13:00:08 sean-k-mooney i think most operator would just like sometihng that works :)
13:00:18 sean-k-mooney where they dont have to think to hard about it
13:01:46 sean-k-mooney ultimatly we need to change all our primary keys to bigints eventually
13:01:55 jangutter[m] Yeah, and the fixes are different whether you’re having an outage or not, of course.
13:04:48 jangutter[m] At least this isn’t as bad as the Y2038 problem. If you’ve been running a cluster long enough for this to affect you, chances are you know a bit about the internals of OpenStack.
13:08:11 sean-k-mooney ya the internal culsterwas running since like osp7 and upgraded along the way so 6-8 years of primarlly a dev/ci workload with many many short lived vms
13:09:22 sean-k-mooney anyeay if melwitt rememebr andy more or still has the script somewhere perhaps she can comment on how feasible it woudl be to have a nova-manage command to "heal" it when needed but that a workaround not a real fix
13:10:03 sean-k-mooney jangutter[m]: have ye fixed this for your current cloud
13:10:41 sean-k-mooney you mentioned you just promoted the column right
13:11:23 sean-k-mooney i dont know that we ever join on that primary key so for that specici table droping that column might also be an option
13:12:22 sean-k-mooney the unique constraitn is the instance id or uuid and the metadata key i think and we alwasy join on the instance id
13:14:32 sean-k-mooney ya its on the instanfce uuid we actuly do not have a proper uniwue constrating on that table...
13:14:35 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L987-L1004
13:15:23 sean-k-mooney i wonder if that would be the better fix in this case add a uniqeconstrat on instnace_uuid + key colum and drop id entirly
13:16:06 sean-k-mooney the instance uuid is effectivly the real "primary key" based on our actul usage which is why its already got its own index
13:21:35 sean-k-mooney ah this has the soft delete mixin
13:22:00 sean-k-mooney so ya we cant actully change the colume name like that or delete the id colume and use a compistie primay key
13:22:15 sean-k-mooney so the only options are reindex or promot to BigInt
13:25:10 sean-k-mooney jangutter[m]: you will need to promote the deleted colume to the same type because of softdelete fyi
13:25:25 sean-k-mooney otherwase you wont be abel to delete if the value exceeds int max
13:38:07 jangutter[m] oh yes, we saw you need to do it in pairs.
13:40:42 jangutter[m] We managed to speedrun it in 2-3 years with a combination of ~50 properties per instance and a tight create/delete loop.
13:41:02 sean-k-mooney yep that will do it
13:41:28 sean-k-mooney we did the reindex by sufflleing all the rows down to avoid needing to mess with the schema
13:43:04 sean-k-mooney as noted we dont actully use the id values
13:43:12 sean-k-mooney so the hack is basiclly
13:43:15 sean-k-mooney SET @num := 0;
13:43:17 sean-k-mooney UPDATE instance_system_metadata SET id = @num := (@num+1);
13:43:19 sean-k-mooney ALTER TABLE instance_system_metadata AUTO_INCREMENT =1;
13:44:35 sean-k-mooney i say basiclly becase if the deleted colum is non 0 you have to set delete=id as well
13:44:49 sean-k-mooney to keep the soft delete state correct
13:45:02 sean-k-mooney im sure you can express that as a single query but i think we didi it in python
13:45:51 jangutter[m] That would be a handy hack to have in nova-manage.
13:46:28 jangutter[m] but you have to be sure the id isn’t referenced elsewhere.
13:51:52 sean-k-mooney ya and there are other tabels that we can or coudl do the same for
13:52:09 sean-k-mooney basiclly a `nova-manage db compact` command
13:52:52 sean-k-mooney i think to do that properly you need to do it with a full table lock and batch it to now block operations
13:53:19 sean-k-mooney or do it offline but but if you do it 1000 rows at a time ectra you could do it online
14:13:20 MaxLamprecht[m] Hey folks, I wanted to ask regarding some attachment fixes I proposed a while ago(we merged 1 out of 3 fixes a while ago). https://review.opendev.org/c/openstack/nova/+/974832 and https://review.opendev.org/c/openstack/nova/+/972045 are still open with valid reproducers/regression tests. Is there something I can do to get more attention on this topic?
14:24:30 opendevreview minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841
14:39:26 tkajinam clif, hj could you try https://review.opendev.org/c/openstack/nova/+/999780 when you have time ?
14:39:41 tkajinam there are a few functional tests still failing but CI shows at least unit tests are passing
14:42:29 clif tkajinam: passes for me: py3: OK (113.29=setup[2.17]+cmd[99.77,7.69,3.66] seconds)
14:42:58 tkajinam clif, that's good. could you also run functional tests there, too ?
14:43:05 clif sure
14:43:06 tkajinam I expect two reshape tests may fail
14:49:07 clif tkajinam: I get two failures in functional: nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev and nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev_mixed
14:49:12 tkajinam yeah
14:49:40 tkajinam it's wired that update_available_resource is called there though it's mocked out
15:03:58 opendevreview Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup (B6) https://review.opendev.org/c/openstack/nova/+/980676
15:06:14 melwitt MaxLamprecht[m]: channel ping is good, both have green CI, i will try to look soon
15:08:47 Uggla Reminder upstream bug triage in ~20mn.
15:12:24 lajoskatona sean-k-mooney: Hi, there is a patch ( https://review.opendev.org/c/openstack/nova/+/982085) for https://bugs.launchpad.net/nova/+bug/2051685 (After repeat of incomplete migration nova applies wrong (status=error) migration context in update_available_resource periodic job)
15:12:48 lajoskatona sean-k-mooney: and a reproducer functional test under the commit. As I remember we had a short chat about it, if you have time can you please check it? Bence / rubasov started to work on it but He was moved to another project so I am kind of responsible for it now :)
15:15:19 opendevreview Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679
15:28:04 opendevreview minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841
15:33:38 Uggla Link reminder: meet.google.com/zjr-rxus-hzj (upstream triage)

Earlier   Later