| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-05 | |||
| 12:43:42 | sean-k-mooney | was manually reindex the records in teh table | |
| 12:44:02 | sean-k-mooney | ya so our internal cloud hit this about 2-3 years ago | |
| 12:44:08 | jangutter[m] | We took down our DB and re-imported (since we could take the downtime) | |
| 12:44:12 | sean-k-mooney | melwitt had the pleasure of helping them fix it | |
| 12:45:05 | sean-k-mooney | i think we did somethign simialr we shuffled the row ids back to the start of the reange fillign the gaps and then reset the increment point | |
| 12:45:06 | jangutter[m] | But yeah, migrating the primary index is not to be taken lightly. | |
| 12:45:48 | sean-k-mooney | the way we woudl do that upstream woudl be to add a new column and index then later hava a contract migration to drop the old integer column | |
| 12:46:21 | sean-k-mooney | so you woudl do the expand schema migration ot add the new colume then upgrade nova then we woudl drop the old primary index column | |
| 12:47:15 | sean-k-mooney | jangutter[m]: i dont know how shareable or robust the scripting we did internaly was but i could see adding a nova-manage command to do this if it was safe to do so | |
| 12:48:14 | jangutter[m] | You’d still run out of indexes, but hopefully other things would bother you first (like the rise of the Zombie people in 2045) | |
| 12:48:14 | sean-k-mooney | but ya the instance system metadata table is the first to hit it because there are many entreis per vm | |
| 12:49:46 | jangutter[m] | Thanks for confirming! | |
| 12:50:15 | sean-k-mooney | im just checking if i have any refence ot the downstream issue its been a while | |
| 12:50:47 | jangutter[m] | Yeah, strangely I didn’t find any launchpad issues | |
| 12:51:28 | sean-k-mooney | we discused it in the ptg a few years ago but i dont know if we created a bug | |
| 12:52:22 | sean-k-mooney | https://redhat.atlassian.net/browse/OSPRH-4216 | |
| 12:53:25 | sean-k-mooney | ok so that was orgially imported form bugzill 2 jira instances ago | |
| 12:54:47 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=2164247 | |
| 12:56:13 | sean-k-mooney | https://etherpad.opendev.org/p/nova-bobcat-ptg#L431 | |
| 12:57:03 | sean-k-mooney | jangutter[m]: so it looks like we agreed new PK will be big ints and to ask operators what there prefence woudl be for upgrades and if they woudl prefer reindexing | |
| 12:57:31 | sean-k-mooney | jangutter[m]: so if you have a prefence let us know :) | |
| 12:58:43 | jangutter[m] | We’re planning our Epoxy migrations now (and looks like most of them will be offline), so we might not be a good example. | |
| 13:00:08 | sean-k-mooney | i think most operator would just like sometihng that works :) | |
| 13:00:18 | sean-k-mooney | where they dont have to think to hard about it | |
| 13:01:46 | sean-k-mooney | ultimatly we need to change all our primary keys to bigints eventually | |
| 13:01:55 | jangutter[m] | Yeah, and the fixes are different whether you’re having an outage or not, of course. | |
| 13:04:48 | jangutter[m] | At least this isn’t as bad as the Y2038 problem. If you’ve been running a cluster long enough for this to affect you, chances are you know a bit about the internals of OpenStack. | |
| 13:08:11 | sean-k-mooney | ya the internal culsterwas running since like osp7 and upgraded along the way so 6-8 years of primarlly a dev/ci workload with many many short lived vms | |
| 13:09:22 | sean-k-mooney | anyeay if melwitt rememebr andy more or still has the script somewhere perhaps she can comment on how feasible it woudl be to have a nova-manage command to "heal" it when needed but that a workaround not a real fix | |
| 13:10:03 | sean-k-mooney | jangutter[m]: have ye fixed this for your current cloud | |
| 13:10:41 | sean-k-mooney | you mentioned you just promoted the column right | |
| 13:11:23 | sean-k-mooney | i dont know that we ever join on that primary key so for that specici table droping that column might also be an option | |
| 13:12:22 | sean-k-mooney | the unique constraitn is the instance id or uuid and the metadata key i think and we alwasy join on the instance id | |
| 13:14:32 | sean-k-mooney | ya its on the instanfce uuid we actuly do not have a proper uniwue constrating on that table... | |
| 13:14:35 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L987-L1004 | |
| 13:15:23 | sean-k-mooney | i wonder if that would be the better fix in this case add a uniqeconstrat on instnace_uuid + key colum and drop id entirly | |
| 13:16:06 | sean-k-mooney | the instance uuid is effectivly the real "primary key" based on our actul usage which is why its already got its own index | |
| 13:21:35 | sean-k-mooney | ah this has the soft delete mixin | |
| 13:22:00 | sean-k-mooney | so ya we cant actully change the colume name like that or delete the id colume and use a compistie primay key | |
| 13:22:15 | sean-k-mooney | so the only options are reindex or promot to BigInt | |
| 13:25:10 | sean-k-mooney | jangutter[m]: you will need to promote the deleted colume to the same type because of softdelete fyi | |
| 13:25:25 | sean-k-mooney | otherwase you wont be abel to delete if the value exceeds int max | |
| 13:38:07 | jangutter[m] | oh yes, we saw you need to do it in pairs. | |
| 13:40:42 | jangutter[m] | We managed to speedrun it in 2-3 years with a combination of ~50 properties per instance and a tight create/delete loop. | |
| 13:41:02 | sean-k-mooney | yep that will do it | |
| 13:41:28 | sean-k-mooney | we did the reindex by sufflleing all the rows down to avoid needing to mess with the schema | |
| 13:43:04 | sean-k-mooney | as noted we dont actully use the id values | |
| 13:43:12 | sean-k-mooney | so the hack is basiclly | |
| 13:43:15 | sean-k-mooney | SET @num := 0; | |
| 13:43:17 | sean-k-mooney | UPDATE instance_system_metadata SET id = @num := (@num+1); | |
| 13:43:19 | sean-k-mooney | ALTER TABLE instance_system_metadata AUTO_INCREMENT =1; | |
| 13:44:35 | sean-k-mooney | i say basiclly becase if the deleted colum is non 0 you have to set delete=id as well | |
| 13:44:49 | sean-k-mooney | to keep the soft delete state correct | |
| 13:45:02 | sean-k-mooney | im sure you can express that as a single query but i think we didi it in python | |
| 13:45:51 | jangutter[m] | That would be a handy hack to have in nova-manage. | |
| 13:46:28 | jangutter[m] | but you have to be sure the id isn’t referenced elsewhere. | |
| 13:51:52 | sean-k-mooney | ya and there are other tabels that we can or coudl do the same for | |
| 13:52:09 | sean-k-mooney | basiclly a `nova-manage db compact` command | |
| 13:52:52 | sean-k-mooney | i think to do that properly you need to do it with a full table lock and batch it to now block operations | |
| 13:53:19 | sean-k-mooney | or do it offline but but if you do it 1000 rows at a time ectra you could do it online | |
| 14:13:20 | MaxLamprecht[m] | Hey folks, I wanted to ask regarding some attachment fixes I proposed a while ago(we merged 1 out of 3 fixes a while ago). https://review.opendev.org/c/openstack/nova/+/974832 and https://review.opendev.org/c/openstack/nova/+/972045 are still open with valid reproducers/regression tests. Is there something I can do to get more attention on this topic? | |
| 14:24:30 | opendevreview | minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841 | |
| 14:39:26 | tkajinam | clif, hj could you try https://review.opendev.org/c/openstack/nova/+/999780 when you have time ? | |
| 14:39:41 | tkajinam | there are a few functional tests still failing but CI shows at least unit tests are passing | |
| 14:42:29 | clif | tkajinam: passes for me: py3: OK (113.29=setup[2.17]+cmd[99.77,7.69,3.66] seconds) | |
| 14:42:58 | tkajinam | clif, that's good. could you also run functional tests there, too ? | |
| 14:43:05 | clif | sure | |
| 14:43:06 | tkajinam | I expect two reshape tests may fail | |
| 14:49:07 | clif | tkajinam: I get two failures in functional: nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev and nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev_mixed | |
| 14:49:12 | tkajinam | yeah | |
| 14:49:40 | tkajinam | it's wired that update_available_resource is called there though it's mocked out | |
| 15:03:58 | opendevreview | Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup (B6) https://review.opendev.org/c/openstack/nova/+/980676 | |
| 15:06:14 | melwitt | MaxLamprecht[m]: channel ping is good, both have green CI, i will try to look soon | |
| 15:08:47 | Uggla | Reminder upstream bug triage in ~20mn. | |
| 15:12:24 | lajoskatona | sean-k-mooney: Hi, there is a patch ( https://review.opendev.org/c/openstack/nova/+/982085) for https://bugs.launchpad.net/nova/+bug/2051685 (After repeat of incomplete migration nova applies wrong (status=error) migration context in update_available_resource periodic job) | |
| 15:12:48 | lajoskatona | sean-k-mooney: and a reproducer functional test under the commit. As I remember we had a short chat about it, if you have time can you please check it? Bence / rubasov started to work on it but He was moved to another project so I am kind of responsible for it now :) | |
| 15:15:19 | opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | |
| 15:28:04 | opendevreview | minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841 | |
| 15:33:38 | Uggla | Link reminder: meet.google.com/zjr-rxus-hzj (upstream triage) | |
| 15:40:34 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 15:40:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 16:37:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add reproducer for bug/2162912 https://review.opendev.org/c/openstack/nova/+/999877 | |
| 17:06:49 | tkajinam | sean-k-mooney dansmith https://review.opendev.org/c/openstack/nova/+/999780 fixes the unit/func tests in amd machine | |
| 17:07:00 | tkajinam | verified in ci by https://review.opendev.org/c/openstack/nova/+/999729 | |
| 17:07:06 | tkajinam | just fyi | |
| 17:07:18 | tkajinam | (but appreciate your reviews, if you have time | |
| 17:08:57 | sean-k-mooney | i looked breifly this morningin the only thing i was debtin is if you shoudl pass in the var when constucting the fixture as you are doign or just have the class var defiend in the fixture and use it direcly | |
| 17:09:37 | sean-k-mooney | that would just avoid proxing it here https://review.opendev.org/c/openstack/nova/+/999780/6/nova/tests/functional/libvirt/base.py | |
| 17:19:26 | sean-k-mooney | but i see why you did it this way | |
| 17:22:03 | sean-k-mooney | +2 form me | |
| 17:46:03 | sean-k-mooney | gmaan: https://review.opendev.org/c/openstack/nova/+/980679 this is the patch that i mentioned today related to usign an executor to Parallelize per-node resource updates | |
| 17:46:32 | sean-k-mooney | its workign the exact same as the _sync_power_executor https://review.opendev.org/c/openstack/nova/+/980679/5/nova/compute/manager.py#669 | |
| 19:06:20 | gmaan | sean-k-mooney: ack | |
| 19:20:54 | opendevreview | Bryan Fraschetti proposed openstack/nova master: Feat: Add Command for Forceful BDM Deletion https://review.opendev.org/c/openstack/nova/+/985717 | |
| 19:20:54 | opendevreview | Bryan Fraschetti proposed openstack/nova master: add task state for attach volume and detach volume https://review.opendev.org/c/openstack/nova/+/985718 | |
| 19:20:55 | opendevreview | Bryan Fraschetti proposed openstack/nova master: Reject to attach and detach volume while task_state is not None https://review.opendev.org/c/openstack/nova/+/758918 | |
| 19:20:55 | opendevreview | Bryan Fraschetti proposed openstack/nova master: Reject Volume Attachment if task_state not Attaching https://review.opendev.org/c/openstack/nova/+/985719 | |
| 20:07:57 | opendevreview | Merged openstack/placement stable/2026.1: Reproduce bug 2160721 https://review.opendev.org/c/openstack/placement/+/998873 | |
| 20:07:59 | opendevreview | Merged openstack/placement stable/2026.1: Check overall capacity before generating candidates https://review.opendev.org/c/openstack/placement/+/998874 | |
| #openstack-nova - 2026-08-06 | |||
| 03:56:02 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |