| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-28 | |||
| 13:52:22 | sean-k-mooney | i was thinkign yes | |
| 13:52:23 | dansmith | either way, that fake deleted instance marker thing probably needs to be cleaned in its own migration and never done again | |
| 13:52:27 | sean-k-mooney | and only keeping the last one | |
| 13:52:36 | sean-k-mooney | but i wanted your input on where to draw the line | |
| 13:52:41 | dansmith | keeping the bobcat one you mean | |
| 13:52:50 | sean-k-mooney | ya so we should keep that one | |
| 13:52:59 | sean-k-mooney | because you can still go form antelope to carical | |
| 13:53:12 | sean-k-mooney | but vitoriay is very much eol | |
| 13:53:24 | sean-k-mooney | so i think we coudl drop that on master | |
| 13:53:44 | dansmith | yeah | |
| 13:54:24 | sean-k-mooney | the reason this is on my mind | |
| 13:54:37 | sean-k-mooney | is there is a mailing list thread | |
| 13:54:42 | sean-k-mooney | on slup upgrades | |
| 13:55:11 | sean-k-mooney | and apprently this is impacting folks so i wanted to highlight this again so we can figure out how to adress this | |
| 13:56:45 | dansmith | yeah | |
| 13:59:29 | sean-k-mooney | by the way i noticed that we only run the online migrations | |
| 13:59:42 | sean-k-mooney | if we pass a var to grenade | |
| 14:00:01 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L83-L86 | |
| 14:00:22 | sean-k-mooney | i have not checked yet but i wonder if we are passing FORCE_ONLINE_MIGRATIONS in our ci jobs | |
| 14:00:23 | dansmith | online migrations are supposed to be post-upgrade idle time work, so the default is to not run them in grenade to make sure they aren't *required* to make things work | |
| 14:00:35 | sean-k-mooney | oh ok | |
| 14:00:46 | sean-k-mooney | so that intentional | |
| 14:00:50 | dansmith | yeah | |
| 14:01:02 | sean-k-mooney | i guess we would only set that for a blocker data migration? | |
| 14:01:29 | sean-k-mooney | i was just trying to think is there any way we could catch the isseu in ci | |
| 14:02:35 | dansmith | no, a blocker migration is something that blocks the schema upgrade, before we've upgraded code | |
| 14:02:52 | dansmith | so the only way we'd hit that is if we were somehow doing a multi-step upgrade in ci | |
| 14:03:04 | dansmith | sean-k-mooney: a functional reproducer for this would be best I think, | |
| 14:03:04 | sean-k-mooney | ok i was not sure if we had both blocker migration for schema and data or not | |
| 14:03:12 | dansmith | as it requires going through old migrations and new ones right? | |
| 14:03:41 | dansmith | like, you have to have run the vif one years ago and *then* run the more recent ones with that deleted entry still in there? | |
| 14:03:43 | sean-k-mooney | maybe we might need to tweak or aovid some of the db fixture | |
| 14:03:49 | dansmith | also doesn't a db archive clean this up? | |
| 14:04:27 | dansmith | oh nm, it gets re-created even if we don't have anything to migrate I guess? ugh, that was such a mistake | |
| 14:04:36 | sean-k-mooney | i think, we might replicate the fake isntance issue | |
| 14:04:53 | sean-k-mooney | if we created the scemea with the latest verion | |
| 14:05:00 | sean-k-mooney | and then did the online migration twice | |
| 14:05:05 | dansmith | yeah, twice | |
| 14:05:06 | sean-k-mooney | but im not sure | |
| 14:05:14 | dansmith | or we can just nuke that old stuff and move on | |
| 14:05:49 | sean-k-mooney | so tryign to replciat this manually might be a good first step just to make sure we can replciate this in say devstack by hand | |
| 14:06:07 | sean-k-mooney | then we can comment out the old migrations and see if it works | |
| 14:06:17 | sean-k-mooney | although i think we will need to clean up the fake isntance | |
| 14:06:44 | sean-k-mooney | im trying not to get nerd sniped into actully fixing this myslef | |
| 14:06:52 | sean-k-mooney | which is why i have not tried that yet | |
| 14:07:06 | sean-k-mooney | but if others dont have tiem i can maybe try that | |
| 14:08:45 | dansmith | $ nova-manage db online_data_migrations | |
| 14:08:45 | dansmith | Running batches of 50 until complete | |
| 14:08:45 | dansmith | ERROR nova.objects.instance [None req-f81c2435-02e7-4cb1-83be-41c1d58c0114 None None] [instance: 730e0a06-3164-4096-8713-4d1f87d3260b] Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found. | |
| 14:08:57 | dansmith | confirmed | |
| 14:09:10 | dansmith | I don't think we need to go overboard here, just nuke that ancient migration and move on right? | |
| 14:09:45 | sean-k-mooney | right the only thing i was thinking fo doing diffently was maybe deleteing or skiping the marker instnace in the migtaion that is failing | |
| 14:10:22 | sean-k-mooney | so maybe 2 patches 1 to make the comptue node migraiton not fail if the fake instance is there by skiping that | |
| 14:10:31 | sean-k-mooney | then a second one to drop the old migrations | |
| 14:11:11 | sean-k-mooney | unless you feel like we coudl backport droping the migration to carical? Antelope? | |
| 14:11:31 | sean-k-mooney | i guess carical because we run the migrtion for the dest release | |
| 14:14:18 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 14:14:37 | dansmith | this ^ plus archive should be enough to get yourself out of a jam, so it seems like that's the best thing to do in the short term to me | |
| 14:15:05 | dansmith | and yes, I think we could backport that being so small a release or two, | |
| 14:15:25 | dansmith | but keeping it small like that also means unmaintainers could snag it as well | |
| 14:15:31 | sean-k-mooney | am i guess. do you agree we shoudl follow up with removing the other old ones in a diffent commit | |
| 14:16:44 | sean-k-mooney | dansmith: by the way i would assume/hope there was some unit test for that that we shoudl drop | |
| 14:16:45 | dansmith | if I apply that ^ and run archive once, then my migrations complete successfully | |
| 14:17:01 | dansmith | sean-k-mooney: yeah probably, we'll find out | |
| 14:17:07 | sean-k-mooney | ack | |
| 14:52:08 | gibi | sean-k-mooney, dansmith: fyi with https://review.opendev.org/c/openstack/nova/+/948311 and https://review.opendev.org/c/openstack/oslo.service/+/948310 nova-scheduler can be started with native threading. It is able to start up without crashing. Reaching "DB_Driver: join new ServiceGroup member aio to the scheduler group". It does not receive the RPC messages yet for some unknown reason. But it is | |
| 14:52:14 | gibi | progress :) | |
| 14:52:27 | dansmith | cool | |
| 14:52:43 | sean-k-mooney | thats an odd failure mode but cool | |
| 14:52:57 | gibi | the oslo.service threading backend is not yet in good shape but I was able to provide a good bunch of feedback to the author | |
| 14:53:12 | sean-k-mooney | did you change oslo.messaging executor to threaded | |
| 14:54:33 | sean-k-mooney | gibi: basicaly this https://github.com/openstack/nova/blob/master/nova/rpc.py#L220-L226 | |
| 14:54:36 | gibi | sean-k-mooney: not yet explicitly. I thought that is automatic | |
| 14:54:38 | gibi | ohh | |
| 14:54:46 | sean-k-mooney | it is if we dont pass one to use | |
| 14:54:48 | gibi | hehe | |
| 14:54:50 | sean-k-mooney | but nova does | |
| 14:54:52 | gibi | then that is the next step | |
| 14:55:00 | gibi | thanks for the pointer | |
| 14:55:09 | sean-k-mooney | i woudl personally prefer to continue to pass one explcitly | |
| 14:55:19 | sean-k-mooney | but in theory either works | |
| 14:55:44 | sean-k-mooney | i think the automatic switch can fail in some cases but im unsure | |
| 14:57:00 | gibi | woot, scheduling works. VM booted with threaded scheduler | |
| 14:57:07 | sean-k-mooney | nice | |
| 14:58:04 | gibi | now I have a long list of cleanups and a long list of testing to make in the series while the oslo folks cleans up the their patch | |
| 14:58:55 | sean-k-mooney | if you can patch the new revision with that chagne we can get some ci ressult and see if 1 it work in ci and 2 if there is any obvious timing differences | |
| 14:59:26 | sean-k-mooney | i know there will be run to run variance but we can at least see if it massively changed things | |
| 14:59:42 | sean-k-mooney | it would be good to look at the memory usage form those runs too | |
| 15:00:06 | sean-k-mooney | im not expecting anything dramatic in either case but it will be a nice datapoint | |
| 15:01:06 | gibi | I need to extend devstack to be able to pass an env variable to nova-scheduler service from the zuul config to enable the threaded mode | |
| 15:01:27 | sean-k-mooney | ah well you could hardcode it for now to on | |
| 15:01:33 | gibi | ahh true | |
| 15:01:42 | sean-k-mooney | but ok ya we need to be able to pass it via the systemd service file | |
| 15:01:42 | gibi | a DNM patch with hardcoded logic | |
| 15:01:51 | gibi | ahh meeting :/ | |
| 15:30:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:37:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:39:43 | gibi | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/948311/comment/0de434cf_ac26e3db/ this should trigger CI with nova-scheduler running in threading mode. | |
| 15:40:42 | sean-k-mooney | dansmith: looks like it passed unit tests... at least on arm | |
| 15:42:31 | sean-k-mooney | it failed the functional test os at least we have some testign for that | |