| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-28 | |||
| 09:18:50 | gibi | now that I'm back to relying on the upstream CI, I will spend more time caring for it and filing bugs j) | |
| 09:18:54 | gibi | :) | |
| 09:31:07 | mkuroha | gibi: Thank you for the confirmation!!:) I have one concern: I'm wondering if there is any concern in the behavior when /etc/nova/provider.yaml is deleted while /applied_provider_config/provider.yaml is present. (Current: the additional.traits of the resource provider and the /applied_provider_config/provider.yaml file remain.). | |
| 09:31:33 | mkuroha | Additionally, in this patch, are there any tests other than the nova's functional tests that should be added? (This is my almost first patch for nova, and I'm not very familiar with it, so I would like to confirm it.) | |
| 09:33:07 | gibi | mkuroha: good point about the edge case of deleting /etc/nova/provider.yaml. I think the expected behavior in this case that all the traits that are listed in the applied file is removed and then the applied file is deleted at the end. | |
| 09:35:25 | gibi | mkuroha: the functional test looks good to me. Adding one for the deleted provider.yaml case would be nice too. If you feel that the merge_provider_config or other changes could benefit from some unit test then that is also OK to add. | |
| 09:36:04 | gibi | but for me the current PoC would be enough to write and approve a spec | |
| 09:37:32 | gibi | so I suggest to write a small spec based on the PoC, so that we can approve that by the rest of the team | |
| 09:40:15 | mkuroha | gibi: I see, thank you for your all comments!! I will start working on fixing the spec side :) | |
| 09:40:25 | gibi | mkuroha: thanks for the effort! | |
| 10:13:47 | opendevreview | Elod Illes proposed openstack/nova stable/2025.1: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/946623 | |
| 10:17:56 | opendevreview | Merged openstack/nova master: Replace eventlet sleep with time.sleep https://review.opendev.org/c/openstack/nova/+/947212 | |
| 11:39:06 | sean-k-mooney | melwitt: dansmith we proably need to go fix https://bugs.launchpad.net/nova/+bug/2076614 | |
| 11:39:57 | sean-k-mooney | is there any reason not to delete all the online migration beofre the bobcat one on master?> | |
| 11:40:29 | sean-k-mooney | the previous one was for victoria when upgrading to wallaby | |
| 11:41:07 | sean-k-mooney | so no modern cloud shoudl be running anythin other then the latest one. even if your coming form yoga | |
| 11:41:18 | sean-k-mooney | or wallaby | |
| 13:52:00 | dansmith | sean-k-mooney: removing the super old ones seems fine.. are you suggesting removing the one from victoria? | |
| 13:52:15 | sean-k-mooney | that im unsure | |
| 13:52:22 | sean-k-mooney | i was thinkign yes | |
| 13:52:23 | dansmith | either way, that fake deleted instance marker thing probably needs to be cleaned in its own migration and never done again | |
| 13:52:27 | sean-k-mooney | and only keeping the last one | |
| 13:52:36 | sean-k-mooney | but i wanted your input on where to draw the line | |
| 13:52:41 | dansmith | keeping the bobcat one you mean | |
| 13:52:50 | sean-k-mooney | ya so we should keep that one | |
| 13:52:59 | sean-k-mooney | because you can still go form antelope to carical | |
| 13:53:12 | sean-k-mooney | but vitoriay is very much eol | |
| 13:53:24 | sean-k-mooney | so i think we coudl drop that on master | |
| 13:53:44 | dansmith | yeah | |
| 13:54:24 | sean-k-mooney | the reason this is on my mind | |
| 13:54:37 | sean-k-mooney | is there is a mailing list thread | |
| 13:54:42 | sean-k-mooney | on slup upgrades | |
| 13:55:11 | sean-k-mooney | and apprently this is impacting folks so i wanted to highlight this again so we can figure out how to adress this | |
| 13:56:45 | dansmith | yeah | |
| 13:59:29 | sean-k-mooney | by the way i noticed that we only run the online migrations | |
| 13:59:42 | sean-k-mooney | if we pass a var to grenade | |
| 14:00:01 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L83-L86 | |
| 14:00:22 | sean-k-mooney | i have not checked yet but i wonder if we are passing FORCE_ONLINE_MIGRATIONS in our ci jobs | |
| 14:00:23 | dansmith | online migrations are supposed to be post-upgrade idle time work, so the default is to not run them in grenade to make sure they aren't *required* to make things work | |
| 14:00:35 | sean-k-mooney | oh ok | |
| 14:00:46 | sean-k-mooney | so that intentional | |
| 14:00:50 | dansmith | yeah | |
| 14:01:02 | sean-k-mooney | i guess we would only set that for a blocker data migration? | |
| 14:01:29 | sean-k-mooney | i was just trying to think is there any way we could catch the isseu in ci | |
| 14:02:35 | dansmith | no, a blocker migration is something that blocks the schema upgrade, before we've upgraded code | |
| 14:02:52 | dansmith | so the only way we'd hit that is if we were somehow doing a multi-step upgrade in ci | |
| 14:03:04 | dansmith | sean-k-mooney: a functional reproducer for this would be best I think, | |
| 14:03:04 | sean-k-mooney | ok i was not sure if we had both blocker migration for schema and data or not | |
| 14:03:12 | dansmith | as it requires going through old migrations and new ones right? | |
| 14:03:41 | dansmith | like, you have to have run the vif one years ago and *then* run the more recent ones with that deleted entry still in there? | |
| 14:03:43 | sean-k-mooney | maybe we might need to tweak or aovid some of the db fixture | |
| 14:03:49 | dansmith | also doesn't a db archive clean this up? | |
| 14:04:27 | dansmith | oh nm, it gets re-created even if we don't have anything to migrate I guess? ugh, that was such a mistake | |
| 14:04:36 | sean-k-mooney | i think, we might replicate the fake isntance issue | |
| 14:04:53 | sean-k-mooney | if we created the scemea with the latest verion | |
| 14:05:00 | sean-k-mooney | and then did the online migration twice | |
| 14:05:05 | dansmith | yeah, twice | |
| 14:05:06 | sean-k-mooney | but im not sure | |
| 14:05:14 | dansmith | or we can just nuke that old stuff and move on | |
| 14:05:49 | sean-k-mooney | so tryign to replciat this manually might be a good first step just to make sure we can replciate this in say devstack by hand | |
| 14:06:07 | sean-k-mooney | then we can comment out the old migrations and see if it works | |
| 14:06:17 | sean-k-mooney | although i think we will need to clean up the fake isntance | |
| 14:06:44 | sean-k-mooney | im trying not to get nerd sniped into actully fixing this myslef | |
| 14:06:52 | sean-k-mooney | which is why i have not tried that yet | |
| 14:07:06 | sean-k-mooney | but if others dont have tiem i can maybe try that | |
| 14:08:45 | dansmith | $ nova-manage db online_data_migrations | |
| 14:08:45 | dansmith | Running batches of 50 until complete | |
| 14:08:45 | dansmith | ERROR nova.objects.instance [None req-f81c2435-02e7-4cb1-83be-41c1d58c0114 None None] [instance: 730e0a06-3164-4096-8713-4d1f87d3260b] Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found. | |
| 14:08:57 | dansmith | confirmed | |
| 14:09:10 | dansmith | I don't think we need to go overboard here, just nuke that ancient migration and move on right? | |
| 14:09:45 | sean-k-mooney | right the only thing i was thinking fo doing diffently was maybe deleteing or skiping the marker instnace in the migtaion that is failing | |
| 14:10:22 | sean-k-mooney | so maybe 2 patches 1 to make the comptue node migraiton not fail if the fake instance is there by skiping that | |
| 14:10:31 | sean-k-mooney | then a second one to drop the old migrations | |
| 14:11:11 | sean-k-mooney | unless you feel like we coudl backport droping the migration to carical? Antelope? | |
| 14:11:31 | sean-k-mooney | i guess carical because we run the migrtion for the dest release | |
| 14:14:18 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 14:14:37 | dansmith | this ^ plus archive should be enough to get yourself out of a jam, so it seems like that's the best thing to do in the short term to me | |
| 14:15:05 | dansmith | and yes, I think we could backport that being so small a release or two, | |
| 14:15:25 | dansmith | but keeping it small like that also means unmaintainers could snag it as well | |
| 14:15:31 | sean-k-mooney | am i guess. do you agree we shoudl follow up with removing the other old ones in a diffent commit | |
| 14:16:44 | sean-k-mooney | dansmith: by the way i would assume/hope there was some unit test for that that we shoudl drop | |
| 14:16:45 | dansmith | if I apply that ^ and run archive once, then my migrations complete successfully | |
| 14:17:01 | dansmith | sean-k-mooney: yeah probably, we'll find out | |
| 14:17:07 | sean-k-mooney | ack | |
| 14:52:08 | gibi | sean-k-mooney, dansmith: fyi with https://review.opendev.org/c/openstack/nova/+/948311 and https://review.opendev.org/c/openstack/oslo.service/+/948310 nova-scheduler can be started with native threading. It is able to start up without crashing. Reaching "DB_Driver: join new ServiceGroup member aio to the scheduler group". It does not receive the RPC messages yet for some unknown reason. But it is | |
| 14:52:14 | gibi | progress :) | |
| 14:52:27 | dansmith | cool | |
| 14:52:43 | sean-k-mooney | thats an odd failure mode but cool | |
| 14:52:57 | gibi | the oslo.service threading backend is not yet in good shape but I was able to provide a good bunch of feedback to the author | |
| 14:53:12 | sean-k-mooney | did you change oslo.messaging executor to threaded | |
| 14:54:33 | sean-k-mooney | gibi: basicaly this https://github.com/openstack/nova/blob/master/nova/rpc.py#L220-L226 | |
| 14:54:36 | gibi | sean-k-mooney: not yet explicitly. I thought that is automatic | |
| 14:54:38 | gibi | ohh | |
| 14:54:46 | sean-k-mooney | it is if we dont pass one to use | |
| 14:54:48 | gibi | hehe | |
| 14:54:50 | sean-k-mooney | but nova does | |
| 14:54:52 | gibi | then that is the next step | |
| 14:55:00 | gibi | thanks for the pointer | |
| 14:55:09 | sean-k-mooney | i woudl personally prefer to continue to pass one explcitly | |
| 14:55:19 | sean-k-mooney | but in theory either works | |