| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-28 | |||
| 08:52:57 | gibi | mkuroha: thanks for proposing that patch. It looks pretty clean and simple, so I like it. How do you feel about this approach? | |
| 08:54:55 | gibi | bauzas: yeah, I want to see the first light locally that nova-scheduler starts without monkey patching and with all the spawn calls actually taking native threads from a threadpool. If it start and not imediately fails that is already success. Then I will try throwing tempest at it and see if it works, does not hang, does not loose threads from the pool, etc. | |
| 08:55:34 | gibi | bauzas: it needs to depend the still open (and not fully working) oslo.sevice patch so this will be mostly local trials and maybe CI runs if I can extend devstack to pass env variable to nova-scheduler at start | |
| 08:56:27 | gibi | I expect that this means less actual reviewable patches proposed, and more WIP patches iterated | |
| 08:56:36 | gibi | this week. | |
| 08:57:01 | gibi | Also this week I'm off Thrusday, Friday | |
| 08:57:10 | gibi | so it is a short week anyhow | |
| 08:58:24 | gibi | so you can review the long series for spotting conceptual errors but that series is mostly WIP and expected to change based on the feedback provided by the local testing of scatter-gather in the scheduler in native threading mode. | |
| 08:58:43 | gibi | I will write up a post about how to replicate the local testing setup I have so others can try | |
| 09:10:27 | bauzas | gibi: cool, sorry wasn't nagged by a notification when you pinged me | |
| 09:10:44 | bauzas | gibi: I'll also be on PTO on Thur and Fri | |
| 09:11:36 | bauzas | gibi: so tbc, that's this gerrit branch that I could look at ? https://review.opendev.org/c/openstack/nova/+/948064/3 | |
| 09:12:20 | gibi | yes that is the one I'm trying to use to test things locally | |
| 09:12:36 | bauzas | noted | |
| 09:12:44 | gibi | there are two independent patches. A small delete https://review.opendev.org/c/openstack/nova/+/947260?usp=search | |
| 09:13:03 | gibi | and the replacement series here https://review.opendev.org/c/openstack/nova/+/947212 | |
| 09:13:29 | gibi | on the latter I expect Kamil to propose more similar patches this week | |
| 09:13:56 | bauzas | I already reviewed the hacking patch | |
| 09:14:10 | gibi | yeah, I just noticed. Thanks | |
| 09:14:41 | bauzas | for the moment, the removals and hacking patches were easy to review and without any concerns | |
| 09:14:45 | gibi | if you look at the recheck history of https://review.opendev.org/c/openstack/nova/+/947212 you see that our gate is not super happy | |
| 09:15:03 | gibi | I filed https://bugs.launchpad.net/glance/+bug/2109428 | |
| 09:15:04 | bauzas | but the futurist branch needs more brain time for me :)= | |
| 09:15:17 | gibi | bauzas: yeah the long barnch is more complicated | |
| 09:15:20 | gibi | and less hashed out | |
| 09:15:50 | bauzas | yeah saw the libvirt failure | |
| 09:15:55 | bauzas | seems a new regression IMHO | |
| 09:16:17 | gibi | also we reverted a pyroute2 bump over the weekend https://bugs.launchpad.net/glance/+bug/2109428 | |
| 09:16:27 | gibi | sorry worng link | |
| 09:16:28 | gibi | https://review.opendev.org/c/openstack/requirements/+/948283 | |
| 09:17:12 | gibi | but the author of pyroute2 already responded in https://bugs.launchpad.net/os-vif/+bug/2109396 so that is nicely moving along | |
| 09:17:42 | bauzas | thanks for the info | |
| 09:18:50 | gibi | now that I'm back to relying on the upstream CI, I will spend more time caring for it and filing bugs j) | |
| 09:18:54 | gibi | :) | |
| 09:31:07 | mkuroha | gibi: Thank you for the confirmation!!:) I have one concern: I'm wondering if there is any concern in the behavior when /etc/nova/provider.yaml is deleted while /applied_provider_config/provider.yaml is present. (Current: the additional.traits of the resource provider and the /applied_provider_config/provider.yaml file remain.). | |
| 09:31:33 | mkuroha | Additionally, in this patch, are there any tests other than the nova's functional tests that should be added? (This is my almost first patch for nova, and I'm not very familiar with it, so I would like to confirm it.) | |
| 09:33:07 | gibi | mkuroha: good point about the edge case of deleting /etc/nova/provider.yaml. I think the expected behavior in this case that all the traits that are listed in the applied file is removed and then the applied file is deleted at the end. | |
| 09:35:25 | gibi | mkuroha: the functional test looks good to me. Adding one for the deleted provider.yaml case would be nice too. If you feel that the merge_provider_config or other changes could benefit from some unit test then that is also OK to add. | |
| 09:36:04 | gibi | but for me the current PoC would be enough to write and approve a spec | |
| 09:37:32 | gibi | so I suggest to write a small spec based on the PoC, so that we can approve that by the rest of the team | |
| 09:40:15 | mkuroha | gibi: I see, thank you for your all comments!! I will start working on fixing the spec side :) | |
| 09:40:25 | gibi | mkuroha: thanks for the effort! | |
| 10:13:47 | opendevreview | Elod Illes proposed openstack/nova stable/2025.1: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/946623 | |
| 10:17:56 | opendevreview | Merged openstack/nova master: Replace eventlet sleep with time.sleep https://review.opendev.org/c/openstack/nova/+/947212 | |
| 11:39:06 | sean-k-mooney | melwitt: dansmith we proably need to go fix https://bugs.launchpad.net/nova/+bug/2076614 | |
| 11:39:57 | sean-k-mooney | is there any reason not to delete all the online migration beofre the bobcat one on master?> | |
| 11:40:29 | sean-k-mooney | the previous one was for victoria when upgrading to wallaby | |
| 11:41:07 | sean-k-mooney | so no modern cloud shoudl be running anythin other then the latest one. even if your coming form yoga | |
| 11:41:18 | sean-k-mooney | or wallaby | |
| 13:52:00 | dansmith | sean-k-mooney: removing the super old ones seems fine.. are you suggesting removing the one from victoria? | |
| 13:52:15 | sean-k-mooney | that im unsure | |
| 13:52:22 | sean-k-mooney | i was thinkign yes | |
| 13:52:23 | dansmith | either way, that fake deleted instance marker thing probably needs to be cleaned in its own migration and never done again | |
| 13:52:27 | sean-k-mooney | and only keeping the last one | |
| 13:52:36 | sean-k-mooney | but i wanted your input on where to draw the line | |
| 13:52:41 | dansmith | keeping the bobcat one you mean | |
| 13:52:50 | sean-k-mooney | ya so we should keep that one | |
| 13:52:59 | sean-k-mooney | because you can still go form antelope to carical | |
| 13:53:12 | sean-k-mooney | but vitoriay is very much eol | |
| 13:53:24 | sean-k-mooney | so i think we coudl drop that on master | |
| 13:53:44 | dansmith | yeah | |
| 13:54:24 | sean-k-mooney | the reason this is on my mind | |
| 13:54:37 | sean-k-mooney | is there is a mailing list thread | |
| 13:54:42 | sean-k-mooney | on slup upgrades | |
| 13:55:11 | sean-k-mooney | and apprently this is impacting folks so i wanted to highlight this again so we can figure out how to adress this | |
| 13:56:45 | dansmith | yeah | |
| 13:59:29 | sean-k-mooney | by the way i noticed that we only run the online migrations | |
| 13:59:42 | sean-k-mooney | if we pass a var to grenade | |
| 14:00:01 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L83-L86 | |
| 14:00:22 | sean-k-mooney | i have not checked yet but i wonder if we are passing FORCE_ONLINE_MIGRATIONS in our ci jobs | |
| 14:00:23 | dansmith | online migrations are supposed to be post-upgrade idle time work, so the default is to not run them in grenade to make sure they aren't *required* to make things work | |
| 14:00:35 | sean-k-mooney | oh ok | |
| 14:00:46 | sean-k-mooney | so that intentional | |
| 14:00:50 | dansmith | yeah | |
| 14:01:02 | sean-k-mooney | i guess we would only set that for a blocker data migration? | |
| 14:01:29 | sean-k-mooney | i was just trying to think is there any way we could catch the isseu in ci | |
| 14:02:35 | dansmith | no, a blocker migration is something that blocks the schema upgrade, before we've upgraded code | |
| 14:02:52 | dansmith | so the only way we'd hit that is if we were somehow doing a multi-step upgrade in ci | |
| 14:03:04 | dansmith | sean-k-mooney: a functional reproducer for this would be best I think, | |
| 14:03:04 | sean-k-mooney | ok i was not sure if we had both blocker migration for schema and data or not | |
| 14:03:12 | dansmith | as it requires going through old migrations and new ones right? | |
| 14:03:41 | dansmith | like, you have to have run the vif one years ago and *then* run the more recent ones with that deleted entry still in there? | |
| 14:03:43 | sean-k-mooney | maybe we might need to tweak or aovid some of the db fixture | |
| 14:03:49 | dansmith | also doesn't a db archive clean this up? | |
| 14:04:27 | dansmith | oh nm, it gets re-created even if we don't have anything to migrate I guess? ugh, that was such a mistake | |
| 14:04:36 | sean-k-mooney | i think, we might replicate the fake isntance issue | |
| 14:04:53 | sean-k-mooney | if we created the scemea with the latest verion | |
| 14:05:00 | sean-k-mooney | and then did the online migration twice | |
| 14:05:05 | dansmith | yeah, twice | |
| 14:05:06 | sean-k-mooney | but im not sure | |
| 14:05:14 | dansmith | or we can just nuke that old stuff and move on | |
| 14:05:49 | sean-k-mooney | so tryign to replciat this manually might be a good first step just to make sure we can replciate this in say devstack by hand | |
| 14:06:07 | sean-k-mooney | then we can comment out the old migrations and see if it works | |
| 14:06:17 | sean-k-mooney | although i think we will need to clean up the fake isntance | |
| 14:06:44 | sean-k-mooney | im trying not to get nerd sniped into actully fixing this myslef | |
| 14:06:52 | sean-k-mooney | which is why i have not tried that yet | |
| 14:07:06 | sean-k-mooney | but if others dont have tiem i can maybe try that | |
| 14:08:45 | dansmith | $ nova-manage db online_data_migrations | |
| 14:08:45 | dansmith | Running batches of 50 until complete | |
| 14:08:45 | dansmith | ERROR nova.objects.instance [None req-f81c2435-02e7-4cb1-83be-41c1d58c0114 None None] [instance: 730e0a06-3164-4096-8713-4d1f87d3260b] Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found. | |