| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-28 | |||
| 14:03:41 | dansmith | like, you have to have run the vif one years ago and *then* run the more recent ones with that deleted entry still in there? | |
| 14:03:43 | sean-k-mooney | maybe we might need to tweak or aovid some of the db fixture | |
| 14:03:49 | dansmith | also doesn't a db archive clean this up? | |
| 14:04:27 | dansmith | oh nm, it gets re-created even if we don't have anything to migrate I guess? ugh, that was such a mistake | |
| 14:04:36 | sean-k-mooney | i think, we might replicate the fake isntance issue | |
| 14:04:53 | sean-k-mooney | if we created the scemea with the latest verion | |
| 14:05:00 | sean-k-mooney | and then did the online migration twice | |
| 14:05:05 | dansmith | yeah, twice | |
| 14:05:06 | sean-k-mooney | but im not sure | |
| 14:05:14 | dansmith | or we can just nuke that old stuff and move on | |
| 14:05:49 | sean-k-mooney | so tryign to replciat this manually might be a good first step just to make sure we can replciate this in say devstack by hand | |
| 14:06:07 | sean-k-mooney | then we can comment out the old migrations and see if it works | |
| 14:06:17 | sean-k-mooney | although i think we will need to clean up the fake isntance | |
| 14:06:44 | sean-k-mooney | im trying not to get nerd sniped into actully fixing this myslef | |
| 14:06:52 | sean-k-mooney | which is why i have not tried that yet | |
| 14:07:06 | sean-k-mooney | but if others dont have tiem i can maybe try that | |
| 14:08:45 | dansmith | $ nova-manage db online_data_migrations | |
| 14:08:45 | dansmith | Running batches of 50 until complete | |
| 14:08:45 | dansmith | ERROR nova.objects.instance [None req-f81c2435-02e7-4cb1-83be-41c1d58c0114 None None] [instance: 730e0a06-3164-4096-8713-4d1f87d3260b] Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found. | |
| 14:08:57 | dansmith | confirmed | |
| 14:09:10 | dansmith | I don't think we need to go overboard here, just nuke that ancient migration and move on right? | |
| 14:09:45 | sean-k-mooney | right the only thing i was thinking fo doing diffently was maybe deleteing or skiping the marker instnace in the migtaion that is failing | |
| 14:10:22 | sean-k-mooney | so maybe 2 patches 1 to make the comptue node migraiton not fail if the fake instance is there by skiping that | |
| 14:10:31 | sean-k-mooney | then a second one to drop the old migrations | |
| 14:11:11 | sean-k-mooney | unless you feel like we coudl backport droping the migration to carical? Antelope? | |
| 14:11:31 | sean-k-mooney | i guess carical because we run the migrtion for the dest release | |
| 14:14:18 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 14:14:37 | dansmith | this ^ plus archive should be enough to get yourself out of a jam, so it seems like that's the best thing to do in the short term to me | |
| 14:15:05 | dansmith | and yes, I think we could backport that being so small a release or two, | |
| 14:15:25 | dansmith | but keeping it small like that also means unmaintainers could snag it as well | |
| 14:15:31 | sean-k-mooney | am i guess. do you agree we shoudl follow up with removing the other old ones in a diffent commit | |
| 14:16:44 | sean-k-mooney | dansmith: by the way i would assume/hope there was some unit test for that that we shoudl drop | |
| 14:16:45 | dansmith | if I apply that ^ and run archive once, then my migrations complete successfully | |
| 14:17:01 | dansmith | sean-k-mooney: yeah probably, we'll find out | |
| 14:17:07 | sean-k-mooney | ack | |
| 14:52:08 | gibi | sean-k-mooney, dansmith: fyi with https://review.opendev.org/c/openstack/nova/+/948311 and https://review.opendev.org/c/openstack/oslo.service/+/948310 nova-scheduler can be started with native threading. It is able to start up without crashing. Reaching "DB_Driver: join new ServiceGroup member aio to the scheduler group". It does not receive the RPC messages yet for some unknown reason. But it is | |
| 14:52:14 | gibi | progress :) | |
| 14:52:27 | dansmith | cool | |
| 14:52:43 | sean-k-mooney | thats an odd failure mode but cool | |
| 14:52:57 | gibi | the oslo.service threading backend is not yet in good shape but I was able to provide a good bunch of feedback to the author | |
| 14:53:12 | sean-k-mooney | did you change oslo.messaging executor to threaded | |
| 14:54:33 | sean-k-mooney | gibi: basicaly this https://github.com/openstack/nova/blob/master/nova/rpc.py#L220-L226 | |
| 14:54:36 | gibi | sean-k-mooney: not yet explicitly. I thought that is automatic | |
| 14:54:38 | gibi | ohh | |
| 14:54:46 | sean-k-mooney | it is if we dont pass one to use | |
| 14:54:48 | gibi | hehe | |
| 14:54:50 | sean-k-mooney | but nova does | |
| 14:54:52 | gibi | then that is the next step | |
| 14:55:00 | gibi | thanks for the pointer | |
| 14:55:09 | sean-k-mooney | i woudl personally prefer to continue to pass one explcitly | |
| 14:55:19 | sean-k-mooney | but in theory either works | |
| 14:55:44 | sean-k-mooney | i think the automatic switch can fail in some cases but im unsure | |
| 14:57:00 | gibi | woot, scheduling works. VM booted with threaded scheduler | |
| 14:57:07 | sean-k-mooney | nice | |
| 14:58:04 | gibi | now I have a long list of cleanups and a long list of testing to make in the series while the oslo folks cleans up the their patch | |
| 14:58:55 | sean-k-mooney | if you can patch the new revision with that chagne we can get some ci ressult and see if 1 it work in ci and 2 if there is any obvious timing differences | |
| 14:59:26 | sean-k-mooney | i know there will be run to run variance but we can at least see if it massively changed things | |
| 14:59:42 | sean-k-mooney | it would be good to look at the memory usage form those runs too | |
| 15:00:06 | sean-k-mooney | im not expecting anything dramatic in either case but it will be a nice datapoint | |
| 15:01:06 | gibi | I need to extend devstack to be able to pass an env variable to nova-scheduler service from the zuul config to enable the threaded mode | |
| 15:01:27 | sean-k-mooney | ah well you could hardcode it for now to on | |
| 15:01:33 | gibi | ahh true | |
| 15:01:42 | sean-k-mooney | but ok ya we need to be able to pass it via the systemd service file | |
| 15:01:42 | gibi | a DNM patch with hardcoded logic | |
| 15:01:51 | gibi | ahh meeting :/ | |
| 15:30:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:37:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:39:43 | gibi | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/948311/comment/0de434cf_ac26e3db/ this should trigger CI with nova-scheduler running in threading mode. | |
| 15:40:42 | sean-k-mooney | dansmith: looks like it passed unit tests... at least on arm | |
| 15:42:31 | sean-k-mooney | it failed the functional test os at least we have some testign for that | |
| 15:42:49 | sean-k-mooney | i guess testing online data migration does make more sense to do in fucntional test | |
| 15:43:43 | sean-k-mooney | although we could have had a test that just mocked each of the migration funciton and asserted all of the expected one were called but we evidently dont | |
| 15:47:13 | sean-k-mooney | ... locks like the cyborg job i sbroken its not able to find the wsgi console script | |
| 15:47:42 | sean-k-mooney | partly becasue they have not converted to pyporject.toml + havign the wsgi entryp point being called via a module path | |
| 16:07:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 17:06:05 | sean-k-mooney | gibi: the ovs hybrid plug job passed so that promising https://zuul.opendev.org/t/openstack/build/4770c559c7514ad48463b7c198f31fa1 | |
| 17:07:06 | sean-k-mooney | https://zuul.opendev.org/t/openstack/buildset/1cd96c3f1a6e405eba4934bc783382db some of the other fast tempet test have also passed | |
| 17:08:47 | sean-k-mooney | gibi: is there any logmessage or anything like that that we can point to to knwo its in threadign mode | |
| 17:22:31 | sean-k-mooney | gibi: ah we shoudl see https://review.opendev.org/c/openstack/nova/+/948311/4/nova/monkey_patch.py#89 if we are monkey patched | |
| 17:22:53 | sean-k-mooney | and this warning if we are not https://review.opendev.org/c/openstack/nova/+/948311/4/nova/monkey_patch.py#119 | |
| 17:24:38 | sean-k-mooney | i do not see either printed | |
| 17:25:01 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 17:31:27 | gibi | sean-k-mooney: yeah I think somehow logging is not properly initialized. If I run then nova-scheduler in devstack from a terminal I see the warning. I will look into this. In parellel I put together a extra logging about the stats of our pools during use. | |
| 17:31:53 | sean-k-mooney | ack | |
| 17:32:08 | sean-k-mooney | when we are monkey patching i dont think we hav initalised oslo | |
| 17:32:15 | sean-k-mooney | or logign fully | |
| 17:32:28 | sean-k-mooney | * or rather logging fullyt | |
| 17:33:39 | sean-k-mooney | gibi: what prorbaly happening is this is likely going to stdout | |
| 17:33:56 | sean-k-mooney | because we have not setup the redirection to the journal yet | |
| 17:34:39 | sean-k-mooney | im not sure we have parsed or cofnig yet when we are monkey patching like this | |
| 17:35:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 17:36:07 | gibi | this will print semi periodically ^^ | |
| 17:36:16 | gibi | so we will see the type and the state of the executors | |
| 17:36:23 | sean-k-mooney | ack | |
| 17:37:12 | sean-k-mooney | ok that include the executor name in the stats | |
| 17:37:19 | sean-k-mooney | ya that shoudl work | |
| 17:37:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 17:38:14 | gibi | I stop here today | |
| 17:38:19 | gibi | it was a good day :) | |
| 17:39:39 | sean-k-mooney | gibi: ack o/ | |