| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:21 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:44:22 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:59 | dansmith | right, hard error now | |
| 21:46:00 | dansmith | right, hard error now | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:14 | andrewbogott | * andrewbogott frantically restarts things | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:49:03 | dansmith | oh, | |
| 21:49:03 | dansmith | oh, | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:48 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:49 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:54 | dansmith | but...good? | |
| 21:51:54 | dansmith | but...good? | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:17 | dansmith | well, | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:54:00 | andrewbogott | they were hidden in the DB | |
| 21:54:00 | dansmith | we want to not regress the minimum version by starting something old, nor do we want to migrate some data that an older service that might get turned on and be confused | |
| 21:54:00 | andrewbogott | they were hidden in the DB | |
| 21:54:00 | dansmith | we want to not regress the minimum version by starting something old, nor do we want to migrate some data that an older service that might get turned on and be confused | |
| 21:54:11 | dansmith | hidden in the db? | |
| 21:54:11 | dansmith | hidden in the db? | |
| 21:54:26 | dansmith | if it's not ignoring deleted records then that's definitely a bug | |
| 21:54:26 | dansmith | if it's not ignoring deleted records then that's definitely a bug | |
| 21:54:26 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |
| 21:54:27 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |
| 21:54:57 | andrewbogott | but I guess since I didn't restart all services after every db change I don't know which bit was causing the problem :( | |
| 21:54:57 | andrewbogott | but I guess since I didn't restart all services after every db change I don't know which bit was causing the problem :( | |
| 21:55:44 | dansmith | well, there might still be an issue with considering deleted records it sounds like, which would be a bug if you want to file it | |
| 21:55:44 | dansmith | well, there might still be an issue with considering deleted records it sounds like, which would be a bug if you want to file it | |
| 21:56:13 | andrewbogott | at the very least there's an issue with <unknown> service still reporting the version of a deleted service, at least until it's restarted | |
| 21:56:13 | andrewbogott | at the very least there's an issue with <unknown> service still reporting the version of a deleted service, at least until it's restarted | |
| 21:56:35 | dansmith | well, do we know if it's deleted or just not being shown in the API? | |
| 21:56:35 | dansmith | well, do we know if it's deleted or just not being shown in the API? | |
| 21:56:45 | andrewbogott | good point, it could be either | |
| 21:56:46 | andrewbogott | good point, it could be either | |
| 21:56:47 | dansmith | I don't know how the api behaves | |
| 21:56:47 | dansmith | I don't know how the api behaves | |
| 21:56:53 | andrewbogott | I guess I need to figure out if I can reproduce it | |
| 21:56:53 | andrewbogott | I guess I need to figure out if I can reproduce it | |
| 21:56:58 | dansmith | but yes, if it's that, then definitely a bug | |
| 21:56:58 | dansmith | but yes, if it's that, then definitely a bug | |
| 21:57:16 | andrewbogott | dansmith: does that minimum_version_check happen via RPC? I stuck a bunch of debug lines in the version check on my compute node and they were never traversed. | |
| 21:57:17 | andrewbogott | dansmith: does that minimum_version_check happen via RPC? I stuck a bunch of debug lines in the version check on my compute node and they were never traversed. | |
| 21:57:44 | dansmith | computes can't talk to the database, so yes | |
| 21:57:44 | dansmith | computes can't talk to the database, so yes | |