| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 21:45:59 | dansmith | right, hard error now | |
| 21:46:00 | dansmith | right, hard error now | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:14 | andrewbogott | * andrewbogott frantically restarts things | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:49:03 | dansmith | oh, | |
| 21:49:03 | dansmith | oh, | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:48 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:49 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:54 | dansmith | but...good? | |
| 21:51:54 | dansmith | but...good? | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:17 | dansmith | well, | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:54:00 | andrewbogott | they were hidden in the DB | |
| 21:54:00 | dansmith | we want to not regress the minimum version by starting something old, nor do we want to migrate some data that an older service that might get turned on and be confused | |
| 21:54:00 | andrewbogott | they were hidden in the DB | |
| 21:54:00 | dansmith | we want to not regress the minimum version by starting something old, nor do we want to migrate some data that an older service that might get turned on and be confused | |
| 21:54:11 | dansmith | hidden in the db? | |
| 21:54:11 | dansmith | hidden in the db? | |
| 21:54:26 | dansmith | if it's not ignoring deleted records then that's definitely a bug | |
| 21:54:26 | dansmith | if it's not ignoring deleted records then that's definitely a bug | |
| 21:54:26 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |
| 21:54:27 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |
| 21:54:57 | andrewbogott | but I guess since I didn't restart all services after every db change I don't know which bit was causing the problem :( | |
| 21:54:57 | andrewbogott | but I guess since I didn't restart all services after every db change I don't know which bit was causing the problem :( | |
| 21:55:44 | dansmith | well, there might still be an issue with considering deleted records it sounds like, which would be a bug if you want to file it | |
| 21:55:44 | dansmith | well, there might still be an issue with considering deleted records it sounds like, which would be a bug if you want to file it | |
| 21:56:13 | andrewbogott | at the very least there's an issue with <unknown> service still reporting the version of a deleted service, at least until it's restarted | |
| 21:56:13 | andrewbogott | at the very least there's an issue with <unknown> service still reporting the version of a deleted service, at least until it's restarted | |
| 21:56:35 | dansmith | well, do we know if it's deleted or just not being shown in the API? | |
| 21:56:35 | dansmith | well, do we know if it's deleted or just not being shown in the API? | |
| 21:56:45 | andrewbogott | good point, it could be either | |
| 21:56:46 | andrewbogott | good point, it could be either | |
| 21:56:47 | dansmith | I don't know how the api behaves | |
| 21:56:47 | dansmith | I don't know how the api behaves | |
| 21:56:53 | andrewbogott | I guess I need to figure out if I can reproduce it | |
| 21:56:53 | andrewbogott | I guess I need to figure out if I can reproduce it | |
| 21:56:58 | dansmith | but yes, if it's that, then definitely a bug | |
| 21:56:58 | dansmith | but yes, if it's that, then definitely a bug | |
| 21:57:16 | andrewbogott | dansmith: does that minimum_version_check happen via RPC? I stuck a bunch of debug lines in the version check on my compute node and they were never traversed. | |
| 21:57:17 | andrewbogott | dansmith: does that minimum_version_check happen via RPC? I stuck a bunch of debug lines in the version check on my compute node and they were never traversed. | |
| 21:57:44 | dansmith | computes can't talk to the database, so yes | |
| 21:57:44 | dansmith | computes can't talk to the database, so yes | |
| 21:57:45 | dansmith | but other services go straight to the db for it | |
| 21:57:45 | dansmith | but other services go straight to the db for it | |
| 21:58:45 | andrewbogott | I don't think I've ever seen @base.remotable_classmethod but I take that to mean 'this happens on a totally different system' | |
| 21:58:45 | andrewbogott | I don't think I've ever seen @base.remotable_classmethod but I take that to mean 'this happens on a totally different system' | |
| 21:58:52 | andrewbogott | or at least can | |
| 21:58:53 | andrewbogott | or at least can | |
| 21:59:04 | dansmith | it does, if coming from compute, | |
| 21:59:05 | dansmith | it does, if coming from compute, | |
| 21:59:11 | andrewbogott | ok | |
| 21:59:11 | andrewbogott | ok | |
| 21:59:18 | clarkb | gmann: ya and it works like 90% of the time | |
| 21:59:18 | andrewbogott | that solves another mystery then | |
| 21:59:18 | clarkb | gmann: ya and it works like 90% of the time | |
| 21:59:18 | andrewbogott | that solves another mystery then | |
| 21:59:19 | dansmith | and yes, you might be right.. you might need to restart conductor which does that on behalf of the compute | |
| 21:59:19 | dansmith | and yes, you might be right.. you might need to restart conductor which does that on behalf of the compute | |
| 21:59:38 | clarkb | gmann: that is why I suspect maybe it has to do with the cache refresh since in theory that only happens if you exceed timeouts or similar | |
| 21:59:38 | clarkb | gmann: that is why I suspect maybe it has to do with the cache refresh since in theory that only happens if you exceed timeouts or similar | |
| 21:59:45 | dansmith | yeah, I hadn't really thought of that, but I guess we'd get conductor's in-memory "cache" of that version | |
| 21:59:46 | dansmith | yeah, I hadn't really thought of that, but I guess we'd get conductor's in-memory "cache" of that version | |
| 22:00:12 | andrewbogott | That seems like the simplest explanation. Not easy to fix though | |
| 22:00:12 | andrewbogott | That seems like the simplest explanation. Not easy to fix though | |