| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:54 | andrewbogott | on startup | |
| 21:39:54 | andrewbogott | on startup | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:21 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:44:22 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:59 | dansmith | right, hard error now | |
| 21:46:00 | dansmith | right, hard error now | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:14 | andrewbogott | * andrewbogott frantically restarts things | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:49:03 | dansmith | oh, | |
| 21:49:03 | dansmith | oh, | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:48 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:49 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:54 | dansmith | but...good? | |
| 21:51:54 | dansmith | but...good? | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:17 | dansmith | well, | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |