| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 21:33:22 | andrewbogott | A totally reasonable warning except I cannot for the live of me figure out /why/ that's the minimum service level. All my nodes are updated, | |
| 21:33:22 | andrewbogott | A totally reasonable warning except I cannot for the live of me figure out /why/ that's the minimum service level. All my nodes are updated, | |
| 21:33:49 | andrewbogott | I've read the code, looked in all my DBs, no idea where that 40 is coming from. Can you tell me what the process is for calculating that? Is it cached someplace? | |
| 21:33:50 | andrewbogott | I've read the code, looked in all my DBs, no idea where that 40 is coming from. Can you tell me what the process is for calculating that? Is it cached someplace? | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:54 | andrewbogott | on startup | |
| 21:39:54 | andrewbogott | on startup | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:21 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:44:22 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:59 | dansmith | right, hard error now | |
| 21:46:00 | dansmith | right, hard error now | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:14 | andrewbogott | * andrewbogott frantically restarts things | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:49:03 | dansmith | oh, | |
| 21:49:03 | dansmith | oh, | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:48 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:49 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:54 | dansmith | but...good? | |
| 21:51:54 | dansmith | but...good? | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:17 | dansmith | well, | |