| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:54 | andrewbogott | on startup | |
| 21:39:54 | andrewbogott | on startup | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:21 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:44:22 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:59 | dansmith | right, hard error now | |
| 21:46:00 | dansmith | right, hard error now | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:14 | andrewbogott | * andrewbogott frantically restarts things | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:46:53 | dansmith | you said 53 above | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:47:47 | andrewbogott | maybe it's a subversion? That's what I see in my db | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:48:18 | dansmith | no, 53 was some time in wallaby | |
| 21:49:03 | dansmith | oh, | |
| 21:49:03 | dansmith | oh, | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:05 | dansmith | the alias is wrong | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:27 | andrewbogott | ok, I just ran a fleet-wide restart of all services (api, scheduler, conductor, api-metadata, and placement-api for good measure) | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:49:36 | andrewbogott | and I think the warning has gone away. Let me double check on another host... | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:40 | andrewbogott | yah, it's gone | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:50:43 | andrewbogott | So it was cached... somewhere! | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:06 | andrewbogott | thanks for talking me through it dansmith, this is my mistake for despairing before observing rule one | |
| 21:51:48 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:49 | dansmith | well, it's really not cached anywhere other than in memory in a process, fwiw | |
| 21:51:54 | dansmith | but...good? | |
| 21:51:54 | dansmith | but...good? | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:05 | dansmith | however, it looks to me like the alias in V is wrong, it should be 53 as you note | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:10 | andrewbogott | you're right, 'cached' isn't really the right word for it | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:12 | dansmith | I don't think it would cause the situation you were seeing | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:38 | andrewbogott | I wonder if there should be some clean-up stage that removes records from obviously-no-longer-running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:52:59 | andrewbogott | Or at least that version check could be restricted to running services | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:16 | dansmith | well, if you do service delete that will happen, but we don't want to do that automatically since we don't know if you've just got something offline for a long time | |
| 21:53:17 | dansmith | well, | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:44 | andrewbogott | yeah, I assume the issue is me decom'ing hardware before doing the service delete | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:53:56 | andrewbogott | although this was for hosts that no longer appeared in the cli | |
| 21:54:00 | andrewbogott | they were hidden in the DB | |
| 21:54:00 | dansmith | we want to not regress the minimum version by starting something old, nor do we want to migrate some data that an older service that might get turned on and be confused | |
| 21:54:00 | andrewbogott | they were hidden in the DB | |
| 21:54:00 | dansmith | we want to not regress the minimum version by starting something old, nor do we want to migrate some data that an older service that might get turned on and be confused | |
| 21:54:11 | dansmith | hidden in the db? | |
| 21:54:11 | dansmith | hidden in the db? | |
| 21:54:26 | dansmith | if it's not ignoring deleted records then that's definitely a bug | |
| 21:54:26 | dansmith | if it's not ignoring deleted records then that's definitely a bug | |
| 21:54:26 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |
| 21:54:27 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |