| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 21:54:27 | andrewbogott | just -- I found references to hosts in the db that didn't appear in 'openstack compute service list' | |
| 21:54:57 | andrewbogott | but I guess since I didn't restart all services after every db change I don't know which bit was causing the problem :( | |
| 21:54:57 | andrewbogott | but I guess since I didn't restart all services after every db change I don't know which bit was causing the problem :( | |
| 21:55:44 | dansmith | well, there might still be an issue with considering deleted records it sounds like, which would be a bug if you want to file it | |
| 21:55:44 | dansmith | well, there might still be an issue with considering deleted records it sounds like, which would be a bug if you want to file it | |
| 21:56:13 | andrewbogott | at the very least there's an issue with <unknown> service still reporting the version of a deleted service, at least until it's restarted | |
| 21:56:13 | andrewbogott | at the very least there's an issue with <unknown> service still reporting the version of a deleted service, at least until it's restarted | |
| 21:56:35 | dansmith | well, do we know if it's deleted or just not being shown in the API? | |
| 21:56:35 | dansmith | well, do we know if it's deleted or just not being shown in the API? | |
| 21:56:45 | andrewbogott | good point, it could be either | |
| 21:56:46 | andrewbogott | good point, it could be either | |
| 21:56:47 | dansmith | I don't know how the api behaves | |
| 21:56:47 | dansmith | I don't know how the api behaves | |
| 21:56:53 | andrewbogott | I guess I need to figure out if I can reproduce it | |
| 21:56:53 | andrewbogott | I guess I need to figure out if I can reproduce it | |
| 21:56:58 | dansmith | but yes, if it's that, then definitely a bug | |
| 21:56:58 | dansmith | but yes, if it's that, then definitely a bug | |
| 21:57:16 | andrewbogott | dansmith: does that minimum_version_check happen via RPC? I stuck a bunch of debug lines in the version check on my compute node and they were never traversed. | |
| 21:57:17 | andrewbogott | dansmith: does that minimum_version_check happen via RPC? I stuck a bunch of debug lines in the version check on my compute node and they were never traversed. | |
| 21:57:44 | dansmith | computes can't talk to the database, so yes | |
| 21:57:44 | dansmith | computes can't talk to the database, so yes | |
| 21:57:45 | dansmith | but other services go straight to the db for it | |
| 21:57:45 | dansmith | but other services go straight to the db for it | |
| 21:58:45 | andrewbogott | I don't think I've ever seen @base.remotable_classmethod but I take that to mean 'this happens on a totally different system' | |
| 21:58:45 | andrewbogott | I don't think I've ever seen @base.remotable_classmethod but I take that to mean 'this happens on a totally different system' | |
| 21:58:52 | andrewbogott | or at least can | |
| 21:58:53 | andrewbogott | or at least can | |
| 21:59:04 | dansmith | it does, if coming from compute, | |
| 21:59:05 | dansmith | it does, if coming from compute, | |
| 21:59:11 | andrewbogott | ok | |
| 21:59:11 | andrewbogott | ok | |
| 21:59:18 | clarkb | gmann: ya and it works like 90% of the time | |
| 21:59:18 | andrewbogott | that solves another mystery then | |
| 21:59:18 | clarkb | gmann: ya and it works like 90% of the time | |
| 21:59:18 | andrewbogott | that solves another mystery then | |
| 21:59:19 | dansmith | and yes, you might be right.. you might need to restart conductor which does that on behalf of the compute | |
| 21:59:19 | dansmith | and yes, you might be right.. you might need to restart conductor which does that on behalf of the compute | |
| 21:59:38 | clarkb | gmann: that is why I suspect maybe it has to do with the cache refresh since in theory that only happens if you exceed timeouts or similar | |
| 21:59:38 | clarkb | gmann: that is why I suspect maybe it has to do with the cache refresh since in theory that only happens if you exceed timeouts or similar | |
| 21:59:45 | dansmith | yeah, I hadn't really thought of that, but I guess we'd get conductor's in-memory "cache" of that version | |
| 21:59:46 | dansmith | yeah, I hadn't really thought of that, but I guess we'd get conductor's in-memory "cache" of that version | |
| 22:00:12 | andrewbogott | That seems like the simplest explanation. Not easy to fix though | |
| 22:00:12 | andrewbogott | That seems like the simplest explanation. Not easy to fix though | |
| 22:00:20 | andrewbogott | other than having the 'delete' command print | |
| 22:00:20 | andrewbogott | other than having the 'delete' command print | |
| 22:00:23 | dansmith | well, | |
| 22:00:26 | andrewbogott | "ok now restart your conductor" | |
| 22:00:26 | andrewbogott | "ok now restart your conductor" | |
| 22:00:27 | dansmith | not sure that needs a fix | |
| 22:00:27 | dansmith | not sure that needs a fix | |
| 22:00:56 | andrewbogott | d'you think the conductor would've refreshed its state eventually? | |
| 22:00:56 | andrewbogott | d'you think the conductor would've refreshed its state eventually? | |
| 22:01:17 | clarkb | gmann: fungi mentioned it could possibly be an openstack(client|sdk) regression where it isn't passing that info properly | |
| 22:01:17 | dansmith | no, all the nodes will hold that value in memory until restart, that's kinda the point | |
| 22:01:17 | dansmith | no, all the nodes will hold that value in memory until restart, that's kinda the point | |
| 22:01:17 | clarkb | gmann: fungi mentioned it could possibly be an openstack(client|sdk) regression where it isn't passing that info properly | |
| 22:01:28 | dansmith | API and scheduler will do the same | |
| 22:01:28 | dansmith | API and scheduler will do the same | |
| 22:01:46 | dansmith | it's just that restarting an api worker will resolve it, whereas computes are dependent on the conductors for it | |
| 22:01:46 | dansmith | it's just that restarting an api worker will resolve it, whereas computes are dependent on the conductors for it | |
| 22:03:21 | andrewbogott | It's really the conductor's job to provide /current/ database state though isn't it? | |
| 22:03:21 | andrewbogott | It's really the conductor's job to provide /current/ database state though isn't it? | |
| 22:03:40 | andrewbogott | Are there reasons why we would want the conductor to not re-read for every query? (Other than performance) | |
| 22:03:40 | andrewbogott | Are there reasons why we would want the conductor to not re-read for every query? (Other than performance) | |
| 22:04:16 | dansmith | yeah, it's expensive, and it also controls conductor's RPC pin, which we don't want to shift at runtime unless you restart or HUP it | |
| 22:04:16 | dansmith | yeah, it's expensive, and it also controls conductor's RPC pin, which we don't want to shift at runtime unless you restart or HUP it | |
| 22:05:43 | dansmith | the conductor also translates things according to the version pin and the versions supported by the compute that is asking for a thing | |
| 22:05:43 | dansmith | the conductor also translates things according to the version pin and the versions supported by the compute that is asking for a thing | |
| 22:05:47 | dansmith | it's not just a transparent proxy | |
| 22:05:47 | dansmith | it's not just a transparent proxy | |
| 22:06:00 | andrewbogott | hmmmm | |
| 22:06:00 | andrewbogott | hmmmm | |
| 22:06:49 | andrewbogott | So really it should be pushed -- when a service is deleted it should force some kind of reload (rather than the conductor re-reading the db periodically) | |
| 22:06:49 | andrewbogott | So really it should be pushed -- when a service is deleted it should force some kind of reload (rather than the conductor re-reading the db periodically) | |
| 22:06:59 | andrewbogott | that sounds like a pain to implement for a tiny gain | |
| 22:06:59 | andrewbogott | that sounds like a pain to implement for a tiny gain | |
| 22:07:44 | dansmith | you want to fan-out to every service in the cluster every time a service is deleted just in case some of them need to re-calculate the min version? | |
| 22:07:45 | dansmith | you want to fan-out to every service in the cluster every time a service is deleted just in case some of them need to re-calculate the min version? | |
| 22:07:50 | dansmith | seems like a lot of pain for little gain | |
| 22:07:50 | dansmith | seems like a lot of pain for little gain | |
| 22:07:56 | dansmith | or maybe that's what you meant. | |
| 22:07:56 | dansmith | or maybe that's what you meant. | |
| 22:08:07 | andrewbogott | yeah, a lot of pain | |
| 22:08:07 | andrewbogott | yeah, a lot of pain | |
| 22:09:18 | andrewbogott | So a real bug but not one that's worth fixing | |
| 22:09:18 | andrewbogott | So a real bug but not one that's worth fixing | |
| 22:09:37 | openstackgerrit | Dmitrii Shcherbakov proposed openstack/nova-specs master: Introduce Transport Nodes https://review.opendev.org/c/openstack/nova-specs/+/787458 | |
| 22:10:24 | andrewbogott | dansmith: I need to run -- thanks again for getting me unstuck! | |
| 22:10:24 | andrewbogott | dansmith: I need to run -- thanks again for getting me unstuck! | |
| 22:15:41 | dansmith | yup | |
| 22:15:42 | dansmith | yup | |
| #openstack-nova - 2021-04-29 | |||
| 00:34:11 | brinzhang | gibi, gmann: the nova-grenade-multinode task always failed, is there a bug tracing? | |
| 00:34:11 | brinzhang | gibi, gmann: the nova-grenade-multinode task always failed, is there a bug tracing? | |
| 00:34:59 | brinzhang | https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode | |
| 00:34:59 | brinzhang | https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode | |
| 00:52:38 | gmann | brinzhang: i see error 'Failed to start rtslib-fb-targetctl.service: Unit rtslib-fb-targetctl.service is not loaded properly: Exec format error.' | |
| 00:52:38 | gmann | brinzhang: i see error 'Failed to start rtslib-fb-targetctl.service: Unit rtslib-fb-targetctl.service is not loaded properly: Exec format error.' | |
| 00:52:47 | gmann | but that is passing sometime | |
| 00:52:47 | gmann | but that is passing sometime | |
| 00:52:51 | gmann | not 100% failure | |