| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-01 | |||
| 17:15:19 | bauzas | actually this sounds doable, I'll come up with something | |
| 17:27:08 | sean-k-mooney | gibi: dansmith: related to the health check converstation. i alrady have a time stamp for when each healthcheck result was recorded so im just going to add one more timestamp for the overall responce | |
| 17:27:32 | dansmith | sean-k-mooney: cool | |
| 17:27:35 | dansmith | bauzas: ack | |
| 17:28:05 | sean-k-mooney | you could figure that out yourslef on the client side but that seams pretty trivial to do and it allow you one extra data point as a clinet, how long did the responce take to get to me | |
| 17:29:07 | sean-k-mooney | we expect this to mainly run on localhost so that should be instant but if its not then it might be useful | |
| 17:30:31 | sean-k-mooney | melwitt: did you push an update to your series by the way. i wont get to it today but ill try and take a look again tomorrow | |
| 17:31:26 | melwitt | sean-k-mooney: no not yet :( I will later today | |
| 17:31:34 | sean-k-mooney | no rush | |
| 17:45:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: check we don't leak in the func test https://review.opendev.org/c/openstack/nova/+/907465 | |
| 17:45:56 | bauzas | dansmith: made the functest, proofing we don't leak ^ | |
| 17:46:27 | bauzas | I'll rebase my series by squashing that one with the change in question | |
| 17:47:28 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 17:47:28 | opendevreview | Sylvain Bauza proposed openstack/nova master: Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 17:47:33 | bauzas | there we go | |
| 18:06:31 | clarkb | sean-k-mooney: no there isn't a good way to run zuul jobs locally | |
| 18:06:57 | clarkb | sean-k-mooney: this is why tools like tox and nox and make etc are valuable and honestly good ideas regardless of your CI system | |
| 18:23:03 | sean-k-mooney | clarkb: well we would not remove them evn if we coudl run zuul jobs locally | |
| 18:24:03 | clarkb | sure, btu if those tools work properly you reduce a lot of problems | |
| 18:24:14 | clarkb | and the problems with not disabling selinux don't go away running a zuul job locally | |
| 18:24:42 | sean-k-mooney | hehe well that is just because bauzas is using an os that is not supported by the too he was trying to od | |
| 18:25:11 | sean-k-mooney | the devstack is_fedora check i think works for centos and rhel but we have other check that check for fedora and centos by name | |
| 18:25:48 | sean-k-mooney | there are docs that tell you to use specific function in devstack to do this but perople done alwasy follow the docs even if they wrote them :) | |
| 18:28:14 | clarkb | more generally there is a balancing act between being so tied to your CI system that the software only really works there and ensuring the software is generally deployable | |
| 18:28:31 | clarkb | a common place we see this is people updating devsatck job vars rather than setting appropriate defaults in devstack proper | |
| 18:28:46 | clarkb | and we should try and avoid those tendencies and ensure the software can stand on its own | |
| 18:29:05 | sean-k-mooney | selunx disbaling is done here https://github.com/openstack/devstack/blob/5c1736b78256f5da86a91c4489f43f8ba1bce224/tools/fixup_stuff.sh#L38-L43 garded by is_fedora so that hsoudl have worked | |
| 18:30:29 | clarkb | where "run this zuul job locally" is potentially useful is determining why builds have failed | |
| 18:30:37 | clarkb | it is less useful as a "run our software" option | |
| 18:32:54 | sean-k-mooney | clarkb: so the issue is i have been using devstack for so long now that i rearly have issues and i can fix them pretty quickly when i do | |
| 18:33:53 | clarkb | yup, we end up with blinders and biases. I'm just saying that in this situation I feel ike the problem exist in devstack and not a lack of tooling in zuul | |
| 18:39:44 | sean-k-mooney | zuul jobs can be wriite so tehy are runablle locally | |
| 18:39:53 | sean-k-mooney | they are just ansible after all | |
| 18:40:26 | sean-k-mooney | the only thing tha tmakes that hard with our devstack ones is bascilly a meta palybooks to run all the pre/post playbooks in the write order | |
| 18:41:00 | sean-k-mooney | if you coud trivially gerneate that you and the inventory you could jus tupdate the ips and run it agaisnt your own vms | |
| 18:41:53 | sean-k-mooney | in the molecule senario and playbooks i worte in my ard repo i basically just did that | |
| 18:42:30 | sean-k-mooney | i reused the roles and statically combined themn to give me somethign pretty close to the upstream jobs | |
| 18:48:27 | clarkb | right. I'm basically asserting that needing to resort to that for users to run devstack is a bug though | |
| 18:48:36 | clarkb | and we should try and make the tool easier to use rather than make CI a stand in | |
| 19:21:25 | sean-k-mooney | right but devstack is not hard to use | |
| 19:22:13 | sean-k-mooney | all the issue bauzas hit where tyring to use it in unsupproted ways (an os that is not supproted after starting in a mode that is not supproted on its nearest supproted cousin) | |
| 19:42:55 | sean-k-mooney | dansmith: i have a terible idea. | |
| 19:43:03 | dansmith | yeah? | |
| 19:43:16 | sean-k-mooney | the dbcounter uses a plugin https://opendev.org/openstack/devstack/src/branch/master/tools/dbcounter/dbcounter.py | |
| 19:43:50 | sean-k-mooney | do you think i could create one for the healthchecks to detect if a conenction resets and is reestablished | |
| 19:44:35 | sean-k-mooney | would that be an insane thing to do even look into or something to condier | |
| 19:45:10 | dansmith | yeah I think that's a bad idea, | |
| 19:45:19 | dansmith | because it has to be in the [db]/connection string to work | |
| 19:45:34 | sean-k-mooney | oh ok ya i didnt know that part | |
| 19:46:01 | sean-k-mooney | never mind then i just saw it pringitn stats since i have journaclt open | |
| 19:46:09 | sean-k-mooney | and then i rememebred it was a plugin | |
| 19:49:11 | sean-k-mooney | the example is litcally just regestirign an evnt with a call back | |
| 19:49:21 | sean-k-mooney | https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.CreateEnginePlugin | |
| 19:50:19 | sean-k-mooney | but i mised the url part | |
| 19:50:49 | sean-k-mooney | although you can enable it a diffent way | |
| 19:51:10 | sean-k-mooney | anyway i was ment to finish a while ago so ill leave that for now | |
| 19:53:34 | dansmith | feels like a monkeypatch to me :) | |
| 19:54:04 | sean-k-mooney | kind of but via a supported interface | |
| 19:54:19 | sean-k-mooney | as in its an intential extention point but | |
| 19:54:23 | JayF | I'll note Ironic uses this event interface directly to set PRAGMA on sqlite usages. | |
| 19:54:25 | sean-k-mooney | the db is the last part on my list | |
| 19:54:33 | JayF | So you could potentially get the same behavior without having to monkey patch | |
| 19:54:52 | sean-k-mooney | JayF: im not planning to monkey patch | |
| 19:55:20 | sean-k-mooney | JayF: what im trying to figure out is a clean way to detect if an database connection is lost and restablished | |
| 19:55:24 | JayF | Oh yeah, I figured that, but just pointing out that event interface is generally useful for stuff like that: https://github.com/openstack/ironic/blob/master/ironic/db/sqlalchemy/__init__.py#L28 | |
| 19:55:33 | sean-k-mooney | without have to do that everywhere we interact wit the db | |
| 19:55:34 | JayF | Yeah, I'm saying: register events in the same way they do in that CreatePlugin | |
| 19:55:56 | JayF | that example in Ironic is actually pretty darn close to half of what you need in terms of registering the connect event | |
| 19:56:14 | sean-k-mooney | ya so i looke at doing that too | |
| 19:56:37 | sean-k-mooney | but its a little tricky to ensure we do that give how we mange our engine fasads | |
| 19:57:14 | JayF | Note that doesn't have a handle on anything | |
| 19:57:35 | JayF | Do you not have a place, ala __init__.py in a relevant module, you're guaranteed to be run in every thread? | |
| 19:57:54 | JayF | the dbapi_connection is provided for you on a ConnectionEvent | |
| 19:59:52 | sean-k-mooney | simple answer no we have at least 2 places like that posble more | |
| 20:00:45 | sean-k-mooney | for the cell db i belive all engin creation goes though this function https://github.com/openstack/nova/blob/master/nova/db/main/api.py#L138 | |
| 20:01:06 | sean-k-mooney | for the api db here https://github.com/openstack/nova/blob/master/nova/db/api/api.py#L50 | |
| 20:01:22 | sean-k-mooney | but there is a lot of code to review | |
| 20:01:34 | JayF | I'm saying I think your assumption that it has to be done *at* engine creation time is inaccurate | |
| 20:01:48 | JayF | that it can be done anytime *before* that engine creation, as well | |
| 20:01:54 | JayF | which is likely a much easier problem to solve | |
| 20:02:07 | JayF | but I'm not super familiar with nova is there's a different dimension here I'm missing | |
| 20:02:11 | sean-k-mooney | proably | |
| 20:02:26 | sean-k-mooney | i just need to review the options and then see where to go form there | |
| 20:02:50 | sean-k-mooney | i was lookign at how we optionally enabel osprofiler.sqlalchemy | |
| 20:02:53 | sean-k-mooney | as well | |
| 20:03:49 | sean-k-mooney | it has a handel_error function | |
| 20:03:51 | sean-k-mooney | https://opendev.org/openstack/osprofiler/src/branch/master/osprofiler/sqlalchemy.py#L99 | |
| 20:04:31 | sean-k-mooney | right now im just trying to see what the options are in this space | |
| 20:05:40 | sean-k-mooney | its uses https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/enginefacade.py#L792 append_on_engine_create | |
| 20:07:50 | JayF | oh, nice! I imagine we didn't use that because we only wanted to catch sqlite connections | |
| 20:07:58 | sean-k-mooney | https://docs.sqlalchemy.org/en/20/core/pooling.html#custom-legacy-pessimistic-ping | |
| 20:08:23 | sean-k-mooney | there are docs for how to lliste for the engine_connect event | |
| 20:09:10 | sean-k-mooney | so i was wondering if i could combine some for those approch to ensure that wehn a engin is create we greister a litener for and engine_connect | |
| 20:09:32 | sean-k-mooney | that just set a db_connection healtcheck to pass/fail | |
| 20:10:18 | sean-k-mooney | anyway ill let this simmer in the back of my mind and come back in a few days | |
| 20:10:29 | sean-k-mooney | thanks for the ironic link | |
| 20:10:29 | JayF | In Ironic, we don't have a handle on engine at all when we setup that event listener; so I'm a little confused as to if it really needs to be done per engine | |
| 20:10:55 | sean-k-mooney | ya im not sure either | |
| 20:11:12 | sean-k-mooney | i should proably just ask zeek about it | |
| 20:13:37 | JayF | My current belief is it is gloriously simple -- just make sure the listener is registered and you don't need to worry about doing something "per enginer". If you find that is not correct, please let me know if you remember :D | |