| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-16 | |||
| 12:48:53 | gibi | if there is one static connection for the in memory DB then that can be the point of the race. And I imagine that it is very different from the DB perspective if one connection starts a transaction within a transaction or two independent connection runs two independent but timewise overlapping transactions | |
| 12:49:49 | gibi | dansmith: ^^ | |
| 12:54:49 | gibi | sqlalchemy.pool.impl.StaticPool this is the pool in the functional test by default | |
| 12:55:43 | gibi | https://github.com/sqlalchemy/sqlalchemy/blob/7dce9fec9d2071351d0c003a607955eca5dcf9ae/lib/sqlalchemy/pool/impl.py#L451 | |
| 12:57:39 | sean-k-mooney | isnt that why oslo.db is overreidng | |
| 13:05:21 | gibi | when the SQLite DB is in memory then we use StaticPool when we switch to file based (for the api DB) then that file based DB uses sqlalchemy.pool.impl.NullPool | |
| 13:05:43 | gibi | this also points to that maybe the source of the race is the StaticPool | |
| 13:06:20 | gibi | null pool basically always opens a new connection | |
| 13:06:23 | gibi | so it cannot race | |
| 13:06:53 | gibi | I'm not sure what oslo_db overrides in this regards | |
| 13:13:19 | bauzas | stephenfin: silly question but I was thinking we were still supporting tox 2.x, am I wrong ? | |
| 13:15:06 | bauzas | hmmm, OK, we were supporting 3.18 | |
| 13:15:45 | stephenfin | bauzas: I assume you mean tox 3.x (2.x hasn't seen a release in nearly a decade). In any case, no. The only reason to have kept tox 3.x around was for Python 2.x support but that's not been an issue for some time | |
| 13:16:34 | stephenfin | That and tempest, which is relying on the shared venv thing that doesn't work in tox 4.x (I've been working on fixing that in the background) but this doesn't affect nova itself | |
| 13:19:28 | gibi | so I do see that with StaticPool we can basically checkout the same connection twice without returning the connection first. So two parallel entity can talk to the same tcp connection object. I guess in eventlet mode in the functional test our yield points was in places where such cross talk did not cause two transaction to overlap inside that single connection. And in production code we don't use | |
| 13:19:34 | gibi | the StaticPool so we never checkout the same tcp connection twice so crosstalk cannot happen < dansmith my current answer to your question | |
| 13:19:46 | bauzas | stephenfin: yeah sorry, typo there, I meant 3.x | |
| 13:19:47 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: Bump min version of pyroute2 https://review.opendev.org/c/openstack/os-vif/+/984934 | |
| 13:20:22 | bauzas | stephenfin: but there is a CLI behavioural change with posargs when upgrading to 4.x, which is pretty impactful for our community, I'd appreciate more than just a file change | |
| 13:21:17 | gibi | (and I have proof that this code is magic. it contains fairys: sqlalchemy.pool.base._ConnectionFairy) | |
| 13:22:34 | gibi | we cannot switch to the null pool with in memory SQLite as the DB is freed as soon as the connection drops and nullpool drops it after use | |
| 13:23:32 | gibi | we cannot really use QueuePool either as two connection cannot address the same in memory SQLite, that will be two different in memory DB for the two connection | |
| 13:24:11 | gibi | so either we use StaticPool with one single never closed connection or we switch away from in memory DB | |
| 13:24:35 | gibi | the former is racy in threading mode when two threads checks out the same connection object and crosstalk | |
| 13:31:21 | stephenfin | bauzas: a CLI change? | |
| 13:34:49 | bauzas | stephenfin: https://tox.wiki/en/4.6.4/upgrading.html isn't quite a short doc | |
| 13:35:18 | bauzas | and my main concern is https://github.com/tox-dev/tox/issues/2183 | |
| 13:38:22 | bauzas | s/ |
|
| 13:38:50 | opendevreview | Kamil Sambor proposed openstack/nova master: Test nova CLI commands with native threading https://review.opendev.org/c/openstack/nova/+/984036 | |
| 13:41:38 | sean-k-mooney | bauzas: we are already likely using 4.28.0 | |
| 13:41:47 | sean-k-mooney | we dont cap tox via upper-constraits | |
| 13:41:53 | bauzas | if you just pip install tox, sure | |
| 13:42:01 | sean-k-mooney | right which is what we do | |
| 13:42:02 | bauzas | like myself | |
| 13:42:30 | dansmith | sean-k-mooney: who is we? :) I don't pip install tox, I brew install it | |
| 13:42:40 | sean-k-mooney | im talkign about the ci jobs | |
| 13:43:07 | dansmith | gibi: sounds like this was an interesting diversion but that led to some understanding about why we're hitting this in tests only and that production should be good yeah? | |
| 13:43:14 | sean-k-mooney | and as i noted in the reivew tox installs a seperve version of tox in a venv and uses htat if the min veriosn is above our current one | |
| 13:43:50 | sean-k-mooney | s/our current one/ the one that is locally installed/ | |
| 13:44:06 | bauzas | I'm doublechecking but I'm almost sure we have somewhere in our docs that says we should use tox 3.x | |
| 13:44:38 | bauzas | of course that's a legacy doc with 4.x but if that's on the nova doc, I'd want to be consistent | |
| 13:45:20 | bauzas | and again, the fact that we bump to 4.x is understandable but I just say we should say it way louder than just a file change to tox.ini | |
| 13:45:23 | sean-k-mooney | devstack does seam to insall na older version but i dont think that is what our tox jobs do but ill doubel check | |
| 13:46:04 | bauzas | thanks | |
| 13:46:16 | sean-k-mooney | https://opendev.org/zuul/zuul-jobs/commit/bda76ac8bb30bb8d8c24552194038db0ff5e6255 | |
| 13:46:24 | sean-k-mooney | ok so it was pinend but that was removed a year ago | |
| 13:46:42 | sean-k-mooney | oh no 4 years ago https://opendev.org/zuul/zuul-jobs/commit/13925d1965aa5658304c9ca88fb0307d6cff2eff | |
| 13:46:46 | bauzas | again, I'm not opposed to the change, I just want us to be careful and explaining about the bump (a reno would be nice) | |
| 13:46:53 | sean-k-mooney | yep | |
| 13:46:58 | sean-k-mooney | i agree on the release note | |
| 13:47:26 | stephenfin | renos are for end-users/operators. This is developer-focused. I don't believe we should document this in release note | |
| 13:47:36 | sean-k-mooney | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L24 | |
| 13:47:46 | sean-k-mooney | stephenfin: it has an implciation for packageser | |
| 13:48:04 | sean-k-mooney | so the other secion could be approrpate | |
| 13:48:08 | stephenfin | how? | |
| 13:48:20 | sean-k-mooney | i dont think it needs to be in upgrade | |
| 13:48:45 | sean-k-mooney | stephenfin: well i guess it depend on how they run unit test but your probly right that it does not actully | |
| 13:48:56 | sean-k-mooney | they are likely runing them directly | |
| 13:49:20 | bauzas | yeah, that's for an other section IMHO | |
| 13:49:23 | sean-k-mooney | sicne they woudl not want to use deps form pypi in the rpm/deb test execution | |
| 13:50:15 | stephenfin | yeah, I'd be shocked if they were running tox rather than stestr directly (if I recall correctly, RDO spec files were still using os-testr for at least some projects) | |
| 13:50:45 | sean-k-mooney | stephenfin: so you need tox 4.28.0 adn our ci is currently caped at 4.48 | |
| 13:51:05 | sean-k-mooney | so we shoudl be testign with a version that suprot it today | |
| 13:54:37 | stephenfin | so do I need a release note? I don't think it makes sense but if it'll help smooth the path to getting this in I can do so | |
| 13:56:10 | dansmith | TBH, a reno for this seems odd to me | |
| 13:56:40 | sean-k-mooney | im ok without one but im fine if other wnat it | |
| 13:57:02 | sean-k-mooney | as you said its mainly a devleoper chnge so other is the only place that seam reasonable if we have one | |
| 13:57:15 | bauzas | okay, then if no reno, we should then mention that in our contrib docs | |
| 13:57:18 | dansmith | I don't see any other tox or unit test related stuff in our renos (from a dumb grep) | |
| 13:57:23 | sean-k-mooney | i just removed my +2 to allow htat converstaion to happen without it there | |
| 13:57:25 | dansmith | contributor docs makes a lot more sense | |
| 13:58:11 | dansmith | we have development-environment.rst which talks about pip installing tox, so that seems like an appropriate place | |
| 13:58:25 | bauzas | we have https://github.com/openstack/nova/blob/master/doc/source/contributor/development-environment.rst | |
| 13:58:32 | bauzas | ah jinxed | |
| 13:58:46 | bauzas | stephenfin: fancy amending this ? | |
| 13:58:52 | sean-k-mooney | stephenfin: the actual thing i think would be more imporant to fix is figure out why devstack is capped to 3.x when our tox jobs are not. and get those in sync so that folks that use tox provdied by there devstack install have somethign less old... | |
| 13:59:13 | dansmith | sean-k-mooney: might have been for grenade or testing older distros at some point | |
| 13:59:13 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 13:59:35 | stephenfin | sean-k-mooney: it's pinned for tempest. From my message earlier: | |
| 13:59:36 | sean-k-mooney | dansmith: ya i suspect its from when we supproted jammy | |
| 13:59:51 | stephenfin | 14:16 stephenfin That and tempest, which is relying on the shared venv thing that doesn't work in tox 4.x (I've been working on fixing that in the background) but this doesn't affect nova itself | |
| 13:59:57 | sean-k-mooney | stephenfin: oh because of the venv thing? | |
| 14:00:01 | stephenfin | yeah | |
| 14:00:01 | sean-k-mooney | ok | |
| 14:00:07 | dansmith | sean-k-mooney: I run 4.4 on jammy, but.. something earlier | |
| 14:00:33 | stephenfin | but that (a) is huge tech debt we really need to fix and (b) doesn't affect nova's use of tox | |
| 14:01:27 | sean-k-mooney | stephenfin: my new thoery is hte hardwrae where i sometimes(often) have the log setup time in tox is just cursed | |
| 14:02:27 | gibi | dansmith: > led to some understanding about why we're hitting this in tests only and that production should be good yeah? | yeah I think I see now why this cannot happen in production | |
| 14:02:28 | sean-k-mooney | nova full unit test sqiute shoudl not exectue in less time then buildign the venv to run it in... py3: OK (508.14=setup[284.11]+cmd[193.47,23.86,6.70] seconds) | |
| 14:02:47 | dansmith | gibi: ack, well, sorry for the goose chase, but I feel better :) | |
| 14:03:25 | stephenfin | bauzas: Can do. What will that look like? I'd rather not duplicate what's in tox.ini in our docs since they're liable to drift, so maybe I can just say you need at least tox 4.x available (since it'll bootstrap itself if it needs a newer version)? | |
| 14:03:30 | gibi | dansmith: don't feel bad about it. Your question actually lead to a deeper understanding of the situation | |
| 14:03:31 | stephenfin | s/will/should/ | |
| 14:03:55 | bauzas | gibi: dansmith: I hadn't yet reviewed, but thanks for the discussion, I'll look at the patch | |
| 14:04:46 | gibi | dansmith: the next similar area I want to understand more is the CellDatabases fixture that implements some weird routing of DB connections via hooking into the target_cell call | |
| 14:05:03 | bauzas | stephenfin: there are two problems to solve here : 1/ documenting the fact that in Hibiscus, we bumped to 4.28.0 (should be easy addition to dev-env.rst), 2/ before merging the patch, making sure that we don't create devstack problems | |
| 14:05:18 | dansmith | gibi: ack, there are some acrobatics in there for sure | |
| 14:06:48 | stephenfin | (2) isn't an issue and will be confirmed by CI and the fact that other projects have already done see (e.g. ironic https://github.com/openstack/ironic/blob/master/tox.ini#L2) | |
| 14:07:22 | stephenfin | I can also walk through how tempest is (wrongly) using tox and why it depends on tox 3.x, but my WIP patches probably explain it better | |
| 14:08:10 | stephenfin | https://review.opendev.org/q/topic:%22rework-tempest-install%22+is:open | |