| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-20 | |||
| 00:38:48 | sean-k-mooney[m] | maybe stephenfin will spot something | |
| 00:39:01 | melwitt | it might only be just me but I tried wiping everything, upgrading tox, re-cloning the nova repo | |
| 00:39:26 | sean-k-mooney[m] | i wonder if there was an sqlalchemy release or soemthing that has chnaged behaivor | |
| 00:52:41 | sean-k-mooney[m] | so ya just quickly ssh to my home server cloned nova and ran the db test and ya it failed for me too | |
| 00:53:01 | sean-k-mooney[m] | i only got 16 failure but it proably not deterministic | |
| 00:53:19 | sean-k-mooney[m] | its likely differnt based on the number of cores/parallel tests we have | |
| 00:53:52 | sean-k-mooney[m] | the server i ran it on had 48 threads so there will be less tests running in the same process since it will use more of them | |
| 00:54:31 | melwitt | oh, yeah ok | |
| 00:55:21 | sean-k-mooney[m] | looks like its coming direclty form the db fixture which is od | |
| 00:55:24 | sean-k-mooney[m] | *odd | |
| 00:55:30 | melwitt | yeah | |
| 00:56:12 | melwitt | I have run tests like this subset a lot of times in the past and this is the first time I see it fail like this. so I figure it must be a fairly recent change | |
| 00:56:57 | sean-k-mooney[m] | im wondering if its a recent nova change or oslo.db or sqlalchmy version change | |
| 00:58:28 | melwitt | yeah I don't find sqlalchemy in my tox env which ... I don't understand | |
| 01:01:48 | sean-k-mooney[m] | i have 1.4. something | |
| 01:02:11 | sean-k-mooney[m] | i downgraded its uppercase in pip freeze | |
| 01:02:32 | sean-k-mooney[m] | but downgrading it and oslo.db had no reall effect so i looks like that is not the issue | |
| 01:03:29 | sean-k-mooney[m] | or rather it happens with oslo.db 11 and 10 and sqlalchmey 1.4.x and 1.3.x | |
| 01:03:52 | sean-k-mooney[m] | so ya maybe a recent nova change | |
| 01:04:18 | sean-k-mooney[m] | ill try and take a look again in the morning. | |
| 01:04:19 | melwitt | oh derp | |
| 01:04:21 | melwitt | haha thanks | |
| 01:04:22 | sean-k-mooney[m] | night all o/ | |
| 01:04:27 | melwitt | SQLAlchemy==1.4.25 | |
| 01:04:39 | melwitt | gnite o/ | |
| 06:13:31 | bauzas | good morning Nova | |
| 07:13:02 | opendevreview | Pierre-Samuel Le Stang proposed openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807551 | |
| 08:30:32 | gibi | morning | |
| 09:33:24 | gibi | bauzas: do we start with the tox.ini python version pinning topic today at 13:00 UTC? | |
| 09:33:47 | bauzas | I was thinking so | |
| 09:36:15 | bauzas | gibi: updated the etherpad to make it clear where we start | |
| 09:36:40 | gibi | bauzas: thanks, I will gather my thoughts then | |
| 09:41:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396 | |
| 09:41:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748 | |
| 09:48:09 | opendevreview | Balazs Gibizer proposed openstack/nova master: Avoid unbound instance_uuid var during delete https://review.opendev.org/c/openstack/nova/+/805605 | |
| 09:48:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: [nova-manage]support extended resource request https://review.opendev.org/c/openstack/nova/+/802060 | |
| 09:48:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate https://review.opendev.org/c/openstack/nova/+/805046 | |
| 09:56:26 | gibi | sean-k-mooney[m]: btw, there is a settings in gerrit "Set new changes to "work in progress" by default" so the gerrit dev also has similar thinking as you, that we need a separate way to signal when a patch is ready and it is not the push | |
| 10:23:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: turn off two unit test cases https://review.opendev.org/c/openstack/nova/+/814735 | |
| 10:24:26 | gibi | sean-k-mooney[m], melwitt: I found the two offending unit test case that causes .AlreadyStartedError error if the db tests are run selectively ^^ | |
| 10:24:47 | gibi | but I don't know why they are breaking our test runs | |
| 10:31:26 | gibi | more funnier. I can fix it, but I don't know why the fix helps :D | |
| 10:37:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix db migration unit test https://review.opendev.org/c/openstack/nova/+/814735 | |
| 10:38:18 | gibi | sean-k-mooney[m], melwitt: after this ^^ I cannot reproduce the problem locally any more. But I need somebody to help with explainig why this helps | |
| 10:40:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix db migration unit test https://review.opendev.org/c/openstack/nova/+/814735 | |
| 10:59:16 | stephenfin | gibi: gdi, good find | |
| 11:02:59 | stephenfin | all that context manager stuff is global. I suspect we're doing something somewhere to break this global-ness but not when we only run those tests | |
| 11:04:16 | stephenfin | https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/test_wsgi_app.py#L71-L77 | |
| 11:35:34 | opendevreview | Wenping Song proposed openstack/nova master: Report gpu devices that only configured https://review.opendev.org/c/openstack/nova/+/814743 | |
| 11:57:55 | sean-k-mooney | gibi: the engie facade mocking is similar to what we did in https://review.opendev.org/c/openstack/nova/+/805663/9/nova/tests/unit/db/test_migration.py | |
| 12:00:00 | sean-k-mooney | gibi: in that case we needed to create our own db instance in the test to allow the url to be modifed since that is read only when the engin is created | |
| 12:00:12 | sean-k-mooney | well configured | |
| 12:02:27 | sean-k-mooney | so it does kind of make sense to me that any test that do manual db sync might need this type of mocking but i would not exepct the db syncy to actill try to restart the engin facade | |
| 12:07:25 | sean-k-mooney | stephenfin: while your looking at gate issues https://review.opendev.org/c/openstack/nova/+/814036 also ill joing the ptg shortly | |
| 12:13:51 | sean-k-mooney | oh right we are currently utc +1 | |
| 12:15:22 | opendevreview | Wenping Song proposed openstack/nova master: Cleanup guest process when vm evacuate failed and then deleted https://review.opendev.org/c/openstack/nova/+/814746 | |
| 12:15:28 | stephenfin | sean-k-mooney: I don't think it's trying to start it. I think it's trying to _configure_ it | |
| 12:16:29 | stephenfin | bauzas: I'll miss the first 30 minutes of the PTG, unfortunately | |
| 12:16:42 | bauzas | stephenfin: OK, then I'll flip some topics | |
| 12:17:05 | bauzas | stephenfin: we'll start with gibi's topic about the tox usage | |
| 12:17:24 | sean-k-mooney | well i think stephen is interested in that | |
| 12:18:02 | stephenfin | yeah, I was going to say. I can miss it though. I think a few people understand the problem so I'm sure you'll settle on something sensible | |
| 12:18:41 | sean-k-mooney | stephenfin: im hoping py39 will become a required testing env for yoga | |
| 12:20:17 | sean-k-mooney | just an fyi so we will need to be able to test with that locally but ya we understand what you were tryign to do | |
| 12:22:25 | gibi | stephenfin: ok, so we have global context_manager objects one for the api db and one for the main db. So that is what gets dirty and leaking between tests | |
| 12:23:24 | gibi | do we have some special handling for them in a fixture we use for every test case so in general test cases are not interacting? And don't we use the same fixture in the db_sync tests for a reason? | |
| 12:28:51 | stephenfin | gibi: Sigh, all the DB tests are using NoDBTestCase, so we're not actually using the Database fixture https://github.com/openstack/nova/blob/master/nova/tests/fixtures/nova.py#L611-L617 | |
| 12:29:31 | stephenfin | I still don't know why running other tests first fixes the issue though... | |
| 12:30:57 | gibi | hm, so the normal way to deal with these globals is the Database fixture but for explicit DB testing we dont use that fixture | |
| 12:32:22 | stephenfin | that's my understanding from a quick inspection, but I'm juggling many things today and haven't had time to investigate it properly yet /o\ | |
| 12:32:35 | stephenfin | it does seem strange that DB tests would not use the DB fixture though | |
| 12:32:36 | gibi | I think we have a poison fixture that prevent us touching the DB from a NoDBTestCase but it does not hit for the db_sync tests | |
| 12:33:08 | stephenfin | hmm, true | |
| 12:39:46 | gibi | I'm lost in oslo_db / sqla I guess our poision specific for the way nova objects are accessing the db and it still allows our direct get_engine calls | |
| 12:50:36 | bauzas | reminder : nova sessions start in 10 mins | |
| 13:02:56 | bauzas | sessions start by now :) | |
| 13:03:12 | bauzas | if people wanna come => https://www.openstack.org/ptg/rooms/newton | |
| 13:53:23 | stephenfin | I assume we're on a break? | |
| 13:53:58 | gibi | yepp | |
| 13:54:05 | gibi | stephenfin: you joined for the break :D | |
| 13:54:11 | stephenfin | classic me | |
| 13:54:15 | gibi | :) | |
| 13:55:11 | bauzas | stephenfin: we basically NACK'd all your ideas while you were gone :p | |
| 13:55:19 | bauzas | terrible me | |
| 13:58:19 | bauzas | belmoreira: we have an issue with the interop session being discussed one hour later that conflicts the vncproxy discussion we were planning to have at 15:40pm | |
| 13:58:32 | bauzas | belmoreira: can we make the discussion start at 4pm ? | |
| 13:58:39 | bauzas | dang | |
| 13:59:01 | bauzas | belmoreira: I meant, can we discuss the vncproxy thing at 3pm ? | |
| 13:59:06 | bauzas | (UTC) | |
| 13:59:28 | stephenfin | bauzas: just another PTG so | |
| 13:59:29 | stephenfin | :) | |
| 13:59:36 | belmoreira | bauzas 3pm utc works for me. thanks | |
| 14:00:41 | bauzas | just hope melwitt can be present at this time | |
| 14:00:56 | bauzas | belmoreira: ack, will flip the sessions | |
| 14:11:49 | bauzas | stephenfin: we'll do neutron things until 3pm UTC | |
| 15:07:48 | bauzas | dansmith: are you available ? belmoreira is discussing about moving instances between projects and we're talking of os-chown | |
| 15:08:12 | dansmith | bauzas: not right this moment, but I can try to join soon | |
| 15:08:19 | bauzas | dansmith: cool, appreciated. | |
| 15:11:24 | dansmith | bauzas: actually, this next topic is something I have to stay for, so it'll be a bit yet | |
| 15:11:37 | bauzas | ok | |