| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-27 | |||
| 13:19:09 | sean-k-mooney | i can try running it locally what patch have you checked out | |
| 13:20:13 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | |
| 13:20:26 | dmitriis | sean-k-mooney: ^ https://review.opendev.org/c/openstack/nova/+/824833 the latest here | |
| 13:21:33 | sean-k-mooney | funcitonal or unit | |
| 13:21:47 | dmitriis | sean-k-mooney: unit: `tox -e py39` | |
| 13:22:08 | sean-k-mooney | ack just running them now under py38 | |
| 13:22:18 | sean-k-mooney | if they pass ill install py39 and test again | |
| 13:23:08 | dmitriis | sean-k-mooney: ack, I'll spin up a container and do a clean check as well. | |
| 13:23:54 | sean-k-mooney | we do ocationally get unit test breakages when a new lib is released | |
| 13:24:36 | sean-k-mooney | although the last release was Oct 21, 2021 | |
| 13:24:41 | sean-k-mooney | so that is likely not the issue here | |
| 13:26:00 | dmitriis | ack | |
| 13:28:12 | sean-k-mooney | there was an oslo.limits release and olslo utils release in the last few days but also does not look like the woudl be at fault | |
| 13:28:25 | sean-k-mooney | so far i think things are pass ing but im gong to grab a drink | |
| 13:28:27 | sean-k-mooney | brb | |
| 13:28:48 | dmitriis | sean-k-mooney: thanks for checking, will try to figure out what's on my env that's causing it | |
| 13:38:13 | sean-k-mooney | got the same error here | |
| 13:38:42 | bauzas | sean-k-mooney: gibi: sorry folks, was on discussion with someone you know | |
| 13:38:46 | sean-k-mooney | https://paste.opendev.org/show/812399/ | |
| 13:38:46 | bauzas | looking at both your changes | |
| 13:39:16 | sean-k-mooney | dmitriis: so its in the block device tests so not related to your patch | |
| 13:39:31 | gibi | bauzas: no worries | |
| 13:40:19 | sean-k-mooney | this is failing here https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L172-L185 | |
| 13:42:14 | dmitriis | sean-k-mooney: yeah, just confusing. Got 1 failure only with py39 in a clean 21.10 container https://paste.opendev.org/show/812400/ | |
| 13:42:40 | dmitriis | hopefully it will just work in CI :^) | |
| 13:43:47 | sean-k-mooney | ah there was a fasterners release 2 days ago | |
| 13:43:54 | sean-k-mooney | https://pypi.org/project/fasteners/ | |
| 13:44:30 | sean-k-mooney | which we picked up | |
| 13:44:32 | sean-k-mooney | https://github.com/openstack/requirements/commit/c086fff066e40f3a87ff81fe72c8c16cdefe7c64 | |
| 13:54:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devices https://review.opendev.org/c/openstack/nova/+/824834 | |
| 13:54:20 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | |
| 13:54:20 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | |
| 13:54:21 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | |
| 13:56:58 | dmitriis | sean-k-mooney: looking | |
| 13:57:20 | dmitriis | sean-k-mooney, chateaulav: os-traits version bump change: https://review.opendev.org/c/openstack/nova/+/826675 | |
| 13:57:32 | chateaulav | thanks | |
| 14:07:27 | sean-k-mooney | gibi: bauzas we might have an impending gate failure https://paste.opendev.org/show/812401/ | |
| 14:07:44 | sean-k-mooney | so looks like even with latest master we now have lock issues | |
| 14:11:49 | sean-k-mooney | looks like os-brick was depending on poslition for calling kw args with is always dangrous | |
| 14:11:51 | sean-k-mooney | https://github.com/openstack/os-brick/blob/master/os_brick/initiator/utils.py#L45-L46 | |
| 14:12:22 | gibi | sean-k-mooney: I cannot reproduce the lockpath failure locally with fresh env on nova master | |
| 14:13:23 | sean-k-mooney | odd i could after checking out dmitriis code and then still could after checking out master | |
| 14:13:36 | sean-k-mooney | ill try cleaning my env and pyc files again | |
| 14:16:54 | sean-k-mooney | i can repoduce ti consitently on my laptop im trying it on my server now too | |
| 14:17:38 | dmitriis | sean-k-mooney: oddly, the latest run on CI https://zuul.opendev.org/t/openstack/build/daf00a97b54f4e349ef609240a956347/log/job-output.txt doesn't have it | |
| 14:20:10 | sean-k-mooney | gibi: odd it passed there | |
| 14:20:16 | bauzas | gibi: sean-k-mooney: do we have logstash yet ? | |
| 14:20:28 | bauzas | to see whether it's a large race condition | |
| 14:20:28 | gibi | bauzas: I used logstash today | |
| 14:21:22 | sean-k-mooney | im going to see if it only happens in a repo after usign dmitriis patch | |
| 14:21:36 | sean-k-mooney | i wonder if we are leaking locks or somehting | |
| 14:22:14 | sean-k-mooney | dmitriis: you patch is still pending in the gate right | |
| 14:23:27 | dmitriis | sean-k-mooney: yes but I can see here https://zuul.opendev.org/t/openstack/status#nova that it already finished running unit tests | |
| 14:24:31 | sean-k-mooney | odd that it passed | |
| 14:24:40 | sean-k-mooney | and we can both repoduce the issue locally | |
| 14:25:16 | sean-k-mooney | dmitriis: what operating system are you using by the way | |
| 14:25:31 | dmitriis | sean-k-mooney: yes, I'm trying to compare how many test cases fail each time and whether it depends on timing | |
| 14:25:33 | opendevreview | Merged openstack/placement master: disable traits count check to allow os-traits 2.7.0 https://review.opendev.org/c/openstack/placement/+/826486 | |
| 14:25:52 | dmitriis | sean-k-mooney: Ubuntu 21.10 | |
| 14:26:04 | sean-k-mooney | for me it was consitent | |
| 14:26:12 | sean-k-mooney | well mostly | |
| 14:26:18 | sean-k-mooney | i coudl re run with --failing | |
| 14:26:31 | dmitriis | sean-k-mooney: in a 21.10 container I consistently get 1 failure | |
| 14:26:34 | sean-k-mooney | that went form 9 to 8 then staied with 8 | |
| 14:28:32 | sean-k-mooney | hum clean repo one failure but its differetn | |
| 14:28:34 | sean-k-mooney | RuntimeError: Reader <_MainThread(MainThread, started 140118297270080)> to writer privilege escalation not allowed | |
| 14:28:51 | dmitriis | sean-k-mooney: yes, that's the one I am getting in a clean container with the patch applied | |
| 14:29:03 | dmitriis | RuntimeError: Reader <_MainThread(MainThread, started 139926055308288)> to writer privilege escalation not allowed | |
| 14:29:10 | sean-k-mooney | ya i have your patch aplied too | |
| 14:30:28 | sean-k-mooney | gibi: bauzas i guess we are not seeing this in the gate | |
| 14:30:35 | sean-k-mooney | but its just odd behavior | |
| 14:31:07 | sean-k-mooney | the issue seam to be related to the fasteners release i think | |
| 14:31:16 | sean-k-mooney | im going to downgrade it to confirm | |
| 14:31:23 | dmitriis | sean-k-mooney: ack | |
| 14:32:33 | dmitriis | sean-k-mooney: tried with the previous (VPD) patch only https://paste.opendev.org/show/812404/ | |
| 14:33:41 | sean-k-mooney | well a uuid is not an rfc1738 url | |
| 14:34:00 | sean-k-mooney | hum same issue with older fasteners | |
| 14:36:59 | dmitriis | sean-k-mooney: the other failures are odd to see because that same patch passed on CI https://review.opendev.org/c/openstack/nova/+/808199. Checking with older fasteners as well. | |
| 14:37:11 | sean-k-mooney | dmitriis: master seam to work ok for me | |
| 14:37:20 | sean-k-mooney | dmitriis: have you treid rebaseing all your patches | |
| 14:37:36 | sean-k-mooney | we might have fix this already on master and you patches might be missing it | |
| 14:37:46 | dmitriis | sean-k-mooney: ack, let me try that | |
| 14:53:35 | bauzas | sean-k-mooney: cool, interesting | |
| 14:53:41 | bauzas | thanks for the follow-up | |
| 14:54:36 | sean-k-mooney | bauzas: gibi by the way we likely need to have a dicussion about what to do about placment,os-traits and os-resouce-providres at some point | |
| 14:54:54 | gibi | sean-k-mooney: ack. I read the scrollback from the release channel | |
| 14:55:32 | sean-k-mooney | perhaps a ptg topic. we coudl start it on the ML or in an etherpad | |
| 14:55:58 | gibi | I do believe that os-traits being a separate lib helps for the other consumers like nova and neutron. So I would not want to merge that back to placement | |
| 14:56:27 | gibi | So I'm on the side to relax the testing and add a release step where we check that the placement is released with the latest os-traits | |
| 14:57:36 | sean-k-mooney | ya long term that is proably the most operator freindly too | |
| 14:57:56 | sean-k-mooney | it would be a lot of work to reabsorb os-trait into placment | |
| 14:58:26 | sean-k-mooney | since we woudl have to modify nova,neutron,cyborg and possibel others + the deployment tooling | |
| 15:01:27 | sean-k-mooney | gibi: i can look at relaxing the test and push a patch for people to consider | |
| 15:01:48 | gibi | sean-k-mooney: OK, thanks | |
| 15:05:06 | sean-k-mooney | i dont know if cdent or others are aroudn that shoudl review | |
| 15:05:17 | sean-k-mooney | are any of the placment core team actully still active | |
| 15:05:50 | dmitriis | sean-k-mooney: hmm, tried master locally, getting the same failures with locking. Tried master on a server machine (21.04) - no failures. | |
| 15:06:05 | gibi | sean-k-mooney: I think we only have those that are active in nova too | |
| 15:06:24 | dmitriis | will try a few more combinations | |
| 15:06:43 | sean-k-mooney | perhaps tetsuro? although i have not seen them in a while | |