| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-27 | |||
| 14:20:28 | bauzas | to see whether it's a large race condition | |
| 14:20:28 | gibi | bauzas: I used logstash today | |
| 14:21:22 | sean-k-mooney | im going to see if it only happens in a repo after usign dmitriis patch | |
| 14:21:36 | sean-k-mooney | i wonder if we are leaking locks or somehting | |
| 14:22:14 | sean-k-mooney | dmitriis: you patch is still pending in the gate right | |
| 14:23:27 | dmitriis | sean-k-mooney: yes but I can see here https://zuul.opendev.org/t/openstack/status#nova that it already finished running unit tests | |
| 14:24:31 | sean-k-mooney | odd that it passed | |
| 14:24:40 | sean-k-mooney | and we can both repoduce the issue locally | |
| 14:25:16 | sean-k-mooney | dmitriis: what operating system are you using by the way | |
| 14:25:31 | dmitriis | sean-k-mooney: yes, I'm trying to compare how many test cases fail each time and whether it depends on timing | |
| 14:25:33 | opendevreview | Merged openstack/placement master: disable traits count check to allow os-traits 2.7.0 https://review.opendev.org/c/openstack/placement/+/826486 | |
| 14:25:52 | dmitriis | sean-k-mooney: Ubuntu 21.10 | |
| 14:26:04 | sean-k-mooney | for me it was consitent | |
| 14:26:12 | sean-k-mooney | well mostly | |
| 14:26:18 | sean-k-mooney | i coudl re run with --failing | |
| 14:26:31 | dmitriis | sean-k-mooney: in a 21.10 container I consistently get 1 failure | |
| 14:26:34 | sean-k-mooney | that went form 9 to 8 then staied with 8 | |
| 14:28:32 | sean-k-mooney | hum clean repo one failure but its differetn | |
| 14:28:34 | sean-k-mooney | RuntimeError: Reader <_MainThread(MainThread, started 140118297270080)> to writer privilege escalation not allowed | |
| 14:28:51 | dmitriis | sean-k-mooney: yes, that's the one I am getting in a clean container with the patch applied | |
| 14:29:03 | dmitriis | RuntimeError: Reader <_MainThread(MainThread, started 139926055308288)> to writer privilege escalation not allowed | |
| 14:29:10 | sean-k-mooney | ya i have your patch aplied too | |
| 14:30:28 | sean-k-mooney | gibi: bauzas i guess we are not seeing this in the gate | |
| 14:30:35 | sean-k-mooney | but its just odd behavior | |
| 14:31:07 | sean-k-mooney | the issue seam to be related to the fasteners release i think | |
| 14:31:16 | sean-k-mooney | im going to downgrade it to confirm | |
| 14:31:23 | dmitriis | sean-k-mooney: ack | |
| 14:32:33 | dmitriis | sean-k-mooney: tried with the previous (VPD) patch only https://paste.opendev.org/show/812404/ | |
| 14:33:41 | sean-k-mooney | well a uuid is not an rfc1738 url | |
| 14:34:00 | sean-k-mooney | hum same issue with older fasteners | |
| 14:36:59 | dmitriis | sean-k-mooney: the other failures are odd to see because that same patch passed on CI https://review.opendev.org/c/openstack/nova/+/808199. Checking with older fasteners as well. | |
| 14:37:11 | sean-k-mooney | dmitriis: master seam to work ok for me | |
| 14:37:20 | sean-k-mooney | dmitriis: have you treid rebaseing all your patches | |
| 14:37:36 | sean-k-mooney | we might have fix this already on master and you patches might be missing it | |
| 14:37:46 | dmitriis | sean-k-mooney: ack, let me try that | |
| 14:53:35 | bauzas | sean-k-mooney: cool, interesting | |
| 14:53:41 | bauzas | thanks for the follow-up | |
| 14:54:36 | sean-k-mooney | bauzas: gibi by the way we likely need to have a dicussion about what to do about placment,os-traits and os-resouce-providres at some point | |
| 14:54:54 | gibi | sean-k-mooney: ack. I read the scrollback from the release channel | |
| 14:55:32 | sean-k-mooney | perhaps a ptg topic. we coudl start it on the ML or in an etherpad | |
| 14:55:58 | gibi | I do believe that os-traits being a separate lib helps for the other consumers like nova and neutron. So I would not want to merge that back to placement | |
| 14:56:27 | gibi | So I'm on the side to relax the testing and add a release step where we check that the placement is released with the latest os-traits | |
| 14:57:36 | sean-k-mooney | ya long term that is proably the most operator freindly too | |
| 14:57:56 | sean-k-mooney | it would be a lot of work to reabsorb os-trait into placment | |
| 14:58:26 | sean-k-mooney | since we woudl have to modify nova,neutron,cyborg and possibel others + the deployment tooling | |
| 15:01:27 | sean-k-mooney | gibi: i can look at relaxing the test and push a patch for people to consider | |
| 15:01:48 | gibi | sean-k-mooney: OK, thanks | |
| 15:05:06 | sean-k-mooney | i dont know if cdent or others are aroudn that shoudl review | |
| 15:05:17 | sean-k-mooney | are any of the placment core team actully still active | |
| 15:05:50 | dmitriis | sean-k-mooney: hmm, tried master locally, getting the same failures with locking. Tried master on a server machine (21.04) - no failures. | |
| 15:06:05 | gibi | sean-k-mooney: I think we only have those that are active in nova too | |
| 15:06:24 | dmitriis | will try a few more combinations | |
| 15:06:43 | sean-k-mooney | perhaps tetsuro? although i have not seen them in a while | |
| 15:07:11 | sean-k-mooney | ill add the patch to open discuss for next weeks meeting once i figure out what to do | |
| 15:07:22 | gibi | ack | |
| 15:24:09 | ade_lee__ | sean-k-mooney, hey -- could you take a look at https://zuul.opendev.org/t/openstack/build/554ffdeb6c3a4cdc8c53c55dddd6c36e please and let me know whats missing? | |
| 15:24:33 | ade_lee__ | sean-k-mooney, this is for https://review.opendev.org/c/openstack/tempest/+/826580 as discussd the other day | |
| 15:26:12 | sean-k-mooney | sure ill take a look quickly and see if anything jumps out | |
| 15:26:24 | sean-k-mooney | you might need to set swap on the subnode too but ill check that | |
| 15:29:14 | sean-k-mooney | ade_lee__: actuly devstack may not have finished on the compute | |
| 15:29:41 | sean-k-mooney | ya ok | |
| 15:29:49 | sean-k-mooney | so on the compute devstack failed to install pip | |
| 15:29:57 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/554ffdeb6c3a4cdc8c53c55dddd6c36e/log/compute1/logs/devstacklog.txt | |
| 15:30:53 | dmitriis | sean-k-mooney: on 21.04 server tests are passing even with patches applied like it happens on the CI. Also not getting any failures in a 21.04 container locally with patches applied (with a rebase to master). Without a rebase I seem to be getting only "writer privilege escalation not allowed". So probably something in 21.10 libs is different and | |
| 15:30:53 | dmitriis | results in the breakage with locks. This hasn't happened recently so I'll try to explore further which update might have caused it. | |
| 15:31:14 | dmitriis | but at least one issue should be covered with rebasing | |
| 15:31:38 | sean-k-mooney | ack | |
| 15:31:58 | sean-k-mooney | if we dont see it in ci for now i would proceed with the rest of your work and we can keep an eye out for it | |
| 15:34:57 | sean-k-mooney | ade_lee__: this might just be an intermient failure on centos 8 | |
| 15:35:17 | sean-k-mooney | so we shoudl recheck but ade_lee__ any plans to move that to centos-9-stream | |
| 15:35:28 | sean-k-mooney | if we dont do that this cycle we shoudl do it next cycle | |
| 15:35:41 | ade_lee__ | sean-k-mooney, maybe -- the relevant code seems to be here -- https://opendev.org/openstack/devstack/src/branch/master/tools/install_pip.sh#L121-L134 | |
| 15:36:33 | sean-k-mooney | it looks like hte rax mirror just did not have a cenots8 compatiable packages | |
| 15:36:58 | ade_lee__ | sean-k-mooney, but yes, the plan is absolutely to move to centos-9-stream | |
| 15:37:10 | sean-k-mooney | the issue with 8 is its using python 3.6 | |
| 15:37:17 | sean-k-mooney | which is now end of life upstream | |
| 15:37:35 | sean-k-mooney | so we are going to start to see pip not working on it sonner rather then later | |
| 15:37:45 | ade_lee__ | sean-k-mooney, the only problem right now is that fips+python+centos-9 is broken | |
| 15:37:56 | ade_lee__ | sean-k-mooney, but its getting fixed .. | |
| 15:38:25 | sean-k-mooney | so we might need to expand that to use pip form packages for cenos 8 stream too | |
| 15:39:09 | ade_lee__ | yup | |
| 15:40:10 | sean-k-mooney | https://github.com/pypa/pip/commit/0252c04a16cd93fe422cebf0b48453b559a2e404 | |
| 15:40:17 | sean-k-mooney | pip droped support for 3.6 | |
| 15:40:43 | sean-k-mooney | https://github.com/pypa/pip/blob/main/setup.py#L83 | |
| 15:40:49 | ade_lee__ | sean-k-mooney, as soon as https://bugzilla.redhat.com/show_bug.cgi?id=1942527 is fixed, I plan to move all these to centos-9-stream | |
| 15:40:51 | sean-k-mooney | so ya that is the issue | |
| 15:41:23 | sean-k-mooney | this will break all centos 8 jobs so we should likely fix devstack to work around it for now | |
| 15:41:29 | ade_lee__ | which is supposed to happen maybe by next week .. | |
| 15:41:47 | ade_lee__ | yeah - I'll expand the conditional and see if that works | |
| 15:42:25 | sean-k-mooney | just have it check is_fedroa | |
| 15:42:27 | sean-k-mooney | drop the rest | |
| 15:42:39 | sean-k-mooney | and it shoudl be correct | |
| 15:43:04 | sean-k-mooney | we just need to ensure python3-pip is in the rpm files in devstack | |
| 15:43:05 | ade_lee__ | sean-k-mooney, ok - trying .. | |
| 15:44:03 | sean-k-mooney | which it is https://github.com/openstack/devstack/blob/master/files/rpms/general#L29 | |
| 15:45:29 | sean-k-mooney | ade_lee__: by the way that is in post but has to go through qe ectra and then be published and buil in centos | |
| 15:45:39 | sean-k-mooney | so it will proably take more then a week | |
| 15:45:52 | ade_lee__ | sigh .. yeah | |
| 15:46:17 | sean-k-mooney | i mean it wont take that much longer there is a merge request up | |
| 15:46:21 | sean-k-mooney | we can still do it this cycle | |