| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-11 | |||
| 10:04:50 | sean-k-mooney | so we do not expect to have any new non-client release in uc between FF and RC-1 unless its an RC-1 fix | |
| 10:05:31 | sean-k-mooney | it looks like the release was actully done 13 days ago | |
| 10:05:40 | sean-k-mooney | so that would have been before FF | |
| 10:05:57 | frickler | yes, it is just the u-c bump that is lagging | |
| 10:06:37 | sean-k-mooney | right which is not good because that kidn of defeats the reason for the eailer releas | |
| 10:06:57 | sean-k-mooney | but i guess in this case it just would have been better to highlihgt this earlier | |
| 10:07:03 | sean-k-mooney | i.e. that it was not passing | |
| 10:08:29 | sean-k-mooney | im running the tests with the older release not and then ill manually unpin and see if this repoduces for me locally | |
| 10:08:30 | frickler | yes, handling by the requirements team could be improved if they had more participants I guess. it also isn't helped by the CI being unstable in general. I'm just trying to fill in some of the gaps where I can | |
| 10:10:21 | sean-k-mooney | so locally passes with 5.2.0 and fails with 5.3.0 | |
| 10:10:29 | sean-k-mooney | so thats a good start | |
| 10:11:13 | sean-k-mooney | looks like the same error too | |
| 10:11:35 | sean-k-mooney | although for some reason my tox execution seams to be hanging... | |
| 10:16:17 | sean-k-mooney | oh right the other test is a timeout exception | |
| 10:16:21 | sean-k-mooney | so it was just waitign on that | |
| 10:45:29 | opendevreview | Merged openstack/python-novaclient stable/2023.2: Update .gitreview for stable/2023.2 https://review.opendev.org/c/openstack/python-novaclient/+/894074 | |
| 11:07:06 | sean-k-mooney | this is kind of insane | |
| 11:07:19 | sean-k-mooney | apprently this LOG.ERROR is causign the test to fail | |
| 11:07:21 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/privsep/utils.py#L77C13-L78C62 | |
| 11:08:05 | sean-k-mooney | this passes https://paste.opendev.org/show/bUyh4M9k01rrYaP58223/ | |
| 11:08:46 | sean-k-mooney | but https://paste.opendev.org/show/b4wfwcpSB6XdjPZzC5UN/ fails | |
| 11:09:11 | sean-k-mooney | but the error is | |
| 11:09:13 | sean-k-mooney | AssertionError: Expected 'write' to not have been called. Called 1 times. | |
| 11:09:15 | sean-k-mooney | Calls: [call(7, b'X')]. | |
| 11:10:08 | sean-k-mooney | the commented out code | |
| 11:10:11 | sean-k-mooney | # m.write(b"x" * align_size) | |
| 11:10:13 | sean-k-mooney | # os.write(fd, m) | |
| 11:10:28 | sean-k-mooney | woudl write X to fd 7 | |
| 11:10:35 | sean-k-mooney | but its commeted out... | |
| 11:10:47 | sean-k-mooney | and the log message is not writng that | |
| 11:30:06 | sean-k-mooney | sothere is somehting funky happening https://github.com/openstack/oslo.log/compare/5.3.0...5.2.0 and https://github.com/openstack/oslo.log/compare/5.2.0...5.3.0 | |
| 11:30:17 | sean-k-mooney | shoudl not both show patches but they do | |
| 11:37:20 | elodilles | sean-k-mooney: now that i'm looking at git tree it shows that we had some issue with oslo.log at antelope release, thus reverted the two patches, but we did it on stable/2023.1 branch and not on master | |
| 11:39:25 | sean-k-mooney | ack | |
| 11:39:35 | sean-k-mooney | perhaps i shoudl be diffeint agaisnt a diffent base | |
| 11:40:05 | sean-k-mooney | the failure is definatly comign form the LOG.error call | |
| 11:40:15 | sean-k-mooney | but i cant see any patch that would casue this in those diffes | |
| 11:40:27 | elodilles | yes, 5.1.0..5.3.0, but it still not shows the reverts that we only merged on stable/2023.1 :/ | |
| 11:41:35 | elodilles | there are 3 patches reverted in 5.2.0 | |
| 11:41:45 | sean-k-mooney | just so we are on the same page https://paste.opendev.org/show/bHorJ0sp6UTAKfu2EbJ2/ passes | |
| 11:42:03 | sean-k-mooney | its the LOG.error on line 34 | |
| 11:42:22 | sean-k-mooney | that is causing the issue but the message it is loging and the assert that is fialing do not make sense | |
| 11:43:42 | sean-k-mooney | no matter what i log it causes Calls: [call(7, b'X')] on os.write | |
| 11:45:30 | elodilles | sean-k-mooney: these were reverted on 5.2.0, but not in master (5.3.0): https://paste.opendev.org/show/bRU8ommJhreoVpdiWMRV/ | |
| 11:45:46 | elodilles | one of them has to be related to the issue | |
| 11:46:51 | sean-k-mooney | so the eventlet fix was added and removed fith | |
| 11:46:54 | sean-k-mooney | *right | |
| 11:47:19 | sean-k-mooney | im going to cloen oslo.log and pip install -e it into the env | |
| 11:47:25 | sean-k-mooney | and i guess do a biset effectivly | |
| 11:48:51 | frickler | so this whole issue is 6 months old and got successfully ignored? cool | |
| 11:49:42 | sean-k-mooney | no idea really. its only causing two test to fail | |
| 11:49:51 | sean-k-mooney | so i dont think its actully breaking real code | |
| 11:50:15 | sean-k-mooney | however im not parcarlly happy with just modifying the test without understadnign why this is breakign | |
| 11:50:27 | sean-k-mooney | as it really not obvious why this is happeining | |
| 11:50:58 | sean-k-mooney | and looking at the patch delta i woudl not expect this type of change between 5.2.0 and 5.3.0 | |
| 11:53:25 | elodilles | stephenfin: do you have any idea what could be a solution for this oslo.log issue? ^^^ | |
| 11:56:34 | sean-k-mooney | so its broken by 94b9dc3 | |
| 11:56:40 | frickler | sean-k-mooney: 5.2.0 is a bugfix release only done on stable/2023.1, 5.1.0 is the initial release for 2023.1 | |
| 11:56:51 | sean-k-mooney | https://github.com/openstack/oslo.log/commit/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17 | |
| 11:56:52 | elodilles | yepp | |
| 11:57:08 | frickler | so 5.1.0..5.3.0 gives a better idea of what happened in master. though iiuc the issue is in 5.0.0..5.1.0 | |
| 11:58:29 | frickler | 94b9dc3 is 5.0.1, ack | |
| 11:59:20 | sean-k-mooney | so with that calls to LOG.error are internally using os.write | |
| 11:59:32 | sean-k-mooney | presumably its realted to the lock its taking | |
| 11:59:44 | sean-k-mooney | here https://github.com/openstack/oslo.log/commit/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17#diff-2c76d41c287653560e6e84f39ce877b4f9ba33c7b36db17194e7435cd54adfb0R41 | |
| 12:00:58 | sean-k-mooney | its this call on release | |
| 12:01:04 | sean-k-mooney | https://github.com/openstack/oslo.log/commit/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17#diff-2c76d41c287653560e6e84f39ce877b4f9ba33c7b36db17194e7435cd54adfb0R108 | |
| 12:01:09 | sean-k-mooney | line 108 | |
| 12:01:19 | sean-k-mooney | that is causing teh test failure sepcifically | |
| 12:01:50 | sean-k-mooney | the next commit maks this conditionaly | |
| 12:01:52 | sean-k-mooney | https://github.com/openstack/oslo.log/commit/de615d9370681a2834cebe88acfa81b919da340c | |
| 12:02:12 | sean-k-mooney | so i guess there are two ways to fix this | |
| 12:02:25 | sean-k-mooney | i could proably disable this fix in this test | |
| 12:02:35 | sean-k-mooney | or i might be able to mock the lock | |
| 12:03:32 | sean-k-mooney | the issue is we have a StandardLogging fixture in our base test clase which handles all or most of the loging config | |
| 12:03:50 | sean-k-mooney | i can also mock the LOG var on the moduel i guess | |
| 12:04:28 | sean-k-mooney | we cocationally do that so i would just replace LOG with a mock explictly | |
| 12:05:13 | elodilles | so this issue only comes up in nova's tests, but could not cause any problem outside the test code, you mean? | |
| 12:07:03 | sean-k-mooney | its not breaking any real code | |
| 12:07:25 | sean-k-mooney | its just breaking the test case as we are mocking os.write and testign a function that uses it directly | |
| 12:07:40 | sean-k-mooney | and oslo.log is not ment ot driectly call os.write | |
| 12:07:47 | sean-k-mooney | or at least id did not previously | |
| 12:08:13 | sean-k-mooney | the "eventlet logging fix" patch add a loc that internally calls os.write | |
| 12:10:14 | elodilles | hmmmm, i see | |
| 12:11:32 | sean-k-mooney | self.useFixture(fixtures.MonkeyPatch("nova.privsep.utils.LOG", mock.Mock())) | |
| 12:11:36 | sean-k-mooney | that shoudl be the fix | |
| 12:11:55 | sean-k-mooney | ya that works | |
| 12:12:04 | sean-k-mooney | ill push that up as a patch | |
| 12:27:06 | elodilles | sean-k-mooney: thanks! \o/ | |
| 13:24:02 | opendevreview | sean mooney proposed openstack/nova master: adapt to oslo.log changes https://review.opendev.org/c/openstack/nova/+/894538 | |
| 13:24:36 | sean-k-mooney | frickler: elodilles: bauzas: ^ that should fix that specific issue | |
| 13:24:56 | bauzas | just saw the patch | |
| 13:25:50 | sean-k-mooney | there may be more generic ways to adress this but i dont really want do do anything more invaisve or global right now | |
| 13:26:22 | bauzas | sean-k-mooney: I wish oslo.log would have had an upgrades relnote | |
| 13:26:50 | sean-k-mooney | well they were not awre i guss that we were sentiive to things like calls to os.write | |
| 13:26:57 | sean-k-mooney | it does not rbeak runtime code | |
| 13:27:09 | sean-k-mooney | with that said i need to test this with our functional tests as well | |
| 13:27:15 | sean-k-mooney | so im going to do that now | |
| 13:28:54 | bauzas | cool | |
| 13:53:33 | opendevreview | Merged openstack/python-novaclient stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2 https://review.opendev.org/c/openstack/python-novaclient/+/894076 | |