| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-24 | |||
| 17:18:15 | sean-k-mooney | could this be related to the use of pthread for the heartbeat | |
| 17:18:52 | melwitt | which heartbeat? the service heartbeats are eventlet, that I saw | |
| 17:19:15 | sean-k-mooney | the only real pthread i know of in nova are teh oslo.messaging heartbeat and the libvirt one | |
| 17:19:28 | sean-k-mooney | although no that would not make sense fo 10/13 | |
| 17:20:03 | dansmith | at one point we changed the ordering of our imports relative to the monkeypatching to "fix" something | |
| 17:20:14 | melwitt | zzzeek: yeah... dansmith pointed out that glance and cinder use eventlet and monkey patch, but yet we don't see this error from them | |
| 17:20:15 | dansmith | and I think that got backported.. I wonder if that's relevant? | |
| 17:21:00 | sean-k-mooney | melwitt: i was refering to https://github.com/openstack/oslo.messaging/blob/5aa645b38b4c1cf08b00e687eb6c7c4b8a0211fc/oslo_messaging/_drivers/impl_rabbit.py#L90-L100 | |
| 17:21:14 | melwitt | * melwitt looks for link | |
| 17:21:31 | sean-k-mooney | dansmith: mdboots change | |
| 17:21:39 | dansmith | sean-k-mooney: right | |
| 17:21:52 | melwitt | I was just looking at that earlier | |
| 17:21:59 | dansmith | sean-k-mooney: I wonder if that ended up with us getting a combination of real and green threads in a way that is problematic.. | |
| 17:22:13 | sean-k-mooney | https://github.com/openstack/nova/commit/3c5e2b0e9fac985294a949852bb8c83d4ed77e04#diff-c2e5ad6353633e738ba126e0f11ea14ed3f6ea94554deec967586fd2dfcf060d | |
| 17:22:16 | sean-k-mooney | that one | |
| 17:22:35 | melwitt | yeah that's it | |
| 17:22:58 | melwitt | sean-k-mooney: ack thanks (pthread) | |
| 17:23:16 | sean-k-mooney | dansmith: well in principal that should have moved the patching eairler so less likely to get a mix | |
| 17:23:28 | sean-k-mooney | but you are suggestign without it we still could be | |
| 17:23:31 | sean-k-mooney | if we have not backported it | |
| 17:23:34 | dansmith | well, | |
| 17:23:45 | melwitt | yeah, we did not backport it | |
| 17:23:49 | dansmith | I think in wsgi mode that will come in at the point at which we hit it due to importing that api module | |
| 17:24:15 | dansmith | melwitt: oh I thought we did.. maybe that's related to the sudden cessation of reports? :) | |
| 17:24:29 | melwitt | could be, yeah | |
| 17:24:45 | dansmith | melwitt: did you say you didn't see it at all in later releases, or just ... less? | |
| 17:25:07 | sean-k-mooney | this was merged in train | |
| 17:25:18 | melwitt | dansmith: I could not find any mention of it past queens/13 when I bugzilla searched everything under nova and pymysql | |
| 17:25:22 | sean-k-mooney | so if we did not abckprot we shoudl see it up to 15 | |
| 17:26:00 | dansmith | melwitt: that seems like it could be a strong contender for being related then | |
| 17:26:14 | sean-k-mooney | didnt we also change the mysql clint at one point | |
| 17:26:17 | melwitt | dansmith: yeah, agree | |
| 17:26:21 | dansmith | sean-k-mooney: long ago | |
| 17:26:27 | sean-k-mooney | i think pymsql is the new one right | |
| 17:26:38 | sean-k-mooney | it used to be mysql_python or something | |
| 17:26:43 | dansmith | it is, but that change was like icehouse or something I think | |
| 17:26:57 | sean-k-mooney | ya ok | |
| 17:27:53 | melwitt | I had been looking at it from the context of it also providing a way to disable monkey patching, but due to my lack of understanding of eventlet and mixing with native threads, it did not click for me to think it could have fixed things to monkey patch earlier | |
| 17:28:27 | melwitt | it makes sense when you say it now though.. | |
| 17:28:31 | dansmith | a combination of references to the un-patched library and the patched one could very much be relevant | |
| 17:28:43 | dansmith | and that's what that change was aabout | |
| 17:28:54 | dansmith | and it's also the argument against monkeypatching altogether of course :P | |
| 17:29:42 | sean-k-mooney | its ok stephenfin will reventyly get around to deleteing all the eventlet code like all the ohter stuff he has deleted :) | |
| 17:29:53 | melwitt | yeah. that change was mostly non understandable by my brain | |
| 17:29:56 | sean-k-mooney | but ya we did this for urllib3 eventully | |
| 17:30:00 | sean-k-mooney | *orginally | |
| 17:30:13 | sean-k-mooney | and som eohter service i guess but it makes sesne | |
| 17:30:40 | dansmith | urllib3 being socket-oriented, along with pymysql ... :) | |
| 17:30:58 | sean-k-mooney | we did actully backport this https://review.opendev.org/c/openstack/nova/+/647310 | |
| 17:31:02 | sean-k-mooney | but only to stien | |
| 17:31:19 | melwitt | ah, ok, my bad | |
| 17:31:27 | melwitt | I had thought it landed in stein | |
| 17:31:30 | dansmith | okay I was sure we did backport it some, but .. fair enough | |
| 17:31:32 | melwitt | (originally) | |
| 17:31:59 | sean-k-mooney | i guess that stien is where the orginal bug was reported | |
| 17:32:08 | sean-k-mooney | and we just did not bring it back before that | |
| 17:33:19 | sean-k-mooney | hum https://bugs.launchpad.net/nova/+bug/1808951 | |
| 17:33:21 | openstack | Launchpad bug 1808951 in tripleo "python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object" [High,Incomplete] | |
| 17:33:34 | sean-k-mooney | oh i miss read SSL as SQL | |
| 17:33:44 | sean-k-mooney | i was going to say it refrence SQL too | |
| 17:34:16 | melwitt | ok, I think it would be interesting if I build them a test package with that change and see if they can try it out | |
| 17:34:30 | melwitt | that would be good proof that it is/was the fix | |
| 17:34:36 | dansmith | melwitt: yeah if they're willing I think that'd be a good test | |
| 17:35:20 | melwitt | I'll get that done and give them the option | |
| 17:55:04 | sean-k-mooney | melwitt: lyarwood so it look like the new resovled is breaking lowerconstraits on stable os-vif branches | |
| 17:55:16 | sean-k-mooney | how is that adressed for stabel brances | |
| 17:55:29 | sean-k-mooney | do we update to the oldest lib that works? | |
| 17:55:51 | sean-k-mooney | hacking seams to be what is breakign things | |
| 17:56:01 | sean-k-mooney | although there coudl be other issues | |
| 17:57:16 | sean-k-mooney | ok on master stephenfin removed any non direct deps | |
| 17:57:18 | sean-k-mooney | https://github.com/openstack/os-vif/commit/44d8937148aac1a61f40e59d3271c45f9fe6aa03 | |
| 17:57:48 | sean-k-mooney | ill see if i can do somethign similar | |
| 17:58:45 | lyarwood | sean-k-mooney: ack yeah or backport a modified version of that? | |
| 17:59:13 | sean-k-mooney | well that is waht i was going to do but with lower min version to match the branch its going too | |
| 17:59:44 | sean-k-mooney | im not sure if starting from that patch will be faster or not | |
| 18:30:24 | openstackgerrit | sean mooney proposed openstack/os-vif stable/victoria: Resolve dependency issues https://review.opendev.org/c/openstack/os-vif/+/792840 | |
| 18:56:29 | sean-k-mooney | elod: lyarwood: am i allowed to squash two change in a backport upstream | |
| 18:57:23 | sean-k-mooney | basicaly i have 2 chocie squash https://review.opendev.org/c/openstack/os-vif/+/716223 and the ussuri version of https://review.opendev.org/c/openstack/os-vif/+/792840 | |
| 18:57:51 | sean-k-mooney | or i can incoperate ignoring W504 | |
| 18:58:52 | sean-k-mooney | oh wait no that is not adding W504 | |
| 18:59:02 | sean-k-mooney | that not the patch i need to add | |
| 18:59:39 | sean-k-mooney | its https://github.com/openstack/os-vif/commit/d57a5f39edcb8ef3de09e80925c8fe628e5e0f3a | |
| 19:01:15 | sean-k-mooney | but since that raise the min version hackign i cant really backport | |
| 19:02:16 | sean-k-mooney | ok ill take a look at this again tomorrow | |
| 19:06:26 | lyarwood | <sean-k-mooney "elod: lyarwood: am i allowed to "> Yes FWIW, if it fixes an otherwise unsolvable problem you can merge multiple. | |
| 19:07:18 | sean-k-mooney | i tought that i need to backport the cleanup patch and merge it with the lower constratint one but that is not what intoduced the w504 skip | |
| 19:07:35 | sean-k-mooney | lyarwood: it was the patch that bumps hacking form 1.x to 3.0 for python 3 support | |
| 19:08:26 | sean-k-mooney | ill see if i can figure out how to cap flak8 and py code style to avoid the hacking bump | |
| 19:09:06 | sean-k-mooney | but if i cant we will have to deciside if we are oke with the bump to hacking in the pep8 tox env | |
| 19:09:27 | sean-k-mooney | given it wont impackt any other test and wont be used at runtime | |
| 19:10:54 | sean-k-mooney | if we are ok with going to hacking 3.0 ill look at squashing thsoe two patches | |
| 19:11:25 | sean-k-mooney | well tomorrow im going to go get dinner now o/ | |
| 22:01:16 | openstackgerrit | Merged openstack/nova master: image_meta: Provide image_ref as the id when fetching from instance https://review.opendev.org/c/openstack/nova/+/790659 | |
| #openstack-nova - 2021-05-25 | |||
| 06:11:00 | icey | hey - any chance of getting a review on https://review.opendev.org/c/openstack/nova/+/761824/ ? It got a +1 from Zuul a while back but has been sitting in limbo since February | |
| 07:43:35 | lyarwood | icey: Apologies, I'll take a look now | |
| 07:46:55 | bauzas | morning folks | |
| 07:47:29 | bauzas | * bauzas starts his specs review day | |
| 07:47:38 | bauzas | if folks want me to look at some spec, ping me | |
| 07:58:22 | openstackgerrit | Merged openstack/nova-specs master: Allow migrating PMEM's data https://review.opendev.org/c/openstack/nova-specs/+/785563 | |
| 08:01:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/wallaby: Add regression test for bug #1928063 https://review.opendev.org/c/openstack/nova/+/792902 | |