| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-20 | |||
| 14:37:47 | gmann | sean-k-mooney: not running glance on uwsgi and rest all service on uwsgi. register service was removed much earlier | |
| 14:38:08 | gmann | sean-k-mooney: this one https://review.opendev.org/#/c/740322/10 | |
| 14:38:10 | sean-k-mooney | oh ok | |
| 14:38:47 | sean-k-mooney | this is for the multi store stuff so the glance import pipelene stuff works | |
| 14:38:50 | gmann | dansmith: setting standalone as false for legacy grenade jobs too - https://review.opendev.org/#/c/741955/ | |
| 14:39:00 | dansmith | gmann: I saw, thanks | |
| 14:39:28 | dansmith | gmann: lots of tentacles... | |
| 14:40:02 | gmann | yeah, legacy jobs fixes are always 'missed-and-fixed-later' :) | |
| 14:43:55 | dansmith | gmann: hah | |
| 14:53:28 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730313 | |
| 14:55:03 | gibi | sean-k-mooney: opened a nova bug for the novnc breakage https://bugs.launchpad.net/nova/+bug/1888237 | |
| 14:55:03 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "novan-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | |
| 14:57:43 | sean-k-mooney | lyarwood: you might be interested in ^ by the way since we were discussing the need to test new version of it last week | |
| 14:58:12 | lyarwood | sean-k-mooney: yup ack, I'll try to look this evening | |
| 14:58:27 | sean-k-mooney | in this case there has not been a new release but its now failing with tls enabled | |
| 14:59:09 | sean-k-mooney | actully | |
| 14:59:26 | sean-k-mooney | there was a new tag of novnc 6 days ago | |
| 14:59:35 | sean-k-mooney | v1.2.0 | |
| 14:59:52 | lyarwood | yeah that did drop a few days ago | |
| 15:00:00 | lyarwood | doesn't nova-next use NOVNC from source? | |
| 15:00:19 | sean-k-mooney | possibly that is what is failing | |
| 15:00:30 | sean-k-mooney | but the error seams to be in websockify | |
| 15:00:34 | gibi | we are pulling 1.1.0 https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/job-output.txt#2154 | |
| 15:00:40 | sean-k-mooney | or eventlet not sure | |
| 15:00:54 | lyarwood | kk then it's not that | |
| 15:04:03 | openstackgerrit | Sergey Galas' proposed openstack/nova stable/train: fixes broken shared live migration of a vm with a vif https://review.opendev.org/741963 | |
| 15:18:58 | jgwentworth | lyarwood: you had tried to get nova-next to work to install novnc from source in the past but didn't finish https://review.opendev.org/671490 | |
| 15:19:53 | melwitt | I think there were some issues but I don't remember now | |
| 15:34:40 | gibi | sean-k-mooney, lyarwood: I was able to reproduce https://bugs.launchpad.net/nova/+bug/1888237 in a local devstack. I've added the needed local.conf config | |
| 15:34:40 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | |
| 15:34:52 | gibi | I mean I added the local.conf info to the bug | |
| 15:35:01 | sean-k-mooney | ok so enabligh the tls proxy or is there more | |
| 15:35:33 | gibi | tls proxy and forcing devstack to configure ssl for the novncproxy | |
| 15:35:37 | sean-k-mooney | oh you need NOVA_CONSOLE_PROXY_COMPUTE_TLS=True as well | |
| 15:35:37 | gibi | USE_SSL=True | |
| 15:35:37 | gibi | ENABLED_SERVICES+=,tls-proxy | |
| 15:35:37 | gibi | GLANCE_STANDALONE=False | |
| 15:35:38 | gibi | NOVA_CONSOLE_PROXY_COMPUTE_TLS=True | |
| 15:35:43 | sean-k-mooney | yep just read the bug | |
| 15:37:46 | sean-k-mooney | given you have repoduced it i thnk its safe to move to confirmed | |
| 15:38:44 | sean-k-mooney | critical instead of high is because its a gate blocker right | |
| 15:39:49 | gibi | right, it is a gate blocker, so I set it to critical | |
| 15:40:01 | gibi | and now it is in confirmed state too | |
| 15:54:05 | melwitt | gibi: saw your latest comment on the lp bug, fwiw urllib* is mentioned on this old eventlet issue (linked by sean-k-mooney) too. so I wonder if patching it/not patching it is related | |
| 15:54:10 | melwitt | https://github.com/eventlet/eventlet/issues/201 | |
| 15:54:50 | gibi | melwitt: I think something changed the import order and now urllib3 in imported before monkey_patch | |
| 15:55:10 | melwitt | ahh | |
| 15:55:46 | melwitt | I might be remembering it wrong but I want to say that was done intentionally a long time ago.. /me looks | |
| 15:56:25 | sean-k-mooney | dont we have a log message for that | |
| 15:56:25 | gibi | could be that I'm mixing that up | |
| 15:56:46 | gibi | sean-k-mooney: yes, and that is what I find as a difference between the failing and the passing job | |
| 15:56:59 | sean-k-mooney | ah ok | |
| 15:57:19 | melwitt | oh wait sorry I misunderstood the original change | |
| 15:57:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561 | |
| 15:57:48 | lyarwood | melwitt: ^ losing my mind with this, hopefully it's slightly better this time around. | |
| 15:57:52 | melwitt | this is what I was looking at the guts of the log message https://github.com/openstack/nova/blob/b7161fe9b92f0045e97c300a80e58d32b6f49be1/nova/monkey_patch.py#L39 | |
| 15:58:34 | melwitt | this made me think we import urllib3 before patching intentionally? | |
| 16:00:00 | gibi | this is the current import order http://paste.openstack.org/show/796133/ | |
| 16:00:13 | melwitt | lyarwood: haha ok. sorry about that, not trying to cause headaches but if all 4 errors are fair game for both live/inactive detach failures then it seemed to me like all possible combos should have test cases. that's why I was asking are we holding to what the code comments say, only some errors are for live and only some are for inactive | |
| 16:01:29 | gibi | it seems eventlet imports greendns that imports requests that imports urllib3 | |
| 16:02:06 | melwitt | I see, I was about to say I didn't see us importing urllib3 directly anywhere | |
| 16:02:09 | lyarwood | melwitt: yeah sorry wasn't a complaint about your review, just a rant :) | |
| 16:02:38 | melwitt | lyarwood: heh ok :) | |
| 16:03:45 | gibi | the passing job uses dnspython===1.16.0 the failing one uses dnspython===2.0.0 | |
| 16:03:55 | gibi | and the dns package is in the import trace aboce | |
| 16:04:01 | gibi | I'm getting close... | |
| 16:06:15 | gibi | yepp pulling dnspython back to 1.16.0 resolves the novnc issue locally | |
| 16:07:00 | gibi | this was the commit bumping that version during the weekned https://review.opendev.org/#/c/741800/ | |
| 16:08:30 | melwitt | gibi: ahhh nice find! I don't think I ever would have figured it out 😆 | |
| 16:10:06 | gibi | melwitt: it was a shoot in the dark based on the import trace | |
| 16:11:28 | gibi | should I propose a pin in the global upper-constraints to fix it? | |
| 16:11:49 | sean-k-mooney | ah that was one of the packages that did change between the runs | |
| 16:13:21 | melwitt | yeah I'm not sure what else we could do, either pin globally or pin nova. I'd ask the requirements peeps: prometheanfire or smcginnis | |
| 16:13:31 | sean-k-mooney | paste.openstack.org/show/796119/ | |
| 16:13:37 | prometheanfire | wat | |
| 16:13:48 | sean-k-mooney | so we went form 1.16.0 to 2.0.0 | |
| 16:14:04 | gibi | prometheanfire: dnspython 1.16.0 -> 2.0.0 bump broke nova novncproxy | |
| 16:14:05 | melwitt | the new dnspython 2.0.0 broke our novnc ssl stuff | |
| 16:14:06 | melwitt | in a way that seems to not be our fault | |
| 16:14:19 | melwitt | https://bugs.launchpad.net/nova/+bug/1888237 | |
| 16:14:19 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,Triaged] | |
| 16:14:29 | prometheanfire | upstream bug in novnc? | |
| 16:14:45 | gibi | prometheanfire: more like a change in behavior (import order) in dnspython | |
| 16:14:53 | gibi | prometheanfire: that breaks eventlet monkey patching for nova | |
| 16:15:27 | prometheanfire | ok, upstream bug in dnspython? | |
| 16:15:46 | prometheanfire | I just want to make sure that if/when we mask this we refrence the right bug :D | |
| 16:16:17 | gibi | prometheanfire: I have no idea if dnspython intenationally change things in 2.0.0 that does not work with eventlet | |
| 16:16:53 | gibi | there is a related eventlet issue is https://github.com/eventlet/eventlet/issues/619 | |
| 16:16:59 | gibi | eventlet is incompatible with dnspython 2.0.0rc1 | |
| 16:17:24 | prometheanfire | my main requirement (hehe, get it) is that we refrence when we can remove or relax the masking | |
| 16:17:46 | gibi | prometheanfire: OK, I will reference the above eventlet issue | |
| 16:17:50 | prometheanfire | the eventlet bug looks promising | |
| 16:18:50 | gibi | prometheanfire: so, can I pin dnspython to 1.16.0 in the requirements repo by referencing the eventlet bug? | |
| 16:19:08 | prometheanfire | ya, eventlet deps on dnspython, so the bug is in eventlet | |
| 16:19:16 | prometheanfire | gibi: and mask dnspython 2.0.0 | |
| 16:20:06 | gibi | prometheanfire: ack, will propose a patch soon | |
| 16:23:13 | openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: DNM: testing focal https://review.opendev.org/741984 | |
| 16:24:31 | sean-k-mooney | gibi do you want to update https://github.com/eventlet/eventlet/issues/632 | |
| 16:24:43 | gibi | sean-k-mooney: I will do | |
| 16:24:58 | gibi | sean-k-mooney: I duplicate it to 619 | |