| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-20 | |||
| 13:32:09 | efried | brinzhang_: Yeah, sorry, I wouldn't be able to make enough time for something the size and scope of nova/cyborg. | |
| 13:33:07 | gibi | sean-k-mooney: thanks for reporting that to eventlet | |
| 13:33:47 | sean-k-mooney | there isnt much for them to go on really but maybe that can suggest something | |
| 13:34:02 | sean-k-mooney | gibi: is this repetable by the way | |
| 13:35:23 | gibi | sean-k-mooney: I saw it in two independent job run | |
| 13:36:43 | gibi | sean-k-mooney: here is another reporduction https://zuul.opendev.org/t/openstack/build/5f7dd591c62843ce9a114276b933a932/log/controller/logs/screen-n-novnc-cell1.txt#68 | |
| 13:36:49 | gibi | from https://review.opendev.org/#/c/673341/50 | |
| 13:43:00 | gibi | I don't have SSL/TLS configured for vnc proxy in my devstack | |
| 14:05:15 | gibi | so I'm restacking right now | |
| 14:10:36 | sean-k-mooney | im not seeign it in logstash so i can see how often its happens | |
| 14:13:54 | gibi | logstash indexing is behind with about 120 hours | |
| 14:14:47 | sean-k-mooney | ok so 5 days | |
| 14:14:55 | sean-k-mooney | and the failures were 3-4 days ago | |
| 14:15:10 | sean-k-mooney | well in the previous 30 days there are no hits | |
| 14:15:38 | gibi | the last patch merged ~3 days ago | |
| 14:22:55 | gibi | sean-k-mooney: here is the nova-next job history https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master | |
| 14:23:50 | sean-k-mooney | yep but if it was in logstash i could have added an elasticrecheck query for it | |
| 14:25:16 | sean-k-mooney | i would expect this to fail on other jobs too | |
| 14:27:10 | gibi | I'm not sure where we run the tempest test test_novnc | |
| 14:27:58 | gibi | hm tempest-integrated-compute also runs that test | |
| 14:28:07 | gibi | and that is green | |
| 14:28:35 | gibi | but that runs without SSL setupo | |
| 14:28:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: tesging nova greande job https://review.opendev.org/741956 | |
| 14:28:47 | gibi | https://zuul.opendev.org/t/openstack/build/675ff69ce6624ddb83b7ff7fec1cef9b/log/controller/logs/screen-n-novnc-cell1.txt#7 | |
| 14:28:53 | gmann | gibi: ^^ testing the grenade jobs failure fix ^^ | |
| 14:28:56 | sean-k-mooney | gibi: ah right we might not have tls or run it in other jobs | |
| 14:29:08 | gibi | gmann: thanks for looking into grenade! | |
| 14:29:10 | gmann | legacy jobs were left for glance stabdalone thing | |
| 14:29:14 | gmann | standalone | |
| 14:32:02 | artom | Am I? | |
| 14:32:09 | sean-k-mooney | ^ | |
| 14:32:11 | artom | Maybe Dalone is just a prick. | |
| 14:32:20 | artom | And deserves to be stabbed? | |
| 14:32:22 | sean-k-mooney | maybe | |
| 14:34:11 | dansmith | artom: always check *after* the threat | |
| 14:34:21 | dansmith | it's funnier (for the rest of us) that way | |
| 14:34:48 | artom | You mean my usual method of 1. open mouth 2. say dumb thing 3. check 4. regret | |
| 14:35:28 | sean-k-mooney | gmann: is standalone glance just without the registry service? | |
| 14:35:41 | sean-k-mooney | or something else | |
| 14:37:47 | gmann | sean-k-mooney: not running glance on uwsgi and rest all service on uwsgi. register service was removed much earlier | |
| 14:38:08 | gmann | sean-k-mooney: this one https://review.opendev.org/#/c/740322/10 | |
| 14:38:10 | sean-k-mooney | oh ok | |
| 14:38:47 | sean-k-mooney | this is for the multi store stuff so the glance import pipelene stuff works | |
| 14:38:50 | gmann | dansmith: setting standalone as false for legacy grenade jobs too - https://review.opendev.org/#/c/741955/ | |
| 14:39:00 | dansmith | gmann: I saw, thanks | |
| 14:39:28 | dansmith | gmann: lots of tentacles... | |
| 14:40:02 | gmann | yeah, legacy jobs fixes are always 'missed-and-fixed-later' :) | |
| 14:43:55 | dansmith | gmann: hah | |
| 14:53:28 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730313 | |
| 14:55:03 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "novan-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | |
| 14:55:03 | gibi | sean-k-mooney: opened a nova bug for the novnc breakage https://bugs.launchpad.net/nova/+bug/1888237 | |
| 14:57:43 | sean-k-mooney | lyarwood: you might be interested in ^ by the way since we were discussing the need to test new version of it last week | |
| 14:58:12 | lyarwood | sean-k-mooney: yup ack, I'll try to look this evening | |
| 14:58:27 | sean-k-mooney | in this case there has not been a new release but its now failing with tls enabled | |
| 14:59:09 | sean-k-mooney | actully | |
| 14:59:26 | sean-k-mooney | there was a new tag of novnc 6 days ago | |
| 14:59:35 | sean-k-mooney | v1.2.0 | |
| 14:59:52 | lyarwood | yeah that did drop a few days ago | |
| 15:00:00 | lyarwood | doesn't nova-next use NOVNC from source? | |
| 15:00:19 | sean-k-mooney | possibly that is what is failing | |
| 15:00:30 | sean-k-mooney | but the error seams to be in websockify | |
| 15:00:34 | gibi | we are pulling 1.1.0 https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/job-output.txt#2154 | |
| 15:00:40 | sean-k-mooney | or eventlet not sure | |
| 15:00:54 | lyarwood | kk then it's not that | |
| 15:04:03 | openstackgerrit | Sergey Galas' proposed openstack/nova stable/train: fixes broken shared live migration of a vm with a vif https://review.opendev.org/741963 | |
| 15:18:58 | jgwentworth | lyarwood: you had tried to get nova-next to work to install novnc from source in the past but didn't finish https://review.opendev.org/671490 | |
| 15:19:53 | melwitt | I think there were some issues but I don't remember now | |
| 15:34:40 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | |
| 15:34:40 | gibi | sean-k-mooney, lyarwood: I was able to reproduce https://bugs.launchpad.net/nova/+bug/1888237 in a local devstack. I've added the needed local.conf config | |
| 15:34:52 | gibi | I mean I added the local.conf info to the bug | |
| 15:35:01 | sean-k-mooney | ok so enabligh the tls proxy or is there more | |
| 15:35:33 | gibi | tls proxy and forcing devstack to configure ssl for the novncproxy | |
| 15:35:37 | gibi | GLANCE_STANDALONE=False | |
| 15:35:37 | gibi | ENABLED_SERVICES+=,tls-proxy | |
| 15:35:37 | gibi | USE_SSL=True | |
| 15:35:37 | sean-k-mooney | oh you need NOVA_CONSOLE_PROXY_COMPUTE_TLS=True as well | |
| 15:35:38 | gibi | NOVA_CONSOLE_PROXY_COMPUTE_TLS=True | |
| 15:35:43 | sean-k-mooney | yep just read the bug | |
| 15:37:46 | sean-k-mooney | given you have repoduced it i thnk its safe to move to confirmed | |
| 15:38:44 | sean-k-mooney | critical instead of high is because its a gate blocker right | |
| 15:39:49 | gibi | right, it is a gate blocker, so I set it to critical | |
| 15:40:01 | gibi | and now it is in confirmed state too | |
| 15:54:05 | melwitt | gibi: saw your latest comment on the lp bug, fwiw urllib* is mentioned on this old eventlet issue (linked by sean-k-mooney) too. so I wonder if patching it/not patching it is related | |
| 15:54:10 | melwitt | https://github.com/eventlet/eventlet/issues/201 | |
| 15:54:50 | gibi | melwitt: I think something changed the import order and now urllib3 in imported before monkey_patch | |
| 15:55:10 | melwitt | ahh | |
| 15:55:46 | melwitt | I might be remembering it wrong but I want to say that was done intentionally a long time ago.. /me looks | |
| 15:56:25 | gibi | could be that I'm mixing that up | |
| 15:56:25 | sean-k-mooney | dont we have a log message for that | |
| 15:56:46 | gibi | sean-k-mooney: yes, and that is what I find as a difference between the failing and the passing job | |
| 15:56:59 | sean-k-mooney | ah ok | |
| 15:57:19 | melwitt | oh wait sorry I misunderstood the original change | |
| 15:57:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561 | |
| 15:57:48 | lyarwood | melwitt: ^ losing my mind with this, hopefully it's slightly better this time around. | |
| 15:57:52 | melwitt | this is what I was looking at the guts of the log message https://github.com/openstack/nova/blob/b7161fe9b92f0045e97c300a80e58d32b6f49be1/nova/monkey_patch.py#L39 | |
| 15:58:34 | melwitt | this made me think we import urllib3 before patching intentionally? | |
| 16:00:00 | gibi | this is the current import order http://paste.openstack.org/show/796133/ | |
| 16:00:13 | melwitt | lyarwood: haha ok. sorry about that, not trying to cause headaches but if all 4 errors are fair game for both live/inactive detach failures then it seemed to me like all possible combos should have test cases. that's why I was asking are we holding to what the code comments say, only some errors are for live and only some are for inactive | |
| 16:01:29 | gibi | it seems eventlet imports greendns that imports requests that imports urllib3 | |
| 16:02:06 | melwitt | I see, I was about to say I didn't see us importing urllib3 directly anywhere | |