| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-20 | |||
| 12:28:02 | openstackgerrit | Merged openstack/nova stable/pike: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/740636 | |
| 12:28:07 | openstackgerrit | Merged openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730312 | |
| 12:29:03 | openstackgerrit | Merged openstack/nova stable/train: libvirt: Mark e1000e VIF as supported https://review.opendev.org/736962 | |
| 12:30:33 | sean-k-mooney | not obvious changes in the reqruiement repo either | |
| 12:35:10 | gibi | if I understand correclty the error coming from the ssl c lib so I cannot really debug it | |
| 12:41:36 | sean-k-mooney | right so eventlet is monkeypatch the ssl socket | |
| 12:42:33 | sean-k-mooney | but it looks like the version of eventlet we are using is not handelign the wrapping correctly | |
| 12:42:40 | openstackgerrit | Merged openstack/nova stable/train: Correct reported system memory https://review.opendev.org/740667 | |
| 12:42:59 | sean-k-mooney | maybe they broke 3.6 compatiablity but there has not been a releae of eventlet in a few months | |
| 12:43:06 | sean-k-mooney | so im not sure why we would only see it now | |
| 12:43:52 | sean-k-mooney | they have had this issue in the past https://github.com/eventlet/eventlet/issues/201 | |
| 12:45:03 | sean-k-mooney | and it happened again https://github.com/eventlet/eventlet/issues/311 | |
| 12:49:02 | gibi | those fixes talk about urllib2 but as far as I see we install urllib3 | |
| 12:49:38 | sean-k-mooney | yes my point was more they have missed monkeypatching the context correctly in the past | |
| 13:01:22 | sean-k-mooney | ill open an eventlet bug and see what they think | |
| 13:01:56 | gibi | sean-k-mooney: thanks | |
| 13:02:24 | gibi | I'm still trying to figure out what changed now that break us as all the sw releases are couple months old | |
| 13:02:44 | sean-k-mooney | maybe the host openssl version | |
| 13:03:02 | sean-k-mooney | or a recent python 3.6.z release | |
| 13:03:11 | gibi | looking... | |
| 13:08:15 | gibi | 1.1.1-1ubuntu2.1~18.04.6 was used for the last passing test as well as the failing test | |
| 13:08:28 | gibi | I mean openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.6). | |
| 13:09:01 | sean-k-mooney | and the python version? | |
| 13:10:30 | gibi | that is also the same python3.6 is already the newest version (3.6.9-1~18.04ubuntu1). | |
| 13:11:09 | gibi | I'm comparing the nova-next run from here (passed) https://review.opendev.org/#/c/739209/ with here (failed) https://review.opendev.org/#/c/728481/ | |
| 13:12:45 | gibi | libssl-dev is also the same | |
| 13:27:49 | gibi | there was some image rebuild during the weekend but it only affected 3.7 and 3.8 images as far as I see https://review.opendev.org/#/c/741789/1 | |
| 13:28:24 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 | |
| 13:29:06 | sean-k-mooney | ya but most of the relevent packages seam to be the same between the passing and failing build | |
| 13:32:09 | efried | brinzhang_: Yeah, sorry, I wouldn't be able to make enough time for something the size and scope of nova/cyborg. | |
| 13:33:07 | gibi | sean-k-mooney: thanks for reporting that to eventlet | |
| 13:33:47 | sean-k-mooney | there isnt much for them to go on really but maybe that can suggest something | |
| 13:34:02 | sean-k-mooney | gibi: is this repetable by the way | |
| 13:35:23 | gibi | sean-k-mooney: I saw it in two independent job run | |
| 13:36:43 | gibi | sean-k-mooney: here is another reporduction https://zuul.opendev.org/t/openstack/build/5f7dd591c62843ce9a114276b933a932/log/controller/logs/screen-n-novnc-cell1.txt#68 | |
| 13:36:49 | gibi | from https://review.opendev.org/#/c/673341/50 | |
| 13:43:00 | gibi | I don't have SSL/TLS configured for vnc proxy in my devstack | |
| 14:05:15 | gibi | so I'm restacking right now | |
| 14:10:36 | sean-k-mooney | im not seeign it in logstash so i can see how often its happens | |
| 14:13:54 | gibi | logstash indexing is behind with about 120 hours | |
| 14:14:47 | sean-k-mooney | ok so 5 days | |
| 14:14:55 | sean-k-mooney | and the failures were 3-4 days ago | |
| 14:15:10 | sean-k-mooney | well in the previous 30 days there are no hits | |
| 14:15:38 | gibi | the last patch merged ~3 days ago | |
| 14:22:55 | gibi | sean-k-mooney: here is the nova-next job history https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master | |
| 14:23:50 | sean-k-mooney | yep but if it was in logstash i could have added an elasticrecheck query for it | |
| 14:25:16 | sean-k-mooney | i would expect this to fail on other jobs too | |
| 14:27:10 | gibi | I'm not sure where we run the tempest test test_novnc | |
| 14:27:58 | gibi | hm tempest-integrated-compute also runs that test | |
| 14:28:07 | gibi | and that is green | |
| 14:28:35 | gibi | but that runs without SSL setupo | |
| 14:28:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: tesging nova greande job https://review.opendev.org/741956 | |
| 14:28:47 | gibi | https://zuul.opendev.org/t/openstack/build/675ff69ce6624ddb83b7ff7fec1cef9b/log/controller/logs/screen-n-novnc-cell1.txt#7 | |
| 14:28:53 | gmann | gibi: ^^ testing the grenade jobs failure fix ^^ | |
| 14:28:56 | sean-k-mooney | gibi: ah right we might not have tls or run it in other jobs | |
| 14:29:08 | gibi | gmann: thanks for looking into grenade! | |
| 14:29:10 | gmann | legacy jobs were left for glance stabdalone thing | |
| 14:29:14 | gmann | standalone | |
| 14:32:02 | artom | Am I? | |
| 14:32:09 | sean-k-mooney | ^ | |
| 14:32:11 | artom | Maybe Dalone is just a prick. | |
| 14:32:20 | artom | And deserves to be stabbed? | |
| 14:32:22 | sean-k-mooney | maybe | |
| 14:34:11 | dansmith | artom: always check *after* the threat | |
| 14:34:21 | dansmith | it's funnier (for the rest of us) that way | |
| 14:34:48 | artom | You mean my usual method of 1. open mouth 2. say dumb thing 3. check 4. regret | |
| 14:35:28 | sean-k-mooney | gmann: is standalone glance just without the registry service? | |
| 14:35:41 | sean-k-mooney | or something else | |
| 14:37:47 | gmann | sean-k-mooney: not running glance on uwsgi and rest all service on uwsgi. register service was removed much earlier | |
| 14:38:08 | gmann | sean-k-mooney: this one https://review.opendev.org/#/c/740322/10 | |
| 14:38:10 | sean-k-mooney | oh ok | |
| 14:38:47 | sean-k-mooney | this is for the multi store stuff so the glance import pipelene stuff works | |
| 14:38:50 | gmann | dansmith: setting standalone as false for legacy grenade jobs too - https://review.opendev.org/#/c/741955/ | |
| 14:39:00 | dansmith | gmann: I saw, thanks | |
| 14:39:28 | dansmith | gmann: lots of tentacles... | |
| 14:40:02 | gmann | yeah, legacy jobs fixes are always 'missed-and-fixed-later' :) | |
| 14:43:55 | dansmith | gmann: hah | |
| 14:53:28 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730313 | |
| 14:55:03 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "novan-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | |
| 14:55:03 | gibi | sean-k-mooney: opened a nova bug for the novnc breakage https://bugs.launchpad.net/nova/+bug/1888237 | |
| 14:57:43 | sean-k-mooney | lyarwood: you might be interested in ^ by the way since we were discussing the need to test new version of it last week | |
| 14:58:12 | lyarwood | sean-k-mooney: yup ack, I'll try to look this evening | |
| 14:58:27 | sean-k-mooney | in this case there has not been a new release but its now failing with tls enabled | |
| 14:59:09 | sean-k-mooney | actully | |
| 14:59:26 | sean-k-mooney | there was a new tag of novnc 6 days ago | |
| 14:59:35 | sean-k-mooney | v1.2.0 | |
| 14:59:52 | lyarwood | yeah that did drop a few days ago | |
| 15:00:00 | lyarwood | doesn't nova-next use NOVNC from source? | |
| 15:00:19 | sean-k-mooney | possibly that is what is failing | |
| 15:00:30 | sean-k-mooney | but the error seams to be in websockify | |
| 15:00:34 | gibi | we are pulling 1.1.0 https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/job-output.txt#2154 | |
| 15:00:40 | sean-k-mooney | or eventlet not sure | |
| 15:00:54 | lyarwood | kk then it's not that | |
| 15:04:03 | openstackgerrit | Sergey Galas' proposed openstack/nova stable/train: fixes broken shared live migration of a vm with a vif https://review.opendev.org/741963 | |
| 15:18:58 | jgwentworth | lyarwood: you had tried to get nova-next to work to install novnc from source in the past but didn't finish https://review.opendev.org/671490 | |
| 15:19:53 | melwitt | I think there were some issues but I don't remember now | |
| 15:34:40 | openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | |
| 15:34:40 | gibi | sean-k-mooney, lyarwood: I was able to reproduce https://bugs.launchpad.net/nova/+bug/1888237 in a local devstack. I've added the needed local.conf config | |
| 15:34:52 | gibi | I mean I added the local.conf info to the bug | |
| 15:35:01 | sean-k-mooney | ok so enabligh the tls proxy or is there more | |