| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-20 | |||
| 07:33:36 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/741883 | |
| 07:37:40 | tony_su | gibi: hello gibi, could you spare some time reviewing provider-config-file patches as I saw its expire date is July 23. Today, I rebased them against master tip. I checked zuul failure and it's NOT related to us. | |
| 07:39:28 | gibi | tony_su: I will try but there is simply too much on my plate | |
| 07:39:33 | gibi | sorry | |
| 07:40:30 | nightmare_unreal | hello, what's the command for zuul to rebuild the patch. Something we put in patch comments and zuul perform it's checks again | |
| 07:41:25 | tony_su | gibi: got it, Thanks. I will ask alex_xu to make a round of review if he has some bandwidth. https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/provider-config-file | |
| 07:45:37 | gibi | tony_su: ack | |
| 07:47:12 | openstackgerrit | Elod Illes proposed openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730312 | |
| 07:48:30 | nightmare_unreal | found it, it was recheck | |
| 07:54:59 | alex_xu | tony_su: willl begin to review tomorrow or tonight if possible | |
| 10:14:33 | gibi | brinzhang_: hi! I left comments in https://review.opendev.org/#/c/716186 | |
| 11:16:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove two-level loop https://review.opendev.org/741920 | |
| 11:22:33 | sean-k-mooney | just an fyi i wont be around as much this week | |
| 11:58:53 | gibi | sean-k-mooney: ack | |
| 12:12:08 | gibi | does somebody knows about the failing novnc tempest test https://c50701f61b79b2f93421-0d0b23b7f62d479822a1b52b605369e3.ssl.cf2.rackcdn.com/673341/50/check/nova-next/5f7dd59/testr_results.html ? | |
| 12:21:39 | gibi | novnc proxy has a stacktrace so I have something to start with | |
| 12:21:40 | gibi | https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/screen-n-novnc-cell1.txt#29 | |
| 12:25:06 | sean-k-mooney | that looks like an interaction issue with eventlet? maybe | |
| 12:25:42 | sean-k-mooney | was there a new releae of either eventlet or websockify | |
| 12:26:26 | sean-k-mooney | not websockify anyway | |
| 12:26:33 | sean-k-mooney | that last release last august | |
| 12:27:04 | sean-k-mooney | eventlet was april | |
| 12:28:02 | openstackgerrit | Merged openstack/nova stable/pike: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/740636 | |
| 12:28:07 | openstackgerrit | Merged openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730312 | |
| 12:29:03 | openstackgerrit | Merged openstack/nova stable/train: libvirt: Mark e1000e VIF as supported https://review.opendev.org/736962 | |
| 12:30:33 | sean-k-mooney | not obvious changes in the reqruiement repo either | |
| 12:35:10 | gibi | if I understand correclty the error coming from the ssl c lib so I cannot really debug it | |
| 12:41:36 | sean-k-mooney | right so eventlet is monkeypatch the ssl socket | |
| 12:42:33 | sean-k-mooney | but it looks like the version of eventlet we are using is not handelign the wrapping correctly | |
| 12:42:40 | openstackgerrit | Merged openstack/nova stable/train: Correct reported system memory https://review.opendev.org/740667 | |
| 12:42:59 | sean-k-mooney | maybe they broke 3.6 compatiablity but there has not been a releae of eventlet in a few months | |
| 12:43:06 | sean-k-mooney | so im not sure why we would only see it now | |
| 12:43:52 | sean-k-mooney | they have had this issue in the past https://github.com/eventlet/eventlet/issues/201 | |
| 12:45:03 | sean-k-mooney | and it happened again https://github.com/eventlet/eventlet/issues/311 | |
| 12:49:02 | gibi | those fixes talk about urllib2 but as far as I see we install urllib3 | |
| 12:49:38 | sean-k-mooney | yes my point was more they have missed monkeypatching the context correctly in the past | |
| 13:01:22 | sean-k-mooney | ill open an eventlet bug and see what they think | |
| 13:01:56 | gibi | sean-k-mooney: thanks | |
| 13:02:24 | gibi | I'm still trying to figure out what changed now that break us as all the sw releases are couple months old | |
| 13:02:44 | sean-k-mooney | maybe the host openssl version | |
| 13:03:02 | sean-k-mooney | or a recent python 3.6.z release | |
| 13:03:11 | gibi | looking... | |
| 13:08:15 | gibi | 1.1.1-1ubuntu2.1~18.04.6 was used for the last passing test as well as the failing test | |
| 13:08:28 | gibi | I mean openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.6). | |
| 13:09:01 | sean-k-mooney | and the python version? | |
| 13:10:30 | gibi | that is also the same python3.6 is already the newest version (3.6.9-1~18.04ubuntu1). | |
| 13:11:09 | gibi | I'm comparing the nova-next run from here (passed) https://review.opendev.org/#/c/739209/ with here (failed) https://review.opendev.org/#/c/728481/ | |
| 13:12:45 | gibi | libssl-dev is also the same | |
| 13:27:49 | gibi | there was some image rebuild during the weekend but it only affected 3.7 and 3.8 images as far as I see https://review.opendev.org/#/c/741789/1 | |
| 13:28:24 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 | |
| 13:29:06 | sean-k-mooney | ya but most of the relevent packages seam to be the same between the passing and failing build | |
| 13:32:09 | efried | brinzhang_: Yeah, sorry, I wouldn't be able to make enough time for something the size and scope of nova/cyborg. | |
| 13:33:07 | gibi | sean-k-mooney: thanks for reporting that to eventlet | |
| 13:33:47 | sean-k-mooney | there isnt much for them to go on really but maybe that can suggest something | |
| 13:34:02 | sean-k-mooney | gibi: is this repetable by the way | |
| 13:35:23 | gibi | sean-k-mooney: I saw it in two independent job run | |
| 13:36:43 | gibi | sean-k-mooney: here is another reporduction https://zuul.opendev.org/t/openstack/build/5f7dd591c62843ce9a114276b933a932/log/controller/logs/screen-n-novnc-cell1.txt#68 | |
| 13:36:49 | gibi | from https://review.opendev.org/#/c/673341/50 | |
| 13:43:00 | gibi | I don't have SSL/TLS configured for vnc proxy in my devstack | |
| 14:05:15 | gibi | so I'm restacking right now | |
| 14:10:36 | sean-k-mooney | im not seeign it in logstash so i can see how often its happens | |
| 14:13:54 | gibi | logstash indexing is behind with about 120 hours | |
| 14:14:47 | sean-k-mooney | ok so 5 days | |
| 14:14:55 | sean-k-mooney | and the failures were 3-4 days ago | |
| 14:15:10 | sean-k-mooney | well in the previous 30 days there are no hits | |
| 14:15:38 | gibi | the last patch merged ~3 days ago | |
| 14:22:55 | gibi | sean-k-mooney: here is the nova-next job history https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master | |
| 14:23:50 | sean-k-mooney | yep but if it was in logstash i could have added an elasticrecheck query for it | |
| 14:25:16 | sean-k-mooney | i would expect this to fail on other jobs too | |
| 14:27:10 | gibi | I'm not sure where we run the tempest test test_novnc | |
| 14:27:58 | gibi | hm tempest-integrated-compute also runs that test | |
| 14:28:07 | gibi | and that is green | |
| 14:28:35 | gibi | but that runs without SSL setupo | |
| 14:28:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: tesging nova greande job https://review.opendev.org/741956 | |
| 14:28:47 | gibi | https://zuul.opendev.org/t/openstack/build/675ff69ce6624ddb83b7ff7fec1cef9b/log/controller/logs/screen-n-novnc-cell1.txt#7 | |
| 14:28:53 | gmann | gibi: ^^ testing the grenade jobs failure fix ^^ | |
| 14:28:56 | sean-k-mooney | gibi: ah right we might not have tls or run it in other jobs | |
| 14:29:08 | gibi | gmann: thanks for looking into grenade! | |
| 14:29:10 | gmann | legacy jobs were left for glance stabdalone thing | |
| 14:29:14 | gmann | standalone | |
| 14:32:02 | artom | Am I? | |
| 14:32:09 | sean-k-mooney | ^ | |
| 14:32:11 | artom | Maybe Dalone is just a prick. | |
| 14:32:20 | artom | And deserves to be stabbed? | |
| 14:32:22 | sean-k-mooney | maybe | |
| 14:34:11 | dansmith | artom: always check *after* the threat | |
| 14:34:21 | dansmith | it's funnier (for the rest of us) that way | |
| 14:34:48 | artom | You mean my usual method of 1. open mouth 2. say dumb thing 3. check 4. regret | |
| 14:35:28 | sean-k-mooney | gmann: is standalone glance just without the registry service? | |
| 14:35:41 | sean-k-mooney | or something else | |
| 14:37:47 | gmann | sean-k-mooney: not running glance on uwsgi and rest all service on uwsgi. register service was removed much earlier | |
| 14:38:08 | gmann | sean-k-mooney: this one https://review.opendev.org/#/c/740322/10 | |
| 14:38:10 | sean-k-mooney | oh ok | |
| 14:38:47 | sean-k-mooney | this is for the multi store stuff so the glance import pipelene stuff works | |
| 14:38:50 | gmann | dansmith: setting standalone as false for legacy grenade jobs too - https://review.opendev.org/#/c/741955/ | |
| 14:39:00 | dansmith | gmann: I saw, thanks | |
| 14:39:28 | dansmith | gmann: lots of tentacles... | |
| 14:40:02 | gmann | yeah, legacy jobs fixes are always 'missed-and-fixed-later' :) | |
| 14:43:55 | dansmith | gmann: hah | |
| 14:53:28 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730313 | |