Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-20
15:35:33 gibi tls proxy and forcing devstack to configure ssl for the novncproxy
15:35:37 gibi GLANCE_STANDALONE=False
15:35:37 gibi ENABLED_SERVICES+=,tls-proxy
15:35:37 gibi USE_SSL=True
15:35:37 sean-k-mooney oh you need NOVA_CONSOLE_PROXY_COMPUTE_TLS=True as well
15:35:38 gibi NOVA_CONSOLE_PROXY_COMPUTE_TLS=True
15:35:43 sean-k-mooney yep just read the bug
15:37:46 sean-k-mooney given you have repoduced it i thnk its safe to move to confirmed
15:38:44 sean-k-mooney critical instead of high is because its a gate blocker right
15:39:49 gibi right, it is a gate blocker, so I set it to critical
15:40:01 gibi and now it is in confirmed state too
15:54:05 melwitt gibi: saw your latest comment on the lp bug, fwiw urllib* is mentioned on this old eventlet issue (linked by sean-k-mooney) too. so I wonder if patching it/not patching it is related
15:54:10 melwitt https://github.com/eventlet/eventlet/issues/201
15:54:50 gibi melwitt: I think something changed the import order and now urllib3 in imported before monkey_patch
15:55:10 melwitt ahh
15:55:46 melwitt I might be remembering it wrong but I want to say that was done intentionally a long time ago.. /me looks
15:56:25 gibi could be that I'm mixing that up
15:56:25 sean-k-mooney dont we have a log message for that
15:56:46 gibi sean-k-mooney: yes, and that is what I find as a difference between the failing and the passing job
15:56:59 sean-k-mooney ah ok
15:57:19 melwitt oh wait sorry I misunderstood the original change
15:57:43 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561
15:57:48 lyarwood melwitt: ^ losing my mind with this, hopefully it's slightly better this time around.
15:57:52 melwitt this is what I was looking at the guts of the log message https://github.com/openstack/nova/blob/b7161fe9b92f0045e97c300a80e58d32b6f49be1/nova/monkey_patch.py#L39
15:58:34 melwitt this made me think we import urllib3 before patching intentionally?
16:00:00 gibi this is the current import order http://paste.openstack.org/show/796133/
16:00:13 melwitt lyarwood: haha ok. sorry about that, not trying to cause headaches but if all 4 errors are fair game for both live/inactive detach failures then it seemed to me like all possible combos should have test cases. that's why I was asking are we holding to what the code comments say, only some errors are for live and only some are for inactive
16:01:29 gibi it seems eventlet imports greendns that imports requests that imports urllib3
16:02:06 melwitt I see, I was about to say I didn't see us importing urllib3 directly anywhere
16:02:09 lyarwood melwitt: yeah sorry wasn't a complaint about your review, just a rant :)
16:02:38 melwitt lyarwood: heh ok :)
16:03:45 gibi the passing job uses dnspython===1.16.0 the failing one uses dnspython===2.0.0
16:03:55 gibi and the dns package is in the import trace aboce
16:04:01 gibi I'm getting close...
16:06:15 gibi yepp pulling dnspython back to 1.16.0 resolves the novnc issue locally
16:07:00 gibi this was the commit bumping that version during the weekned https://review.opendev.org/#/c/741800/
16:08:30 melwitt gibi: ahhh nice find! I don't think I ever would have figured it out 😆
16:10:06 gibi melwitt: it was a shoot in the dark based on the import trace
16:11:28 gibi should I propose a pin in the global upper-constraints to fix it?
16:11:49 sean-k-mooney ah that was one of the packages that did change between the runs
16:13:21 melwitt yeah I'm not sure what else we could do, either pin globally or pin nova. I'd ask the requirements peeps: prometheanfire or smcginnis
16:13:31 sean-k-mooney paste.openstack.org/show/796119/
16:13:37 prometheanfire wat
16:13:48 sean-k-mooney so we went form 1.16.0 to 2.0.0
16:14:04 gibi prometheanfire: dnspython 1.16.0 -> 2.0.0 bump broke nova novncproxy
16:14:05 melwitt the new dnspython 2.0.0 broke our novnc ssl stuff
16:14:06 melwitt in a way that seems to not be our fault
16:14:19 openstack Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,Triaged]
16:14:19 melwitt https://bugs.launchpad.net/nova/+bug/1888237
16:14:29 prometheanfire upstream bug in novnc?
16:14:45 gibi prometheanfire: more like a change in behavior (import order) in dnspython
16:14:53 gibi prometheanfire: that breaks eventlet monkey patching for nova
16:15:27 prometheanfire ok, upstream bug in dnspython?
16:15:46 prometheanfire I just want to make sure that if/when we mask this we refrence the right bug :D
16:16:17 gibi prometheanfire: I have no idea if dnspython intenationally change things in 2.0.0 that does not work with eventlet
16:16:53 gibi there is a related eventlet issue is https://github.com/eventlet/eventlet/issues/619
16:16:59 gibi eventlet is incompatible with dnspython 2.0.0rc1
16:17:24 prometheanfire my main requirement (hehe, get it) is that we refrence when we can remove or relax the masking
16:17:46 gibi prometheanfire: OK, I will reference the above eventlet issue
16:17:50 prometheanfire the eventlet bug looks promising
16:18:50 gibi prometheanfire: so, can I pin dnspython to 1.16.0 in the requirements repo by referencing the eventlet bug?
16:19:08 prometheanfire ya, eventlet deps on dnspython, so the bug is in eventlet
16:19:16 prometheanfire gibi: and mask dnspython 2.0.0
16:20:06 gibi prometheanfire: ack, will propose a patch soon
16:23:13 openstackgerrit Ghanshyam Mann proposed openstack/nova-specs master: DNM: testing focal https://review.opendev.org/741984
16:24:31 sean-k-mooney gibi do you want to update https://github.com/eventlet/eventlet/issues/632
16:24:43 gibi sean-k-mooney: I will do
16:24:58 gibi sean-k-mooney: I duplicate it to 619
16:25:15 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595
16:25:16 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596
16:26:15 gibi sean-k-mooney: I think I have no rights to mark it as duplicate but I left a comment stating it
16:26:36 sean-k-mooney i can close it
16:27:11 gibi go ahead
16:27:21 sean-k-mooney just commenting on the other bug is enough to cross link them so ya its closed
16:27:30 gibi cool, thanks
16:32:22 gibi sean-k-mooney, prometheanfire, melwitt, lyarwood: pushed https://review.opendev.org/#/c/741986/
16:32:45 sean-k-mooney 27 secods ago
16:32:59 sean-k-mooney oh you pushed
16:33:05 sean-k-mooney i missread
16:33:27 sean-k-mooney i thougt you ment lyarwood pushed it and you noticed and were telling us
16:33:35 gibi sorry :)
16:39:13 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
16:39:14 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
16:39:14 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance https://review.opendev.org/714706
16:39:15 openstackgerrit Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
16:39:15 openstackgerrit Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
16:39:16 openstackgerrit Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395
16:44:24 gibi OK, I'm ending my workday see you tomorrow
16:47:12 melwitt o/
17:42:05 openstackgerrit Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934
17:42:06 openstackgerrit Stephen Finucane proposed openstack/nova master: Don't unset Instance.old_flavor, new_flavor until necessary https://review.opendev.org/741995
17:45:09 stephenfin huaqiang: Have +2d those two patches. However, we're having issues with the gate so I wouldn't recheck yet
21:34:27 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in FIP policies https://review.opendev.org/742030
21:43:18 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in baremetal nodes policies https://review.opendev.org/741716
21:43:31 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in baremetal nodes policy https://review.opendev.org/741717
21:53:22 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Correct the check_str and pass actual target in FIP pools policy https://review.opendev.org/741797
#openstack-nova - 2020-07-21
08:00:45 bauzas good morning Nova
08:02:54 gibi bauzas: good morning
08:07:46 alex_xu stephenfin: do you remember why we return cpuset string instead of json list in metadata API?https://review.opendev.org/#/c/688936/30/nova/api/metadata/base.py@453, feel like json list is more natual there
08:18:31 openstackgerrit Rico Lin proposed openstack/nova master: [TEST][DNM] Validate ARM64 Unittests https://review.opendev.org/742094

Earlier   Later