Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-25
20:03:11 mriedem weeee https://bugs.launchpad.net/nova/+bug/1826382
20:03:12 openstack Launchpad bug 1826382 in OpenStack Compute (nova) "Updates to placement api fail if placement endpoint changes" [Undecided,Triaged] - Assigned to Liam Young (gnuoy)
20:03:14 mriedem we're too smart for our own good
20:03:24 mriedem jaypipes: ^ thoughts on how to handle and auto-detect/correct that?
20:04:12 mriedem tl;dr if don't configure nova to talk to placement with a specific endpoint url, we look it up via the service catalog and ksa; if the endpoint changes and we're using a stale client adapter, all requests fail (http to https)
20:04:19 mriedem you have to restart all the computes to fix it
20:05:19 efried mriedem: There was some attempt to address that via safe_connect
20:06:01 mriedem that probably gets a bit confusing though b/c we raise a nova-specific exception,
20:06:02 efried mriedem: One thing we could do is rebuild the adapter on SIGHUP
20:06:21 mriedem in this case ResourceProviderRetrievalFailed
20:06:25 mriedem heh, i mentioned that in the bug,
20:06:29 mriedem but if sighup were fixed
20:06:42 efried mriedem: Well, there was *one* path through @safe_connect that swallowed the error and rebuilt the adapter and retried. But I don't think it was the right one.
20:06:53 mriedem still, let's say you have 1000 computes and you change the placement endpoint, do you want to sighup 1k computes? maybe that's normal
20:07:05 mriedem right that's the EndpointNotFound case
20:07:06 efried yeah, why tf are you changing the endpoint
20:07:13 mriedem ksa doesn't raise an exception here, we just get a 400 response back
20:07:15 efried you deserve to have to sighup 1000 computes
20:07:24 efried whoah
20:07:30 mriedem true, i can't really decide if this is low priority or what
20:07:57 efried Because the endpoint is there, just no longer accepting http? Is that what you meant by (http to https)?
20:08:14 artom Yeah, I'd have thunk if you change the endpoint you put a redirect in place first
20:08:19 mriedem "In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started."
20:08:29 efried because if the old endpoint is gone gone, that should be a connect failure, which ksa converts.
20:08:36 efried Yeah, that makes... some sense I guess?
20:08:43 efried but yeah, a redirect would be nice
20:09:41 artom I don't really think it's reasonable to expect Nova to pull the endpoint list form the catalog for every single request.
20:09:42 efried mriedem: btw, I got ultra screwed on the school runs, won't be back until even later than originally anticipated.
20:09:53 efried artom: ++
20:10:04 artom What do we do for Neutron and Cinder?
20:10:06 mriedem artom: i don't think anyone is suggesting that
20:10:24 efried mriedem: actually, the fact that we get a 400 from the API is the kicker here.
20:10:28 mriedem right
20:10:36 efried If it were any kind of 500, I would say we might should do something about it.
20:10:44 mriedem we could put a decorator on the get/put/post/delete methods to sniff the response for a 400 and if "Reason: You're speaking plain HTTP to an SSL-enabled server port." is in the text, but that's super hacky
20:10:47 efried but the fact that there's still something at that endpoint, just ain't working anymore, is not our fault.
20:10:54 efried yeah, that's bs
20:11:17 jroll I've never seen an https web server return a response when you try to talk http to it
20:11:18 artom Wait, they switched to HTTPS *on the same port*?
20:11:24 artom Is that a thing?
20:11:26 jroll the client just fails to negotiate a tls connection
20:11:54 fungi i *think* you can do starttls that way, but it's more common to see it on the assigned plaintext port equivalent
20:11:57 efried artom: Suspect portless therefore using defaults (80/443)
20:12:14 efried gotta run
20:12:20 fungi also more common for other protocols than http
20:12:21 jroll huh, looking at the bug and apparently it's a thing, wtf
20:12:31 artom fungi, ah, true
20:13:38 fungi e.g., you can initiate a tls negotiation blind on the smtps port, or on the smtp port you can issue a starttls command (if its advertised on ehlo) and then start tls negotiation
20:14:21 fungi https://en.wikipedia.org/wiki/Opportunistic_TLS
20:14:52 fungi leads to https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header
20:15:47 melwitt efried: tinyurl for what now?
20:15:50 fungi so possible, and has a standard, but... not common in the wild
20:18:15 artom fungi, yeah, though in this case it's almost inverted: the server responds with a 400 because the request *isn't* HTTPS
20:18:58 fungi oh, so protocol downgrade? that's generally just considered a security vulnerability
20:19:44 fungi ahh, or client trying to make plaintext connection on tls port? i think the phrase for that is "broken configuration"
20:22:01 jroll fungi: right, I'm surprised that apache actually responds to that, rather than the tls negotiation failing
20:26:08 fungi i'd have come up with a more humorous term, but my comedy circuits are currently overridden by the fact that my mower just kicked up a clam shell, and i'm certain there's a joke in there somewhere if i can just shuck it out
20:26:44 mriedem i see what you did there
20:38:12 melwitt mriedem: I was just reading into the ML thread and looking at your results here https://review.opendev.org/654468 and wondered, don't you need LIBS_FROM_GIT=oslo.config to avoid it using the installed version from pypi? I see it checking out the commit you Depends-On in the job output but then see oslo.config installed later? http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_27
20:38:13 melwitt 0275
20:38:49 melwitt http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_270275
20:41:44 mriedem i was under the impression zuulv3 handled that for us
20:42:07 mriedem but maybe that's only if it's in required_projects in zuul.yaml
20:43:01 mriedem pip freeze is showing a tag rather than a git hash though http://logs.openstack.org/68/654468/6/check/openstack-tox-py36/6c5ead4/tox/py36-4.log
20:43:05 mriedem oslo.config==6.8.1
20:44:33 melwitt yeah that was my concern
20:45:08 mriedem not sure what this is about "cannot set the recursion limit to 10 at the recursion depth 55: the limit is too low"
20:45:20 mriedem the python docs don't say anything about a lower bound
20:46:34 melwitt where are you seeing that?
20:46:41 mriedem http://logs.openstack.org/68/654468/6/check/openstack-tox-py36/6c5ead4/testr_results.html.gz
20:47:19 melwitt oh different job
20:48:19 mriedem yeah the bug only hits on py36 for some reason
20:49:05 fungi mriedem: melwitt: yes, a repository will only be checked out alongside if it's in the required-projects list
20:49:24 mriedem ack, ok, that's easily fixable in this hack patch
20:49:59 fungi otherwise depends-on means "don't merge until this other change merges"
20:50:45 fungi (which also applies to a required project, but an rp will also be tested in conjunction with your change)
20:51:14 melwitt mriedem: I guess the doc does say "Changed in version 3.5.1: A RecursionError exception is now raised if the new limit is too low at the current recursion depth." https://docs.python.org/3/library/sys.html#sys.setrecursionlimit
20:51:26 melwitt so it has to be at least 55?
20:51:32 melwitt based on the error message it put
20:52:07 mriedem hmm
20:52:07 mriedem b'RecursionError: cannot set the recursion limit to 56 at the recursion depth 55: the limit is too low'
20:52:22 melwitt lol, what are we supposed to
20:52:27 melwitt do
20:53:50 melwitt I guess that means at that moment, the depth is 55
20:54:03 mriedem yeah
20:54:09 mriedem i bumped to 100 and then it passed
20:54:09 melwitt so the limit has to be some headroom above that to allow for more
20:54:16 melwitt ok, cool
20:54:29 mriedem it was 1000 in my local py3.6
20:54:39 mriedem i mean the default or whatever
20:54:41 mriedem which is a bionic node
20:54:57 melwitt ok, I see
20:59:36 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468
20:59:41 mriedem fungi: melwitt: ^ hopefully that does it
21:00:25 mriedem nova meeting
21:00:39 melwitt good call excluding all the other jobs too
21:27:02 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Move legacy-grenade-dsvm-neutron-multinode-live-migration in-tree https://review.opendev.org/640207
21:28:49 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove deprecated 'default_flavor' config option https://review.opendev.org/645476
21:29:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.opendev.org/637716
21:31:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Fix nova-grenade-live-migration run book for opendev migration https://review.opendev.org/655800

Earlier   Later