Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-25
18:48:30 mriedem melwitt: you can toggle that in the dashboard
18:48:35 melwitt oh cool, I didn't know that
18:48:41 mriedem top right
18:48:45 melwitt oh I see it now
18:48:48 melwitt niiiiiice
18:48:53 mriedem looks like the neutron one is default 7 days
18:49:17 mriedem i'm more interested in why the nova graph doesn't go beyond .... oh
18:49:19 mriedem opendev rename
18:49:30 mriedem the graph won't go beyond 4/20
18:49:39 mriedem when infra smoked a bowl and renamed everything
18:49:40 melwitt yeah, bummer
18:49:51 melwitt haha
18:50:00 mriedem but the 7 day trend seems normalish
18:50:09 mriedem http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-7d&to=now
18:50:18 melwitt the cinder graph is whack
18:50:34 mriedem http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=3&from=now-7d&to=now
18:50:35 melwitt yeah 7 day looks reasonable
18:50:37 mriedem yeah...
18:50:42 mriedem not sure what's going on there
18:50:57 mriedem maybe ask eharney if he knows?
18:51:17 mriedem i don't really know why these would be different between the projects
18:51:31 dansmith cooper6581: not sure I understand the test.. "...if the base URLs are set" -- what does that mean?
18:51:36 melwitt yeah, me neither, that's what I'm stuck on
18:53:05 cooper6581 I just copied the comment from the test test_non_formatted_url_with_no_base :p - I'm assuming what is referred to as base URL ends up being CONF.transport_url (I'm totally keyboard dog right now though) https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145
18:53:47 cooper6581 this is the test a copied https://github.com/openstack/nova/blob/ca6c32f279cf62915a11b32339cbac8128a8656e/nova/tests/unit/objects/test_cell_mapping.py#L252
18:55:10 dansmith cooper6581: those assertions are just to handle a case where you have templated a url in the database but don't have anything to parse from the config to format those things.. I'm not sure what that has to do with your case
19:00:15 cooper6581 Ahh, I see. I understand better now. Thanks!
19:02:14 melwitt mriedem: hm, I'm seeing the snapshot tests failing in cinder land http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/testr_results.html.gz which looks like our glance issue we just fixed on stable recently on the surface
19:02:59 melwitt http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_16_18_32_53_335460
19:03:35 melwitt it's the same thing, but how if it's been fixed on master for a long time
19:03:51 melwitt oh that's stable/rocky
19:04:11 melwitt bad example on my part
19:06:35 dansmith cooper6581: would it be better (than what we have) if we catch ValueError and explicitly log "yo dawg, this looks like a template because it has templating characters in it, but it failed to work with python's format() so you might want to fix it" ?
19:06:40 melwitt I looked at a couple of failures on master and they looked random
19:07:32 dansmith cooper6581: we could also treat a ValueError as "probably not a template", but then someone trying to use templates loses the checking until they realize that nova is trying to connect to rabbit as "worker-{user"
19:35:31 openstackgerrit melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.opendev.org/651947
19:57:25 efried melwitt: I made a grokkable tinyurl and put it on the master. lmk if you hate it.
20:01:16 cooper6581 dansmith: I still need to double check with ccstone, but I think the code is totally fine. I think the problem was when we stepped to newton, that logic wasn't in there, so it got inserted into the DB
20:03:11 mriedem weeee https://bugs.launchpad.net/nova/+bug/1826382
20:03:12 openstack Launchpad bug 1826382 in OpenStack Compute (nova) "Updates to placement api fail if placement endpoint changes" [Undecided,Triaged] - Assigned to Liam Young (gnuoy)
20:03:14 mriedem we're too smart for our own good
20:03:24 mriedem jaypipes: ^ thoughts on how to handle and auto-detect/correct that?
20:04:12 mriedem tl;dr if don't configure nova to talk to placement with a specific endpoint url, we look it up via the service catalog and ksa; if the endpoint changes and we're using a stale client adapter, all requests fail (http to https)
20:04:19 mriedem you have to restart all the computes to fix it
20:05:19 efried mriedem: There was some attempt to address that via safe_connect
20:06:01 mriedem that probably gets a bit confusing though b/c we raise a nova-specific exception,
20:06:02 efried mriedem: One thing we could do is rebuild the adapter on SIGHUP
20:06:21 mriedem in this case ResourceProviderRetrievalFailed
20:06:25 mriedem heh, i mentioned that in the bug,
20:06:29 mriedem but if sighup were fixed
20:06:42 efried mriedem: Well, there was *one* path through @safe_connect that swallowed the error and rebuilt the adapter and retried. But I don't think it was the right one.
20:06:53 mriedem still, let's say you have 1000 computes and you change the placement endpoint, do you want to sighup 1k computes? maybe that's normal
20:07:05 mriedem right that's the EndpointNotFound case
20:07:06 efried yeah, why tf are you changing the endpoint
20:07:13 mriedem ksa doesn't raise an exception here, we just get a 400 response back
20:07:15 efried you deserve to have to sighup 1000 computes
20:07:24 efried whoah
20:07:30 mriedem true, i can't really decide if this is low priority or what
20:07:57 efried Because the endpoint is there, just no longer accepting http? Is that what you meant by (http to https)?
20:08:14 artom Yeah, I'd have thunk if you change the endpoint you put a redirect in place first
20:08:19 mriedem "In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started."
20:08:29 efried because if the old endpoint is gone gone, that should be a connect failure, which ksa converts.
20:08:36 efried Yeah, that makes... some sense I guess?
20:08:43 efried but yeah, a redirect would be nice
20:09:41 artom I don't really think it's reasonable to expect Nova to pull the endpoint list form the catalog for every single request.
20:09:42 efried mriedem: btw, I got ultra screwed on the school runs, won't be back until even later than originally anticipated.
20:09:53 efried artom: ++
20:10:04 artom What do we do for Neutron and Cinder?
20:10:06 mriedem artom: i don't think anyone is suggesting that
20:10:24 efried mriedem: actually, the fact that we get a 400 from the API is the kicker here.
20:10:28 mriedem right
20:10:36 efried If it were any kind of 500, I would say we might should do something about it.
20:10:44 mriedem we could put a decorator on the get/put/post/delete methods to sniff the response for a 400 and if "Reason: You're speaking plain HTTP to an SSL-enabled server port." is in the text, but that's super hacky
20:10:47 efried but the fact that there's still something at that endpoint, just ain't working anymore, is not our fault.
20:10:54 efried yeah, that's bs
20:11:17 jroll I've never seen an https web server return a response when you try to talk http to it
20:11:18 artom Wait, they switched to HTTPS *on the same port*?
20:11:24 artom Is that a thing?
20:11:26 jroll the client just fails to negotiate a tls connection
20:11:54 fungi i *think* you can do starttls that way, but it's more common to see it on the assigned plaintext port equivalent
20:11:57 efried artom: Suspect portless therefore using defaults (80/443)
20:12:14 efried gotta run
20:12:20 fungi also more common for other protocols than http
20:12:21 jroll huh, looking at the bug and apparently it's a thing, wtf
20:12:31 artom fungi, ah, true
20:13:38 fungi e.g., you can initiate a tls negotiation blind on the smtps port, or on the smtp port you can issue a starttls command (if its advertised on ehlo) and then start tls negotiation
20:14:21 fungi https://en.wikipedia.org/wiki/Opportunistic_TLS
20:14:52 fungi leads to https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header
20:15:47 melwitt efried: tinyurl for what now?
20:15:50 fungi so possible, and has a standard, but... not common in the wild
20:18:15 artom fungi, yeah, though in this case it's almost inverted: the server responds with a 400 because the request *isn't* HTTPS
20:18:58 fungi oh, so protocol downgrade? that's generally just considered a security vulnerability
20:19:44 fungi ahh, or client trying to make plaintext connection on tls port? i think the phrase for that is "broken configuration"
20:22:01 jroll fungi: right, I'm surprised that apache actually responds to that, rather than the tls negotiation failing
20:26:08 fungi i'd have come up with a more humorous term, but my comedy circuits are currently overridden by the fact that my mower just kicked up a clam shell, and i'm certain there's a joke in there somewhere if i can just shuck it out
20:26:44 mriedem i see what you did there
20:38:12 melwitt mriedem: I was just reading into the ML thread and looking at your results here https://review.opendev.org/654468 and wondered, don't you need LIBS_FROM_GIT=oslo.config to avoid it using the installed version from pypi? I see it checking out the commit you Depends-On in the job output but then see oslo.config installed later? http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_27
20:38:13 melwitt 0275
20:38:49 melwitt http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_270275
20:41:44 mriedem i was under the impression zuulv3 handled that for us

Earlier   Later