Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-25
17:58:54 mriedem oh right cancel() just returns false if it's already executing and you can't stop it
17:59:04 melwitt yeah, so far I'm not seeing in the futurist source that it event does a green thread kill() when cancel() is called with the GreenThreadPoolExecutor
17:59:09 melwitt *even
17:59:20 melwitt I can't tell that cancel() ever does anything
18:03:36 dansmith ...because it's just a wrapper around a work queue :)
18:17:03 melwitt here's what cancel() does in concurrent.futures https://github.com/python/cpython/blob/d7befad328ad1a6d1f812be2bf154c1cd1e01fbc/Lib/concurrent/futures/_base.py#L352
18:18:22 dansmith "I'm guessing that this cancel is actually a condition variable"
18:18:39 dansmith some dude said ^
18:31:42 melwitt dansmith: you.... win!
18:36:46 melwitt I like futures, this is fun
18:40:24 openstackgerrit melanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells https://review.opendev.org/650172
18:44:30 cooper6581 mriedem: Quick follow up for the { in the password for the transport_url yesterday during our upgrade. I don't know enough if this is a bug or a documentation issue, but I wrote a quick test that reproduces the issue - https://gist.github.com/cooper6581/467f982a0a44494ca32d0c2f755112bc
18:45:16 efried gibi: FYI I copied the berlin onboarding deck and updated it for denver/train https://docs.google.com/presentation/d/1W81DsZGG_bqns2JEdSKwqPCLdAQlR5gTtss-kiGT7Pk/edit?usp=sharing
18:45:43 cooper6581 Specifically, I don't know if we should have ever gotten into a state where transport_url was set, and the config had a non templated url. Maybe it could have been an error on our side during the upgrade v0v
18:46:56 melwitt lyarwood, mriedem: it's alive... http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 (and not tracking tempest-full as nicely as we would hope)
18:47:38 mriedem cooper6581: hmm, i'm not sure, that would require some brain power for sorting out the templated stuff which i don't have loaded up right now, dansmith might know off hand though
18:47:52 mriedem i have to re-learn how the templating code each time i look at it b/c i'm slow
18:47:59 mriedem *code works
18:48:15 mriedem melwitt: that's just 6 hours though
18:48:17 melwitt I'm gonna propose a new change to make the graphs cover more time (make look more like http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1) and add more projects and stable branches
18:48:18 mriedem http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-30d&to=now
18:48:30 mriedem melwitt: you can toggle that in the dashboard
18:48:35 melwitt oh cool, I didn't know that
18:48:41 mriedem top right
18:48:45 melwitt oh I see it now
18:48:48 melwitt niiiiiice
18:48:53 mriedem looks like the neutron one is default 7 days
18:49:17 mriedem i'm more interested in why the nova graph doesn't go beyond .... oh
18:49:19 mriedem opendev rename
18:49:30 mriedem the graph won't go beyond 4/20
18:49:39 mriedem when infra smoked a bowl and renamed everything
18:49:40 melwitt yeah, bummer
18:49:51 melwitt haha
18:50:00 mriedem but the 7 day trend seems normalish
18:50:09 mriedem http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-7d&to=now
18:50:18 melwitt the cinder graph is whack
18:50:34 mriedem http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=3&from=now-7d&to=now
18:50:35 melwitt yeah 7 day looks reasonable
18:50:37 mriedem yeah...
18:50:42 mriedem not sure what's going on there
18:50:57 mriedem maybe ask eharney if he knows?
18:51:17 mriedem i don't really know why these would be different between the projects
18:51:31 dansmith cooper6581: not sure I understand the test.. "...if the base URLs are set" -- what does that mean?
18:51:36 melwitt yeah, me neither, that's what I'm stuck on
18:53:05 cooper6581 I just copied the comment from the test test_non_formatted_url_with_no_base :p - I'm assuming what is referred to as base URL ends up being CONF.transport_url (I'm totally keyboard dog right now though) https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145
18:53:47 cooper6581 this is the test a copied https://github.com/openstack/nova/blob/ca6c32f279cf62915a11b32339cbac8128a8656e/nova/tests/unit/objects/test_cell_mapping.py#L252
18:55:10 dansmith cooper6581: those assertions are just to handle a case where you have templated a url in the database but don't have anything to parse from the config to format those things.. I'm not sure what that has to do with your case
19:00:15 cooper6581 Ahh, I see. I understand better now. Thanks!
19:02:14 melwitt mriedem: hm, I'm seeing the snapshot tests failing in cinder land http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/testr_results.html.gz which looks like our glance issue we just fixed on stable recently on the surface
19:02:59 melwitt http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_16_18_32_53_335460
19:03:35 melwitt it's the same thing, but how if it's been fixed on master for a long time
19:03:51 melwitt oh that's stable/rocky
19:04:11 melwitt bad example on my part
19:06:35 dansmith cooper6581: would it be better (than what we have) if we catch ValueError and explicitly log "yo dawg, this looks like a template because it has templating characters in it, but it failed to work with python's format() so you might want to fix it" ?
19:06:40 melwitt I looked at a couple of failures on master and they looked random
19:07:32 dansmith cooper6581: we could also treat a ValueError as "probably not a template", but then someone trying to use templates loses the checking until they realize that nova is trying to connect to rabbit as "worker-{user"
19:35:31 openstackgerrit melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.opendev.org/651947
19:57:25 efried melwitt: I made a grokkable tinyurl and put it on the master. lmk if you hate it.
20:01:16 cooper6581 dansmith: I still need to double check with ccstone, but I think the code is totally fine. I think the problem was when we stepped to newton, that logic wasn't in there, so it got inserted into the DB
20:03:11 mriedem weeee https://bugs.launchpad.net/nova/+bug/1826382
20:03:12 openstack Launchpad bug 1826382 in OpenStack Compute (nova) "Updates to placement api fail if placement endpoint changes" [Undecided,Triaged] - Assigned to Liam Young (gnuoy)
20:03:14 mriedem we're too smart for our own good
20:03:24 mriedem jaypipes: ^ thoughts on how to handle and auto-detect/correct that?
20:04:12 mriedem tl;dr if don't configure nova to talk to placement with a specific endpoint url, we look it up via the service catalog and ksa; if the endpoint changes and we're using a stale client adapter, all requests fail (http to https)
20:04:19 mriedem you have to restart all the computes to fix it
20:05:19 efried mriedem: There was some attempt to address that via safe_connect
20:06:01 mriedem that probably gets a bit confusing though b/c we raise a nova-specific exception,
20:06:02 efried mriedem: One thing we could do is rebuild the adapter on SIGHUP
20:06:21 mriedem in this case ResourceProviderRetrievalFailed
20:06:25 mriedem heh, i mentioned that in the bug,
20:06:29 mriedem but if sighup were fixed
20:06:42 efried mriedem: Well, there was *one* path through @safe_connect that swallowed the error and rebuilt the adapter and retried. But I don't think it was the right one.
20:06:53 mriedem still, let's say you have 1000 computes and you change the placement endpoint, do you want to sighup 1k computes? maybe that's normal
20:07:05 mriedem right that's the EndpointNotFound case
20:07:06 efried yeah, why tf are you changing the endpoint
20:07:13 mriedem ksa doesn't raise an exception here, we just get a 400 response back
20:07:15 efried you deserve to have to sighup 1000 computes
20:07:24 efried whoah
20:07:30 mriedem true, i can't really decide if this is low priority or what
20:07:57 efried Because the endpoint is there, just no longer accepting http? Is that what you meant by (http to https)?
20:08:14 artom Yeah, I'd have thunk if you change the endpoint you put a redirect in place first
20:08:19 mriedem "In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started."
20:08:29 efried because if the old endpoint is gone gone, that should be a connect failure, which ksa converts.
20:08:36 efried Yeah, that makes... some sense I guess?
20:08:43 efried but yeah, a redirect would be nice
20:09:41 artom I don't really think it's reasonable to expect Nova to pull the endpoint list form the catalog for every single request.
20:09:42 efried mriedem: btw, I got ultra screwed on the school runs, won't be back until even later than originally anticipated.
20:09:53 efried artom: ++
20:10:04 artom What do we do for Neutron and Cinder?
20:10:06 mriedem artom: i don't think anyone is suggesting that
20:10:24 efried mriedem: actually, the fact that we get a 400 from the API is the kicker here.
20:10:28 mriedem right
20:10:36 efried If it were any kind of 500, I would say we might should do something about it.
20:10:44 mriedem we could put a decorator on the get/put/post/delete methods to sniff the response for a 400 and if "Reason: You're speaking plain HTTP to an SSL-enabled server port." is in the text, but that's super hacky
20:10:47 efried but the fact that there's still something at that endpoint, just ain't working anymore, is not our fault.
20:10:54 efried yeah, that's bs
20:11:17 jroll I've never seen an https web server return a response when you try to talk http to it
20:11:18 artom Wait, they switched to HTTPS *on the same port*?
20:11:24 artom Is that a thing?
20:11:26 jroll the client just fails to negotiate a tls connection

Earlier   Later