| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-25 | |||
| 17:44:17 | melwitt | yeah, it's not but it's all included in a separate futures package containing backports https://pypi.org/project/futures/ | |
| 17:44:19 | dansmith | which is why I said what I did | |
| 17:44:25 | dansmith | aye | |
| 17:44:43 | melwitt | yeah, I imagine the same, using same low-level primitives underneath | |
| 17:51:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert "Wait for network-vif-plugged on resize revert" https://review.opendev.org/639396 | |
| 17:51:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Run revert resize tests in nova-live-migration https://review.opendev.org/653498 | |
| 17:51:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert resize: wait for external events in compute manager https://review.opendev.org/644881 | |
| 17:51:59 | openstackgerrit | Merged openstack/os-vif master: Remove IP proxy methods https://review.opendev.org/655695 | |
| 17:57:27 | mriedem | the cancel() thing came up in kevin's changes for canceling queued live migrations, | |
| 17:57:33 | mriedem | there is probably something in the compute manager code around that | |
| 17:58:54 | mriedem | oh right cancel() just returns false if it's already executing and you can't stop it | |
| 17:59:04 | melwitt | yeah, so far I'm not seeing in the futurist source that it event does a green thread kill() when cancel() is called with the GreenThreadPoolExecutor | |
| 17:59:09 | melwitt | *even | |
| 17:59:20 | melwitt | I can't tell that cancel() ever does anything | |
| 18:03:36 | dansmith | ...because it's just a wrapper around a work queue :) | |
| 18:17:03 | melwitt | here's what cancel() does in concurrent.futures https://github.com/python/cpython/blob/d7befad328ad1a6d1f812be2bf154c1cd1e01fbc/Lib/concurrent/futures/_base.py#L352 | |
| 18:18:22 | dansmith | "I'm guessing that this cancel is actually a condition variable" | |
| 18:18:39 | dansmith | some dude said ^ | |
| 18:31:42 | melwitt | dansmith: you.... win! | |
| 18:36:46 | melwitt | I like futures, this is fun | |
| 18:40:24 | openstackgerrit | melanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells https://review.opendev.org/650172 | |
| 18:44:30 | cooper6581 | mriedem: Quick follow up for the { in the password for the transport_url yesterday during our upgrade. I don't know enough if this is a bug or a documentation issue, but I wrote a quick test that reproduces the issue - https://gist.github.com/cooper6581/467f982a0a44494ca32d0c2f755112bc | |
| 18:45:16 | efried | gibi: FYI I copied the berlin onboarding deck and updated it for denver/train https://docs.google.com/presentation/d/1W81DsZGG_bqns2JEdSKwqPCLdAQlR5gTtss-kiGT7Pk/edit?usp=sharing | |
| 18:45:43 | cooper6581 | Specifically, I don't know if we should have ever gotten into a state where transport_url was set, and the config had a non templated url. Maybe it could have been an error on our side during the upgrade v0v | |
| 18:46:56 | melwitt | lyarwood, mriedem: it's alive... http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 (and not tracking tempest-full as nicely as we would hope) | |
| 18:47:38 | mriedem | cooper6581: hmm, i'm not sure, that would require some brain power for sorting out the templated stuff which i don't have loaded up right now, dansmith might know off hand though | |
| 18:47:52 | mriedem | i have to re-learn how the templating code each time i look at it b/c i'm slow | |
| 18:47:59 | mriedem | *code works | |
| 18:48:15 | mriedem | melwitt: that's just 6 hours though | |
| 18:48:17 | melwitt | I'm gonna propose a new change to make the graphs cover more time (make look more like http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1) and add more projects and stable branches | |
| 18:48:18 | mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-30d&to=now | |
| 18:48:30 | mriedem | melwitt: you can toggle that in the dashboard | |
| 18:48:35 | melwitt | oh cool, I didn't know that | |
| 18:48:41 | mriedem | top right | |
| 18:48:45 | melwitt | oh I see it now | |
| 18:48:48 | melwitt | niiiiiice | |
| 18:48:53 | mriedem | looks like the neutron one is default 7 days | |
| 18:49:17 | mriedem | i'm more interested in why the nova graph doesn't go beyond .... oh | |
| 18:49:19 | mriedem | opendev rename | |
| 18:49:30 | mriedem | the graph won't go beyond 4/20 | |
| 18:49:39 | mriedem | when infra smoked a bowl and renamed everything | |
| 18:49:40 | melwitt | yeah, bummer | |
| 18:49:51 | melwitt | haha | |
| 18:50:00 | mriedem | but the 7 day trend seems normalish | |
| 18:50:09 | mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-7d&to=now | |
| 18:50:18 | melwitt | the cinder graph is whack | |
| 18:50:34 | mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=3&from=now-7d&to=now | |
| 18:50:35 | melwitt | yeah 7 day looks reasonable | |
| 18:50:37 | mriedem | yeah... | |
| 18:50:42 | mriedem | not sure what's going on there | |
| 18:50:57 | mriedem | maybe ask eharney if he knows? | |
| 18:51:17 | mriedem | i don't really know why these would be different between the projects | |
| 18:51:31 | dansmith | cooper6581: not sure I understand the test.. "...if the base URLs are set" -- what does that mean? | |
| 18:51:36 | melwitt | yeah, me neither, that's what I'm stuck on | |
| 18:53:05 | cooper6581 | I just copied the comment from the test test_non_formatted_url_with_no_base :p - I'm assuming what is referred to as base URL ends up being CONF.transport_url (I'm totally keyboard dog right now though) https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145 | |
| 18:53:47 | cooper6581 | this is the test a copied https://github.com/openstack/nova/blob/ca6c32f279cf62915a11b32339cbac8128a8656e/nova/tests/unit/objects/test_cell_mapping.py#L252 | |
| 18:55:10 | dansmith | cooper6581: those assertions are just to handle a case where you have templated a url in the database but don't have anything to parse from the config to format those things.. I'm not sure what that has to do with your case | |
| 19:00:15 | cooper6581 | Ahh, I see. I understand better now. Thanks! | |
| 19:02:14 | melwitt | mriedem: hm, I'm seeing the snapshot tests failing in cinder land http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/testr_results.html.gz which looks like our glance issue we just fixed on stable recently on the surface | |
| 19:02:59 | melwitt | http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_16_18_32_53_335460 | |
| 19:03:35 | melwitt | it's the same thing, but how if it's been fixed on master for a long time | |
| 19:03:51 | melwitt | oh that's stable/rocky | |
| 19:04:11 | melwitt | bad example on my part | |
| 19:06:35 | dansmith | cooper6581: would it be better (than what we have) if we catch ValueError and explicitly log "yo dawg, this looks like a template because it has templating characters in it, but it failed to work with python's format() so you might want to fix it" ? | |
| 19:06:40 | melwitt | I looked at a couple of failures on master and they looked random | |
| 19:07:32 | dansmith | cooper6581: we could also treat a ValueError as "probably not a template", but then someone trying to use templates loses the checking until they realize that nova is trying to connect to rabbit as "worker-{user" | |
| 19:35:31 | openstackgerrit | melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.opendev.org/651947 | |
| 19:57:25 | efried | melwitt: I made a grokkable tinyurl and put it on the master. lmk if you hate it. | |
| 20:01:16 | cooper6581 | dansmith: I still need to double check with ccstone, but I think the code is totally fine. I think the problem was when we stepped to newton, that logic wasn't in there, so it got inserted into the DB | |
| 20:03:11 | mriedem | weeee https://bugs.launchpad.net/nova/+bug/1826382 | |
| 20:03:12 | openstack | Launchpad bug 1826382 in OpenStack Compute (nova) "Updates to placement api fail if placement endpoint changes" [Undecided,Triaged] - Assigned to Liam Young (gnuoy) | |
| 20:03:14 | mriedem | we're too smart for our own good | |
| 20:03:24 | mriedem | jaypipes: ^ thoughts on how to handle and auto-detect/correct that? | |
| 20:04:12 | mriedem | tl;dr if don't configure nova to talk to placement with a specific endpoint url, we look it up via the service catalog and ksa; if the endpoint changes and we're using a stale client adapter, all requests fail (http to https) | |
| 20:04:19 | mriedem | you have to restart all the computes to fix it | |
| 20:05:19 | efried | mriedem: There was some attempt to address that via safe_connect | |
| 20:06:01 | mriedem | that probably gets a bit confusing though b/c we raise a nova-specific exception, | |
| 20:06:02 | efried | mriedem: One thing we could do is rebuild the adapter on SIGHUP | |
| 20:06:21 | mriedem | in this case ResourceProviderRetrievalFailed | |
| 20:06:25 | mriedem | heh, i mentioned that in the bug, | |
| 20:06:29 | mriedem | but if sighup were fixed | |
| 20:06:42 | efried | mriedem: Well, there was *one* path through @safe_connect that swallowed the error and rebuilt the adapter and retried. But I don't think it was the right one. | |
| 20:06:53 | mriedem | still, let's say you have 1000 computes and you change the placement endpoint, do you want to sighup 1k computes? maybe that's normal | |
| 20:07:05 | mriedem | right that's the EndpointNotFound case | |
| 20:07:06 | efried | yeah, why tf are you changing the endpoint | |
| 20:07:13 | mriedem | ksa doesn't raise an exception here, we just get a 400 response back | |
| 20:07:15 | efried | you deserve to have to sighup 1000 computes | |
| 20:07:24 | efried | whoah | |
| 20:07:30 | mriedem | true, i can't really decide if this is low priority or what | |
| 20:07:57 | efried | Because the endpoint is there, just no longer accepting http? Is that what you meant by (http to https)? | |
| 20:08:14 | artom | Yeah, I'd have thunk if you change the endpoint you put a redirect in place first | |
| 20:08:19 | mriedem | "In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started." | |
| 20:08:29 | efried | because if the old endpoint is gone gone, that should be a connect failure, which ksa converts. | |
| 20:08:36 | efried | Yeah, that makes... some sense I guess? | |
| 20:08:43 | efried | but yeah, a redirect would be nice | |
| 20:09:41 | artom | I don't really think it's reasonable to expect Nova to pull the endpoint list form the catalog for every single request. | |
| 20:09:42 | efried | mriedem: btw, I got ultra screwed on the school runs, won't be back until even later than originally anticipated. | |
| 20:09:53 | efried | artom: ++ | |
| 20:10:04 | artom | What do we do for Neutron and Cinder? | |
| 20:10:06 | mriedem | artom: i don't think anyone is suggesting that | |