| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-17 | |||
| 18:44:39 | dansmith | yeah | |
| 18:45:09 | dansmith | I hadn't scrolled over enough | |
| 18:45:28 | dansmith | well, I guess that's it | |
| 18:45:34 | dansmith | and needing of a backport | |
| 18:45:38 | dansmith | I just dunno why we're never gc'ing those | |
| 18:45:41 | melwitt | ah, I had hoped it would show the scrolling I did to ppl shared with | |
| 18:46:06 | melwitt | yeah, I don't get it either. I thought it should have worked fine the way it was | |
| 18:46:30 | mriedem | melwitt: for your todo in the wip you could use oslo.cache which i think has a default timer | |
| 18:46:55 | mriedem | and can be configured for memcache | |
| 18:47:01 | dansmith | mriedem: will that put it in memcache or dogpile? | |
| 18:47:03 | dansmith | because we probably don't want that | |
| 18:47:08 | mriedem | defaults to dogpile | |
| 18:47:09 | mriedem | in memory | |
| 18:47:16 | mriedem | well, i think | |
| 18:47:31 | mriedem | https://docs.openstack.org/developer/oslo.cache/opts.html | |
| 18:47:39 | dansmith | we just don't want those shared across machines in any way, and not serialized either | |
| 18:49:09 | melwitt | mriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova | |
| 18:49:36 | dansmith | well, for the backport we can ignore the cache purge, | |
| 18:49:45 | dansmith | since the other lists aren't purged either | |
| 18:51:37 | melwitt | dansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used | |
| 18:52:23 | dansmith | right, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting | |
| 18:52:34 | dansmith | I wouldn't even hook up SIGHUP since the other lists are't either | |
| 18:53:17 | melwitt | okay | |
| 18:55:01 | dansmith | related tangent, | |
| 18:55:16 | dansmith | reporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance | |
| 18:55:29 | dansmith | i.e "ending tcp connections is >100% higher than expected" | |
| 18:55:46 | dansmith | re: the perf-monitoring forum session | |
| 18:55:58 | melwitt | yeah, I was thinking the same | |
| 18:56:30 | dansmith | I also wonder if we should open a bug against oslo-db for this | |
| 18:56:50 | melwitt | going to un-WIP the devstack patch. sdague, where is the non-csv dstat in the logs? I was wondering should we put --tcp in for both csv and non-csv? I did but I wasn't sure | |
| 18:58:26 | melwitt | dansmith: yeah, should at least ask zzzeek about it and open a bug if it's not an expected thing | |
| 19:00:16 | sdague | melwitt: it's in a screen | |
| 19:00:22 | sdague | well, it's in the journal :) | |
| 19:00:26 | sdague | but we dump it out to screen | |
| 19:00:31 | sdague | files | |
| 19:00:34 | sdague | for compat in gate runs | |
| 19:01:35 | melwitt | cool, thanks | |
| 19:06:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:06:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:06:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:06:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:06:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:06:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:06:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:10:56 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 19:11:46 | melwitt | sdague: patch has been un-WIP https://review.openstack.org/#/c/465194 | |
| 19:12:03 | sdague | melwitt: +2, thanks | |
| 19:12:33 | melwitt | thx | |
| 19:16:21 | cdent | dansmith, bauzas, edleafe I tried to leave some questions on https://review.openstack.org/#/c/465175/ to capture my concerns, but not sure if they are clear | |
| 19:17:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance https://review.openstack.org/391060 | |
| 19:40:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:40:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:40:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:40:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:40:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:40:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:40:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:45:21 | melwitt | cfriesen: had you filed a bug for the increase in DB connections in Newton? I'll need one to refer to in my patch | |
| 19:56:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 20:01:01 | openstackgerrit | melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042 | |
| 20:06:22 | zzzeek | melwitt: can you tell me about https://review.openstack.org/#/c/465042/3/nova/context.py | |
| 20:07:42 | melwitt | zzzeek: yes! I was hoping you would be able to comment on this. recently we've seen a significant increase in number of DB connections since we've been doing DB connection switching for cells | |
| 20:08:01 | zzzeek | melwitt: so...did you try changing pool_size and max_overflow ? | |
| 20:08:50 | zzzeek | melwitt: also when you say "switching" - are you retrieving independent connections per cell DB or are you "switching" a single connection to refer to different databases within a single MySQL server? | |
| 20:09:10 | zzzeek | IIUC "cells" would be in different DB servers altogether so they'd have to have independent DB connections | |
| 20:09:27 | dansmith | zzzeek: correct, we're just surprised that when we create and stop using one, they don't seem to be gc'd | |
| 20:09:35 | melwitt | zzzeek: independent connection. to make a connection we create a new oslo.db transaction context manager | |
| 20:09:42 | dansmith | zzzeek: i.e. every time we do that switch (i.e. new connection), we get a connection that never goes away | |
| 20:09:50 | zzzeek | dansmith / melwitt OK, so, the SQLAlchemy engine by default uses a connection pool | |
| 20:10:01 | zzzeek | if you don't want it to pool anything, you need to turn it off | |
| 20:10:23 | dansmith | zzzeek: the pool is keeping references to all the connections then? | |
| 20:10:51 | zzzeek | dansmith: it keeps refernces to as many connections as you've told it to. our default in oslo.db is probably 5 at the moment | |
| 20:11:10 | dansmith | zzzeek: we end up with ~1400 connections after a tempest run | |
| 20:11:38 | dansmith | I'm guessing it doesn't know that each re-connection to the same db is reusable though? so it's keeping 5 connections of every transaction manager? | |
| 20:11:50 | zzzeek | dansmith: we end up with 1.4K connections on an openstack even w/o separate cell DBs if we have dozens of processes. it's kind of a problem to which there are a few solutions | |
| 20:11:53 | dansmith | if that's the case, we should just not do that thing (i.e. do what melwitt has proposed) | |
| 20:11:57 | melwitt | zzzeek: I had thought the pool would hand back an object that is the same DB as before | |
| 20:12:10 | dansmith | zzzeek: well, it doesn't happen with melwitt's patch | |
| 20:12:25 | zzzeek | dansmith: right but, you don't need a patch. just turn down the pooling | |
| 20:12:53 | zzzeek | dansmith: with one exception, which is that to disable the pool entirely, you need to use the "nullpool" implementation and we might need to add a hook to oslo.db if you really want that. but that is not what melwitt's patch is doing. | |
| 20:13:02 | dansmith | well, we get different behavior than what we have today.. meaning we defeat the benefits of the connection pooling , if there are benefits | |
| 20:13:39 | zzzeek | dansmith: just in the last few IRC lines im not sure if all three of us are clear on every concept here, for example, I don't know what a "Transaction manager" is | |
| 20:13:52 | dansmith | zzzeek: the oslo.db context | |
| 20:14:02 | melwitt | it's the _TransactionContextManager class in oslo.db | |
| 20:14:06 | dansmith | https://review.openstack.org/#/c/465042/3/nova/context.py | |
| 20:14:10 | dansmith | L384 there ^ | |
| 20:14:12 | zzzeek | also, the idea of a "pool" is that yes, it has a connection that it will reuse as needed | |
| 20:14:34 | melwitt | okay, so I don't know what we're doing to not get the reusability | |
| 20:15:05 | dansmith | melwitt: the pool manager has to know that a repeated connection attempt is "the same" in order to re-use an existing connection | |
| 20:15:19 | dansmith | so I'm wondering if we're confusing it such that it isn't re-using every time we create a new context there | |
| 20:15:34 | zzzeek | dansmith: ok more terms that confuse me, what is "the pool manager" | |
| 20:15:35 | melwitt | right. how does it normally know that I guess? yeah | |
| 20:16:04 | dansmith | zzzeek: I'm sure I'm not using the right term.. whatever the thing is in sqla that does the pooling.. you said "nullpool" above, which would be one of those things | |
| 20:16:21 | dansmith | melwitt: well, it can do connection pooling with one context easily | |
| 20:16:31 | melwitt | right ... | |
| 20:16:39 | dansmith | and that's all we've needed in the past | |
| 20:17:01 | dansmith | but now we're creating lots of those things | |
| 20:17:03 | zzzeek | dansmith: OK. so we are talking aobut SQLAlchemy QueuePool or NullPool. it doesn't know about connection attempts being "the same" or anything like that, it basically is asked for a DB connection, and it gives you one from its pool. if it doesnt have any, it either decides to give you an ad-hoc connection based on what you've configured "max_overflow" towards, or, it makes you wait | |
| 20:17:06 | melwitt | zzzeek: this is how we establish db connections (static) and what we did to "switch" is just create a new one of those each time https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L79-L80 | |
| 20:17:07 | mriedem | i'd think it would key off the connection url | |