| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-17 | |||
| 18:37:39 | edleafe | cdent: my understanding is that if N instances are requested, we'd pick N hosts from the list of weighed hosts. Then for each, we'd get the cell uuid and from the remaining, find enough in the same cell to satisfy the number of alternates needed | |
| 18:37:49 | gyee_ | mriedem: two bugs then | |
| 18:37:50 | mriedem | so if you're using microversion<2.29 then force is None | |
| 18:37:52 | mriedem | yup | |
| 18:38:01 | EmilienM | cdent: I need to go now, but you can PM me if you have more comments, i'll catch up later | |
| 18:38:03 | edleafe | So I didn't envision overlap at all | |
| 18:38:26 | edleafe | but I suppose it wouldn't be terrible, since the claim would simply fail on a retry | |
| 18:40:09 | melwitt | dansmith, sdague: I created a WIP devstack patch to add the --tcp option to dstat and added Depends-On it for the caching db connections patch, and compared the output (using active tcp as a hint to db connections) | |
| 18:40:30 | sdague | melwitt: cool, honestly, just add that if you like | |
| 18:40:35 | melwitt | I put them in a spreadsheet here https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing | |
| 18:40:52 | sdague | dstat is really there to cross check all these things for runs, so no reason to not report it all the time | |
| 18:42:31 | melwitt | cool. it makes the csv less human readable but I think it's probably worth it for us to be able to have a high level indicator of how many connections we're using http://logs.openstack.org/94/465194/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a6b700/logs/dstat-csv_log.txt.gz | |
| 18:43:14 | dansmith | melwitt: not sure I get it.. both tabs top out at ~40 right? | |
| 18:43:16 | sdague | yeh, the non csv version is still emitted as well | |
| 18:44:24 | dansmith | oh | |
| 18:44:33 | melwitt | dansmith: that's listening. I highlighted the "active/established" column, col AC. comparing a gate-tempest-dsvm-neutron-full-ubuntu-xenial in both cases, without caching active connections were 1495 at the end of the run, with caching it was 445 | |
| 18:44:39 | dansmith | yeah | |
| 18:45:09 | dansmith | I hadn't scrolled over enough | |
| 18:45:28 | dansmith | well, I guess that's it | |
| 18:45:34 | dansmith | and needing of a backport | |
| 18:45:38 | dansmith | I just dunno why we're never gc'ing those | |
| 18:45:41 | melwitt | ah, I had hoped it would show the scrolling I did to ppl shared with | |
| 18:46:06 | melwitt | yeah, I don't get it either. I thought it should have worked fine the way it was | |
| 18:46:30 | mriedem | melwitt: for your todo in the wip you could use oslo.cache which i think has a default timer | |
| 18:46:55 | mriedem | and can be configured for memcache | |
| 18:47:01 | dansmith | mriedem: will that put it in memcache or dogpile? | |
| 18:47:03 | dansmith | because we probably don't want that | |
| 18:47:08 | mriedem | defaults to dogpile | |
| 18:47:09 | mriedem | in memory | |
| 18:47:16 | mriedem | well, i think | |
| 18:47:31 | mriedem | https://docs.openstack.org/developer/oslo.cache/opts.html | |
| 18:47:39 | dansmith | we just don't want those shared across machines in any way, and not serialized either | |
| 18:49:09 | melwitt | mriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova | |
| 18:49:36 | dansmith | well, for the backport we can ignore the cache purge, | |
| 18:49:45 | dansmith | since the other lists aren't purged either | |
| 18:51:37 | melwitt | dansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used | |
| 18:52:23 | dansmith | right, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting | |
| 18:52:34 | dansmith | I wouldn't even hook up SIGHUP since the other lists are't either | |
| 18:53:17 | melwitt | okay | |
| 18:55:01 | dansmith | related tangent, | |
| 18:55:16 | dansmith | reporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance | |
| 18:55:29 | dansmith | i.e "ending tcp connections is >100% higher than expected" | |
| 18:55:46 | dansmith | re: the perf-monitoring forum session | |
| 18:55:58 | melwitt | yeah, I was thinking the same | |
| 18:56:30 | dansmith | I also wonder if we should open a bug against oslo-db for this | |
| 18:56:50 | melwitt | going to un-WIP the devstack patch. sdague, where is the non-csv dstat in the logs? I was wondering should we put --tcp in for both csv and non-csv? I did but I wasn't sure | |
| 18:58:26 | melwitt | dansmith: yeah, should at least ask zzzeek about it and open a bug if it's not an expected thing | |
| 19:00:16 | sdague | melwitt: it's in a screen | |
| 19:00:22 | sdague | well, it's in the journal :) | |
| 19:00:26 | sdague | but we dump it out to screen | |
| 19:00:31 | sdague | files | |
| 19:00:34 | sdague | for compat in gate runs | |
| 19:01:35 | melwitt | cool, thanks | |
| 19:06:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:06:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:06:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:06:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:06:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:06:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:06:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:10:56 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 19:11:46 | melwitt | sdague: patch has been un-WIP https://review.openstack.org/#/c/465194 | |
| 19:12:03 | sdague | melwitt: +2, thanks | |
| 19:12:33 | melwitt | thx | |
| 19:16:21 | cdent | dansmith, bauzas, edleafe I tried to leave some questions on https://review.openstack.org/#/c/465175/ to capture my concerns, but not sure if they are clear | |
| 19:17:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance https://review.openstack.org/391060 | |
| 19:40:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:40:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:40:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:40:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:40:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:40:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:40:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:45:21 | melwitt | cfriesen: had you filed a bug for the increase in DB connections in Newton? I'll need one to refer to in my patch | |
| 19:56:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 20:01:01 | openstackgerrit | melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042 | |
| 20:06:22 | zzzeek | melwitt: can you tell me about https://review.openstack.org/#/c/465042/3/nova/context.py | |
| 20:07:42 | melwitt | zzzeek: yes! I was hoping you would be able to comment on this. recently we've seen a significant increase in number of DB connections since we've been doing DB connection switching for cells | |
| 20:08:01 | zzzeek | melwitt: so...did you try changing pool_size and max_overflow ? | |
| 20:08:50 | zzzeek | melwitt: also when you say "switching" - are you retrieving independent connections per cell DB or are you "switching" a single connection to refer to different databases within a single MySQL server? | |
| 20:09:10 | zzzeek | IIUC "cells" would be in different DB servers altogether so they'd have to have independent DB connections | |
| 20:09:27 | dansmith | zzzeek: correct, we're just surprised that when we create and stop using one, they don't seem to be gc'd | |
| 20:09:35 | melwitt | zzzeek: independent connection. to make a connection we create a new oslo.db transaction context manager | |
| 20:09:42 | dansmith | zzzeek: i.e. every time we do that switch (i.e. new connection), we get a connection that never goes away | |
| 20:09:50 | zzzeek | dansmith / melwitt OK, so, the SQLAlchemy engine by default uses a connection pool | |
| 20:10:01 | zzzeek | if you don't want it to pool anything, you need to turn it off | |
| 20:10:23 | dansmith | zzzeek: the pool is keeping references to all the connections then? | |
| 20:10:51 | zzzeek | dansmith: it keeps refernces to as many connections as you've told it to. our default in oslo.db is probably 5 at the moment | |
| 20:11:10 | dansmith | zzzeek: we end up with ~1400 connections after a tempest run | |
| 20:11:38 | dansmith | I'm guessing it doesn't know that each re-connection to the same db is reusable though? so it's keeping 5 connections of every transaction manager? | |
| 20:11:50 | zzzeek | dansmith: we end up with 1.4K connections on an openstack even w/o separate cell DBs if we have dozens of processes. it's kind of a problem to which there are a few solutions | |
| 20:11:53 | dansmith | if that's the case, we should just not do that thing (i.e. do what melwitt has proposed) | |
| 20:11:57 | melwitt | zzzeek: I had thought the pool would hand back an object that is the same DB as before | |
| 20:12:10 | dansmith | zzzeek: well, it doesn't happen with melwitt's patch | |
| 20:12:25 | zzzeek | dansmith: right but, you don't need a patch. just turn down the pooling | |
| 20:12:53 | zzzeek | dansmith: with one exception, which is that to disable the pool entirely, you need to use the "nullpool" implementation and we might need to add a hook to oslo.db if you really want that. but that is not what melwitt's patch is doing. | |
| 20:13:02 | dansmith | well, we get different behavior than what we have today.. meaning we defeat the benefits of the connection pooling , if there are benefits | |
| 20:13:39 | zzzeek | dansmith: just in the last few IRC lines im not sure if all three of us are clear on every concept here, for example, I don't know what a "Transaction manager" is | |
| 20:13:52 | dansmith | zzzeek: the oslo.db context | |
| 20:14:02 | melwitt | it's the _TransactionContextManager class in oslo.db | |