| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-17 | |||
| 18:31:49 | dansmith | I dunno, I guess I'd wait for either bauzas or johnthetubaguy to comment on that | |
| 18:32:24 | cdent | thanks dansmith, I'll have to read through it a bit more | |
| 18:32:31 | cdent | it feels like it could be simpler, but feelings may not matter | |
| 18:33:16 | dansmith | well, FWIW, when I first read that code I was kinda horrified with the nested dicting, | |
| 18:33:32 | dansmith | but I didn't complain because I think it probably makes the smallest behavioral change | |
| 18:33:52 | dansmith | it's more code, but I'm not sure it's _that_ much more complex | |
| 18:34:20 | dansmith | it's not an unreasonable opinion though, IMHO | |
| 18:34:44 | dansmith | recording it, even if it is shot down with technical reasons is probably a good idea | |
| 18:34:57 | cdent | roger that | |
| 18:35:04 | gyee_ | mriedem: maybe none != False? | |
| 18:35:06 | EmilienM | mriedem, cdent: super quick fyi: I tested again nova/wsgi and it works fine | |
| 18:35:21 | cdent | yay! | |
| 18:35:30 | EmilienM | this is our config: http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache_config/10-nova_api_wsgi.conf.txt.gz | |
| 18:35:37 | EmilienM | if you have any suggestion, feel free to let me know | |
| 18:35:47 | EmilienM | the proof it works http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache/nova_api_wsgi_access_ssl.txt.gz | |
| 18:36:56 | EmilienM | mwhahaha: ^ fyi also | |
| 18:37:04 | mriedem | gyee_: i think you might be right https://github.com/openstack/nova/blob/stable/newton/nova/api/openstack/compute/evacuate.py#L92 | |
| 18:37:13 | cdent | EmilienM: is cgi-bin/nova/nova-api the new wsgi script that pbr creates? | |
| 18:37:26 | EmilienM | cdent: yes, nova-api-wsgi or something | |
| 18:37:35 | mriedem | gyee_: gah, yeah, that's it | |
| 18:37:39 | edleafe | cdent: my understanding is that if N instances are requested, we'd pick N hosts from the list of weighed hosts. Then for each, we'd get the cell uuid and from the remaining, find enough in the same cell to satisfy the number of alternates needed | |
| 18:37:39 | EmilienM | cdent: /usr/bin/nova-api-wsgi | |
| 18:37:49 | gyee_ | mriedem: two bugs then | |
| 18:37:50 | mriedem | so if you're using microversion<2.29 then force is None | |
| 18:37:52 | mriedem | yup | |
| 18:38:01 | EmilienM | cdent: I need to go now, but you can PM me if you have more comments, i'll catch up later | |
| 18:38:03 | edleafe | So I didn't envision overlap at all | |
| 18:38:26 | edleafe | but I suppose it wouldn't be terrible, since the claim would simply fail on a retry | |
| 18:40:09 | melwitt | dansmith, sdague: I created a WIP devstack patch to add the --tcp option to dstat and added Depends-On it for the caching db connections patch, and compared the output (using active tcp as a hint to db connections) | |
| 18:40:30 | sdague | melwitt: cool, honestly, just add that if you like | |
| 18:40:35 | melwitt | I put them in a spreadsheet here https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing | |
| 18:40:52 | sdague | dstat is really there to cross check all these things for runs, so no reason to not report it all the time | |
| 18:42:31 | melwitt | cool. it makes the csv less human readable but I think it's probably worth it for us to be able to have a high level indicator of how many connections we're using http://logs.openstack.org/94/465194/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a6b700/logs/dstat-csv_log.txt.gz | |
| 18:43:14 | dansmith | melwitt: not sure I get it.. both tabs top out at ~40 right? | |
| 18:43:16 | sdague | yeh, the non csv version is still emitted as well | |
| 18:44:24 | dansmith | oh | |
| 18:44:33 | melwitt | dansmith: that's listening. I highlighted the "active/established" column, col AC. comparing a gate-tempest-dsvm-neutron-full-ubuntu-xenial in both cases, without caching active connections were 1495 at the end of the run, with caching it was 445 | |
| 18:44:39 | dansmith | yeah | |
| 18:45:09 | dansmith | I hadn't scrolled over enough | |
| 18:45:28 | dansmith | well, I guess that's it | |
| 18:45:34 | dansmith | and needing of a backport | |
| 18:45:38 | dansmith | I just dunno why we're never gc'ing those | |
| 18:45:41 | melwitt | ah, I had hoped it would show the scrolling I did to ppl shared with | |
| 18:46:06 | melwitt | yeah, I don't get it either. I thought it should have worked fine the way it was | |
| 18:46:30 | mriedem | melwitt: for your todo in the wip you could use oslo.cache which i think has a default timer | |
| 18:46:55 | mriedem | and can be configured for memcache | |
| 18:47:01 | dansmith | mriedem: will that put it in memcache or dogpile? | |
| 18:47:03 | dansmith | because we probably don't want that | |
| 18:47:08 | mriedem | defaults to dogpile | |
| 18:47:09 | mriedem | in memory | |
| 18:47:16 | mriedem | well, i think | |
| 18:47:31 | mriedem | https://docs.openstack.org/developer/oslo.cache/opts.html | |
| 18:47:39 | dansmith | we just don't want those shared across machines in any way, and not serialized either | |
| 18:49:09 | melwitt | mriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova | |
| 18:49:36 | dansmith | well, for the backport we can ignore the cache purge, | |
| 18:49:45 | dansmith | since the other lists aren't purged either | |
| 18:51:37 | melwitt | dansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used | |
| 18:52:23 | dansmith | right, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting | |
| 18:52:34 | dansmith | I wouldn't even hook up SIGHUP since the other lists are't either | |
| 18:53:17 | melwitt | okay | |
| 18:55:01 | dansmith | related tangent, | |
| 18:55:16 | dansmith | reporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance | |
| 18:55:29 | dansmith | i.e "ending tcp connections is >100% higher than expected" | |
| 18:55:46 | dansmith | re: the perf-monitoring forum session | |
| 18:55:58 | melwitt | yeah, I was thinking the same | |
| 18:56:30 | dansmith | I also wonder if we should open a bug against oslo-db for this | |
| 18:56:50 | melwitt | going to un-WIP the devstack patch. sdague, where is the non-csv dstat in the logs? I was wondering should we put --tcp in for both csv and non-csv? I did but I wasn't sure | |
| 18:58:26 | melwitt | dansmith: yeah, should at least ask zzzeek about it and open a bug if it's not an expected thing | |
| 19:00:16 | sdague | melwitt: it's in a screen | |
| 19:00:22 | sdague | well, it's in the journal :) | |
| 19:00:26 | sdague | but we dump it out to screen | |
| 19:00:31 | sdague | files | |
| 19:00:34 | sdague | for compat in gate runs | |
| 19:01:35 | melwitt | cool, thanks | |
| 19:06:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:06:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:06:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:06:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:06:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:06:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:06:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:10:56 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 19:11:46 | melwitt | sdague: patch has been un-WIP https://review.openstack.org/#/c/465194 | |
| 19:12:03 | sdague | melwitt: +2, thanks | |
| 19:12:33 | melwitt | thx | |
| 19:16:21 | cdent | dansmith, bauzas, edleafe I tried to leave some questions on https://review.openstack.org/#/c/465175/ to capture my concerns, but not sure if they are clear | |
| 19:17:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance https://review.openstack.org/391060 | |
| 19:40:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:40:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:40:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:40:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:40:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:40:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:40:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:45:21 | melwitt | cfriesen: had you filed a bug for the increase in DB connections in Newton? I'll need one to refer to in my patch | |
| 19:56:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 20:01:01 | openstackgerrit | melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042 | |
| 20:06:22 | zzzeek | melwitt: can you tell me about https://review.openstack.org/#/c/465042/3/nova/context.py | |
| 20:07:42 | melwitt | zzzeek: yes! I was hoping you would be able to comment on this. recently we've seen a significant increase in number of DB connections since we've been doing DB connection switching for cells | |
| 20:08:01 | zzzeek | melwitt: so...did you try changing pool_size and max_overflow ? | |