| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-17 | |||
| 18:27:34 | cdent | or to put it another way: what is the reason for not having simply "more hosts" instead of the nested "alternatives"? | |
| 18:27:35 | dansmith | cdent: each host could satisfy the request.. meaning all the primaries and alternates | |
| 18:28:01 | dansmith | cdent: I had thought about it in terms of just a list of hosts, and not the hierarchy, however: | |
| 18:28:28 | dansmith | cdent: I think the point is that we will, for a single num_instances>1 request, not choose the same primary host for two instances, but the alternates can overlap | |
| 18:28:41 | dansmith | cdent: so the distinction is made such that we consume the primaries | |
| 18:29:00 | mriedem | gyee_: i'm still a bit confused about how this happens if that force flag isn't specified | |
| 18:29:12 | cdent | dansmith: is the additional code complexity worth the win? | |
| 18:30:16 | dansmith | cdent: well, if we don't, then we'll choose hostA, (hostB, C) for the first instance and hostB, (HostA, C) for the second, right? that's a bad decision if we fail on hostB and the first instance filled up hostA | |
| 18:30:34 | dansmith | cdent: also, I think we have to consume the hosts as primaries to keep the existing scheduler behavior, right? | |
| 18:31:01 | dansmith | the previous code did that such that num_instances>1 would not pack them all onto the same host, IIUC | |
| 18:31:49 | dansmith | I dunno, I guess I'd wait for either bauzas or johnthetubaguy to comment on that | |
| 18:32:24 | cdent | thanks dansmith, I'll have to read through it a bit more | |
| 18:32:31 | cdent | it feels like it could be simpler, but feelings may not matter | |
| 18:33:16 | dansmith | well, FWIW, when I first read that code I was kinda horrified with the nested dicting, | |
| 18:33:32 | dansmith | but I didn't complain because I think it probably makes the smallest behavioral change | |
| 18:33:52 | dansmith | it's more code, but I'm not sure it's _that_ much more complex | |
| 18:34:20 | dansmith | it's not an unreasonable opinion though, IMHO | |
| 18:34:44 | dansmith | recording it, even if it is shot down with technical reasons is probably a good idea | |
| 18:34:57 | cdent | roger that | |
| 18:35:04 | gyee_ | mriedem: maybe none != False? | |
| 18:35:06 | EmilienM | mriedem, cdent: super quick fyi: I tested again nova/wsgi and it works fine | |
| 18:35:21 | cdent | yay! | |
| 18:35:30 | EmilienM | this is our config: http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache_config/10-nova_api_wsgi.conf.txt.gz | |
| 18:35:37 | EmilienM | if you have any suggestion, feel free to let me know | |
| 18:35:47 | EmilienM | the proof it works http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache/nova_api_wsgi_access_ssl.txt.gz | |
| 18:36:56 | EmilienM | mwhahaha: ^ fyi also | |
| 18:37:04 | mriedem | gyee_: i think you might be right https://github.com/openstack/nova/blob/stable/newton/nova/api/openstack/compute/evacuate.py#L92 | |
| 18:37:13 | cdent | EmilienM: is cgi-bin/nova/nova-api the new wsgi script that pbr creates? | |
| 18:37:26 | EmilienM | cdent: yes, nova-api-wsgi or something | |
| 18:37:35 | mriedem | gyee_: gah, yeah, that's it | |
| 18:37:39 | edleafe | cdent: my understanding is that if N instances are requested, we'd pick N hosts from the list of weighed hosts. Then for each, we'd get the cell uuid and from the remaining, find enough in the same cell to satisfy the number of alternates needed | |
| 18:37:39 | EmilienM | cdent: /usr/bin/nova-api-wsgi | |
| 18:37:49 | gyee_ | mriedem: two bugs then | |
| 18:37:50 | mriedem | so if you're using microversion<2.29 then force is None | |
| 18:37:52 | mriedem | yup | |
| 18:38:01 | EmilienM | cdent: I need to go now, but you can PM me if you have more comments, i'll catch up later | |
| 18:38:03 | edleafe | So I didn't envision overlap at all | |
| 18:38:26 | edleafe | but I suppose it wouldn't be terrible, since the claim would simply fail on a retry | |
| 18:40:09 | melwitt | dansmith, sdague: I created a WIP devstack patch to add the --tcp option to dstat and added Depends-On it for the caching db connections patch, and compared the output (using active tcp as a hint to db connections) | |
| 18:40:30 | sdague | melwitt: cool, honestly, just add that if you like | |
| 18:40:35 | melwitt | I put them in a spreadsheet here https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing | |
| 18:40:52 | sdague | dstat is really there to cross check all these things for runs, so no reason to not report it all the time | |
| 18:42:31 | melwitt | cool. it makes the csv less human readable but I think it's probably worth it for us to be able to have a high level indicator of how many connections we're using http://logs.openstack.org/94/465194/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a6b700/logs/dstat-csv_log.txt.gz | |
| 18:43:14 | dansmith | melwitt: not sure I get it.. both tabs top out at ~40 right? | |
| 18:43:16 | sdague | yeh, the non csv version is still emitted as well | |
| 18:44:24 | dansmith | oh | |
| 18:44:33 | melwitt | dansmith: that's listening. I highlighted the "active/established" column, col AC. comparing a gate-tempest-dsvm-neutron-full-ubuntu-xenial in both cases, without caching active connections were 1495 at the end of the run, with caching it was 445 | |
| 18:44:39 | dansmith | yeah | |
| 18:45:09 | dansmith | I hadn't scrolled over enough | |
| 18:45:28 | dansmith | well, I guess that's it | |
| 18:45:34 | dansmith | and needing of a backport | |
| 18:45:38 | dansmith | I just dunno why we're never gc'ing those | |
| 18:45:41 | melwitt | ah, I had hoped it would show the scrolling I did to ppl shared with | |
| 18:46:06 | melwitt | yeah, I don't get it either. I thought it should have worked fine the way it was | |
| 18:46:30 | mriedem | melwitt: for your todo in the wip you could use oslo.cache which i think has a default timer | |
| 18:46:55 | mriedem | and can be configured for memcache | |
| 18:47:01 | dansmith | mriedem: will that put it in memcache or dogpile? | |
| 18:47:03 | dansmith | because we probably don't want that | |
| 18:47:08 | mriedem | defaults to dogpile | |
| 18:47:09 | mriedem | in memory | |
| 18:47:16 | mriedem | well, i think | |
| 18:47:31 | mriedem | https://docs.openstack.org/developer/oslo.cache/opts.html | |
| 18:47:39 | dansmith | we just don't want those shared across machines in any way, and not serialized either | |
| 18:49:09 | melwitt | mriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova | |
| 18:49:36 | dansmith | well, for the backport we can ignore the cache purge, | |
| 18:49:45 | dansmith | since the other lists aren't purged either | |
| 18:51:37 | melwitt | dansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used | |
| 18:52:23 | dansmith | right, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting | |
| 18:52:34 | dansmith | I wouldn't even hook up SIGHUP since the other lists are't either | |
| 18:53:17 | melwitt | okay | |
| 18:55:01 | dansmith | related tangent, | |
| 18:55:16 | dansmith | reporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance | |
| 18:55:29 | dansmith | i.e "ending tcp connections is >100% higher than expected" | |
| 18:55:46 | dansmith | re: the perf-monitoring forum session | |
| 18:55:58 | melwitt | yeah, I was thinking the same | |
| 18:56:30 | dansmith | I also wonder if we should open a bug against oslo-db for this | |
| 18:56:50 | melwitt | going to un-WIP the devstack patch. sdague, where is the non-csv dstat in the logs? I was wondering should we put --tcp in for both csv and non-csv? I did but I wasn't sure | |
| 18:58:26 | melwitt | dansmith: yeah, should at least ask zzzeek about it and open a bug if it's not an expected thing | |
| 19:00:16 | sdague | melwitt: it's in a screen | |
| 19:00:22 | sdague | well, it's in the journal :) | |
| 19:00:26 | sdague | but we dump it out to screen | |
| 19:00:31 | sdague | files | |
| 19:00:34 | sdague | for compat in gate runs | |
| 19:01:35 | melwitt | cool, thanks | |
| 19:06:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:06:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:06:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 19:06:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 19:06:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 19:06:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 19:06:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 19:10:56 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 19:11:46 | melwitt | sdague: patch has been un-WIP https://review.openstack.org/#/c/465194 | |
| 19:12:03 | sdague | melwitt: +2, thanks | |
| 19:12:33 | melwitt | thx | |
| 19:16:21 | cdent | dansmith, bauzas, edleafe I tried to leave some questions on https://review.openstack.org/#/c/465175/ to capture my concerns, but not sure if they are clear | |
| 19:17:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance https://review.openstack.org/391060 | |
| 19:40:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 19:40:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 19:40:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |