| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-17 | |||
| 18:16:14 | mriedem | where you change the image and the new image meta for the rebuild doesn't work with the original host, but we don't validate that | |
| 18:16:56 | mriedem | gyee_: then i think all you need to do is add an "or recreate" to this conditional https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2767 | |
| 18:17:05 | mriedem | gyee_: do you have a recreate of the bug? | |
| 18:19:01 | gyee_ | mriedem: I haven't file a bug for Nova yet | |
| 18:19:04 | gyee_ | about to do so | |
| 18:19:18 | mriedem | gyee_: oh we have a bug | |
| 18:19:20 | mriedem | well MOS does | |
| 18:19:23 | mriedem | but we can add nova to that | |
| 18:19:49 | cfriesen | question about RPC (with standard rabbitmq/kombu messaging): are incoming messages on nova-compute immediately grabbed by an eventlet? I'm seeing huge lock contention on the 'refresh_cache-%s' lock between the live migration code and external events coming in from neutron | |
| 18:19:57 | gyee_ | do I have perm to do that? Sorry I am still new to Nova. | |
| 18:20:01 | mriedem | gyee_: https://bugs.launchpad.net/mos/+bug/1658070 | |
| 18:20:01 | openstack | Launchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] | |
| 18:20:32 | gyee_ | oh good :-) | |
| 18:24:43 | cdent | bauzas, dansmith: sorry for beating a dead horse, but I need some help understanding https://review.openstack.org/#/c/465175/ (returning alternatives) | |
| 18:25:08 | cdent | edleafe too | |
| 18:25:12 | gyee_ | mriedem: thanks for the help! | |
| 18:25:34 | dansmith | cdent: what about it? | |
| 18:25:36 | mriedem | gyee_: np, you did the hard debug already, | |
| 18:25:39 | cdent | in the collection of hosts being returned from select_destinations there is a list of dicts | |
| 18:25:47 | cdent | in each dict is one host which is the chosen host | |
| 18:25:49 | mriedem | gyee_: if you get a recreate and have that fix handy ping me to review | |
| 18:26:04 | cdent | and then a key 'alternatives' which are other hosts that might work | |
| 18:26:56 | cdent | is each chosen host (across the entire selected_hosts) different or does every single host satisfy the request_spec? | |
| 18:27:20 | gyee_ | mriedem: definitely, just need to add unit tests before pushing it upstream | |
| 18:27:34 | cdent | or to put it another way: what is the reason for not having simply "more hosts" instead of the nested "alternatives"? | |
| 18:27:35 | dansmith | cdent: each host could satisfy the request.. meaning all the primaries and alternates | |
| 18:28:01 | dansmith | cdent: I had thought about it in terms of just a list of hosts, and not the hierarchy, however: | |
| 18:28:28 | dansmith | cdent: I think the point is that we will, for a single num_instances>1 request, not choose the same primary host for two instances, but the alternates can overlap | |
| 18:28:41 | dansmith | cdent: so the distinction is made such that we consume the primaries | |
| 18:29:00 | mriedem | gyee_: i'm still a bit confused about how this happens if that force flag isn't specified | |
| 18:29:12 | cdent | dansmith: is the additional code complexity worth the win? | |
| 18:30:16 | dansmith | cdent: well, if we don't, then we'll choose hostA, (hostB, C) for the first instance and hostB, (HostA, C) for the second, right? that's a bad decision if we fail on hostB and the first instance filled up hostA | |
| 18:30:34 | dansmith | cdent: also, I think we have to consume the hosts as primaries to keep the existing scheduler behavior, right? | |
| 18:31:01 | dansmith | the previous code did that such that num_instances>1 would not pack them all onto the same host, IIUC | |
| 18:31:49 | dansmith | I dunno, I guess I'd wait for either bauzas or johnthetubaguy to comment on that | |
| 18:32:24 | cdent | thanks dansmith, I'll have to read through it a bit more | |
| 18:32:31 | cdent | it feels like it could be simpler, but feelings may not matter | |
| 18:33:16 | dansmith | well, FWIW, when I first read that code I was kinda horrified with the nested dicting, | |
| 18:33:32 | dansmith | but I didn't complain because I think it probably makes the smallest behavioral change | |
| 18:33:52 | dansmith | it's more code, but I'm not sure it's _that_ much more complex | |
| 18:34:20 | dansmith | it's not an unreasonable opinion though, IMHO | |
| 18:34:44 | dansmith | recording it, even if it is shot down with technical reasons is probably a good idea | |
| 18:34:57 | cdent | roger that | |
| 18:35:04 | gyee_ | mriedem: maybe none != False? | |
| 18:35:06 | EmilienM | mriedem, cdent: super quick fyi: I tested again nova/wsgi and it works fine | |
| 18:35:21 | cdent | yay! | |
| 18:35:30 | EmilienM | this is our config: http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache_config/10-nova_api_wsgi.conf.txt.gz | |
| 18:35:37 | EmilienM | if you have any suggestion, feel free to let me know | |
| 18:35:47 | EmilienM | the proof it works http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache/nova_api_wsgi_access_ssl.txt.gz | |
| 18:36:56 | EmilienM | mwhahaha: ^ fyi also | |
| 18:37:04 | mriedem | gyee_: i think you might be right https://github.com/openstack/nova/blob/stable/newton/nova/api/openstack/compute/evacuate.py#L92 | |
| 18:37:13 | cdent | EmilienM: is cgi-bin/nova/nova-api the new wsgi script that pbr creates? | |
| 18:37:26 | EmilienM | cdent: yes, nova-api-wsgi or something | |
| 18:37:35 | mriedem | gyee_: gah, yeah, that's it | |
| 18:37:39 | edleafe | cdent: my understanding is that if N instances are requested, we'd pick N hosts from the list of weighed hosts. Then for each, we'd get the cell uuid and from the remaining, find enough in the same cell to satisfy the number of alternates needed | |
| 18:37:39 | EmilienM | cdent: /usr/bin/nova-api-wsgi | |
| 18:37:49 | gyee_ | mriedem: two bugs then | |
| 18:37:50 | mriedem | so if you're using microversion<2.29 then force is None | |
| 18:37:52 | mriedem | yup | |
| 18:38:01 | EmilienM | cdent: I need to go now, but you can PM me if you have more comments, i'll catch up later | |
| 18:38:03 | edleafe | So I didn't envision overlap at all | |
| 18:38:26 | edleafe | but I suppose it wouldn't be terrible, since the claim would simply fail on a retry | |
| 18:40:09 | melwitt | dansmith, sdague: I created a WIP devstack patch to add the --tcp option to dstat and added Depends-On it for the caching db connections patch, and compared the output (using active tcp as a hint to db connections) | |
| 18:40:30 | sdague | melwitt: cool, honestly, just add that if you like | |
| 18:40:35 | melwitt | I put them in a spreadsheet here https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing | |
| 18:40:52 | sdague | dstat is really there to cross check all these things for runs, so no reason to not report it all the time | |
| 18:42:31 | melwitt | cool. it makes the csv less human readable but I think it's probably worth it for us to be able to have a high level indicator of how many connections we're using http://logs.openstack.org/94/465194/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a6b700/logs/dstat-csv_log.txt.gz | |
| 18:43:14 | dansmith | melwitt: not sure I get it.. both tabs top out at ~40 right? | |
| 18:43:16 | sdague | yeh, the non csv version is still emitted as well | |
| 18:44:24 | dansmith | oh | |
| 18:44:33 | melwitt | dansmith: that's listening. I highlighted the "active/established" column, col AC. comparing a gate-tempest-dsvm-neutron-full-ubuntu-xenial in both cases, without caching active connections were 1495 at the end of the run, with caching it was 445 | |
| 18:44:39 | dansmith | yeah | |
| 18:45:09 | dansmith | I hadn't scrolled over enough | |
| 18:45:28 | dansmith | well, I guess that's it | |
| 18:45:34 | dansmith | and needing of a backport | |
| 18:45:38 | dansmith | I just dunno why we're never gc'ing those | |
| 18:45:41 | melwitt | ah, I had hoped it would show the scrolling I did to ppl shared with | |
| 18:46:06 | melwitt | yeah, I don't get it either. I thought it should have worked fine the way it was | |
| 18:46:30 | mriedem | melwitt: for your todo in the wip you could use oslo.cache which i think has a default timer | |
| 18:46:55 | mriedem | and can be configured for memcache | |
| 18:47:01 | dansmith | mriedem: will that put it in memcache or dogpile? | |
| 18:47:03 | dansmith | because we probably don't want that | |
| 18:47:08 | mriedem | defaults to dogpile | |
| 18:47:09 | mriedem | in memory | |
| 18:47:16 | mriedem | well, i think | |
| 18:47:31 | mriedem | https://docs.openstack.org/developer/oslo.cache/opts.html | |
| 18:47:39 | dansmith | we just don't want those shared across machines in any way, and not serialized either | |
| 18:49:09 | melwitt | mriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova | |
| 18:49:36 | dansmith | well, for the backport we can ignore the cache purge, | |
| 18:49:45 | dansmith | since the other lists aren't purged either | |
| 18:51:37 | melwitt | dansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used | |
| 18:52:23 | dansmith | right, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting | |
| 18:52:34 | dansmith | I wouldn't even hook up SIGHUP since the other lists are't either | |
| 18:53:17 | melwitt | okay | |
| 18:55:01 | dansmith | related tangent, | |
| 18:55:16 | dansmith | reporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance | |
| 18:55:29 | dansmith | i.e "ending tcp connections is >100% higher than expected" | |
| 18:55:46 | dansmith | re: the perf-monitoring forum session | |
| 18:55:58 | melwitt | yeah, I was thinking the same | |
| 18:56:30 | dansmith | I also wonder if we should open a bug against oslo-db for this | |