| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-15 | |||
| 16:22:27 | mdbooth | mriedem: I saw that, just looking for myself. | |
| 16:22:30 | mriedem | ok | |
| 16:22:30 | lyarwood | yeah, I assumed we were setting it via connection_info in LibvirtNetVolumeDriver but I can't find anything on the cinder side | |
| 16:22:43 | openstackgerrit | Dan Smith proposed openstack/nova master: Make compute auto-disable itself if builds are failing https://review.openstack.org/463597 | |
| 16:22:57 | mriedem | lyarwood: the source_protocol is set in nova | |
| 16:23:03 | mriedem | in the volume driver's get_config method | |
| 16:23:25 | lyarwood | mriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L46 | |
| 16:23:26 | mriedem | we could potentially remove the netfs stuff after this since it appears to be dead code now | |
| 16:23:27 | mdbooth | nova/virt/libvirt/volume/net.py: source_protocol = connection_info['driver_volume_type'] | |
| 16:24:00 | lyarwood | argh wrong line number | |
| 16:24:08 | mriedem | mdbooth: ok, i think that's just probably wrong | |
| 16:24:12 | mriedem | i.e. bad var name | |
| 16:24:43 | mdbooth | mriedem: It seems to be used in get_config there | |
| 16:24:48 | lyarwood | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L96 | |
| 16:24:55 | mdbooth | Yeah, that one | |
| 16:25:10 | mdbooth | So that means it could come from cinder and end up in the libvirt xml | |
| 16:26:12 | mdbooth | cinder doesn't appear to set driver_volume_type to netfs anywhere, though | |
| 16:26:17 | mriedem | blech | |
| 16:26:35 | mriedem | btw, the rbd and iscsi net stuff crammed into the same driver code is not fun | |
| 16:26:48 | mdbooth | Yep | |
| 16:27:16 | mriedem | there could have been a cinder driver years ago that had netfs | |
| 16:28:05 | openstackgerrit | Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265 | |
| 16:28:39 | mriedem | i checked grizzly, icehouse and kilo though and found nothing | |
| 16:30:52 | dansmith | sdague: check it out: http://logs.openstack.org/37/458537/19/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e290ec2/logs/screen-n-api.txt.gz?level=TRACE#_May_11_20_08_20_211256 | |
| 16:31:13 | dansmith | I assume that's the mysql server complaining | |
| 16:33:14 | sdague | yeh, probably | |
| 16:33:31 | sdague | I'm pretty sure when you see that it's coming back from the server | |
| 16:33:38 | sdague | this the superconductor fleet? | |
| 16:33:45 | dansmith | sdague: can I raise that in devstack easily or do I need to make something new? | |
| 16:33:52 | dansmith | sdague: yeah but single node | |
| 16:34:00 | sdague | you can change it in devstack, let me find the line | |
| 16:34:04 | dansmith | sdague: do you know if I can control the size of our connection pool? | |
| 16:34:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service https://review.openstack.org/461519 | |
| 16:34:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use uuid for id in Services API https://review.openstack.org/464280 | |
| 16:34:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 16:34:26 | mriedem | dansmith: you can | |
| 16:34:38 | mriedem | we had to do that for postgres at some point | |
| 16:34:44 | sdague | https://github.com/openstack-dev/devstack/blob/27df725179045a2e91a34eb91e29e52d0fe5f663/lib/databases/mysql#L99 .... | |
| 16:34:48 | mriedem | https://github.com/openstack-dev/devstack/tree/master/lib/databases | |
| 16:34:51 | sdague | we're exceeding 1024? | |
| 16:34:59 | sdague | that seems suspicious to me | |
| 16:35:27 | dansmith | well, what's our connection pool size? | |
| 16:36:26 | mriedem | we used to save off the mysql config i thought | |
| 16:36:28 | mriedem | but i can't find it | |
| 16:36:41 | dansmith | connection pool size would be something in our own config | |
| 16:37:06 | dansmith | just noticed by the way, | |
| 16:37:13 | dansmith | that we don't get the config dump at the beginning of n-api logging anymore | |
| 16:37:18 | sdague | dansmith: I don't know that we're setting the pool | |
| 16:37:19 | dansmith | maybe because of the wsgi patch? | |
| 16:38:46 | dansmith | looks like 5 per worker maybe | |
| 16:38:49 | dansmith | which is pretty low | |
| 16:39:06 | dansmith | that would be ~204 workers to exhaust 1024 | |
| 16:39:11 | sdague | dansmith: I still see the dump | |
| 16:39:15 | sdague | http://logs.openstack.org/26/464326/1/check/gate-tempest-dsvm-neutron-src-oslo.context-ubuntu-xenial/4fc411c/logs/screen-n-api.txt.gz#_May_14_11_27_57_370073 | |
| 16:39:18 | dansmith | which seems unlikely on a single node | |
| 16:39:21 | sdague | it's just a little later | |
| 16:39:53 | dansmith | oh okay, that seems odd given we're loading extensions before that, but okay | |
| 16:40:25 | sdague | I think the new route system compiles early | |
| 16:40:33 | dansmith | okay | |
| 16:40:35 | sdague | the stuff that completely does away with the extension loader | |
| 16:40:53 | sdague | we honestly could probably turn off debug on those | |
| 16:41:11 | dansmith | now that th3y | |
| 16:41:16 | dansmith | they are all implicit yeah | |
| 16:41:25 | dansmith | anyway, I dunno what to do about this connection thing | |
| 16:42:14 | dansmith | presumably if we go higher we also need to set maxfds | |
| 16:42:20 | dansmith | unless we already have for something else | |
| 16:43:02 | sdague | yeh, I don't know, but I was pretty sure that was the db rejecting things. Any chance we're leaking db connections or something? | |
| 16:44:04 | dansmith | nothing that fundamental has changed recently, so I dunno | |
| 16:46:29 | dansmith | looks like we may be limited below max_connections due to ulimit on descriptors anyway | |
| 16:46:34 | dansmith | so maybe we're not getting 1024 even | |
| 16:49:59 | cfriesen | ralonsoh: here now, you still have a question or has it been dealt with (I'm checking history now) | |
| 16:51:40 | ralonsoh | cfriesen: well, I have an answer from libvirt guys | |
| 16:52:31 | ralonsoh | cfriesen: http://paste.openstack.org/show/609591/ | |
| 16:52:57 | ralonsoh | cfriesen: at the first time, the suggest me to do what is NOT recommended in openstack manuals | |
| 16:53:52 | ralonsoh | cfriesen: they recommended to modify the kernel module | |
| 16:53:54 | ralonsoh | cfriesen: https://github.com/openstack/openstack-manuals/blob/master/doc/networking-guide/source/config-sriov.rst | |
| 16:57:46 | cfriesen | ralonsoh: technically you're not modifying the kernel module itself, just the parameter on loading it. Arguably either the module parameter or setting it via /sys/class/net would work fine, it's just that they may at some point remove the module parameter in the future. | |
| 16:58:05 | melwitt | dansmith, sdague: on the db connection thing, coincidentally I'm building a new devstack right now to look into it. I have a guess at something that might be related but need to verify | |
| 16:58:28 | dansmith | melwitt: and you won't share until confirmed? | |
| 16:58:32 | sdague | melwitt: ok, cool. | |
| 16:58:38 | dansmith | can I have a teaser for next week's episode? | |
| 16:58:42 | cfriesen | ralonsoh: the nice thing about setting it via /sys/class/net is that it gives you flexibility if you have different hardware using the same module. | |
| 16:59:05 | ralonsoh | cfriesen: I know and I agree. The problem is how to store it | |
| 16:59:07 | dansmith | sdague: melwitt: I just pushed up a ulimit hack to see if that affects anything, fwiw | |
| 16:59:26 | ralonsoh | cfriesen: do you think the idea of adding a pre-up command is enough? | |
| 16:59:46 | sdague | melwitt: yeh, I would feel much more comfortable twiddling knobs if we understood why we were hitting that | |
| 17:01:08 | melwitt | dansmith: heh. well, I could be totally wrong, but I think it might be because we aren't caching the oslo.db transaction context managers. I had thought oslo.db connection pooling underneath took care of that, but now I want to try and see if there's a difference when doing it | |
| 17:01:27 | dansmith | melwitt: hmm | |
| 17:01:44 | dansmith | melwitt: so we also disable some amount of tpooling, which isn't connection pooling that I know of, but.. | |
| 17:01:58 | dansmith | so maybe we're already breaking things and what I'm doing is just pushing us over a limit? | |
| 17:02:24 | melwitt | that's what I'm thinking, but it's a total guess | |
| 17:03:44 | dansmith | hmm, that would be like bad and stuff | |
| 17:03:49 | sdague | lyarwood - https://review.openstack.org/#/c/464664/ approved and in gate, so hopefully you'll be back to working in ~ 1 hr | |
| 17:03:51 | dansmith | not really sure how we'd not be gc'ing them, but.. | |
| 17:03:57 | melwitt | I mean, I think cfriesen or somebody had mentioned recently they saw total db connections go way up since newton or ocata compared with before cells v2. so that had me thinking already that I wanted to look into this | |
| 17:04:03 | dansmith | maybe we're just hitting a rate that makes it a problem | |
| 17:04:31 | dansmith | melwitt: well, we would have an increase because of the api db | |
| 17:04:55 | cfriesen | ralonsoh: in our environment if a PF has VFs we only ever pass the VFs through to the guest so no personal experience with what you're trying to do. In the case where you're using VFs are you "up"ing the PF? | |
| 17:05:06 | dansmith | I guess if we're leaking somehow, we'd get worse the more targeting we do, | |
| 17:05:08 | dansmith | but.. | |