Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-15
16:22:26 mriedem in https://review.openstack.org/#/c/463992/
16:22:27 mdbooth mriedem: I saw that, just looking for myself.
16:22:30 mriedem ok
16:22:30 lyarwood yeah, I assumed we were setting it via connection_info in LibvirtNetVolumeDriver but I can't find anything on the cinder side
16:22:43 openstackgerrit Dan Smith proposed openstack/nova master: Make compute auto-disable itself if builds are failing https://review.openstack.org/463597
16:22:57 mriedem lyarwood: the source_protocol is set in nova
16:23:03 mriedem in the volume driver's get_config method
16:23:25 lyarwood mriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L46
16:23:26 mriedem we could potentially remove the netfs stuff after this since it appears to be dead code now
16:23:27 mdbooth nova/virt/libvirt/volume/net.py: source_protocol = connection_info['driver_volume_type']
16:24:00 lyarwood argh wrong line number
16:24:08 mriedem mdbooth: ok, i think that's just probably wrong
16:24:12 mriedem i.e. bad var name
16:24:43 mdbooth mriedem: It seems to be used in get_config there
16:24:48 lyarwood https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L96
16:24:55 mdbooth Yeah, that one
16:25:10 mdbooth So that means it could come from cinder and end up in the libvirt xml
16:26:12 mdbooth cinder doesn't appear to set driver_volume_type to netfs anywhere, though
16:26:17 mriedem blech
16:26:35 mriedem btw, the rbd and iscsi net stuff crammed into the same driver code is not fun
16:26:48 mdbooth Yep
16:27:16 mriedem there could have been a cinder driver years ago that had netfs
16:28:05 openstackgerrit Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265
16:28:39 mriedem i checked grizzly, icehouse and kilo though and found nothing
16:30:52 dansmith sdague: check it out: http://logs.openstack.org/37/458537/19/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e290ec2/logs/screen-n-api.txt.gz?level=TRACE#_May_11_20_08_20_211256
16:31:13 dansmith I assume that's the mysql server complaining
16:33:14 sdague yeh, probably
16:33:31 sdague I'm pretty sure when you see that it's coming back from the server
16:33:38 sdague this the superconductor fleet?
16:33:45 dansmith sdague: can I raise that in devstack easily or do I need to make something new?
16:33:52 dansmith sdague: yeah but single node
16:34:00 sdague you can change it in devstack, let me find the line
16:34:04 dansmith sdague: do you know if I can control the size of our connection pool?
16:34:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service https://review.openstack.org/461519
16:34:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Use uuid for id in Services API https://review.openstack.org/464280
16:34:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532
16:34:26 mriedem dansmith: you can
16:34:38 mriedem we had to do that for postgres at some point
16:34:44 sdague https://github.com/openstack-dev/devstack/blob/27df725179045a2e91a34eb91e29e52d0fe5f663/lib/databases/mysql#L99 ....
16:34:48 mriedem https://github.com/openstack-dev/devstack/tree/master/lib/databases
16:34:51 sdague we're exceeding 1024?
16:34:59 sdague that seems suspicious to me
16:35:27 dansmith well, what's our connection pool size?
16:36:26 mriedem we used to save off the mysql config i thought
16:36:28 mriedem but i can't find it
16:36:41 dansmith connection pool size would be something in our own config
16:37:06 dansmith just noticed by the way,
16:37:13 dansmith that we don't get the config dump at the beginning of n-api logging anymore
16:37:18 sdague dansmith: I don't know that we're setting the pool
16:37:19 dansmith maybe because of the wsgi patch?
16:38:46 dansmith looks like 5 per worker maybe
16:38:49 dansmith which is pretty low
16:39:06 dansmith that would be ~204 workers to exhaust 1024
16:39:11 sdague dansmith: I still see the dump
16:39:15 sdague http://logs.openstack.org/26/464326/1/check/gate-tempest-dsvm-neutron-src-oslo.context-ubuntu-xenial/4fc411c/logs/screen-n-api.txt.gz#_May_14_11_27_57_370073
16:39:18 dansmith which seems unlikely on a single node
16:39:21 sdague it's just a little later
16:39:53 dansmith oh okay, that seems odd given we're loading extensions before that, but okay
16:40:25 sdague I think the new route system compiles early
16:40:33 dansmith okay
16:40:35 sdague the stuff that completely does away with the extension loader
16:40:53 sdague we honestly could probably turn off debug on those
16:41:11 dansmith now that th3y
16:41:16 dansmith they are all implicit yeah
16:41:25 dansmith anyway, I dunno what to do about this connection thing
16:42:14 dansmith presumably if we go higher we also need to set maxfds
16:42:20 dansmith unless we already have for something else
16:43:02 sdague yeh, I don't know, but I was pretty sure that was the db rejecting things. Any chance we're leaking db connections or something?
16:44:04 dansmith nothing that fundamental has changed recently, so I dunno
16:46:29 dansmith looks like we may be limited below max_connections due to ulimit on descriptors anyway
16:46:34 dansmith so maybe we're not getting 1024 even
16:49:59 cfriesen ralonsoh: here now, you still have a question or has it been dealt with (I'm checking history now)
16:51:40 ralonsoh cfriesen: well, I have an answer from libvirt guys
16:52:31 ralonsoh cfriesen: http://paste.openstack.org/show/609591/
16:52:57 ralonsoh cfriesen: at the first time, the suggest me to do what is NOT recommended in openstack manuals
16:53:52 ralonsoh cfriesen: they recommended to modify the kernel module
16:53:54 ralonsoh cfriesen: https://github.com/openstack/openstack-manuals/blob/master/doc/networking-guide/source/config-sriov.rst
16:57:46 cfriesen ralonsoh: technically you're not modifying the kernel module itself, just the parameter on loading it. Arguably either the module parameter or setting it via /sys/class/net would work fine, it's just that they may at some point remove the module parameter in the future.
16:58:05 melwitt dansmith, sdague: on the db connection thing, coincidentally I'm building a new devstack right now to look into it. I have a guess at something that might be related but need to verify
16:58:28 dansmith melwitt: and you won't share until confirmed?
16:58:32 sdague melwitt: ok, cool.
16:58:38 dansmith can I have a teaser for next week's episode?
16:58:42 cfriesen ralonsoh: the nice thing about setting it via /sys/class/net is that it gives you flexibility if you have different hardware using the same module.
16:59:05 ralonsoh cfriesen: I know and I agree. The problem is how to store it
16:59:07 dansmith sdague: melwitt: I just pushed up a ulimit hack to see if that affects anything, fwiw
16:59:26 ralonsoh cfriesen: do you think the idea of adding a pre-up command is enough?
16:59:46 sdague melwitt: yeh, I would feel much more comfortable twiddling knobs if we understood why we were hitting that
17:01:08 melwitt dansmith: heh. well, I could be totally wrong, but I think it might be because we aren't caching the oslo.db transaction context managers. I had thought oslo.db connection pooling underneath took care of that, but now I want to try and see if there's a difference when doing it
17:01:27 dansmith melwitt: hmm
17:01:44 dansmith melwitt: so we also disable some amount of tpooling, which isn't connection pooling that I know of, but..
17:01:58 dansmith so maybe we're already breaking things and what I'm doing is just pushing us over a limit?
17:02:24 melwitt that's what I'm thinking, but it's a total guess
17:03:44 dansmith hmm, that would be like bad and stuff
17:03:49 sdague lyarwood - https://review.openstack.org/#/c/464664/ approved and in gate, so hopefully you'll be back to working in ~ 1 hr
17:03:51 dansmith not really sure how we'd not be gc'ing them, but..
17:03:57 melwitt I mean, I think cfriesen or somebody had mentioned recently they saw total db connections go way up since newton or ocata compared with before cells v2. so that had me thinking already that I wanted to look into this
17:04:03 dansmith maybe we're just hitting a rate that makes it a problem
17:04:31 dansmith melwitt: well, we would have an increase because of the api db
17:04:55 cfriesen ralonsoh: in our environment if a PF has VFs we only ever pass the VFs through to the guest so no personal experience with what you're trying to do. In the case where you're using VFs are you "up"ing the PF?
17:05:06 dansmith I guess if we're leaking somehow, we'd get worse the more targeting we do,

Earlier   Later