Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-15
16:19:39 mdbooth lyarwood: Bizarrely, I'm currently looking at exactly the same thing
16:20:52 lyarwood haha great
16:20:55 mdbooth lyarwood: We don't appear to have a volume driver which sets netfs
16:21:15 mdbooth Although I'm just looking at all uses of source_protocol
16:21:26 mdbooth Possible it could be imported from somewhere
16:21:54 mriedem mdbooth: yeah, that was added when gluster was added,
16:22:07 mriedem i didn't remove it because i didn't know what might rely on it and break
16:22:26 mriedem in https://review.openstack.org/#/c/463992/
16:22:27 mdbooth mriedem: I saw that, just looking for myself.
16:22:30 mriedem ok
16:22:30 lyarwood yeah, I assumed we were setting it via connection_info in LibvirtNetVolumeDriver but I can't find anything on the cinder side
16:22:43 openstackgerrit Dan Smith proposed openstack/nova master: Make compute auto-disable itself if builds are failing https://review.openstack.org/463597
16:22:57 mriedem lyarwood: the source_protocol is set in nova
16:23:03 mriedem in the volume driver's get_config method
16:23:25 lyarwood mriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L46
16:23:26 mriedem we could potentially remove the netfs stuff after this since it appears to be dead code now
16:23:27 mdbooth nova/virt/libvirt/volume/net.py: source_protocol = connection_info['driver_volume_type']
16:24:00 lyarwood argh wrong line number
16:24:08 mriedem mdbooth: ok, i think that's just probably wrong
16:24:12 mriedem i.e. bad var name
16:24:43 mdbooth mriedem: It seems to be used in get_config there
16:24:48 lyarwood https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L96
16:24:55 mdbooth Yeah, that one
16:25:10 mdbooth So that means it could come from cinder and end up in the libvirt xml
16:26:12 mdbooth cinder doesn't appear to set driver_volume_type to netfs anywhere, though
16:26:17 mriedem blech
16:26:35 mriedem btw, the rbd and iscsi net stuff crammed into the same driver code is not fun
16:26:48 mdbooth Yep
16:27:16 mriedem there could have been a cinder driver years ago that had netfs
16:28:05 openstackgerrit Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265
16:28:39 mriedem i checked grizzly, icehouse and kilo though and found nothing
16:30:52 dansmith sdague: check it out: http://logs.openstack.org/37/458537/19/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e290ec2/logs/screen-n-api.txt.gz?level=TRACE#_May_11_20_08_20_211256
16:31:13 dansmith I assume that's the mysql server complaining
16:33:14 sdague yeh, probably
16:33:31 sdague I'm pretty sure when you see that it's coming back from the server
16:33:38 sdague this the superconductor fleet?
16:33:45 dansmith sdague: can I raise that in devstack easily or do I need to make something new?
16:33:52 dansmith sdague: yeah but single node
16:34:00 sdague you can change it in devstack, let me find the line
16:34:04 dansmith sdague: do you know if I can control the size of our connection pool?
16:34:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service https://review.openstack.org/461519
16:34:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Use uuid for id in Services API https://review.openstack.org/464280
16:34:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532
16:34:26 mriedem dansmith: you can
16:34:38 mriedem we had to do that for postgres at some point
16:34:44 sdague https://github.com/openstack-dev/devstack/blob/27df725179045a2e91a34eb91e29e52d0fe5f663/lib/databases/mysql#L99 ....
16:34:48 mriedem https://github.com/openstack-dev/devstack/tree/master/lib/databases
16:34:51 sdague we're exceeding 1024?
16:34:59 sdague that seems suspicious to me
16:35:27 dansmith well, what's our connection pool size?
16:36:26 mriedem we used to save off the mysql config i thought
16:36:28 mriedem but i can't find it
16:36:41 dansmith connection pool size would be something in our own config
16:37:06 dansmith just noticed by the way,
16:37:13 dansmith that we don't get the config dump at the beginning of n-api logging anymore
16:37:18 sdague dansmith: I don't know that we're setting the pool
16:37:19 dansmith maybe because of the wsgi patch?
16:38:46 dansmith looks like 5 per worker maybe
16:38:49 dansmith which is pretty low
16:39:06 dansmith that would be ~204 workers to exhaust 1024
16:39:11 sdague dansmith: I still see the dump
16:39:15 sdague http://logs.openstack.org/26/464326/1/check/gate-tempest-dsvm-neutron-src-oslo.context-ubuntu-xenial/4fc411c/logs/screen-n-api.txt.gz#_May_14_11_27_57_370073
16:39:18 dansmith which seems unlikely on a single node
16:39:21 sdague it's just a little later
16:39:53 dansmith oh okay, that seems odd given we're loading extensions before that, but okay
16:40:25 sdague I think the new route system compiles early
16:40:33 dansmith okay
16:40:35 sdague the stuff that completely does away with the extension loader
16:40:53 sdague we honestly could probably turn off debug on those
16:41:11 dansmith now that th3y
16:41:16 dansmith they are all implicit yeah
16:41:25 dansmith anyway, I dunno what to do about this connection thing
16:42:14 dansmith presumably if we go higher we also need to set maxfds
16:42:20 dansmith unless we already have for something else
16:43:02 sdague yeh, I don't know, but I was pretty sure that was the db rejecting things. Any chance we're leaking db connections or something?
16:44:04 dansmith nothing that fundamental has changed recently, so I dunno
16:46:29 dansmith looks like we may be limited below max_connections due to ulimit on descriptors anyway
16:46:34 dansmith so maybe we're not getting 1024 even
16:49:59 cfriesen ralonsoh: here now, you still have a question or has it been dealt with (I'm checking history now)
16:51:40 ralonsoh cfriesen: well, I have an answer from libvirt guys
16:52:31 ralonsoh cfriesen: http://paste.openstack.org/show/609591/
16:52:57 ralonsoh cfriesen: at the first time, the suggest me to do what is NOT recommended in openstack manuals
16:53:52 ralonsoh cfriesen: they recommended to modify the kernel module
16:53:54 ralonsoh cfriesen: https://github.com/openstack/openstack-manuals/blob/master/doc/networking-guide/source/config-sriov.rst
16:57:46 cfriesen ralonsoh: technically you're not modifying the kernel module itself, just the parameter on loading it. Arguably either the module parameter or setting it via /sys/class/net would work fine, it's just that they may at some point remove the module parameter in the future.
16:58:05 melwitt dansmith, sdague: on the db connection thing, coincidentally I'm building a new devstack right now to look into it. I have a guess at something that might be related but need to verify
16:58:28 dansmith melwitt: and you won't share until confirmed?
16:58:32 sdague melwitt: ok, cool.
16:58:38 dansmith can I have a teaser for next week's episode?
16:58:42 cfriesen ralonsoh: the nice thing about setting it via /sys/class/net is that it gives you flexibility if you have different hardware using the same module.
16:59:05 ralonsoh cfriesen: I know and I agree. The problem is how to store it
16:59:07 dansmith sdague: melwitt: I just pushed up a ulimit hack to see if that affects anything, fwiw
16:59:26 ralonsoh cfriesen: do you think the idea of adding a pre-up command is enough?
16:59:46 sdague melwitt: yeh, I would feel much more comfortable twiddling knobs if we understood why we were hitting that
17:01:08 melwitt dansmith: heh. well, I could be totally wrong, but I think it might be because we aren't caching the oslo.db transaction context managers. I had thought oslo.db connection pooling underneath took care of that, but now I want to try and see if there's a difference when doing it
17:01:27 dansmith melwitt: hmm
17:01:44 dansmith melwitt: so we also disable some amount of tpooling, which isn't connection pooling that I know of, but..
17:01:58 dansmith so maybe we're already breaking things and what I'm doing is just pushing us over a limit?
17:02:24 melwitt that's what I'm thinking, but it's a total guess
17:03:44 dansmith hmm, that would be like bad and stuff

Earlier   Later