Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-15
16:34:48 mriedem https://github.com/openstack-dev/devstack/tree/master/lib/databases
16:34:51 sdague we're exceeding 1024?
16:34:59 sdague that seems suspicious to me
16:35:27 dansmith well, what's our connection pool size?
16:36:26 mriedem we used to save off the mysql config i thought
16:36:28 mriedem but i can't find it
16:36:41 dansmith connection pool size would be something in our own config
16:37:06 dansmith just noticed by the way,
16:37:13 dansmith that we don't get the config dump at the beginning of n-api logging anymore
16:37:18 sdague dansmith: I don't know that we're setting the pool
16:37:19 dansmith maybe because of the wsgi patch?
16:38:46 dansmith looks like 5 per worker maybe
16:38:49 dansmith which is pretty low
16:39:06 dansmith that would be ~204 workers to exhaust 1024
16:39:11 sdague dansmith: I still see the dump
16:39:15 sdague http://logs.openstack.org/26/464326/1/check/gate-tempest-dsvm-neutron-src-oslo.context-ubuntu-xenial/4fc411c/logs/screen-n-api.txt.gz#_May_14_11_27_57_370073
16:39:18 dansmith which seems unlikely on a single node
16:39:21 sdague it's just a little later
16:39:53 dansmith oh okay, that seems odd given we're loading extensions before that, but okay
16:40:25 sdague I think the new route system compiles early
16:40:33 dansmith okay
16:40:35 sdague the stuff that completely does away with the extension loader
16:40:53 sdague we honestly could probably turn off debug on those
16:41:11 dansmith now that th3y
16:41:16 dansmith they are all implicit yeah
16:41:25 dansmith anyway, I dunno what to do about this connection thing
16:42:14 dansmith presumably if we go higher we also need to set maxfds
16:42:20 dansmith unless we already have for something else
16:43:02 sdague yeh, I don't know, but I was pretty sure that was the db rejecting things. Any chance we're leaking db connections or something?
16:44:04 dansmith nothing that fundamental has changed recently, so I dunno
16:46:29 dansmith looks like we may be limited below max_connections due to ulimit on descriptors anyway
16:46:34 dansmith so maybe we're not getting 1024 even
16:49:59 cfriesen ralonsoh: here now, you still have a question or has it been dealt with (I'm checking history now)
16:51:40 ralonsoh cfriesen: well, I have an answer from libvirt guys
16:52:31 ralonsoh cfriesen: http://paste.openstack.org/show/609591/
16:52:57 ralonsoh cfriesen: at the first time, the suggest me to do what is NOT recommended in openstack manuals
16:53:52 ralonsoh cfriesen: they recommended to modify the kernel module
16:53:54 ralonsoh cfriesen: https://github.com/openstack/openstack-manuals/blob/master/doc/networking-guide/source/config-sriov.rst
16:57:46 cfriesen ralonsoh: technically you're not modifying the kernel module itself, just the parameter on loading it. Arguably either the module parameter or setting it via /sys/class/net would work fine, it's just that they may at some point remove the module parameter in the future.
16:58:05 melwitt dansmith, sdague: on the db connection thing, coincidentally I'm building a new devstack right now to look into it. I have a guess at something that might be related but need to verify
16:58:28 dansmith melwitt: and you won't share until confirmed?
16:58:32 sdague melwitt: ok, cool.
16:58:38 dansmith can I have a teaser for next week's episode?
16:58:42 cfriesen ralonsoh: the nice thing about setting it via /sys/class/net is that it gives you flexibility if you have different hardware using the same module.
16:59:05 ralonsoh cfriesen: I know and I agree. The problem is how to store it
16:59:07 dansmith sdague: melwitt: I just pushed up a ulimit hack to see if that affects anything, fwiw
16:59:26 ralonsoh cfriesen: do you think the idea of adding a pre-up command is enough?
16:59:46 sdague melwitt: yeh, I would feel much more comfortable twiddling knobs if we understood why we were hitting that
17:01:08 melwitt dansmith: heh. well, I could be totally wrong, but I think it might be because we aren't caching the oslo.db transaction context managers. I had thought oslo.db connection pooling underneath took care of that, but now I want to try and see if there's a difference when doing it
17:01:27 dansmith melwitt: hmm
17:01:44 dansmith melwitt: so we also disable some amount of tpooling, which isn't connection pooling that I know of, but..
17:01:58 dansmith so maybe we're already breaking things and what I'm doing is just pushing us over a limit?
17:02:24 melwitt that's what I'm thinking, but it's a total guess
17:03:44 dansmith hmm, that would be like bad and stuff
17:03:49 sdague lyarwood - https://review.openstack.org/#/c/464664/ approved and in gate, so hopefully you'll be back to working in ~ 1 hr
17:03:51 dansmith not really sure how we'd not be gc'ing them, but..
17:03:57 melwitt I mean, I think cfriesen or somebody had mentioned recently they saw total db connections go way up since newton or ocata compared with before cells v2. so that had me thinking already that I wanted to look into this
17:04:03 dansmith maybe we're just hitting a rate that makes it a problem
17:04:31 dansmith melwitt: well, we would have an increase because of the api db
17:04:55 cfriesen ralonsoh: in our environment if a PF has VFs we only ever pass the VFs through to the guest so no personal experience with what you're trying to do. In the case where you're using VFs are you "up"ing the PF?
17:05:06 dansmith I guess if we're leaking somehow, we'd get worse the more targeting we do,
17:05:08 dansmith but..
17:05:25 melwitt yeah. so what I'm thinking might not be related, so I didn't want to get anyone worried about it untiI checked
17:05:38 cfriesen melwitt: we hit increased DB connections going from Mitaka to Newton
17:05:59 dansmith cfriesen: did you look at where those were coming from? i.e. only from api/conductor workers?
17:06:23 ralonsoh cfriesen: it's a user case (and a possible bug). You can use both, PF and VFs, but not at the same time. But if you start a VM with a PF and then you need to use the VFs, those are lost
17:06:26 dansmith well, I guess if we're leaking those would be the services anyway, so.. that might not tell us anything
17:09:07 cfriesen dansmith: I'm still catching up on email, but I don't think we've bottomed out exactly where they're coming from. I think we just bumped the number of connections for now.
17:09:41 dansmith alright
17:10:26 cfriesen dansmith: got any clever ways to figure out what code has open connections?
17:10:34 dansmith netstat?
17:10:37 jmlowe I've been having a problem with nova, I'm pretty sure bugs have been reported years ago (can't find them atm) but have languished, when an instance goes into error state the underlying disk is deleted, this is exactly the wrong thing to do
17:11:02 cfriesen dansmith: that's the easy part. what about within the python code?
17:11:23 dansmith cfriesen: well, knowing what binary has the excess is a start
17:11:37 dansmith we'd need extra instrumentation to know which code I think
17:12:22 mriedem jmlowe: never heard of that
17:12:22 jmlowe for instance, Wednesday during the summit a user resumed a suspended instance that didn't have enough memory so qemu wouldn't start, this is eminently fixable unless nova comes along and deletes the underlying block device for the instance
17:13:36 jmlowe If I dig back in my logs I even have errors from nova repeatedly deleting the same rbd device after the first wrongful deletion is succeeds
17:14:55 jmlowe no probably not just a gross conceptual error in the state diagram but also a bad implementation
17:15:43 jmlowe also kind of hard to replicate reliably on a 300 compute node production system
17:15:46 mriedem jmlowe: what you might be thinking of i,s
17:15:51 mriedem resume in the libvirt driver calls this https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L2562
17:16:12 mriedem if that fails in some unexpected way, we go here https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L5120
17:16:22 jmlowe will also trigger on error state due to migration error
17:17:14 mriedem destroy_disks_on_failure defaults to False here
17:17:18 mriedem and isn't set to True for the resume case
17:17:40 mriedem so when calling cleanup, destroy_disks would be False https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L5042
17:18:44 mriedem jmlowe: i'm confused, are you doing a migration or a resume?
17:19:08 jmlowe happens for both, seemingly any time an instance goes into error
17:19:10 mriedem because if is_shared_block_storage=True, then https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L1014 would delete instance files on disk
17:19:30 jmlowe in my case ceph backed
17:19:35 mriedem the instance going to error state is not a related issue for deleting the disks in the virt driver
17:20:11 mriedem jmlowe: which release of nova are you on?
17:20:17 jmlowe newton
17:20:27 jmlowe don't think it was doing this in mitaka
17:20:39 mriedem i don't really see anything in newton when resume would fail that would destroy the disks
17:21:24 mriedem now, i think that _cleanup_failed_start method was new in ocata and backported to newton...
17:21:47 mriedem yeah https://review.openstack.org/#/q/Iab5afdf1b5b8d107ea0e5895c24d50712e7dc7b1
17:23:13 mriedem hmm, destroy disks wasn't passed through at that point
17:23:15 mriedem checking something

Earlier   Later