| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-16 | |||
| 15:11:35 | dansmith | sdague: I'm about finished with a second run to get another sample, which appears to also have worked | |
| 15:11:44 | mriedem | shouldn't ulimit be 64k in deployment now? | |
| 15:11:48 | sdague | ok, cool, because our ulimit is 1024 by default or something | |
| 15:11:51 | mriedem | deployment == devstack? | |
| 15:12:00 | dansmith | sdague: this bit: https://review.openstack.org/#/c/436094/28/lib/databases/mysql | |
| 15:13:40 | sdague | ok cool | |
| 15:14:28 | dansmith | sdague: so, I dunno whether to just (ask you to allow) that into devstack or what | |
| 15:14:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use plain routes list for os-services endpoint instead of stevedore https://review.openstack.org/460818 | |
| 15:14:48 | dansmith | sdague: 1024 still seems high, but we do have a bunch of databases, tables, and connections going on... | |
| 15:15:21 | sdague | dansmith: yeh, if we've got 1024 db connections it feels like ulimit should at least be double that | |
| 15:15:38 | dansmith | okay, I'll split that out and make it 2048 | |
| 15:15:43 | sdague | otherwise it's just shooting ourselves in the foot | |
| 15:15:47 | dansmith | right | |
| 15:16:24 | sdague | dansmith: so... here is what I think we should do | |
| 15:16:36 | sdague | add a new stackrc variable so that's tunable | |
| 15:16:42 | sdague | default to 2048 | |
| 15:16:45 | dansmith | okay | |
| 15:16:50 | sdague | do the change early in stack.sh for the system | |
| 15:17:07 | sdague | then if people want to play around with other values for constraints, it's easy | |
| 15:17:15 | dansmith | okay | |
| 15:17:38 | dansmith | by early does that mean in stack.sh or in functions somewhere? | |
| 15:18:27 | dansmith | and by "for the system" you mean in limits.conf? | |
| 15:19:02 | dansmith | and hope that systemd just inherits that properly when we start mysql? | |
| 15:19:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: rbd: Remove unnecessary 'encode' calls https://review.openstack.org/412356 | |
| 15:21:03 | sdague | dansmith: well probably build a function for it, but call it before we start any services in stack.sh directly | |
| 15:21:08 | sdague | yeh, limits.conf | |
| 15:21:15 | dansmith | okay | |
| 15:21:23 | sdague | and if set, I'm hoping it inherits through for the world | |
| 15:22:42 | dansmith | yeah, limits.conf is read by pam, so I'm not sure | |
| 15:24:30 | dansmith | sdague: just tested, doesn't seem to have any effect | |
| 15:24:39 | dansmith | system limits on the service Imean | |
| 15:24:47 | dansmith | even with a daemon-reload | |
| 15:25:17 | mdbooth | https://access.redhat.com/solutions/1257953 | |
| 15:25:27 | mdbooth | Although I think that's paywalled | |
| 15:25:34 | dansmith | mdbooth: I already set it with systemd and it works | |
| 15:25:42 | dansmith | we were hoping we could do it generically with limits.conf | |
| 15:25:49 | mdbooth | But yeah, systemd doesn't read limits.conf | |
| 15:25:57 | dansmith | sdague: ^ | |
| 15:26:19 | dansmith | sdague: so I'm guessing I just make that stackrc var mysql-specific | |
| 15:27:10 | dansmith | or I can just comment over it that it's only enforced for things that look at it (like mysql) | |
| 15:29:10 | cfriesen | dansmith: would something like this be okay, or is there some reason it would be terrible? http://paste.openstack.org/show/609690/ We're seeing the cache update be somewhat expensive and wondering if we can just batch the cache updates when there are multiple network-changed events in a batch. | |
| 15:30:26 | dansmith | cfriesen: terrible? you'd never notice cache changes past the first one | |
| 15:31:47 | mdbooth | cfriesen: Another problem with this kind of cache is that they grow unbounded. Possibly not a practical issue with instance uuids, but still icky. | |
| 15:34:43 | cfriesen | dansmith: could we remove all network-changed events for this instance from the "events" list, then call self.network_api.get_instance_nw_info()? | |
| 15:35:05 | sfinucan | mriedem: Could you look at the 'reserve-numa-with-pci' patches at some point this week or next? https://review.openstack.org/#/q/status:open+project:openstack/nova+topic:bp/reserve-numa-with-pci | |
| 15:35:26 | mriedem | maybe, artom is in front of you | |
| 15:35:32 | cfriesen | dansmith: it seems pointless to call it multiple times if it's going to give us the same answer each time | |
| 15:35:34 | dansmith | cfriesen: eh? the events list is transient.. things get removed from there when processed | |
| 15:35:42 | sdague | hmmmm | |
| 15:35:43 | dansmith | cfriesen: that's the nature of a cache refresh | |
| 15:36:09 | sdague | dansmith: well, what's the set with systemd option? Build that unit file? | |
| 15:36:28 | dansmith | cfriesen: maybe I can't see enough of your change.. are you saying you see a single request to refresh an instance multiple times in the same go? | |
| 15:36:30 | dansmith | sdague: yeah | |
| 15:36:51 | sdague | dansmith: I'm fine fixing it just for the systemd case | |
| 15:36:58 | sdague | we're default there everywhere | |
| 15:37:07 | dansmith | sdague: okay, but it'll only be for mysql | |
| 15:37:19 | sdague | oh ... let me go look again | |
| 15:37:26 | cfriesen | dansmith: we are coming into this code and there are multiple network-changed events for the same instance, so we end up calling self.network_api.get_instance_nw_info() multiple times | |
| 15:37:45 | dansmith | cfriesen: ah, okay I assumed this was in the periodic | |
| 15:37:49 | dansmith | cfriesen: why is that happening? | |
| 15:39:36 | cfriesen | dansmith: let me get more context, this was proposed by one of our other guys. | |
| 15:39:44 | dansmith | *facepalm* | |
| 15:39:54 | sdague | dansmith: what about DefaultLimitNOFILE= in /etc/systemd/system.conf | |
| 15:39:58 | sdague | then a daemon restart | |
| 15:40:11 | dansmith | sdague: will that get picked up by a daemon restart? | |
| 15:40:16 | dansmith | I mean reload | |
| 15:40:21 | dansmith | restart is a full re-exec right? | |
| 15:40:24 | bauzas | dansmith: quick question, should you think we would need a new RPC version for adding a "alternatives" key to the returned dict when calling the scheduler API method select_destinations() ? | |
| 15:40:29 | sdague | dansmith: I don't know | |
| 15:40:42 | sdague | but that's where the google's are leading me to | |
| 15:40:56 | dansmith | sdague: okay will try that | |
| 15:41:15 | bauzas | dansmith: my own opinion is that given we say to first upgrading the conductor, then the conductor is anyway newer that the scheduler, so it would know the new key | |
| 15:41:31 | sdague | dansmith: daemon-reload probably should be tried first | |
| 15:41:37 | dansmith | sdague: yeah | |
| 15:41:44 | openstackgerrit | dane-fichter proposed openstack/nova master: Add trusted certificates to InstanceExtras https://review.openstack.org/457711 | |
| 15:41:49 | dansmith | bauzas: of course, if you're changing the rpc api... you should version that change :) | |
| 15:41:56 | bauzas | dansmith: but I could just add a new RPC API version for not returning the new key in case the asked version is not cool | |
| 15:42:16 | bauzas | dansmith: I'm not modifying the interface | |
| 15:42:23 | bauzas | dansmith: so it's not needing a version | |
| 15:42:36 | bauzas | dansmith: I'm just adding some new key for what we return | |
| 15:42:55 | bauzas | but lemme just upload the change | |
| 15:43:00 | bauzas | and you'll see | |
| 15:43:39 | dansmith | changing the return is changing the api, period. | |
| 15:44:25 | bauzas | okay | |
| 15:50:56 | cfriesen | dansmith: with regards to the multiple network-changed events for the same instance, this is in the context of a live-migration on a busy system. Adding to the pain is that the "with lockutils.lock" took a long time to be acquired (which seems to be due to the lack of a "fair" lock in oslo). | |
| 15:54:46 | dansmith | cfriesen: not sure I get it, but if that is just avoiding processing the same instance in a single go, then I guess it makes sense | |
| 15:54:55 | dansmith | (but should use a set() of course) | |
| 16:02:38 | cfriesen | dansmith: yes, it's just to try to process each instance once. I guess neutron must be batching multiple events in a single call. | |
| 16:02:53 | dansmith | I'm pretty sure they don't | |
| 16:03:30 | cfriesen | dansmith: then why is the code written to handle multiple events per call? | |
| 16:03:35 | dansmith | tbh, I think I'd want an analysis of that first.. like "neutron started batching things in commit <foo> therefore we need this" | |
| 16:03:36 | dansmith | otherwise I'd be highly suspect that something else is going on | |
| 16:04:00 | dansmith | cfriesen: because the whole thing was written to be high bandwidth, but AFAIK, they never used it that way | |
| 16:04:24 | dansmith | and multiple events of the same instance in a single batch hardly makes sense | |
| 16:04:47 | dansmith | if they're doing that, then there needs to be a fix on both sides, but if you just paper over it here, we'll never find that | |
| 16:08:44 | melwitt | dansmith, sdague: I did some tests where I ran a subset of tempest locally with concurrency of 4 with and without caching oslo.db transaction context managers and with, it topped out around 90 Threads_connected in mysql and without it was around 140. so it did seem to make a difference. this is the patch I tried https://review.openstack.org/#/c/465042 | |
| 16:09:16 | openstackgerrit | Gage Hugo proposed openstack/nova master: Admin API Policy contingent on is_admin_project https://review.openstack.org/384148 | |
| 16:10:26 | sdague | melwitt: ok, cool, so that's definitely helpful | |
| 16:10:55 | dansmith | sdague: melwitt: I'm not sure I'm surprised by that, up to the limit of the connection pool | |
| 16:11:07 | sdague | melwitt: it feels like we should do that in addition to the raise of the open fds, right? | |