Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-29
19:33:16 mriedem btw is this the heartbeat thing we'd expect to see in the n-api logs for that bug http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/screen-n-api.txt.gz?level=TRACE#_May_29_18_00_29_897709 ?
19:33:25 mriedem because it's all over these tempest-pg-full logs
19:36:15 melwitt I haven't actually seen that log for this.. when I reproduced it in devstack I only saw the "connection reset" error message from oslo.messaging. and some other messages about retrying/reconnecting
19:37:35 mriedem i see lots of "missed heartbeats from client, timeout: 60s" in the rabbit logs
19:37:38 mriedem http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/rabbitmq/rabbit@ubuntu-bionic-rax-iad-0006739854_log.txt.gz
19:37:42 sean-k-mooney it looks like the tcp connection is also being closed
19:38:11 sean-k-mooney mriedem: ya that is expected
19:39:22 sean-k-mooney if we dont disabel monkey patching or disabel the heartbeat then we will have missed heartbeat message in rabbitmq
19:40:41 mriedem i guess my point is there was concern about not hitting this in the gate
19:40:55 mriedem even though we run with uwsgi in devstack and eventlet monkey-patched
19:41:38 sean-k-mooney well in the gate we should see this since we have 1 thread per wsgi process
19:41:48 sean-k-mooney but the api sould actuly still work
19:42:12 melwitt yeah, the gate is normally too busy with activity to cause the heartbeat interruption where the wsgi app pauses bc of lack of activity
19:43:20 melwitt so... I dunno how that's showing up in tempest-pg-full :\
19:43:40 zigo mriedem: Hey matt, thanks again for your patch, it fixed it for us. I've uploaded the fix to Sid, and hopefully it will be part of Buster (currently filling-up an unblock bug to the Debian release team).
19:44:35 mriedem zigo: yw. as noted on the review, i'll likely be modifying that patch a bit.
19:44:37 sean-k-mooney melwitt: wll its not printing constantly so it posible that ocationally we do time out even in the gate
19:45:17 melwitt o
19:48:52 mriedem brb
19:50:14 efried aspiers, kashyap: Minor tweaks to https://review.opendev.org/#/c/655193/ if either of you is around.
19:50:24 aspiers kashyap's gone I think
19:50:31 aspiers and of course I'm not here
19:50:38 efried okay, nbd
19:50:46 aspiers I wouldn't be so crazy as to be working at 9pm
19:51:00 aspiers he said, while writing functional tests :-/
19:53:26 sean-k-mooney melwitt: you know one thing people could try is setting the ideal time for the wsgi server to like 300s instead of whatever teh default is
19:54:11 sean-k-mooney if you are not reciving at least 1 api request every 5 minutes to keep it alive you proabley dont need the performace boost from moneypatching
19:55:09 sean-k-mooney you will endup useing more ram as the wsgi app will be loaded more often but its tradoffs nomater what we do
19:55:11 melwitt well... what about in the middle of the night, when no one's using the api potentially? I feels like any deployment, no matter how large or busy, will have some down time at some point during the day
19:56:25 sean-k-mooney in that case it will suspend after 5 mins and then the next request it recived it will reconnect
19:56:57 melwitt ok, so you're still talking in the context of threads=1 and just avoiding the scary heartbeat messages more
19:57:11 sean-k-mooney but if people want to reduce how often that log message is printed they can tweak how agress the wsgi server is at sleeping
19:57:17 sean-k-mooney ya
19:57:25 melwitt I thought you were suggesting increasing the idle timeout as a way of allowing threads > 1 to work
19:57:27 melwitt ok
19:57:35 aspiers efried: I finally found MAX_INT in placement.db.constants, but that's not available from the unit testing testenvs
19:57:55 efried aspiers: Swear there was somewhere else we were using that.
19:57:56 aspiers using nova.db.constants for now
19:58:01 aspiers but that's not really correct
19:58:06 aspiers even if they happen to be the same value
19:58:41 aspiers efried: it's available to functional testenvs, but I need it in one unit test also
19:58:51 melwitt mriedem: I can take a stab at the stein reno and then have sean-k-mooney correct me because sean knows more than I do about it at this point
19:58:52 efried aspiers: You need it in your code too
19:59:10 sean-k-mooney aspiers: you could just use 2**64 or 2**32
19:59:27 aspiers efried: well sure, but presumably it's safe to use placement from nova code?
19:59:34 efried not at all
19:59:41 aspiers huh
19:59:58 sean-k-mooney aspiers: we cant import random parts of placement in nova
20:00:04 aspiers OK
20:00:14 efried Right. Placement, from nova's perspective, is supposed to be purely a REST interface. Absolutely nothing else.
20:00:25 efried I think we're importing a fixture to make some testing easier
20:00:36 efried but prod code should be purely REST touchpoints.
20:00:38 aspiers Seems totally reasonable to me if it exported constants at least, if nothing else
20:00:48 efried yeah, I could get behind that in theory.
20:00:56 efried There has been talk of making a placement
20:01:00 efried placement-lib project
20:01:01 sean-k-mooney why not just creat the constant in nova
20:01:12 melwitt sean-k-mooney: aside, I wonder did we ever get this sorted downstream in the bz's now that we know we should _not_ disable heartbeats
20:01:18 aspiers sean-k-mooney: like I said, I'm just using nova.db.constants.MAX_INT
20:01:21 sean-k-mooney if its a constant its not going to change often one would hope
20:01:23 efried which would contain things like os-traits and os-resource-classes as well as constants and test fixtures.
20:01:45 artom Wouldn't test fixtures go into oslo_test?
20:01:57 efried not placement test fixtures
20:01:58 sean-k-mooney melwitt: i think martin was going to try testing it with heatbeats disabel to see what happens
20:02:09 aspiers nova.db.constants.MAX_INT isn't technically correct, but it's so damn close that I think it's good enough
20:02:28 efried aspiers: Yeah, was gonna say, as long as you use something you're sure is <= what placement can handle, "arbitrarily large" could be pretty much anything.
20:02:34 artom 31337 :D
20:02:36 efried You could hard code it to 2000 and that would probably be fine.
20:03:09 aspiers efried: actually answering that might breach an NDA ...
20:03:18 melwitt sean-k-mooney: hm... well, it will work on the surface but the issues dansmith pointed out were that rabbit will leave stale connections lying around in that case, so overall it's not a good idea to disable. hopefully martin will test that out as part of it
20:03:59 sean-k-mooney oh i did not know about the stale connection issue?
20:04:28 aspiers artom: probably safer than 1337 ;)
20:04:35 sean-k-mooney melwitt: we we ar confident we should not disable heatbeats then we can save martin time and do?
20:04:58 aspiers artom: which was one of the codes in one of the Denver escape rooms XD
20:05:07 artom Noice.
20:05:13 sean-k-mooney nothing? supress log? make ideal time 5 minuets for mod_wsgi?
20:06:36 sean-k-mooney i think the current situation is more or less lets figure out how to fix it upstream and then backport that fix upstream and downstream
20:07:36 melwitt sean-k-mooney: yes, based on the fact that rabbit will not clean up stale connections in that case, I don't think we should disable it. admittedly I don't understand a ton about this, but the docs recommend against disabling unless tcp keepalives are used in their place https://www.rabbitmq.com/heartbeats.html#disabling
20:08:03 sean-k-mooney we do use tcp keepalives
20:08:17 sean-k-mooney and we set it to like 1 second intervals
20:08:58 sean-k-mooney triplos defalts are pretty agressive in trying to keep the connection alive
20:09:22 melwitt the only possible thing we could do with the logging, if oslo.messaging ppl are agreeable, would be to downgrade the "connection reset" message to WARNING and then set our log level to ERROR to hide it
20:09:47 sean-k-mooney melwitt: i was looking it python log filters a bit too
20:09:57 melwitt and then obviously we have to have threads=1
20:10:18 sean-k-mooney i think we could intall a logfilter that would match on the oslo meesge and drop them too if oslo do not want to change the level
20:10:38 mriedem melwitt: ok, i'm torn between waffling over a reno on the issue and figuring out just the right words vs releasing what we have (which is quite a bit unreleased in stable/stein) and just letting the reno happen later if at all
20:10:40 melwitt oh, I see
20:10:47 mriedem b/c i'm not an authority to review that reno
20:11:23 melwitt yeah, that's fair
20:11:49 melwitt we don't need to do a release to get the reno out there right? it will just get published to the docs automatically
20:11:51 sean-k-mooney mriedem: i dont think the reno is critcal jsut nice to have and sice we are still debating what the best solution is im not sure we should hold up the release for it
20:15:48 melwitt sean-k-mooney: ok, well, I guess according to that doc it's maybe probably ok to disable heartbeats if we have tcp keepalives in place https://www.rabbitmq.com/heartbeats.html#tcp-keepalives I think either way we'd want thorough testing of it before we go ahead and put the change out there
20:19:26 sean-k-mooney melwitt: this is what triplo configures https://github.com/openstack/tripleo-heat-templates/blob/d90434187315e96d3ebc184866dcd91954c0471d/deployment/kernel/kernel-baremetal-puppet.yaml#L117-L122
20:21:48 sean-k-mooney it looks like that is even more agressive then rabbitmqs suggestions https://www.rabbitmq.com/networking.html#tcp-keepalives
20:22:30 melwitt mriedem: ack
20:22:49 melwitt oops
20:22:51 melwitt sean-k-mooney: ack
20:22:55 sean-k-mooney :)
20:25:39 melwitt mriedem: I'm fine either way releasing the reno now vs "soon" since AFAIK it doesn't have to be connected with the release, reno gets published independently. I was mostly saying I think we should publish a known issue vs not
20:26:11 mriedem i removed the -WIP so the release gods can do as they please

Earlier   Later