| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-29 | |||
| 18:53:42 | sean-k-mooney | its proably suppor fragile if the host is in two aggrates with different values | |
| 18:54:57 | mriedem | you can have per-aggregate weights | |
| 18:55:27 | sean-k-mooney | well from the code yes but if a host is in two aggrates and they have differnt values im not sure what we do | |
| 18:55:43 | konetzed | melwitt: sean-k-mooney: thanks for the quick answer. | |
| 18:55:54 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 18:56:00 | mriedem | "If more than one value is found for a host in aggregate metadata, the minimum value will be used." | |
| 18:56:02 | mriedem | depends on the weigher | |
| 18:56:05 | mriedem | but the docs are all tehre | |
| 18:56:06 | mriedem | *there | |
| 18:56:10 | mriedem | per-aggregate weights is new in stein | |
| 18:56:13 | mriedem | yikun added it | |
| 18:56:27 | melwitt | konetzed ^ updated answer :P sorry | |
| 18:57:12 | konetzed | mriedem: thanks. Looks like i need to update from Rocky ;) | |
| 19:26:34 | mriedem | melwitt: ack i'll get back to it in a bit | |
| 19:26:42 | melwitt | thanks | |
| 19:27:01 | mriedem | unrelated plug for a question about the first stable stein release for anyone that has input http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006617.html | |
| 19:27:11 | mriedem | otherwise i'll probably just punt and un-WIP the release patch | |
| 19:27:52 | sean-k-mooney | mriedem: oh ya i ment to reply. i think adding a known issuse release not makes sense | |
| 19:28:01 | sean-k-mooney | im just not sure what it shoudl say | |
| 19:33:16 | mriedem | btw is this the heartbeat thing we'd expect to see in the n-api logs for that bug http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/screen-n-api.txt.gz?level=TRACE#_May_29_18_00_29_897709 ? | |
| 19:33:25 | mriedem | because it's all over these tempest-pg-full logs | |
| 19:36:15 | melwitt | I haven't actually seen that log for this.. when I reproduced it in devstack I only saw the "connection reset" error message from oslo.messaging. and some other messages about retrying/reconnecting | |
| 19:37:35 | mriedem | i see lots of "missed heartbeats from client, timeout: 60s" in the rabbit logs | |
| 19:37:38 | mriedem | http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/rabbitmq/rabbit@ubuntu-bionic-rax-iad-0006739854_log.txt.gz | |
| 19:37:42 | sean-k-mooney | it looks like the tcp connection is also being closed | |
| 19:38:11 | sean-k-mooney | mriedem: ya that is expected | |
| 19:39:22 | sean-k-mooney | if we dont disabel monkey patching or disabel the heartbeat then we will have missed heartbeat message in rabbitmq | |
| 19:40:41 | mriedem | i guess my point is there was concern about not hitting this in the gate | |
| 19:40:55 | mriedem | even though we run with uwsgi in devstack and eventlet monkey-patched | |
| 19:41:38 | sean-k-mooney | well in the gate we should see this since we have 1 thread per wsgi process | |
| 19:41:48 | sean-k-mooney | but the api sould actuly still work | |
| 19:42:12 | melwitt | yeah, the gate is normally too busy with activity to cause the heartbeat interruption where the wsgi app pauses bc of lack of activity | |
| 19:43:20 | melwitt | so... I dunno how that's showing up in tempest-pg-full :\ | |
| 19:43:40 | zigo | mriedem: Hey matt, thanks again for your patch, it fixed it for us. I've uploaded the fix to Sid, and hopefully it will be part of Buster (currently filling-up an unblock bug to the Debian release team). | |
| 19:44:35 | mriedem | zigo: yw. as noted on the review, i'll likely be modifying that patch a bit. | |
| 19:44:37 | sean-k-mooney | melwitt: wll its not printing constantly so it posible that ocationally we do time out even in the gate | |
| 19:45:17 | melwitt | o | |
| 19:48:52 | mriedem | brb | |
| 19:50:14 | efried | aspiers, kashyap: Minor tweaks to https://review.opendev.org/#/c/655193/ if either of you is around. | |
| 19:50:24 | aspiers | kashyap's gone I think | |
| 19:50:31 | aspiers | and of course I'm not here | |
| 19:50:38 | efried | okay, nbd | |
| 19:50:46 | aspiers | I wouldn't be so crazy as to be working at 9pm | |
| 19:51:00 | aspiers | he said, while writing functional tests :-/ | |
| 19:53:26 | sean-k-mooney | melwitt: you know one thing people could try is setting the ideal time for the wsgi server to like 300s instead of whatever teh default is | |
| 19:54:11 | sean-k-mooney | if you are not reciving at least 1 api request every 5 minutes to keep it alive you proabley dont need the performace boost from moneypatching | |
| 19:55:09 | sean-k-mooney | you will endup useing more ram as the wsgi app will be loaded more often but its tradoffs nomater what we do | |
| 19:55:11 | melwitt | well... what about in the middle of the night, when no one's using the api potentially? I feels like any deployment, no matter how large or busy, will have some down time at some point during the day | |
| 19:56:25 | sean-k-mooney | in that case it will suspend after 5 mins and then the next request it recived it will reconnect | |
| 19:56:57 | melwitt | ok, so you're still talking in the context of threads=1 and just avoiding the scary heartbeat messages more | |
| 19:57:11 | sean-k-mooney | but if people want to reduce how often that log message is printed they can tweak how agress the wsgi server is at sleeping | |
| 19:57:17 | sean-k-mooney | ya | |
| 19:57:25 | melwitt | I thought you were suggesting increasing the idle timeout as a way of allowing threads > 1 to work | |
| 19:57:27 | melwitt | ok | |
| 19:57:35 | aspiers | efried: I finally found MAX_INT in placement.db.constants, but that's not available from the unit testing testenvs | |
| 19:57:55 | efried | aspiers: Swear there was somewhere else we were using that. | |
| 19:57:56 | aspiers | using nova.db.constants for now | |
| 19:58:01 | aspiers | but that's not really correct | |
| 19:58:06 | aspiers | even if they happen to be the same value | |
| 19:58:41 | aspiers | efried: it's available to functional testenvs, but I need it in one unit test also | |
| 19:58:51 | melwitt | mriedem: I can take a stab at the stein reno and then have sean-k-mooney correct me because sean knows more than I do about it at this point | |
| 19:58:52 | efried | aspiers: You need it in your code too | |
| 19:59:10 | sean-k-mooney | aspiers: you could just use 2**64 or 2**32 | |
| 19:59:27 | aspiers | efried: well sure, but presumably it's safe to use placement from nova code? | |
| 19:59:34 | efried | not at all | |
| 19:59:41 | aspiers | huh | |
| 19:59:58 | sean-k-mooney | aspiers: we cant import random parts of placement in nova | |
| 20:00:04 | aspiers | OK | |
| 20:00:14 | efried | Right. Placement, from nova's perspective, is supposed to be purely a REST interface. Absolutely nothing else. | |
| 20:00:25 | efried | I think we're importing a fixture to make some testing easier | |
| 20:00:36 | efried | but prod code should be purely REST touchpoints. | |
| 20:00:38 | aspiers | Seems totally reasonable to me if it exported constants at least, if nothing else | |
| 20:00:48 | efried | yeah, I could get behind that in theory. | |
| 20:00:56 | efried | There has been talk of making a placement | |
| 20:01:00 | efried | placement-lib project | |
| 20:01:01 | sean-k-mooney | why not just creat the constant in nova | |
| 20:01:12 | melwitt | sean-k-mooney: aside, I wonder did we ever get this sorted downstream in the bz's now that we know we should _not_ disable heartbeats | |
| 20:01:18 | aspiers | sean-k-mooney: like I said, I'm just using nova.db.constants.MAX_INT | |
| 20:01:21 | sean-k-mooney | if its a constant its not going to change often one would hope | |
| 20:01:23 | efried | which would contain things like os-traits and os-resource-classes as well as constants and test fixtures. | |
| 20:01:45 | artom | Wouldn't test fixtures go into oslo_test? | |
| 20:01:57 | efried | not placement test fixtures | |
| 20:01:58 | sean-k-mooney | melwitt: i think martin was going to try testing it with heatbeats disabel to see what happens | |
| 20:02:09 | aspiers | nova.db.constants.MAX_INT isn't technically correct, but it's so damn close that I think it's good enough | |
| 20:02:28 | efried | aspiers: Yeah, was gonna say, as long as you use something you're sure is <= what placement can handle, "arbitrarily large" could be pretty much anything. | |
| 20:02:34 | artom | 31337 :D | |
| 20:02:36 | efried | You could hard code it to 2000 and that would probably be fine. | |
| 20:03:09 | aspiers | efried: actually answering that might breach an NDA ... | |
| 20:03:18 | melwitt | sean-k-mooney: hm... well, it will work on the surface but the issues dansmith pointed out were that rabbit will leave stale connections lying around in that case, so overall it's not a good idea to disable. hopefully martin will test that out as part of it | |
| 20:03:59 | sean-k-mooney | oh i did not know about the stale connection issue? | |
| 20:04:28 | aspiers | artom: probably safer than 1337 ;) | |
| 20:04:35 | sean-k-mooney | melwitt: we we ar confident we should not disable heatbeats then we can save martin time and do? | |
| 20:04:58 | aspiers | artom: which was one of the codes in one of the Denver escape rooms XD | |
| 20:05:07 | artom | Noice. | |
| 20:05:13 | sean-k-mooney | nothing? supress log? make ideal time 5 minuets for mod_wsgi? | |
| 20:06:36 | sean-k-mooney | i think the current situation is more or less lets figure out how to fix it upstream and then backport that fix upstream and downstream | |
| 20:07:36 | melwitt | sean-k-mooney: yes, based on the fact that rabbit will not clean up stale connections in that case, I don't think we should disable it. admittedly I don't understand a ton about this, but the docs recommend against disabling unless tcp keepalives are used in their place https://www.rabbitmq.com/heartbeats.html#disabling | |
| 20:08:03 | sean-k-mooney | we do use tcp keepalives | |
| 20:08:17 | sean-k-mooney | and we set it to like 1 second intervals | |
| 20:08:58 | sean-k-mooney | triplos defalts are pretty agressive in trying to keep the connection alive | |