Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-24
15:50:55 mriedem gibi: replied, i think your suggestion is ok
15:52:42 finucannot mriedem, efried: https://review.opendev.org/661344
15:53:25 dansmith melwitt: I think disabling heartbeats will increase recovery time if the connection goes stale, and will cause stale connections to stay open on the rabbit side
15:53:34 dansmith melwitt: so I think threads=1, leave heartbeat behavior
15:54:23 melwitt sean-k-mooney, mdbooth ^
15:54:52 aspiers efried: I'm pondering whether my helpers fall foul of https://docs.python.org/3/library/unittest.mock.html#where-to-patch
15:54:52 melwitt dansmith: thanks
15:56:06 aspiers efried: although that wouldn't explain why they have been working so far
15:56:33 gibi mriedem: thanks.
15:57:07 sean-k-mooney dansmith: if we leave teh hearbeat enable should we at least change the log level in the nova-api for the oslo messaging logs?
15:57:19 sean-k-mooney or the heart beat on in partiaclar
15:59:16 efried finucannot: why does starlingx get different zebra stripe?
15:59:18 dansmith I dunno, but we might want to soften the log level in o.msg where it raises holy hell about the reconnct
15:59:22 efried *I* want special zebra stripe
16:00:09 efried aspiers: I'm going to have to get back to you (possibly much) later
16:00:35 aspiers efried: it's OK. Attacking it with a debugger now
16:00:51 sean-k-mooney dansmith: so we suggested that and got some push back to droping it to info in oslo
16:01:38 finucannot efried: Whoops, missed that
16:01:41 finucannot fixed
16:01:49 dansmith sean-k-mooney: so what is your thought, to specifically set o.msg level to error in nova-api?
16:02:24 sean-k-mooney dansmith: effectivly but i think we can be more granular then that. we do this for some privesep stuff right?
16:02:43 dansmith sean-k-mooney: we can set defaults on the logger name level, I think
16:02:59 sean-k-mooney dansmith: actully setting it to error wont help as o.msg is currently loggin it as an error
16:03:16 sean-k-mooney so more we woudl have to install a log filter to speficaly drop that error
16:04:07 dansmith sean-k-mooney: well, I dunno, it *really* seems like that shouldn't be an error, it's just a reconnect, which is exactly what it's supposed to be doing
16:04:21 dansmith so whatever.. however you make it not show up, or leave it and document it or whatever, I care about that less
16:04:39 dansmith what I do care about is that we not disable heartbeats just because it's making a statement in the logs we don't want
16:04:55 mriedem monkeypatch the oslo.messaging code!
16:05:14 beekneemech mriedem: Crazy talk! :-P
16:05:21 sean-k-mooney dansmith: ya i agree it not an error. maybe we could make it a warning in o.mesg and then set the log level to error for o.msg in nova
16:05:34 dansmith sean-k-mooney: yeah
16:07:30 mriedem if only DEFAULT_LOG_LEVELS wasn't a gd list
16:08:05 mriedem makes it a pita to parse and override from code
16:08:54 melwitt dansmith: IIUC from past conversation, the ideal way to handle the eventlet monkey patch issue longer term would be to separate the wsgi app part of nova-api from the rest of it, so we don't have behavior that doesn't fit in with the wsgi app happening inside of it. what would that look like? a separate process for each of nova-api-wsgi and nova-api?
16:10:53 dansmith melwitt: yeah
16:11:10 dansmith that's how wsgi apps area really supposed to be done, AFAIK,
16:11:26 dansmith the wsgi bit is just the "view" and you delegate everything to the controller
16:11:43 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Delete allocations even if _confirm_resize raises https://review.opendev.org/652153
16:11:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional confirm_migration_error test https://review.opendev.org/658136
16:11:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: [stable-only] Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/661349
16:12:50 mriedem delicious MVC
16:13:07 mriedem ganso: ^ likely needs some work b/c the functional test in queens isn't failing when i remove the fix
16:13:21 mriedem struts baby!
16:13:29 mriedem that's the last time i did UI Work
16:13:31 mriedem *work
16:13:49 mriedem Faces and AJAX was a brand new world to me when i left that dept
16:14:19 melwitt dansmith: ok, cool. I wanted to get an idea about what's involved to see if we could make that happen this cycle.
16:14:53 melwitt to do it properly for future
16:15:05 cdent melwitt, dansmith : are you talking about a "controller" as a separate process?
16:15:28 dansmith melwitt: uh, that would be a significant undertaking
16:15:38 dansmith cdent: yeah
16:15:39 melwitt oh :(
16:15:42 cdent yeah, that's why I ask. It would be significant
16:15:49 dansmith cdent: definitely
16:16:36 cdent a somewhat shorter step would be to make a "scatter-gatther-cells-agent" that can be threaded (in whatever way) that the wsgi app talks to over a unix socket
16:16:46 dansmith melwitt: going from a thing designed to be eventlet-based to just slapping a wsgi frontend on it is the kind of thing that doesn't end with a small refactor to substantially change how it works :)
16:17:08 cdent but even that agent idea would be an undertaking
16:17:11 melwitt the concern was, the near term things we can do are to somehow hide the log message and set threads=1. but what's the long term thing to do, or is the near term fix going to be the forever fix
16:17:24 dansmith cdent: yeah, could do that, but that's a lot of deployment change for just that thing
16:17:57 cdent dansmith: yeah, no doubt. (I guess you could also fork it as needed)
16:18:53 cdent call scatter-gather-cells over rpc to the conductor ?
16:19:19 dansmith cdent: that will slow down instance list quite a bit
16:19:58 cdent Yeah, I don't really have much in the way of ideas. We've got impedance mismatches galore.
16:20:14 cdent s/ideas/ideas that can be realized/
16:20:20 melwitt yeah... just wanted to understand whether we'll have a next step here or if this is it. sounds like realistically, there's not more we could do
16:20:23 dansmith yep.. IMHO, fix the short term and then let's properly design the bigger change after we see how that goes
16:20:37 dansmith and not armchair quarterback it the day before a holiday weekend (for some of us) :)
16:21:00 melwitt haha, no, I was only looking for a high-level ballpark idea of what it would look like
16:21:23 melwitt because when you said "separate wsgi part" initially, I didn't really know what that meant
16:50:26 openstackgerrit Merged openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.opendev.org/556751
16:58:51 aspiers gibi: you around? I'm trying to get a functional test to restart the compute service in a way which triggers init_host()
16:59:05 aspiers gibi: and I see you worked on restart_compute_service()
16:59:35 aspiers unfortunately that only restarts the Host, not the Service
17:02:52 aspiers I tried self.compute.stop(); self.start_compute() but the latter invokes nova.test.TestCase.start_service() which borks by trying to create a duplicate HostMapping
17:03:23 aspiers dansmith: I see your fingerprints on this code too :)
17:03:58 aspiers This is about the point where my newbie-ness makes it pretty hard to figure out a way forward
17:10:22 openstackgerrit Adam Spiers proposed openstack/nova master: Provide HW_CPU_AMD_SEV trait when SEV is supported https://review.opendev.org/638680
17:11:40 melwitt aspiers: I don't know much about that but you reminded me of a patch of gibi's that might be of help https://review.opendev.org/512552
17:12:55 aspiers melwitt: thanks! yeah that's the exact area of code I'm talking about
17:13:02 aspiers maybe that change will yield some clues
17:13:10 melwitt that's what I'm hoping :)
17:14:45 dansmith aspiers: "restart the host not the service" does not make sense to me, fwiw
17:15:29 dansmith aspiers: calling start_service() again is not the right thing to do, because it's really create_and_start_service() I think
17:15:45 aspiers yes, I think that's the problem
17:15:46 dansmith aspiers: if you want to re-run init_host() I would just run it
17:17:48 dansmith aspiers: the restart_compute_service() comment explains why doing a restart in a functional test isn't super realistic, and that's why it saves/restores the RT across the start/stop
17:18:01 dansmith so yeah, I think you just want to re-run init if that's what you're trying to do
17:18:05 aspiers OK, thanks
17:18:42 aspiers I wasn't sure if init_host() depended on a bunch of other stuff being run before it in the same context
17:19:09 dansmith well, you can see what it's doing
17:19:38 aspiers I guess self.basic_config_check() is the only real thing done before
17:19:48 aspiers and I don't need to re-run that
17:20:03 dansmith init_host doesn't really do much other than run the driver's init really.. what about it do you need to restart?
17:20:12 dansmith re-run the init_instance parts?
17:20:16 aspiers the SEV check I've added to it
17:20:23 dansmith oh okay
17:20:39 aspiers oh, actually I don't think this will be good enough by itself
17:20:47 aspiers because the SEV check only sets an instance variable
17:21:00 aspiers which is later consumed in u_p_t() to provide the trait
17:21:10 aspiers but maybe _run_periodics() will take care of that

Earlier   Later