Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-24
15:42:06 finucannot sean-k-mooney: So there's more to it than this? https://review.opendev.org/#/c/661251/
15:42:09 finucannot efried: ^
15:42:33 sean-k-mooney that removed it from kolla
15:42:44 sean-k-mooney but they also need to remove it in kolla ansible
15:42:53 sean-k-mooney e.g. you just deleted teh container
15:43:16 sean-k-mooney but the upgrade playbook need to koll all running instance of the contain
15:43:21 sean-k-mooney *kill
15:43:54 finucannot Ah, I figured there was more to it than that. I can try drafting those patches based on what they did in OSA. I'd need one to kill the running instances (which would be backported to stable/stein) then a follow up to remove all references entirely
15:44:33 sean-k-mooney we need to kill this handeler https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/handlers/main.yml#L112-L136
15:44:48 efried finucannot: Okay, mriedem, melwitt, and sean-k-mooney are all +1, I'm approving the bp assuming the above is fairly easy and will be taken care of before we pull the trigger.
15:44:54 sean-k-mooney and remove like 2 or 3 other things but its a fairly small patch
15:46:08 finucannot sean-k-mooney: I'll do that before I clock off
15:47:55 sean-k-mooney ok you need to kill the service default https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/defaults/main.yml#L83-L93 and as i sad just update https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/rolling_upgrade.yml and https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/legacy_upgrade.yml to nuke the running container on
15:47:57 sean-k-mooney upgrade
15:49:02 sean-k-mooney let me know if you need any help navagaing kolla-ansible
15:50:55 mriedem gibi: replied, i think your suggestion is ok
15:52:42 finucannot mriedem, efried: https://review.opendev.org/661344
15:53:25 dansmith melwitt: I think disabling heartbeats will increase recovery time if the connection goes stale, and will cause stale connections to stay open on the rabbit side
15:53:34 dansmith melwitt: so I think threads=1, leave heartbeat behavior
15:54:23 melwitt sean-k-mooney, mdbooth ^
15:54:52 aspiers efried: I'm pondering whether my helpers fall foul of https://docs.python.org/3/library/unittest.mock.html#where-to-patch
15:54:52 melwitt dansmith: thanks
15:56:06 aspiers efried: although that wouldn't explain why they have been working so far
15:56:33 gibi mriedem: thanks.
15:57:07 sean-k-mooney dansmith: if we leave teh hearbeat enable should we at least change the log level in the nova-api for the oslo messaging logs?
15:57:19 sean-k-mooney or the heart beat on in partiaclar
15:59:16 efried finucannot: why does starlingx get different zebra stripe?
15:59:18 dansmith I dunno, but we might want to soften the log level in o.msg where it raises holy hell about the reconnct
15:59:22 efried *I* want special zebra stripe
16:00:09 efried aspiers: I'm going to have to get back to you (possibly much) later
16:00:35 aspiers efried: it's OK. Attacking it with a debugger now
16:00:51 sean-k-mooney dansmith: so we suggested that and got some push back to droping it to info in oslo
16:01:38 finucannot efried: Whoops, missed that
16:01:41 finucannot fixed
16:01:49 dansmith sean-k-mooney: so what is your thought, to specifically set o.msg level to error in nova-api?
16:02:24 sean-k-mooney dansmith: effectivly but i think we can be more granular then that. we do this for some privesep stuff right?
16:02:43 dansmith sean-k-mooney: we can set defaults on the logger name level, I think
16:02:59 sean-k-mooney dansmith: actully setting it to error wont help as o.msg is currently loggin it as an error
16:03:16 sean-k-mooney so more we woudl have to install a log filter to speficaly drop that error
16:04:07 dansmith sean-k-mooney: well, I dunno, it *really* seems like that shouldn't be an error, it's just a reconnect, which is exactly what it's supposed to be doing
16:04:21 dansmith so whatever.. however you make it not show up, or leave it and document it or whatever, I care about that less
16:04:39 dansmith what I do care about is that we not disable heartbeats just because it's making a statement in the logs we don't want
16:04:55 mriedem monkeypatch the oslo.messaging code!
16:05:14 beekneemech mriedem: Crazy talk! :-P
16:05:21 sean-k-mooney dansmith: ya i agree it not an error. maybe we could make it a warning in o.mesg and then set the log level to error for o.msg in nova
16:05:34 dansmith sean-k-mooney: yeah
16:07:30 mriedem if only DEFAULT_LOG_LEVELS wasn't a gd list
16:08:05 mriedem makes it a pita to parse and override from code
16:08:54 melwitt dansmith: IIUC from past conversation, the ideal way to handle the eventlet monkey patch issue longer term would be to separate the wsgi app part of nova-api from the rest of it, so we don't have behavior that doesn't fit in with the wsgi app happening inside of it. what would that look like? a separate process for each of nova-api-wsgi and nova-api?
16:10:53 dansmith melwitt: yeah
16:11:10 dansmith that's how wsgi apps area really supposed to be done, AFAIK,
16:11:26 dansmith the wsgi bit is just the "view" and you delegate everything to the controller
16:11:43 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Delete allocations even if _confirm_resize raises https://review.opendev.org/652153
16:11:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional confirm_migration_error test https://review.opendev.org/658136
16:11:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: [stable-only] Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/661349
16:12:50 mriedem delicious MVC
16:13:07 mriedem ganso: ^ likely needs some work b/c the functional test in queens isn't failing when i remove the fix
16:13:21 mriedem struts baby!
16:13:29 mriedem that's the last time i did UI Work
16:13:31 mriedem *work
16:13:49 mriedem Faces and AJAX was a brand new world to me when i left that dept
16:14:19 melwitt dansmith: ok, cool. I wanted to get an idea about what's involved to see if we could make that happen this cycle.
16:14:53 melwitt to do it properly for future
16:15:05 cdent melwitt, dansmith : are you talking about a "controller" as a separate process?
16:15:28 dansmith melwitt: uh, that would be a significant undertaking
16:15:38 dansmith cdent: yeah
16:15:39 melwitt oh :(
16:15:42 cdent yeah, that's why I ask. It would be significant
16:15:49 dansmith cdent: definitely
16:16:36 cdent a somewhat shorter step would be to make a "scatter-gatther-cells-agent" that can be threaded (in whatever way) that the wsgi app talks to over a unix socket
16:16:46 dansmith melwitt: going from a thing designed to be eventlet-based to just slapping a wsgi frontend on it is the kind of thing that doesn't end with a small refactor to substantially change how it works :)
16:17:08 cdent but even that agent idea would be an undertaking
16:17:11 melwitt the concern was, the near term things we can do are to somehow hide the log message and set threads=1. but what's the long term thing to do, or is the near term fix going to be the forever fix
16:17:24 dansmith cdent: yeah, could do that, but that's a lot of deployment change for just that thing
16:17:57 cdent dansmith: yeah, no doubt. (I guess you could also fork it as needed)
16:18:53 cdent call scatter-gather-cells over rpc to the conductor ?
16:19:19 dansmith cdent: that will slow down instance list quite a bit
16:19:58 cdent Yeah, I don't really have much in the way of ideas. We've got impedance mismatches galore.
16:20:14 cdent s/ideas/ideas that can be realized/
16:20:20 melwitt yeah... just wanted to understand whether we'll have a next step here or if this is it. sounds like realistically, there's not more we could do
16:20:23 dansmith yep.. IMHO, fix the short term and then let's properly design the bigger change after we see how that goes
16:20:37 dansmith and not armchair quarterback it the day before a holiday weekend (for some of us) :)
16:21:00 melwitt haha, no, I was only looking for a high-level ballpark idea of what it would look like
16:21:23 melwitt because when you said "separate wsgi part" initially, I didn't really know what that meant
16:50:26 openstackgerrit Merged openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.opendev.org/556751
16:58:51 aspiers gibi: you around? I'm trying to get a functional test to restart the compute service in a way which triggers init_host()
16:59:05 aspiers gibi: and I see you worked on restart_compute_service()
16:59:35 aspiers unfortunately that only restarts the Host, not the Service
17:02:52 aspiers I tried self.compute.stop(); self.start_compute() but the latter invokes nova.test.TestCase.start_service() which borks by trying to create a duplicate HostMapping
17:03:23 aspiers dansmith: I see your fingerprints on this code too :)
17:03:58 aspiers This is about the point where my newbie-ness makes it pretty hard to figure out a way forward
17:10:22 openstackgerrit Adam Spiers proposed openstack/nova master: Provide HW_CPU_AMD_SEV trait when SEV is supported https://review.opendev.org/638680
17:11:40 melwitt aspiers: I don't know much about that but you reminded me of a patch of gibi's that might be of help https://review.opendev.org/512552
17:12:55 aspiers melwitt: thanks! yeah that's the exact area of code I'm talking about
17:13:02 aspiers maybe that change will yield some clues
17:13:10 melwitt that's what I'm hoping :)
17:14:45 dansmith aspiers: "restart the host not the service" does not make sense to me, fwiw
17:15:29 dansmith aspiers: calling start_service() again is not the right thing to do, because it's really create_and_start_service() I think
17:15:45 aspiers yes, I think that's the problem
17:15:46 dansmith aspiers: if you want to re-run init_host() I would just run it

Earlier   Later