Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-24
15:36:15 mriedem if terminate is just racing with build and the copy of the instance during terminate just doesn't have the info cache data from the db, then the refresh should fix your problem i think,
15:36:35 mriedem but if the info cache in the db is empty when you refresh, that bottom change isn't going to help
15:36:47 amorin the cache is populated in the DB
15:36:47 mriedem it won't make things worse of course
15:36:55 mriedem ok, well in that case you might be saved,
15:37:00 amorin but after the copy of instance is used for deletion
15:37:14 amorin yup, anyway I will double check that point
15:37:15 melwitt dansmith: do you remember where we landed when we were talking about disabling oslo.messaging heartbeats in nova-api, as far as whether we would recommend running that configuration in our upstream docs?
15:37:20 mriedem because https://github.com/openstack/nova/blob/976d1b89c2b754729903291050e01c9bf49704b9/nova/network/neutronv2/api.py#L1757
15:37:33 sean-k-mooney efried: sorry was on a call for the last 90 mins
15:37:43 amorin _gather_port_ids_and_networks is using the nova cache DB right?
15:37:53 sean-k-mooney efried: did ye reach an agreement on the nova-console auth blueprint
15:37:56 openstackgerrit Adam Spiers proposed openstack/nova master: Move selective patching of open() to nova.test for reuse https://review.opendev.org/661266
15:38:12 melwitt dansmith: this is about the idle nova-api wsgi app heartbeat error messages stuff
15:38:26 mriedem amorin: yeah
15:38:47 amorin mriedem, by the way, thanks for the tip about mocking in setup, I was pretty sure that this was possible
15:38:51 mriedem amorin: yeah so tl;dr if you're just racing to refresh from the db on delete then your fix is probably ok
15:38:55 amorin but I am beginning in that subject
15:38:57 mriedem np
15:39:03 mriedem i've only recently-ish started using that trick
15:39:09 efried sean-k-mooney: It sounds like all the deployment projects are either sorted or have patches in flight to sort. I was looking for you & others to vet and +1 basically so I could approve the bp
15:39:54 melwitt efried, finucannot: I'm also +1 on removing nova-consoleauth since it looks like everything is sorted with deployment tools
15:40:02 sean-k-mooney efried: kolla has an open bug but no progess in a while
15:40:22 sean-k-mooney if that is the only blocker i can go write the patches needed
15:40:47 sean-k-mooney it is already optional but it would jsut invovle deleting the container on upgrade an removing the deployment code
15:41:28 efried finucannot: do you have any further words about kolla ^ ?
15:41:57 sean-k-mooney so yes im +1 on removing it and can help the kolla folks as needed since i used to work on kolla and kolla ansibel a few release ago
15:42:06 finucannot sean-k-mooney: So there's more to it than this? https://review.opendev.org/#/c/661251/
15:42:09 finucannot efried: ^
15:42:33 sean-k-mooney that removed it from kolla
15:42:44 sean-k-mooney but they also need to remove it in kolla ansible
15:42:53 sean-k-mooney e.g. you just deleted teh container
15:43:16 sean-k-mooney but the upgrade playbook need to koll all running instance of the contain
15:43:21 sean-k-mooney *kill
15:43:54 finucannot Ah, I figured there was more to it than that. I can try drafting those patches based on what they did in OSA. I'd need one to kill the running instances (which would be backported to stable/stein) then a follow up to remove all references entirely
15:44:33 sean-k-mooney we need to kill this handeler https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/handlers/main.yml#L112-L136
15:44:48 efried finucannot: Okay, mriedem, melwitt, and sean-k-mooney are all +1, I'm approving the bp assuming the above is fairly easy and will be taken care of before we pull the trigger.
15:44:54 sean-k-mooney and remove like 2 or 3 other things but its a fairly small patch
15:46:08 finucannot sean-k-mooney: I'll do that before I clock off
15:47:55 sean-k-mooney ok you need to kill the service default https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/defaults/main.yml#L83-L93 and as i sad just update https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/rolling_upgrade.yml and https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/legacy_upgrade.yml to nuke the running container on
15:47:57 sean-k-mooney upgrade
15:49:02 sean-k-mooney let me know if you need any help navagaing kolla-ansible
15:50:55 mriedem gibi: replied, i think your suggestion is ok
15:52:42 finucannot mriedem, efried: https://review.opendev.org/661344
15:53:25 dansmith melwitt: I think disabling heartbeats will increase recovery time if the connection goes stale, and will cause stale connections to stay open on the rabbit side
15:53:34 dansmith melwitt: so I think threads=1, leave heartbeat behavior
15:54:23 melwitt sean-k-mooney, mdbooth ^
15:54:52 aspiers efried: I'm pondering whether my helpers fall foul of https://docs.python.org/3/library/unittest.mock.html#where-to-patch
15:54:52 melwitt dansmith: thanks
15:56:06 aspiers efried: although that wouldn't explain why they have been working so far
15:56:33 gibi mriedem: thanks.
15:57:07 sean-k-mooney dansmith: if we leave teh hearbeat enable should we at least change the log level in the nova-api for the oslo messaging logs?
15:57:19 sean-k-mooney or the heart beat on in partiaclar
15:59:16 efried finucannot: why does starlingx get different zebra stripe?
15:59:18 dansmith I dunno, but we might want to soften the log level in o.msg where it raises holy hell about the reconnct
15:59:22 efried *I* want special zebra stripe
16:00:09 efried aspiers: I'm going to have to get back to you (possibly much) later
16:00:35 aspiers efried: it's OK. Attacking it with a debugger now
16:00:51 sean-k-mooney dansmith: so we suggested that and got some push back to droping it to info in oslo
16:01:38 finucannot efried: Whoops, missed that
16:01:41 finucannot fixed
16:01:49 dansmith sean-k-mooney: so what is your thought, to specifically set o.msg level to error in nova-api?
16:02:24 sean-k-mooney dansmith: effectivly but i think we can be more granular then that. we do this for some privesep stuff right?
16:02:43 dansmith sean-k-mooney: we can set defaults on the logger name level, I think
16:02:59 sean-k-mooney dansmith: actully setting it to error wont help as o.msg is currently loggin it as an error
16:03:16 sean-k-mooney so more we woudl have to install a log filter to speficaly drop that error
16:04:07 dansmith sean-k-mooney: well, I dunno, it *really* seems like that shouldn't be an error, it's just a reconnect, which is exactly what it's supposed to be doing
16:04:21 dansmith so whatever.. however you make it not show up, or leave it and document it or whatever, I care about that less
16:04:39 dansmith what I do care about is that we not disable heartbeats just because it's making a statement in the logs we don't want
16:04:55 mriedem monkeypatch the oslo.messaging code!
16:05:14 beekneemech mriedem: Crazy talk! :-P
16:05:21 sean-k-mooney dansmith: ya i agree it not an error. maybe we could make it a warning in o.mesg and then set the log level to error for o.msg in nova
16:05:34 dansmith sean-k-mooney: yeah
16:07:30 mriedem if only DEFAULT_LOG_LEVELS wasn't a gd list
16:08:05 mriedem makes it a pita to parse and override from code
16:08:54 melwitt dansmith: IIUC from past conversation, the ideal way to handle the eventlet monkey patch issue longer term would be to separate the wsgi app part of nova-api from the rest of it, so we don't have behavior that doesn't fit in with the wsgi app happening inside of it. what would that look like? a separate process for each of nova-api-wsgi and nova-api?
16:10:53 dansmith melwitt: yeah
16:11:10 dansmith that's how wsgi apps area really supposed to be done, AFAIK,
16:11:26 dansmith the wsgi bit is just the "view" and you delegate everything to the controller
16:11:43 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Delete allocations even if _confirm_resize raises https://review.opendev.org/652153
16:11:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional confirm_migration_error test https://review.opendev.org/658136
16:11:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: [stable-only] Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/661349
16:12:50 mriedem delicious MVC
16:13:07 mriedem ganso: ^ likely needs some work b/c the functional test in queens isn't failing when i remove the fix
16:13:21 mriedem struts baby!
16:13:29 mriedem that's the last time i did UI Work
16:13:31 mriedem *work
16:13:49 mriedem Faces and AJAX was a brand new world to me when i left that dept
16:14:19 melwitt dansmith: ok, cool. I wanted to get an idea about what's involved to see if we could make that happen this cycle.
16:14:53 melwitt to do it properly for future
16:15:05 cdent melwitt, dansmith : are you talking about a "controller" as a separate process?
16:15:28 dansmith melwitt: uh, that would be a significant undertaking
16:15:38 dansmith cdent: yeah
16:15:39 melwitt oh :(
16:15:42 cdent yeah, that's why I ask. It would be significant
16:15:49 dansmith cdent: definitely
16:16:36 cdent a somewhat shorter step would be to make a "scatter-gatther-cells-agent" that can be threaded (in whatever way) that the wsgi app talks to over a unix socket
16:16:46 dansmith melwitt: going from a thing designed to be eventlet-based to just slapping a wsgi frontend on it is the kind of thing that doesn't end with a small refactor to substantially change how it works :)
16:17:08 cdent but even that agent idea would be an undertaking
16:17:11 melwitt the concern was, the near term things we can do are to somehow hide the log message and set threads=1. but what's the long term thing to do, or is the near term fix going to be the forever fix

Earlier   Later