| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-23 | |||
| 19:45:15 | efried | tasker: Does | |
| 19:45:15 | efried | [DEFAULT] | |
| 19:45:15 | efried | default_log_levels = nova=WARNING | |
| 19:45:15 | efried | not work? | |
| 19:46:00 | efried | sean-k-mooney: ack, and shrug. I hope we can soon deprecate *all* of [pci]passthrough_whitelist and [pci]alias. | |
| 19:46:14 | sean-k-mooney | efried: maybe in U | |
| 19:46:37 | sean-k-mooney | of course then we woudl have a cyborg whitelist and device profile | |
| 19:46:46 | sean-k-mooney | but then its there problem :) | |
| 19:46:50 | efried | heh | |
| 19:47:24 | sean-k-mooney | if we get the cyborg integration done fully this cycle that might be a nice thing to port over early next cycle | |
| 19:48:09 | efried | sean-k-mooney: cyborg/fpga + providers.yaml we might have all we need. | |
| 19:48:41 | sean-k-mooney | well i was thinig more generic pci passhtough rather then smart nic | |
| 19:49:29 | sean-k-mooney | e.g. replaceing all the uses of the pci tracker in nova with a very basic generic pci driver in cyborg | |
| 19:50:13 | sean-k-mooney | but making that work with hardare offload ovs and all the other legacy things we have would be a lot of work | |
| 19:50:31 | sean-k-mooney | manyly on the nova side to make it transparent | |
| 19:51:28 | sean-k-mooney | efried: anyway thanks ill send an email to the list | |
| 19:51:49 | efried | coo | |
| 20:02:03 | tasker | efried: did not. I still see a bunch of "HEAD / => generated 0 bytes" | |
| 20:02:40 | efried | tasker: Coming from nova? | |
| 20:03:21 | tasker | yes. specifically the "nova-api-os-compute" container ( if that distinction matters. ) | |
| 20:04:03 | efried | tasker: Can you paste me the entire log line please? | |
| 20:04:32 | efried | I'm thinking that might be coming from the wsgi layer | |
| 20:04:47 | tasker | http://dpaste.com/2C5P2PY | |
| 20:05:31 | tasker | I concur. to complicate things, I don't know how to turn that thing down, either. | |
| 20:05:33 | efried | in which case you may wish to try adding ,eventlet.wsgi.server=WARNING | |
| 20:06:59 | efried | I think the left-hand-side of that '=' is a substring match. So if you say foo.bar=WARNING then log lines coming from foo.bar and foo.bar.baz and foo.bar.etc will all get filtered at a WARNING level. | |
| 20:08:00 | efried | hmph, no indication in that line where it's coming from. Is there a log line template that will add that? Looking... | |
| 20:09:49 | efried | tasker: Looks like you might be able to add %(pathname)s to your logging_default_format_string to get a clue where that line is being generated from; then use the prefix trick above to suppress it. | |
| 20:17:54 | tasker | efried: did not work. "%(pathname)s" was applied to another set of messages, but not the one I'm trying to filter out. | |
| 20:18:20 | efried | interesting | |
| 20:18:56 | tasker | wait .. scratch that. it wasn't applied at all. | |
| 20:19:12 | efried | assume you're restarting services each time | |
| 20:19:31 | tasker | yup. | |
| 20:19:58 | tasker | http://dpaste.com/1ZSB78D -- what the "wsgi" logs look like from "nova-api-wsgi.log" when all testing options are removed. this is the original behaviour. | |
| 20:20:14 | efried | guess you could try adding the pathname to logging_debug_format_suffix | |
| 20:20:29 | tasker | when I set anything to the config file, the lines in nova-api-wsgi.log stop, but not to nova-api-os-compute.log | |
| 20:20:53 | tasker | the latter file contains lines of the first paste I sent you. | |
| 20:21:52 | efried | hmph. I'm afraid I'm out of ideas. | |
| 20:22:01 | efried | Course, I'm not what you'd call an "operator". | |
| 20:22:12 | tasker | hehe. can we bring back "verbose = False"? | |
| 20:23:34 | tasker | efried: I appreciate you taking the time to help me out. You've shed some light on some things I did not know and given me a lot of good information to play around with. | |
| 20:23:59 | efried | you're welcome, sorry I couldn't be more help. | |
| 20:30:29 | tasker | I did match the "pid" in the log file to the PIDs of the uWSGI helpers that spawn whenever the nova service starts. | |
| 20:30:40 | tasker | that lends weight to it being the WSGI module(s). | |
| 20:31:03 | tasker | are the modules using the same file independent of Nova? | |
| 20:32:08 | efried | tasker: You're out of my wheelhouse at this point. Throwing it around earlier like I knew what I was talking about, but I can barely spell WSGI. | |
| 20:33:10 | tasker | it's alright. those were mostly musings. I tend to "stream of conciousness" debug and use any open chat window to get ideas out. | |
| 20:33:32 | efried | heh, wfm. | |
| 20:34:04 | efried | tasker: I've got a guy who might know a bit more. He's on his way into the channel, if you're going to hang around for a bit. | |
| 20:34:20 | tasker | yeah. thanks! | |
| 20:34:56 | efried | o/ donnyd | |
| 20:35:10 | efried | donnyd: meet tasker. tasker: likewise | |
| 20:35:17 | donnyd | Hi | |
| 20:35:18 | efried | donnyd has actual experience deploying openstack | |
| 20:35:19 | tasker | donnyd: helo! | |
| 20:35:37 | donnyd | I'm just getting caught up on what the issue is | |
| 20:36:38 | tasker | trying to turn down all logging in Nova to WARNING because I'm getting these messages that indicate the API endpoints are hit at ( what I guess is INFO ) level. efried was helping and about all we can do is determine that they're coming from WSGI. | |
| 20:37:22 | donnyd | Do you have an example of the log entry you don't want to see? | |
| 20:37:32 | tasker | http://dpaste.com/2C5P2PY | |
| 20:39:06 | tasker | this is Pike, and prior to it, I could set "verbose = False" in nova.conf and call it a day. | |
| 20:39:27 | donnyd | Next question is what deployment method did you use? | |
| 20:39:51 | tasker | OSA | |
| 20:42:02 | donnyd | That surely looks like a wsgi log to me. (I see the same log entry type in my ister server). | |
| 20:43:07 | tasker | know if there's a comparable "verbose = False" flag for uWSGI? | |
| 20:43:20 | tasker | their docs don't hint at such. | |
| 20:50:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Link versioned notification talk into docs https://review.opendev.org/661115 | |
| 20:53:12 | tasker | appending "--disable-logging" to the uwsgi ExecStart string in the service file killed the messages, but that's too heavy-handed. | |
| 20:55:42 | donnyd | Still searching | |
| 20:56:48 | tasker | and having "--disable-logging" and having Nova at "debug_log_levels = nova=INFO | |
| 20:56:55 | tasker | " I can still get Nova's notices of API hits. | |
| 20:57:28 | openstackgerrit | Merged openstack/nova master: refactor nova-manage archive_deleted_rows https://review.opendev.org/643779 | |
| 20:58:09 | tasker | I'm stepping out for some "fresh air". back in a few. | |
| 20:58:12 | efried | Nova meeting in 2 minutes in #openstack-meeting | |
| 21:00:10 | efried | pawing through uwsgi docs, I'm not actually finding any hints that they *have* different log levels. | |
| 21:00:14 | efried | I think it's just on or off. | |
| 21:01:29 | efried | in which case your heavy-handed solution might be the only solution, beyond `grep` :P | |
| 21:01:38 | donnyd | Thats what it looks like to me @efried | |
| 21:02:25 | efried | tasker: Is there anything in nova-api-os-compute.log that you actually care about? | |
| 21:03:54 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118 | |
| 21:07:07 | tasker | efried: errors and warnings. . ) what are you thinking? | |
| 21:09:52 | efried | tasker: I was just wondering whether you could filter the whole thing out :) | |
| 21:10:05 | openstackgerrit | Merged openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.opendev.org/651647 | |
| 21:10:05 | donnyd | https://github.com/unbit/uwsgi-docs/blob/master/ManagementFlag.rst | |
| 21:10:17 | donnyd | 0loggingenable/disable logging | |
| 21:10:33 | donnyd | Looks like its mostly an off or on kinda thing | |
| 21:15:17 | tasker | efried: donnyd: thank you both very much for your assistance. with your help, I think I have a solution that will work. I'll be leaving it overnight and trying to trigger various warnings and events to verify it does what I need. | |
| 21:15:40 | efried | good luck tasker | |
| 21:34:56 | sean-k-mooney | on the nova console auth topic kolla-ansibel still defualt to deploying it on master | |
| 21:35:50 | efried | whee | |
| 21:37:07 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661123 | |
| 21:38:20 | sean-k-mooney | osa looks like id does not but i only quickly looked | |
| 21:39:05 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118 | |
| 21:39:19 | sean-k-mooney | and tripleo remvoed it 14 days ago https://github.com/openstack/tripleo-heat-templates/blob/c33f8cb1174146e3cbdca6241914a78a623ea505/releasenotes/notes/nova_remove_nova-consoleauth-227cbeb44e9ab5ef.yaml | |
| 21:41:00 | melwitt | nice detective work :) | |
| 21:42:31 | sean-k-mooney | melwitt: what was the console auth service used for with cells v1 again | |
| 21:43:26 | melwitt | all console token auth. for cells v1 the database backend stuff was _not_ implemented | |
| 21:44:07 | sean-k-mooney | ah right so its jsust issueing the tokens to connect to the websockets? | |
| 21:44:27 | mriedem | osa is done https://github.com/openstack/openstack-ansible-os_nova/commit/1df0bd51ee5031039d3852cefbb63162e29b22a5 | |
| 21:45:20 | melwitt | sean-k-mooney: console auth service? yeah it stores the token auths locally and responds yea/nay when a request to connect to console proxy comes in | |
| 21:46:04 | melwitt | and the change we did for non-cells v1 was store the token auths in the cell databases and have console proxies (websockets) run per cell so they can query the database to validate token auths | |