Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-23
19:48:41 sean-k-mooney well i was thinig more generic pci passhtough rather then smart nic
19:49:29 sean-k-mooney e.g. replaceing all the uses of the pci tracker in nova with a very basic generic pci driver in cyborg
19:50:13 sean-k-mooney but making that work with hardare offload ovs and all the other legacy things we have would be a lot of work
19:50:31 sean-k-mooney manyly on the nova side to make it transparent
19:51:28 sean-k-mooney efried: anyway thanks ill send an email to the list
19:51:49 efried coo
20:02:03 tasker efried: did not. I still see a bunch of "HEAD / => generated 0 bytes"
20:02:40 efried tasker: Coming from nova?
20:03:21 tasker yes. specifically the "nova-api-os-compute" container ( if that distinction matters. )
20:04:03 efried tasker: Can you paste me the entire log line please?
20:04:32 efried I'm thinking that might be coming from the wsgi layer
20:04:47 tasker http://dpaste.com/2C5P2PY
20:05:31 tasker I concur. to complicate things, I don't know how to turn that thing down, either.
20:05:33 efried in which case you may wish to try adding ,eventlet.wsgi.server=WARNING
20:06:59 efried I think the left-hand-side of that '=' is a substring match. So if you say foo.bar=WARNING then log lines coming from foo.bar and foo.bar.baz and foo.bar.etc will all get filtered at a WARNING level.
20:08:00 efried hmph, no indication in that line where it's coming from. Is there a log line template that will add that? Looking...
20:09:49 efried tasker: Looks like you might be able to add %(pathname)s to your logging_default_format_string to get a clue where that line is being generated from; then use the prefix trick above to suppress it.
20:17:54 tasker efried: did not work. "%(pathname)s" was applied to another set of messages, but not the one I'm trying to filter out.
20:18:20 efried interesting
20:18:56 tasker wait .. scratch that. it wasn't applied at all.
20:19:12 efried assume you're restarting services each time
20:19:31 tasker yup.
20:19:58 tasker http://dpaste.com/1ZSB78D -- what the "wsgi" logs look like from "nova-api-wsgi.log" when all testing options are removed. this is the original behaviour.
20:20:14 efried guess you could try adding the pathname to logging_debug_format_suffix
20:20:29 tasker when I set anything to the config file, the lines in nova-api-wsgi.log stop, but not to nova-api-os-compute.log
20:20:53 tasker the latter file contains lines of the first paste I sent you.
20:21:52 efried hmph. I'm afraid I'm out of ideas.
20:22:01 efried Course, I'm not what you'd call an "operator".
20:22:12 tasker hehe. can we bring back "verbose = False"?
20:23:34 tasker efried: I appreciate you taking the time to help me out. You've shed some light on some things I did not know and given me a lot of good information to play around with.
20:23:59 efried you're welcome, sorry I couldn't be more help.
20:30:29 tasker I did match the "pid" in the log file to the PIDs of the uWSGI helpers that spawn whenever the nova service starts.
20:30:40 tasker that lends weight to it being the WSGI module(s).
20:31:03 tasker are the modules using the same file independent of Nova?
20:32:08 efried tasker: You're out of my wheelhouse at this point. Throwing it around earlier like I knew what I was talking about, but I can barely spell WSGI.
20:33:10 tasker it's alright. those were mostly musings. I tend to "stream of conciousness" debug and use any open chat window to get ideas out.
20:33:32 efried heh, wfm.
20:34:04 efried tasker: I've got a guy who might know a bit more. He's on his way into the channel, if you're going to hang around for a bit.
20:34:20 tasker yeah. thanks!
20:34:56 efried o/ donnyd
20:35:10 efried donnyd: meet tasker. tasker: likewise
20:35:17 donnyd Hi
20:35:18 efried donnyd has actual experience deploying openstack
20:35:19 tasker donnyd: helo!
20:35:37 donnyd I'm just getting caught up on what the issue is
20:36:38 tasker trying to turn down all logging in Nova to WARNING because I'm getting these messages that indicate the API endpoints are hit at ( what I guess is INFO ) level. efried was helping and about all we can do is determine that they're coming from WSGI.
20:37:22 donnyd Do you have an example of the log entry you don't want to see?
20:37:32 tasker http://dpaste.com/2C5P2PY
20:39:06 tasker this is Pike, and prior to it, I could set "verbose = False" in nova.conf and call it a day.
20:39:27 donnyd Next question is what deployment method did you use?
20:39:51 tasker OSA
20:42:02 donnyd That surely looks like a wsgi log to me. (I see the same log entry type in my ister server).
20:43:07 tasker know if there's a comparable "verbose = False" flag for uWSGI?
20:43:20 tasker their docs don't hint at such.
20:50:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Link versioned notification talk into docs https://review.opendev.org/661115
20:53:12 tasker appending "--disable-logging" to the uwsgi ExecStart string in the service file killed the messages, but that's too heavy-handed.
20:55:42 donnyd Still searching
20:56:48 tasker and having "--disable-logging" and having Nova at "debug_log_levels = nova=INFO
20:56:55 tasker " I can still get Nova's notices of API hits.
20:57:28 openstackgerrit Merged openstack/nova master: refactor nova-manage archive_deleted_rows https://review.opendev.org/643779
20:58:09 tasker I'm stepping out for some "fresh air". back in a few.
20:58:12 efried Nova meeting in 2 minutes in #openstack-meeting
21:00:10 efried pawing through uwsgi docs, I'm not actually finding any hints that they *have* different log levels.
21:00:14 efried I think it's just on or off.
21:01:29 efried in which case your heavy-handed solution might be the only solution, beyond `grep` :P
21:01:38 donnyd Thats what it looks like to me @efried
21:02:25 efried tasker: Is there anything in nova-api-os-compute.log that you actually care about?
21:03:54 openstackgerrit melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118
21:07:07 tasker efried: errors and warnings. . ) what are you thinking?
21:09:52 efried tasker: I was just wondering whether you could filter the whole thing out :)
21:10:05 donnyd https://github.com/unbit/uwsgi-docs/blob/master/ManagementFlag.rst
21:10:05 openstackgerrit Merged openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.opendev.org/651647
21:10:17 donnyd 0loggingenable/disable logging
21:10:33 donnyd Looks like its mostly an off or on kinda thing
21:15:17 tasker efried: donnyd: thank you both very much for your assistance. with your help, I think I have a solution that will work. I'll be leaving it overnight and trying to trigger various warnings and events to verify it does what I need.
21:15:40 efried good luck tasker
21:34:56 sean-k-mooney on the nova console auth topic kolla-ansibel still defualt to deploying it on master
21:35:50 efried whee
21:37:07 openstackgerrit melanie witt proposed openstack/nova stable/stein: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661123
21:38:20 sean-k-mooney osa looks like id does not but i only quickly looked
21:39:05 openstackgerrit melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118
21:39:19 sean-k-mooney and tripleo remvoed it 14 days ago https://github.com/openstack/tripleo-heat-templates/blob/c33f8cb1174146e3cbdca6241914a78a623ea505/releasenotes/notes/nova_remove_nova-consoleauth-227cbeb44e9ab5ef.yaml
21:41:00 melwitt nice detective work :)
21:42:31 sean-k-mooney melwitt: what was the console auth service used for with cells v1 again
21:43:26 melwitt all console token auth. for cells v1 the database backend stuff was _not_ implemented
21:44:07 sean-k-mooney ah right so its jsust issueing the tokens to connect to the websockets?
21:44:27 mriedem osa is done https://github.com/openstack/openstack-ansible-os_nova/commit/1df0bd51ee5031039d3852cefbb63162e29b22a5
21:45:20 melwitt sean-k-mooney: console auth service? yeah it stores the token auths locally and responds yea/nay when a request to connect to console proxy comes in
21:46:04 melwitt and the change we did for non-cells v1 was store the token auths in the cell databases and have console proxies (websockets) run per cell so they can query the database to validate token auths
21:46:20 sean-k-mooney melwitt ok so looking at the patches to remove it in osa and triplo if we just dont deploy it will it work with nova today?
21:47:17 sean-k-mooney i dont see any config cahnge in those patches so it seam like nova just does the right thing now?
21:47:22 melwitt sean-k-mooney: yeah, the default behavior is to use the database as the console token auth storage. so running it is not necessary. unless you're running cells v1
21:47:37 sean-k-mooney which kolla never supported
21:48:18 sean-k-mooney so i can write a quick patch to delete the container form kolla and remove it form the kolla-ansibel playbook if that is all thats needed
21:48:53 sean-k-mooney well the upgade playbook would also need to stop any running containers
21:49:09 sean-k-mooney but that seams trivial too
21:50:26 melwitt I don't know if this is relevant at this point but during a live, rolling upgrade from queens to rocky, you would need to keep nova-consoleauth running until everything is rocky
21:50:49 sean-k-mooney looks like https://bugs.launchpad.net/kolla-ansible/+bug/1804474 is open but i am gessing its not been worked on
21:50:49 melwitt if you were not OK with disrupting existing console sessions
21:50:50 openstack Launchpad bug 1804474 in kolla-ansible "nova-consoleauth is deprecated" [Undecided,In progress] - Assigned to caoyuan (cao-yuan)

Earlier   Later