| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-23 | |||
| 19:38:42 | sean-k-mooney | efried: ya basically | |
| 19:38:46 | efried | are there not scenarios where identifying a device by name *doesn't* break? | |
| 19:39:02 | efried | (s/not// for same meaning with fewer negatives) | |
| 19:39:21 | sean-k-mooney | it only works if the device is reliable bound back to the nic driver when nolonger used by the guest and the name does not change whne it rebound to the host | |
| 19:39:37 | sean-k-mooney | so it can work but its really fragile | |
| 19:40:28 | sean-k-mooney | using vendor id and product id or useing the pci address are both more robost as they dont depned on teh driver or the semanitc of the nic vendor | |
| 19:40:39 | efried | sean-k-mooney: I guess what I would do in this case is write the patch and start a thread on the ML with your intent, soliciting opinions on whether this is okay to kill. Link the patch and ML thread from each other. IMO that's sufficient paperwork. And if someone thinks otherwise, you can always write a bug/bp later. | |
| 19:41:21 | efried | though it sounds like what you're talking about actually *is* a bug, and this is just one possible remedy. | |
| 19:41:33 | sean-k-mooney | efried: ok sound good. | |
| 19:41:54 | efried | In my ignorance, another remedy could be "blow up if devname is used for devices it has the potential to break on" | |
| 19:42:02 | sean-k-mooney | it depend on how you configre your driver at the kernel levle | |
| 19:42:17 | sean-k-mooney | and to some extent the nic vendor | |
| 19:42:20 | efried | no way to detect it, other than going all the way through until you hit the bug?? | |
| 19:42:47 | sean-k-mooney | i belive intel nics prefer the networkign driver over vfio-pci but i think it depend on the model to some extend and your udev rules | |
| 19:43:57 | sean-k-mooney | efried: the main issue comes down to will udev/systemd chose to bind the driver to i40e or leave it bound to vfio-pci | |
| 19:44:27 | sean-k-mooney | so its not really a nova issue | |
| 19:44:43 | sean-k-mooney | its more a distro/packaging/install/confituion thing on the host | |
| 19:45:13 | sean-k-mooney | which is what make it hard for nova to dectet and or warn about | |
| 19:45:15 | efried | tasker: Does | |
| 19:45:15 | efried | [DEFAULT] | |
| 19:45:15 | efried | default_log_levels = nova=WARNING | |
| 19:45:15 | efried | not work? | |
| 19:46:00 | efried | sean-k-mooney: ack, and shrug. I hope we can soon deprecate *all* of [pci]passthrough_whitelist and [pci]alias. | |
| 19:46:14 | sean-k-mooney | efried: maybe in U | |
| 19:46:37 | sean-k-mooney | of course then we woudl have a cyborg whitelist and device profile | |
| 19:46:46 | sean-k-mooney | but then its there problem :) | |
| 19:46:50 | efried | heh | |
| 19:47:24 | sean-k-mooney | if we get the cyborg integration done fully this cycle that might be a nice thing to port over early next cycle | |
| 19:48:09 | efried | sean-k-mooney: cyborg/fpga + providers.yaml we might have all we need. | |
| 19:48:41 | sean-k-mooney | well i was thinig more generic pci passhtough rather then smart nic | |
| 19:49:29 | sean-k-mooney | e.g. replaceing all the uses of the pci tracker in nova with a very basic generic pci driver in cyborg | |
| 19:50:13 | sean-k-mooney | but making that work with hardare offload ovs and all the other legacy things we have would be a lot of work | |
| 19:50:31 | sean-k-mooney | manyly on the nova side to make it transparent | |
| 19:51:28 | sean-k-mooney | efried: anyway thanks ill send an email to the list | |
| 19:51:49 | efried | coo | |
| 20:02:03 | tasker | efried: did not. I still see a bunch of "HEAD / => generated 0 bytes" | |
| 20:02:40 | efried | tasker: Coming from nova? | |
| 20:03:21 | tasker | yes. specifically the "nova-api-os-compute" container ( if that distinction matters. ) | |
| 20:04:03 | efried | tasker: Can you paste me the entire log line please? | |
| 20:04:32 | efried | I'm thinking that might be coming from the wsgi layer | |
| 20:04:47 | tasker | http://dpaste.com/2C5P2PY | |
| 20:05:31 | tasker | I concur. to complicate things, I don't know how to turn that thing down, either. | |
| 20:05:33 | efried | in which case you may wish to try adding ,eventlet.wsgi.server=WARNING | |
| 20:06:59 | efried | I think the left-hand-side of that '=' is a substring match. So if you say foo.bar=WARNING then log lines coming from foo.bar and foo.bar.baz and foo.bar.etc will all get filtered at a WARNING level. | |
| 20:08:00 | efried | hmph, no indication in that line where it's coming from. Is there a log line template that will add that? Looking... | |
| 20:09:49 | efried | tasker: Looks like you might be able to add %(pathname)s to your logging_default_format_string to get a clue where that line is being generated from; then use the prefix trick above to suppress it. | |
| 20:17:54 | tasker | efried: did not work. "%(pathname)s" was applied to another set of messages, but not the one I'm trying to filter out. | |
| 20:18:20 | efried | interesting | |
| 20:18:56 | tasker | wait .. scratch that. it wasn't applied at all. | |
| 20:19:12 | efried | assume you're restarting services each time | |
| 20:19:31 | tasker | yup. | |
| 20:19:58 | tasker | http://dpaste.com/1ZSB78D -- what the "wsgi" logs look like from "nova-api-wsgi.log" when all testing options are removed. this is the original behaviour. | |
| 20:20:14 | efried | guess you could try adding the pathname to logging_debug_format_suffix | |
| 20:20:29 | tasker | when I set anything to the config file, the lines in nova-api-wsgi.log stop, but not to nova-api-os-compute.log | |
| 20:20:53 | tasker | the latter file contains lines of the first paste I sent you. | |
| 20:21:52 | efried | hmph. I'm afraid I'm out of ideas. | |
| 20:22:01 | efried | Course, I'm not what you'd call an "operator". | |
| 20:22:12 | tasker | hehe. can we bring back "verbose = False"? | |
| 20:23:34 | tasker | efried: I appreciate you taking the time to help me out. You've shed some light on some things I did not know and given me a lot of good information to play around with. | |
| 20:23:59 | efried | you're welcome, sorry I couldn't be more help. | |
| 20:30:29 | tasker | I did match the "pid" in the log file to the PIDs of the uWSGI helpers that spawn whenever the nova service starts. | |
| 20:30:40 | tasker | that lends weight to it being the WSGI module(s). | |
| 20:31:03 | tasker | are the modules using the same file independent of Nova? | |
| 20:32:08 | efried | tasker: You're out of my wheelhouse at this point. Throwing it around earlier like I knew what I was talking about, but I can barely spell WSGI. | |
| 20:33:10 | tasker | it's alright. those were mostly musings. I tend to "stream of conciousness" debug and use any open chat window to get ideas out. | |
| 20:33:32 | efried | heh, wfm. | |
| 20:34:04 | efried | tasker: I've got a guy who might know a bit more. He's on his way into the channel, if you're going to hang around for a bit. | |
| 20:34:20 | tasker | yeah. thanks! | |
| 20:34:56 | efried | o/ donnyd | |
| 20:35:10 | efried | donnyd: meet tasker. tasker: likewise | |
| 20:35:17 | donnyd | Hi | |
| 20:35:18 | efried | donnyd has actual experience deploying openstack | |
| 20:35:19 | tasker | donnyd: helo! | |
| 20:35:37 | donnyd | I'm just getting caught up on what the issue is | |
| 20:36:38 | tasker | trying to turn down all logging in Nova to WARNING because I'm getting these messages that indicate the API endpoints are hit at ( what I guess is INFO ) level. efried was helping and about all we can do is determine that they're coming from WSGI. | |
| 20:37:22 | donnyd | Do you have an example of the log entry you don't want to see? | |
| 20:37:32 | tasker | http://dpaste.com/2C5P2PY | |
| 20:39:06 | tasker | this is Pike, and prior to it, I could set "verbose = False" in nova.conf and call it a day. | |
| 20:39:27 | donnyd | Next question is what deployment method did you use? | |
| 20:39:51 | tasker | OSA | |
| 20:42:02 | donnyd | That surely looks like a wsgi log to me. (I see the same log entry type in my ister server). | |
| 20:43:07 | tasker | know if there's a comparable "verbose = False" flag for uWSGI? | |
| 20:43:20 | tasker | their docs don't hint at such. | |
| 20:50:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Link versioned notification talk into docs https://review.opendev.org/661115 | |
| 20:53:12 | tasker | appending "--disable-logging" to the uwsgi ExecStart string in the service file killed the messages, but that's too heavy-handed. | |
| 20:55:42 | donnyd | Still searching | |
| 20:56:48 | tasker | and having "--disable-logging" and having Nova at "debug_log_levels = nova=INFO | |
| 20:56:55 | tasker | " I can still get Nova's notices of API hits. | |
| 20:57:28 | openstackgerrit | Merged openstack/nova master: refactor nova-manage archive_deleted_rows https://review.opendev.org/643779 | |
| 20:58:09 | tasker | I'm stepping out for some "fresh air". back in a few. | |
| 20:58:12 | efried | Nova meeting in 2 minutes in #openstack-meeting | |
| 21:00:10 | efried | pawing through uwsgi docs, I'm not actually finding any hints that they *have* different log levels. | |
| 21:00:14 | efried | I think it's just on or off. | |
| 21:01:29 | efried | in which case your heavy-handed solution might be the only solution, beyond `grep` :P | |
| 21:01:38 | donnyd | Thats what it looks like to me @efried | |
| 21:02:25 | efried | tasker: Is there anything in nova-api-os-compute.log that you actually care about? | |
| 21:03:54 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118 | |
| 21:07:07 | tasker | efried: errors and warnings. . ) what are you thinking? | |
| 21:09:52 | efried | tasker: I was just wondering whether you could filter the whole thing out :) | |
| 21:10:05 | openstackgerrit | Merged openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.opendev.org/651647 | |