| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-23 | |||
| 20:19:31 | tasker | yup. | |
| 20:19:58 | tasker | http://dpaste.com/1ZSB78D -- what the "wsgi" logs look like from "nova-api-wsgi.log" when all testing options are removed. this is the original behaviour. | |
| 20:20:14 | efried | guess you could try adding the pathname to logging_debug_format_suffix | |
| 20:20:29 | tasker | when I set anything to the config file, the lines in nova-api-wsgi.log stop, but not to nova-api-os-compute.log | |
| 20:20:53 | tasker | the latter file contains lines of the first paste I sent you. | |
| 20:21:52 | efried | hmph. I'm afraid I'm out of ideas. | |
| 20:22:01 | efried | Course, I'm not what you'd call an "operator". | |
| 20:22:12 | tasker | hehe. can we bring back "verbose = False"? | |
| 20:23:34 | tasker | efried: I appreciate you taking the time to help me out. You've shed some light on some things I did not know and given me a lot of good information to play around with. | |
| 20:23:59 | efried | you're welcome, sorry I couldn't be more help. | |
| 20:30:29 | tasker | I did match the "pid" in the log file to the PIDs of the uWSGI helpers that spawn whenever the nova service starts. | |
| 20:30:40 | tasker | that lends weight to it being the WSGI module(s). | |
| 20:31:03 | tasker | are the modules using the same file independent of Nova? | |
| 20:32:08 | efried | tasker: You're out of my wheelhouse at this point. Throwing it around earlier like I knew what I was talking about, but I can barely spell WSGI. | |
| 20:33:10 | tasker | it's alright. those were mostly musings. I tend to "stream of conciousness" debug and use any open chat window to get ideas out. | |
| 20:33:32 | efried | heh, wfm. | |
| 20:34:04 | efried | tasker: I've got a guy who might know a bit more. He's on his way into the channel, if you're going to hang around for a bit. | |
| 20:34:20 | tasker | yeah. thanks! | |
| 20:34:56 | efried | o/ donnyd | |
| 20:35:10 | efried | donnyd: meet tasker. tasker: likewise | |
| 20:35:17 | donnyd | Hi | |
| 20:35:18 | efried | donnyd has actual experience deploying openstack | |
| 20:35:19 | tasker | donnyd: helo! | |
| 20:35:37 | donnyd | I'm just getting caught up on what the issue is | |
| 20:36:38 | tasker | trying to turn down all logging in Nova to WARNING because I'm getting these messages that indicate the API endpoints are hit at ( what I guess is INFO ) level. efried was helping and about all we can do is determine that they're coming from WSGI. | |
| 20:37:22 | donnyd | Do you have an example of the log entry you don't want to see? | |
| 20:37:32 | tasker | http://dpaste.com/2C5P2PY | |
| 20:39:06 | tasker | this is Pike, and prior to it, I could set "verbose = False" in nova.conf and call it a day. | |
| 20:39:27 | donnyd | Next question is what deployment method did you use? | |
| 20:39:51 | tasker | OSA | |
| 20:42:02 | donnyd | That surely looks like a wsgi log to me. (I see the same log entry type in my ister server). | |
| 20:43:07 | tasker | know if there's a comparable "verbose = False" flag for uWSGI? | |
| 20:43:20 | tasker | their docs don't hint at such. | |
| 20:50:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Link versioned notification talk into docs https://review.opendev.org/661115 | |
| 20:53:12 | tasker | appending "--disable-logging" to the uwsgi ExecStart string in the service file killed the messages, but that's too heavy-handed. | |
| 20:55:42 | donnyd | Still searching | |
| 20:56:48 | tasker | and having "--disable-logging" and having Nova at "debug_log_levels = nova=INFO | |
| 20:56:55 | tasker | " I can still get Nova's notices of API hits. | |
| 20:57:28 | openstackgerrit | Merged openstack/nova master: refactor nova-manage archive_deleted_rows https://review.opendev.org/643779 | |
| 20:58:09 | tasker | I'm stepping out for some "fresh air". back in a few. | |
| 20:58:12 | efried | Nova meeting in 2 minutes in #openstack-meeting | |
| 21:00:10 | efried | pawing through uwsgi docs, I'm not actually finding any hints that they *have* different log levels. | |
| 21:00:14 | efried | I think it's just on or off. | |
| 21:01:29 | efried | in which case your heavy-handed solution might be the only solution, beyond `grep` :P | |
| 21:01:38 | donnyd | Thats what it looks like to me @efried | |
| 21:02:25 | efried | tasker: Is there anything in nova-api-os-compute.log that you actually care about? | |
| 21:03:54 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118 | |
| 21:07:07 | tasker | efried: errors and warnings. . ) what are you thinking? | |
| 21:09:52 | efried | tasker: I was just wondering whether you could filter the whole thing out :) | |
| 21:10:05 | donnyd | https://github.com/unbit/uwsgi-docs/blob/master/ManagementFlag.rst | |
| 21:10:05 | openstackgerrit | Merged openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.opendev.org/651647 | |
| 21:10:17 | donnyd | 0loggingenable/disable logging | |
| 21:10:33 | donnyd | Looks like its mostly an off or on kinda thing | |
| 21:15:17 | tasker | efried: donnyd: thank you both very much for your assistance. with your help, I think I have a solution that will work. I'll be leaving it overnight and trying to trigger various warnings and events to verify it does what I need. | |
| 21:15:40 | efried | good luck tasker | |
| 21:34:56 | sean-k-mooney | on the nova console auth topic kolla-ansibel still defualt to deploying it on master | |
| 21:35:50 | efried | whee | |
| 21:37:07 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661123 | |
| 21:38:20 | sean-k-mooney | osa looks like id does not but i only quickly looked | |
| 21:39:05 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: [stable-only] Improve the reno for consoleauth upgrade check https://review.opendev.org/661118 | |
| 21:39:19 | sean-k-mooney | and tripleo remvoed it 14 days ago https://github.com/openstack/tripleo-heat-templates/blob/c33f8cb1174146e3cbdca6241914a78a623ea505/releasenotes/notes/nova_remove_nova-consoleauth-227cbeb44e9ab5ef.yaml | |
| 21:41:00 | melwitt | nice detective work :) | |
| 21:42:31 | sean-k-mooney | melwitt: what was the console auth service used for with cells v1 again | |
| 21:43:26 | melwitt | all console token auth. for cells v1 the database backend stuff was _not_ implemented | |
| 21:44:07 | sean-k-mooney | ah right so its jsust issueing the tokens to connect to the websockets? | |
| 21:44:27 | mriedem | osa is done https://github.com/openstack/openstack-ansible-os_nova/commit/1df0bd51ee5031039d3852cefbb63162e29b22a5 | |
| 21:45:20 | melwitt | sean-k-mooney: console auth service? yeah it stores the token auths locally and responds yea/nay when a request to connect to console proxy comes in | |
| 21:46:04 | melwitt | and the change we did for non-cells v1 was store the token auths in the cell databases and have console proxies (websockets) run per cell so they can query the database to validate token auths | |
| 21:46:20 | sean-k-mooney | melwitt ok so looking at the patches to remove it in osa and triplo if we just dont deploy it will it work with nova today? | |
| 21:47:17 | sean-k-mooney | i dont see any config cahnge in those patches so it seam like nova just does the right thing now? | |
| 21:47:22 | melwitt | sean-k-mooney: yeah, the default behavior is to use the database as the console token auth storage. so running it is not necessary. unless you're running cells v1 | |
| 21:47:37 | sean-k-mooney | which kolla never supported | |
| 21:48:18 | sean-k-mooney | so i can write a quick patch to delete the container form kolla and remove it form the kolla-ansibel playbook if that is all thats needed | |
| 21:48:53 | sean-k-mooney | well the upgade playbook would also need to stop any running containers | |
| 21:49:09 | sean-k-mooney | but that seams trivial too | |
| 21:50:26 | melwitt | I don't know if this is relevant at this point but during a live, rolling upgrade from queens to rocky, you would need to keep nova-consoleauth running until everything is rocky | |
| 21:50:49 | sean-k-mooney | looks like https://bugs.launchpad.net/kolla-ansible/+bug/1804474 is open but i am gessing its not been worked on | |
| 21:50:49 | melwitt | if you were not OK with disrupting existing console sessions | |
| 21:50:50 | openstack | Launchpad bug 1804474 in kolla-ansible "nova-consoleauth is deprecated" [Undecided,In progress] - Assigned to caoyuan (cao-yuan) | |
| 21:51:23 | sean-k-mooney | melwitt: well if i just remove it in the train branch that should not be an issue right | |
| 21:51:36 | melwitt | yeah should not be | |
| 21:51:37 | sean-k-mooney | e.g. going form stien to train it wont be used anyway | |
| 21:51:55 | sean-k-mooney | and it woudl be safe to backprot technically to rock->stien too | |
| 21:53:04 | sean-k-mooney | ... that said im not ment to be starting new things until i finish some of my curent in flight tasks. | |
| 21:53:29 | melwitt | hah | |
| 22:30:58 | openstackgerrit | Merged openstack/nova master: Do not start nova-network in the notification func test https://review.opendev.org/660975 | |
| 22:34:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Link versioned notification talk into docs https://review.opendev.org/661115 | |
| 22:36:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | |
| 23:57:55 | openstackgerrit | Merged openstack/python-novaclient stable/stein: Set the lower limit of api_version for volume_type https://review.opendev.org/661024 | |
| #openstack-nova - 2019-05-24 | |||
| 02:03:02 | openstackgerrit | melanie witt proposed openstack/nova master: Add documentation for counting quota usage from placement https://review.opendev.org/653845 | |
| 02:03:02 | openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.opendev.org/638324 | |
| 02:42:38 | openstackgerrit | Merged openstack/nova stable/stein: Disable limit if affinity(anti)/same(different)host is requested https://review.opendev.org/659239 | |
| 03:23:19 | yaawang | bauzas: Hi, in spec expose-auto-converge-post-copy(https://review.opendev.org/#/c/651681/), what do you mean by "preffering"? | |
| 04:58:25 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.opendev.org/603955 | |
| 05:10:07 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | |
| 05:10:07 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.opendev.org/631242 | |
| 05:10:08 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | |
| 05:10:09 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.opendev.org/631245 | |
| 05:49:11 | eandersson | What is the general recommended value for heal_instance_info_cache_interval on a large deployment (~1k nodes)? | |
| 05:50:01 | eandersson | With the default we couldn't scale neutron fast enough to actually handle the load. | |