I have an issue where I’m unable to control the verbosity of logs in parts of my Python scripts – it’s probably somehow related to either selenium or seleniumwire. I have a main python file run.py which is used to initiate other python scripts.

My goal was to launch scripts from run.py and log everything under one main log file, ideally called run.log and then separate log files which contain all the logs for the other scripts that are launched by run.py, e.g. other_script would have it’s own log file called other_script.log. Essentially logs related to other_script would be found in both files, but run.py would have logs about other scripts as well.

The way I have this set up currently is that I have a function setup_logging that is initiated under run.py:

def setup_logging(python_file_name: str = None):
    if python_file_name:
        python_file_names = [python_file_name]
    else:
        python_file_names = ['run', 'other_script', 'other_script_2']
    for python_file_name in python_file_names:
        log_file = os.path.join(os.env['GOOGLE_DRIVE_PATH'], 'logs', f'{python_file_name}.log')
        file_handler = TimedRotatingFileHandler(log_file, when='W0', interval=1)
        file_handler.setLevel(logging.INFO)
        formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(funcName)s - %(message)s')
        file_handler.setFormatter(formatter)
        logging.getLogger().addHandler(file_handler)

def main():
    setup_logging()
    logger = logging.getLogger(__name__)
    # do stuff
    logger.info('Running other_script')
    os.system('python3 other_script.py')

Then in other_script.py

import logging
logger = logging.getLogger(__name__)
def main():
    driver = get_webdriver()
    driver.get('www.some_url.com')
    request = driver.wait_for_request('/data?')
    logger.info(f'Request successful for {url}')
    for request in driver.requests:
        json_response = get_json(request)
        logger.debug(f'Logging info about this request {json_response}')
        # etc...

So my logger is set to level INFO which I want to keep as is, because there are a lot of print statements I wish to keep. What happens is that all the logs end up in __main__.log and the last part (other_script.py) creates a lot of verbose logs under INFO stemming presumably from selenium/seleniumwire.

Example of verbose logs I want removed

2024-01-21 21:02:21,758 - INFO - request - Capturing request: https://www.long-url.com/...
2024-01-21 21:02:21,784 - INFO - response - Capturing response: https://www.long-url.com/... 200 

I have tried the following

• in setup_logging change the log level to WARNING, but now I don’t get any of the INFO-level print statements I actually want to see in the logs (e.g. logger.info(f'Request successful for {url}')) .
• set --log-level 2 or above under chrome_options for the webdriver
• in other_script.py try to specify a different logging level for selenium.webdriver.remote.remote_connection with this

from selenium.webdriver.remote.remote_connection import LOGGER as seleniumLogger
seleniumLogger.setLevel(logging.WARNING)
# Set the threshold for urllib3 to WARNING
from urllib3.connectionpool import log as urllibLogger
urllibLogger.setLevel(logging.WARNING)
logger = logging.getLogger(__name__)