Silence Kedro Logs-拾光赋

kedro (18 Part Series)

1 🤷‍️ What is Kedro (The Parts)
2 Create New Kedro Pipeline (kedro new)
… 14 more parts…
3 Announcing find-kedro
4 Kedro Hooks Intro – creating the kedro-preflight hook
5 Create Configurable Kedro Hooks
6 Kedro Static Viz 0.3.0 is out with Hooks Support
7 How Kedro handles your inputs
8 Kedro – My Data Is Not A Table
9 Minimal Kedro Pipeline
10 Kedro pipeline_registry.py
11 Silence Kedro Logs
12 Using Kedro In Scripts
13 Kedro Spaceflights – part 1
14 Incremental Versioned Datasets in Kedro
15 Use Kedro Catalog Create to boost your pipeline development productivity.
16 Data Engineering in The Terminal | How I Kedro
17 What is Kedro
18 Creating new Data Pipelines from the command line

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output.

Silence a Python log

First, how does one silence a python log? Python loggers can be retrieved by the logging module’s getLogger function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways.

python logging levels

Level	Numeric value
CRITICAL	50
ERROR	40
WARNING	30
INFO	20
DEBUG	10
NOTSET	0

Get or Create a logger

Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in.


<span>logger</span> <span>=</span> <span>logging</span><span>.</span><span>getLogger</span><span>(</span><span>'</span><span>kedro</span><span>'</span><span>)</span>
<span>logger</span> <span>=</span> <span>logging</span><span>.</span><span>getLogger</span><span>(</span><span>'</span><span>kedro</span><span>'</span><span>)</span>
logger = logging.getLogger('kedro')

Enter fullscreen mode Exit fullscreen mode

If a logger doesn’t exist under the passed in name, it will create one for you.

Set Level

Once we get the logger, we need to silence it by setting the log level. Typically it’s not appropriate to completely turn off loggers as you would still want information in the case of a complete failure. If you are building a cli such as one that prints out the pipelines to the console, you may not want to see logs that happen during regular operation as this would make it more challenging to integrate with other shell applications.


<span>logger</span><span>.</span><span>setLevel</span><span>(</span><span>logging</span><span>.</span><span>ERROR</span><span>)</span>
<span>logger</span><span>.</span><span>setLevel</span><span>(</span><span>logging</span><span>.</span><span>ERROR</span><span>)</span>
logger.setLevel(logging.ERROR)

Enter fullscreen mode Exit fullscreen mode

Be sure to leave some logging left. After the point of error, you are not going to get a clean output anyways. So let the user see what happened.

It is possible to set the log level before kedro even registers the logger, if there is no logger currently setup under getLogger, it will create one.

Silent all kedro loggers

As of kedro==0.17.3 this function covers every logger issued by kedro. I generated this list of known_kedro_loggers by looking through their codebase and filling in a few others I found by running it.


<span>def</span> <span>silent_loggers</span><span>()</span> <span>-></span> <span>None</span><span>:</span>
    <span>"""</span><span>All logs need to be silent in order for a clean kedro diff output.</span><span>"""</span>
    <span>known_kedro_loggers</span> <span>=</span> <span>[</span>
        <span>"</span><span>ProfileTimeTransformer</span><span>"</span><span>,</span>
        <span>"</span><span>hooks_handler</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.__init__</span><span>"</span><span>,</span>
        <span>"</span><span>kedro</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.config</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.config.config</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.extras.decorators.memory_profiler</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.cli</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.session.session</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.session.store</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.session</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.io.cached_dataset</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.io.data_catalog</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.io</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.journal</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline.decorators</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline.node</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline.pipeline</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.runner</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.runner.runner</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.versioning.journal</span><span>"</span><span>,</span>
        <span>"</span><span>py4</span><span>"</span><span>,</span>
    <span>]</span>
    <span>for</span> <span>logger</span> <span>in</span> <span>[</span>
        <span>*</span><span>known_kedro_loggers</span><span>,</span>
        <span>*</span><span>list</span><span>(</span><span>logging</span><span>.</span><span>root</span><span>.</span><span>manager</span><span>.</span><span>loggerDict</span><span>.</span><span>keys</span><span>()),</span>  <span># type: ignore </span>    <span>]:</span>
        <span>logging</span><span>.</span><span>getLogger</span><span>(</span><span>logger</span><span>).</span><span>setLevel</span><span>(</span><span>logging</span><span>.</span><span>ERROR</span><span>)</span>
<span>def</span> <span>silent_loggers</span><span>()</span> <span>-></span> <span>None</span><span>:</span>
    <span>"""</span><span>All logs need to be silent in order for a clean kedro diff output.</span><span>"""</span>
    <span>known_kedro_loggers</span> <span>=</span> <span>[</span>
        <span>"</span><span>ProfileTimeTransformer</span><span>"</span><span>,</span>
        <span>"</span><span>hooks_handler</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.__init__</span><span>"</span><span>,</span>
        <span>"</span><span>kedro</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.config</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.config.config</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.extras.decorators.memory_profiler</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.cli</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.session.session</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.session.store</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.framework.session</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.io.cached_dataset</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.io.data_catalog</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.io</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.journal</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline.decorators</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline.node</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.pipeline.pipeline</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.runner</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.runner.runner</span><span>"</span><span>,</span>
        <span>"</span><span>kedro.versioning.journal</span><span>"</span><span>,</span>
        <span>"</span><span>py4</span><span>"</span><span>,</span>
    <span>]</span>
    <span>for</span> <span>logger</span> <span>in</span> <span>[</span>
        <span>*</span><span>known_kedro_loggers</span><span>,</span>
        <span>*</span><span>list</span><span>(</span><span>logging</span><span>.</span><span>root</span><span>.</span><span>manager</span><span>.</span><span>loggerDict</span><span>.</span><span>keys</span><span>()),</span>  <span># type: ignore </span>    <span>]:</span>
        <span>logging</span><span>.</span><span>getLogger</span><span>(</span><span>logger</span><span>).</span><span>setLevel</span><span>(</span><span>logging</span><span>.</span><span>ERROR</span><span>)</span>
def silent_loggers() -> None:
    """All logs need to be silent in order for a clean kedro diff output."""
    known_kedro_loggers = [
        "ProfileTimeTransformer",
        "hooks_handler",
        "kedro.__init__",
        "kedro",
        "kedro.config",
        "kedro.config.config",
        "kedro.extras.decorators.memory_profiler",
        "kedro.framework.cli",
        "kedro.framework.session.session",
        "kedro.framework.session.store",
        "kedro.framework.session",
        "kedro.io.cached_dataset",
        "kedro.io.data_catalog",
        "kedro.io",
        "kedro.journal",
        "kedro.pipeline",
        "kedro.pipeline.decorators",
        "kedro.pipeline.node",
        "kedro.pipeline.pipeline",
        "kedro.runner",
        "kedro.runner.runner",
        "kedro.versioning.journal",
        "py4",
    ]
    for logger in [
        *known_kedro_loggers,
        *list(logging.root.manager.loggerDict.keys()),  # type: ignore     ]:
        logging.getLogger(logger).setLevel(logging.ERROR)

Enter fullscreen mode Exit fullscreen mode

This function comes right from a plugin I am currently working on kedro-diff. Check it out, give it a star, and watch it for release.

https://waylonwalker.com/what-is-kedro/

Not familiar with kedro, check out this article to see what it’s all about.

Master the log

Python logs can seem super confusing at first, understanding how to get a logger and set its level are the first steps to mastering it.

Check Out These Related Posts

https://waylonwalker.com/if_name_main/

https://waylonwalker.com/install-micromamba/

https://waylonwalker.com/kedro172_replit/

kedro (18 Part Series)

原文链接：Silence Kedro Logs

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END