Python Logging¶
Logging by Print¶
Logging to console by print is a common starting point for building out code, however, that has some cons:
- No categorization - You cannot set catagories (debug, info, etc.) to the log.
- Only sent to console - No easy way to send to logging services.
- No metadata - No native way to add metadata, such as timestamps to it
- Clutters - Generally seen as left over code.
Python Standard Logging¶
Python has a standard logging facility that will allow you to work on several of those cons. In order to setup the looking it you need to setup a logger instance.
Here is an example of what that has:
That is, create a logging instance (usually assigned to the variable logger).
The normal construct is to use the special attribute __name__ as the file name, to make the code the same in each file, but for it to evaluate to the same thing.
Let's look at some of the methods with the object instance.
['addFilter', 'addHandler', 'callHandlers', 'critical', 'debug', 'disabled', 'error', 'exception', 'fatal', 'filter', 'filters', 'findCaller', 'getChild', 'getEffectiveLevel', 'handle', 'handlers', 'hasHandlers', 'info', 'isEnabledFor', 'level', 'log', 'makeRecord', 'manager', 'name', 'parent', 'propagate', 'removeFilter', 'removeHandler', 'root', 'setLevel', 'warn', 'warning']
Let's also see some constant definition (the ones that are capitalized) on logging (not logger).
The constants will be used to set flags, so keep them in mind for later in the lesson.
Let's add a log message.
Now, when we use a variable in logging, the best practices is to use the %s method and not the f-string or format methods. This is primarily because logs should not cause an error, and the %s is lazy loaded. Which means that it is only evaluated at runtime. This allows you to not cause an issue if there is a bug in your log
Instead it should be:
logger.warning('There was an warning %s', 'non existing value')
logger.warning('There was an warning %s', non_existing_value)
There are some other reasons for lazy loading, such as having an expensive call, but the primary reason is to not have issues started from the logging.
Python Standard Logging - LAB¶
Go through you your code of choice (capstone, napalm challenge, etc) and add logging to it. We will just use logger.warning for all logs for now.
Logger Levels¶
The reality is you do not want to always have the same logging levels all the time.
Python's built-in logging module provides several logging levels to indicate the severity of an event. The levels, from least to most severe, are as follows:
- DEBUG: Detailed information, typically of interest only when diagnosing problems.
- INFO: Confirmation that things are working as expected.
- WARNING: An indication that something unexpected happened or there may be a problem in the near future (e.g., 'disk space low'). The software is still working as expected.
- ERROR: Due to a more serious problem, the software has not been able to perform some function.
- CRITICAL: A very serious error, indicating that the program itself may be unable to continue running.
Each of these levels has an associated integer value:
- DEBUG: 10
- INFO: 20
- WARNING: 30
- ERROR: 40
- CRITICAL: 50
When you set a logging level for your logger, it will log messages from that level and all levels above it. For example, if you set the level to WARNING, it will capture WARNING, ERROR, and CRITICAL messages but will ignore DEBUG and INFO messages.
These values are set in the logging constants we reviewed earlier.
By default, it is set to WARNING, you can see this by calling the getEffectiveLevel() method.
You can set the logging level with the setLevel() method.
The common construct is to always use the logging constant value.
Logger Levels - LAB¶
Update your logging to:
- Set the logging level to be info
- Change the logs to the appropriate logging level vs all being warning
Logger Format¶
The logging module allows you to specify the format of your log messages using the Formatter class. The format string uses placeholders, wrapped in %(name)s, to insert log record attributes like the log level, message, timestamp, and more.
Common Format Attributes:
%(asctime)s: Human-readable time when the LogRecord was created.%(levelname)s: Text logging level (e.g., 'DEBUG', 'ERROR').%(message)s: The logged message itself.%(name)s: Logger's name.%(filename)s: Filename where the log message originated.%(lineno)d: Line number in the file where the log call was made.
Here's a simple example that demonstrates the use of formatting:
import logging
# Setting up the logger and handler
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# Create a console handler
handler = logging.StreamHandler()
# Set a format for the handler
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
# Add the handler to the logger
logger.addHandler(handler)
# Example log messages
logger.debug("This is a debug message.")
logger.error("This is an error message.")
When you run the above code, you might get output like:
>>> logger.debug("This is a debug message.")
2023-07-14 11:05:31,997 - __main__ - DEBUG - This is a debug message.
>>> logger.error("This is an error message.")
2023-07-14 11:05:32,609 - __main__ - ERROR - This is an error message.
>>>
The Formatter has added the current timestamp, logger's name, log level, and the message itself. Adjusting the format string allows you to customize this output as needed.
Logger Formats - LAB¶
Update your script with the logger format of your choice.
Advanced Topics¶
There is a lot of other features to consider in logging:
- Using an external logging configuration
- Logging to an external file or logging service.
- Logging to multiple places
- Logging format with third party libraries such as
logstruct.