logging skill updates
This commit is contained in:
John Lancaster
@@ -0,0 +1,176 @@
|
||||
# JSON File Logging Pattern (Queue + Rotation)
|
||||
|
||||
Use this reference when you need machine-readable JSON logs written to rotating files without blocking caller threads.
|
||||
|
||||
This page captures the pattern used in the logging notebook example: configure a queue-backed root logger, route queued records to a rotating JSON file handler, and explicitly start and stop the `QueueListener` around workload execution.
|
||||
|
||||
|
||||
## Pattern Overview
|
||||
|
||||
Use this topology:
|
||||
|
||||
```text
|
||||
application code -> named logger/root logger -> QueueHandler -> QueueListener -> RotatingFileHandler(JSON)
|
||||
```
|
||||
|
||||
Why this shape:
|
||||
|
||||
- `QueueHandler` keeps file I/O off the main execution path. See [Dealing with handlers that block](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block).
|
||||
- `RotatingFileHandler` bounds disk usage and preserves recent history in backups. See [RotatingFileHandler](https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler).
|
||||
- A JSON formatter makes logs easy to parse for automation and analytics. See [python-json-logger](https://nhairs.github.io/python-json-logger/latest/).
|
||||
|
||||
## Configuration Example
|
||||
|
||||
```python title="logging_config.py"
|
||||
import logging
|
||||
import logging.config
|
||||
|
||||
LOGGING = {
|
||||
"version": 1,
|
||||
"disable_existing_loggers": False,
|
||||
"formatters": {
|
||||
"console": {
|
||||
"format": "%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
|
||||
"datefmt": "%Y-%m-%d %H:%M:%S",
|
||||
},
|
||||
"json": {
|
||||
"()": "pythonjsonlogger.json.JsonFormatter",
|
||||
"format": "pathname,lineno,taskName,created,name,levelname,message,args",
|
||||
"style": ",",
|
||||
"rename_fields": {"levelname": "level"},
|
||||
},
|
||||
},
|
||||
"handlers": {
|
||||
"console": {
|
||||
"class": "logging.StreamHandler",
|
||||
"formatter": "console",
|
||||
"level": "INFO",
|
||||
},
|
||||
"queue": {
|
||||
"class": "logging.handlers.QueueHandler",
|
||||
"handlers": ["file"],
|
||||
},
|
||||
"file": {
|
||||
"class": "logging.handlers.RotatingFileHandler",
|
||||
"filename": "app.log",
|
||||
"maxBytes": 1024**2 * 5,
|
||||
"backupCount": 5,
|
||||
"formatter": "json",
|
||||
},
|
||||
},
|
||||
"root": {"level": "DEBUG", "handlers": ["queue", "console"]},
|
||||
}
|
||||
|
||||
logging.config.dictConfig(LOGGING)
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Queue/listener configuration through `dictConfig` is documented in [Configuring QueueHandler and QueueListener](https://docs.python.org/3/library/logging.config.html#configuring-queuehandler-and-queuelistener).
|
||||
- `disable_existing_loggers: False` is usually safer unless you intentionally want to disable existing non-root loggers.
|
||||
|
||||
## Listener Lifecycle Pattern
|
||||
|
||||
When using queue-backed logging, treat listener startup and shutdown as explicit lifecycle responsibilities.
|
||||
|
||||
```python title="listener_lifecycle.py"
|
||||
import logging
|
||||
import warnings
|
||||
from collections.abc import Callable
|
||||
from contextlib import contextmanager
|
||||
from functools import cache
|
||||
from logging.handlers import QueueHandler, QueueListener
|
||||
|
||||
|
||||
@cache
|
||||
def get_listener(queue_handler_name: str) -> QueueListener | None:
|
||||
match logging.getHandlerByName(queue_handler_name):
|
||||
case QueueHandler(listener=QueueListener() as listener):
|
||||
return listener
|
||||
|
||||
|
||||
def _listener_action(queue_handler_name: str, action: Callable[[QueueListener], None]):
|
||||
match get_listener(queue_handler_name):
|
||||
case QueueListener() as listener:
|
||||
action(listener)
|
||||
return listener
|
||||
|
||||
|
||||
def start_listener(queue_handler_name: str) -> None:
|
||||
listener = _listener_action(queue_handler_name, lambda listener: listener.start())
|
||||
if listener is None:
|
||||
warnings.warn(f"{queue_handler_name} is not set up correctly", stacklevel=2)
|
||||
return
|
||||
|
||||
|
||||
def stop_listener(queue_handler_name: str) -> None:
|
||||
_listener_action(queue_handler_name, lambda listener: listener.stop())
|
||||
|
||||
|
||||
@contextmanager
|
||||
def listener_lifespan(queue_handler_name: str):
|
||||
start_listener(queue_handler_name)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
stop_listener(queue_handler_name)
|
||||
|
||||
|
||||
with listener_lifespan("queue"):
|
||||
logging.info("Started")
|
||||
for _ in range(10**6):
|
||||
logging.debug("Hello world")
|
||||
logging.info("Done")
|
||||
logging.info("Console only")
|
||||
```
|
||||
|
||||
This demonstrates deterministic listener startup/shutdown around the active workload
|
||||
|
||||
Docs for APIs used above:
|
||||
|
||||
- [`logging.getHandlerByName`](https://docs.python.org/3/library/logging.html#logging.getHandlerByName)
|
||||
- [`QueueHandler`](https://docs.python.org/3/library/logging.handlers.html#queuehandler)
|
||||
- [`QueueListener`](https://docs.python.org/3/library/logging.handlers.html#queuelistener)
|
||||
- [`contextlib.contextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)
|
||||
- [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)
|
||||
|
||||
|
||||
## Reading JSON Logs Back
|
||||
|
||||
For quick validation, read recent lines and deserialize JSON:
|
||||
|
||||
```python title="inspect_logs.py"
|
||||
import json
|
||||
from collections import deque
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def read_last_n_lines(file: str | Path, *, n: int):
|
||||
with Path(file).open("r") as f:
|
||||
return deque(f, maxlen=n)
|
||||
|
||||
|
||||
lines = read_last_n_lines("app.log", n=5)
|
||||
records = list(map(json.loads, lines))
|
||||
```
|
||||
|
||||
For rotated logs, enumerate files by basename and sort by modification time before reading.
|
||||
|
||||
## Practical Checks
|
||||
|
||||
Before calling this done:
|
||||
|
||||
1. Confirm listener startup and shutdown run for the workload lifecycle.
|
||||
2. Confirm `app.log` receives JSON lines, not plain text.
|
||||
3. Confirm rotation occurs at the expected size and backup count.
|
||||
4. Confirm console output still appears at the desired level.
|
||||
5. Confirm exceptions and key context fields are preserved in JSON output.
|
||||
|
||||
## Source Links
|
||||
|
||||
- [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html)
|
||||
- [logging.config reference](https://docs.python.org/3/library/logging.config.html)
|
||||
- [Configuring QueueHandler and QueueListener](https://docs.python.org/3/library/logging.config.html#configuring-queuehandler-and-queuelistener)
|
||||
- [logging handlers reference](https://docs.python.org/3/library/logging.handlers.html)
|
||||
- [LogRecord attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes)
|
||||
- [python-json-logger docs](https://nhairs.github.io/python-json-logger/latest/)
|
||||
Reference in New Issue
Block a user