prompts/docs/skills/python-logging/references/json-file-logging.md

# JSON File Logging Pattern (Queue + Rotation)

Use this reference when you need machine-readable JSON logs written to rotating files without blocking caller threads.

This page captures the pattern used in the logging notebook example: configure a queue-backed root logger, route queued records to a rotating JSON file handler, and explicitly start and stop the `QueueListener` around workload execution.


## Pattern Overview

Use this topology:

```text
application code -> named logger/root logger -> QueueHandler -> QueueListener -> RotatingFileHandler(JSON)
```

Why this shape:

- `QueueHandler` keeps file I/O off the main execution path. See [Dealing with handlers that block](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block).
- `RotatingFileHandler` bounds disk usage and preserves recent history in backups. See [RotatingFileHandler](https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler).
- A JSON formatter makes logs easy to parse for automation and analytics. See [python-json-logger](https://nhairs.github.io/python-json-logger/latest/).

## Configuration Example

```python title="logging_config.py"
import logging
import logging.config

LOGGING = {
	"version": 1,
	"disable_existing_loggers": False,
	"formatters": {
		"console": {
			"format": "%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
			"datefmt": "%Y-%m-%d %H:%M:%S",
		},
		"json": {
			"()": "pythonjsonlogger.json.JsonFormatter",
			"format": "pathname,lineno,taskName,created,name,levelname,message,args",
			"style": ",",
			"rename_fields": {"levelname": "level"},
		},
	},
	"handlers": {
		"console": {
			"class": "logging.StreamHandler",
			"formatter": "console",
			"level": "INFO",
		},
		"queue": {
			"class": "logging.handlers.QueueHandler",
			"handlers": ["file"],
		},
		"file": {
			"class": "logging.handlers.RotatingFileHandler",
			"filename": "app.log",
			"maxBytes": 1024**2 * 5,
			"backupCount": 5,
			"formatter": "json",
		},
	},
	"root": {"level": "DEBUG", "handlers": ["queue", "console"]},
}

logging.config.dictConfig(LOGGING)
```

Notes:

- Queue/listener configuration through `dictConfig` is documented in [Configuring QueueHandler and QueueListener](https://docs.python.org/3/library/logging.config.html#configuring-queuehandler-and-queuelistener).
- `disable_existing_loggers: False` is usually safer unless you intentionally want to disable existing non-root loggers.

## Listener Lifecycle Pattern

When using queue-backed logging, treat listener startup and shutdown as explicit lifecycle responsibilities.

```python title="listener_lifecycle.py"
import logging
import warnings
from collections.abc import Callable
from contextlib import contextmanager
from functools import cache
from logging.handlers import QueueHandler, QueueListener


@cache
def get_listener(queue_handler_name: str) -> QueueListener | None:
	match logging.getHandlerByName(queue_handler_name):
		case QueueHandler(listener=QueueListener() as listener):
			return listener


def _listener_action(queue_handler_name: str, action: Callable[[QueueListener], None]):
	match get_listener(queue_handler_name):
		case QueueListener() as listener:
			action(listener)
			return listener


def start_listener(queue_handler_name: str) -> None:
	listener = _listener_action(queue_handler_name, lambda listener: listener.start())
	if listener is None:
		warnings.warn(f"{queue_handler_name} is not set up correctly", stacklevel=2)
		return


def stop_listener(queue_handler_name: str) -> None:
	_listener_action(queue_handler_name, lambda listener: listener.stop())


@contextmanager
def listener_lifespan(queue_handler_name: str):
	start_listener(queue_handler_name)
	try:
		yield
	finally:
		stop_listener(queue_handler_name)


with listener_lifespan("queue"):
	logging.info("Started")
    for _ in range(10**6):
        logging.debug("Hello world")
	logging.info("Done")
logging.info("Console only")
```

This demonstrates deterministic listener startup/shutdown around the active workload

Docs for APIs used above:

- [`logging.getHandlerByName`](https://docs.python.org/3/library/logging.html#logging.getHandlerByName)
- [`QueueHandler`](https://docs.python.org/3/library/logging.handlers.html#queuehandler)
- [`QueueListener`](https://docs.python.org/3/library/logging.handlers.html#queuelistener)
- [`contextlib.contextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)
- [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)


## Reading JSON Logs Back

For quick validation, read recent lines and deserialize JSON:

```python title="inspect_logs.py"
import json
from collections import deque
from pathlib import Path


def read_last_n_lines(file: str | Path, *, n: int):
    with Path(file).open("r") as f:
        return deque(f, maxlen=n)


lines = read_last_n_lines("app.log", n=5)
records = list(map(json.loads, lines))
```

For rotated logs, enumerate files by basename and sort by modification time before reading.

## Practical Checks

Before calling this done:

1. Confirm listener startup and shutdown run for the workload lifecycle.
2. Confirm `app.log` receives JSON lines, not plain text.
3. Confirm rotation occurs at the expected size and backup count.
4. Confirm console output still appears at the desired level.
5. Confirm exceptions and key context fields are preserved in JSON output.

## Source Links

- [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html)
- [logging.config reference](https://docs.python.org/3/library/logging.config.html)
- [Configuring QueueHandler and QueueListener](https://docs.python.org/3/library/logging.config.html#configuring-queuehandler-and-queuelistener)
- [logging handlers reference](https://docs.python.org/3/library/logging.handlers.html)
- [LogRecord attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes)
- [python-json-logger docs](https://nhairs.github.io/python-json-logger/latest/)