JSON File Logging Pattern (Queue + Rotation)

Use this reference when you need machine-readable JSON logs written to rotating files without blocking caller threads.

This page captures the pattern used in the logging notebook example: configure a queue-backed root logger, route queued records to a rotating JSON file handler, and explicitly start and stop the QueueListener around workload execution.

Pattern Overview

Use this topology:

application code -> named logger/root logger -> QueueHandler -> QueueListener -> RotatingFileHandler(JSON)

Why this shape:

QueueHandler keeps file I/O off the main execution path. See Dealing with handlers that block.
RotatingFileHandler bounds disk usage and preserves recent history in backups. See RotatingFileHandler.
A JSON formatter makes logs easy to parse for automation and analytics. See python-json-logger.

Configuration Example

import logging
import logging.config

LOGGING = {
	"version": 1,
	"disable_existing_loggers": False,
	"formatters": {
		"console": {
			"format": "%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
			"datefmt": "%Y-%m-%d %H:%M:%S",
		},
		"json": {
			"()": "pythonjsonlogger.json.JsonFormatter",
			"format": "pathname,lineno,taskName,created,name,levelname,message,args",
			"style": ",",
			"rename_fields": {"levelname": "level"},
		},
	},
	"handlers": {
		"console": {
			"class": "logging.StreamHandler",
			"formatter": "console",
			"level": "INFO",
		},
		"queue": {
			"class": "logging.handlers.QueueHandler",
			"handlers": ["file"],
		},
		"file": {
			"class": "logging.handlers.RotatingFileHandler",
			"filename": "app.log",
			"maxBytes": 1024**2 * 5,
			"backupCount": 5,
			"formatter": "json",
		},
	},
	"root": {"level": "DEBUG", "handlers": ["queue", "console"]},
}

logging.config.dictConfig(LOGGING)

Notes:

Queue/listener configuration through dictConfig is documented in Configuring QueueHandler and QueueListener.
disable_existing_loggers: False is usually safer unless you intentionally want to disable existing non-root loggers.

Listener Lifecycle Pattern

When using queue-backed logging, treat listener startup and shutdown as explicit lifecycle responsibilities.

import logging
import warnings
from collections.abc import Callable
from contextlib import contextmanager
from functools import cache
from logging.handlers import QueueHandler, QueueListener


@cache
def get_listener(queue_handler_name: str) -> QueueListener | None:
	match logging.getHandlerByName(queue_handler_name):
		case QueueHandler(listener=QueueListener() as listener):
			return listener


def _listener_action(queue_handler_name: str, action: Callable[[QueueListener], None]):
	match get_listener(queue_handler_name):
		case QueueListener() as listener:
			action(listener)
			return listener


def start_listener(queue_handler_name: str) -> None:
	listener = _listener_action(queue_handler_name, lambda listener: listener.start())
	if listener is None:
		warnings.warn(f"{queue_handler_name} is not set up correctly", stacklevel=2)
		return


def stop_listener(queue_handler_name: str) -> None:
	_listener_action(queue_handler_name, lambda listener: listener.stop())


@contextmanager
def listener_lifespan(queue_handler_name: str):
	start_listener(queue_handler_name)
	try:
		yield
	finally:
		stop_listener(queue_handler_name)


with listener_lifespan("queue"):
	logging.info("Started")
    for _ in range(10**6):
        logging.debug("Hello world")
	logging.info("Done")
logging.info("Console only")

This demonstrates deterministic listener startup/shutdown around the active workload

Docs for APIs used above:

Reading JSON Logs Back

For quick validation, read recent lines and deserialize JSON:

import json
from collections import deque
from pathlib import Path


def read_last_n_lines(file: str | Path, *, n: int):
    with Path(file).open("r") as f:
        return deque(f, maxlen=n)


lines = read_last_n_lines("app.log", n=5)
records = list(map(json.loads, lines))

For rotated logs, enumerate files by basename and sort by modification time before reading.

Practical Checks

Before calling this done:

Confirm listener startup and shutdown run for the workload lifecycle.
Confirm app.log receives JSON lines, not plain text.
Confirm rotation occurs at the expected size and backup count.
Confirm console output still appears at the desired level.
Confirm exceptions and key context fields are preserved in JSON output.

5.9 KiB Raw Blame History