john/prompts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

declarative logging

Browse Source
This commit is contained in:
John Lancaster
2026-07-08 22:39:46 -05:00
parent 963805c551
commit 70dd0f45d9
4 changed files with 229 additions and 454 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+2 -1
View File
@@ -254,6 +254,7 @@ Existing markdown reference sets are valid examples of authored source material
1. docs/skills/pytesting/references/pytest-docs.md
2. docs/skills/python-logging/references/python-logging-docs.md
3. docs/skills/fastapi-uv-docker/references/fastapi-best-practices.md
3. docs/skills/python-logging/references/json-file-logging.md
4. docs/skills/fastapi-uv-docker/references/fastapi-best-practices.md
These inputs are treated as content sources, while resource URIs and catalog payloads remain the machine-facing contracts.
docs/mcp_layout.md
+2 -1
View File
@@ -195,6 +195,7 @@ Existing reference docs remain valid content inputs in this pattern:
1. docs/skills/pytesting/references/pytest-docs.md
2. docs/skills/python-logging/references/python-logging-docs.md
3. docs/skills/fastapi-uv-docker/references/fastapi-best-practices.md
3. docs/skills/python-logging/references/json-file-logging.md
4. docs/skills/fastapi-uv-docker/references/fastapi-best-practices.md
These are source documents, not deployment artifacts.
docs/skills/python-logging/references/httpx-logging-handler-example.md
+100 -249
View File
@@ -1,26 +1,97 @@
# HTTPX Logging Handler Example
Use this reference when a Python application needs to send log records to an HTTP endpoint with [`httpx`](https://www.python-httpx.org/). The example follows the same boundaries as the TCP network example: feature modules use normal named loggers, application startup configures logging once, a queue keeps HTTP I/O off the caller path, and the receiver or collector owns final routing.
Use this reference when an application should emit JSON logs to an HTTP collector while keeping startup logging configuration declarative.
This page follows the top-level skill pattern:
- define one `LOGGING` dictionary
- apply it once with `logging.config.dictConfig(LOGGING)`
- keep modules focused on logger calls
Source docs to keep nearby:
- [HTTPX clients](https://www.python-httpx.org/advanced/clients/) for connection pooling and shared request configuration.
- [HTTPX timeouts](https://www.python-httpx.org/advanced/timeouts/) for connect, read, write, and pool timeout behavior.
- [HTTPX JSON requests](https://www.python-httpx.org/quickstart/#sending-json-encoded-data) for posting JSON payloads.
- [HTTPX exceptions](https://www.python-httpx.org/quickstart/#exceptions) for `RequestError`, `HTTPStatusError`, and `HTTPError` handling.
- [Dealing with handlers that block](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block) for why slow handlers should sit behind `QueueHandler` and `QueueListener`.
- [`QueueHandler`](https://docs.python.org/3/library/logging.handlers.html#queuehandler) and [`QueueListener`](https://docs.python.org/3/library/logging.handlers.html#queuelistener) for queue-backed logging mechanics.
- [`logging.makeLogRecord`](https://docs.python.org/3/library/logging.html#logging.makeLogRecord) for rebuilding records from serialized fields.
- [HTTPX clients](https://www.python-httpx.org/advanced/clients/)
- [HTTPX timeouts](https://www.python-httpx.org/advanced/timeouts/)
- [`logging.config.dictConfig`](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig)
## Minimal Topology
Send each record as a JSON HTTP request to a collector endpoint. Use `httpx.Client`, not the top-level `httpx.post`, because a handler may send many records to the same host and should reuse connections.
```text
application code -> named logger -> QueueHandler -> QueueListener -> HTTPX JSON handler -> HTTP receiver or collector
application code -> named logger -> HttpxJsonLogHandler -> HTTP collector
```
Application modules stay ordinary:
## Reusable Handler Type
Keep transport behavior in one handler class and wire it declaratively through `dictConfig`.
```python title="httpx_json_handler.py"
import logging
import httpx
class HttpxJsonLogHandler(logging.Handler):
def __init__(self, collector_url: str, timeout_seconds: float = 2.0, token: str | None = None) -> None:
super().__init__()
headers = {"content-type": "application/json"}
if token is not None:
headers["authorization"] = f"Bearer {token}"
timeout = httpx.Timeout(timeout_seconds)
self.client = httpx.Client(base_url=collector_url, headers=headers, timeout=timeout)
def emit(self, record: logging.LogRecord) -> None:
payload = {
"name": record.name,
"levelname": record.levelname,
"levelno": record.levelno,
"pathname": record.pathname,
"lineno": record.lineno,
"funcName": record.funcName,
"created": record.created,
"message": record.getMessage(),
}
try:
response = self.client.post("/logs", json=payload)
response.raise_for_status()
except httpx.HTTPError:
self.handleError(record)
def close(self) -> None:
self.client.close()
super().close()
```
## Application Logging Configuration (Declarative)
```python title="logging_config.py"
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"httpx": {
"class": "httpx_json_handler.HttpxJsonLogHandler",
"collector_url": "http://127.0.0.1:9021",
"timeout_seconds": 2.0,
"token": None,
},
"console": {
"class": "logging.StreamHandler",
"level": "INFO",
"stream": "ext://sys.stdout",
},
},
"root": {
"level": "INFO",
"handlers": ["httpx", "console"],
},
}
def configure_logging() -> None:
logging.config.dictConfig(LOGGING)
```
```python title="feature.py"
import logging
@@ -32,249 +103,29 @@ def sync_customer(customer_id: str) -> None:
logger.info("Syncing customer %s", customer_id)
```
The module does not know whether logs are written locally, sent over HTTP, or forwarded by a platform collector.
## Minimal Receiver
This receiver is for local testing. A production deployment would usually send the same JSON shape to a managed log collector, OpenTelemetry collector, service endpoint, or internal ingestion API.
```python title="log_http_receiver.py"
import json
import logging
import logging.config
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"console": {
"format": "%(asctime)s %(levelname)s %(name)s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "console",
"stream": "ext://sys.stdout",
}
},
"root": {"level": "INFO", "handlers": ["console"]},
}
class LogRecordRequestHandler(BaseHTTPRequestHandler):
def do_POST(self) -> None:
if self.path != "/logs":
self.send_error(404)
return
length = int(self.headers.get("Content-Length", "0"))
body = self.rfile.read(length)
try:
payload = json.loads(body.decode("utf-8"))
record = logging.makeLogRecord(payload)
except Exception:
logging.getLogger(__name__).exception("Dropped malformed log record")
self.send_error(400)
return
logger = logging.getLogger(record.name)
if logger.isEnabledFor(record.levelno):
logger.handle(record)
self.send_response(204)
self.end_headers()
def log_message(self, format: str, *args: object) -> None:
logging.getLogger("http.server").debug(format, *args)
def main() -> None:
logging.config.dictConfig(LOGGING)
server = ThreadingHTTPServer(("127.0.0.1", 9021), LogRecordRequestHandler)
with server:
logging.getLogger(__name__).info("Listening for HTTP log records")
server.serve_forever()
if __name__ == "__main__":
main()
```
### Receiver Mechanics
- The receiver accepts `POST /logs` with a JSON body and returns `204` when the record is accepted.
- `makeLogRecord` turns the JSON dictionary back into a standard `LogRecord`, so the receiver can use the normal logger hierarchy.
- Receiver-side filtering still works, but client-side filtering is better when volume matters because it avoids serializing and transmitting records that will be discarded.
- `log_message` is redirected into the logging system at `DEBUG` so access logs do not pollute normal output.
- Binding to `127.0.0.1` keeps the demo local. If the receiver is reachable across a network, put authentication, TLS, rate limits, and request size limits in front of it.
## Client Configuration
The client side uses a queue-backed logging handler. The listener thread owns the `httpx.Client`, posts JSON records, and closes the connection pool during shutdown.
```python title="logging_config.py"
import copy
import logging
import logging.handlers
import queue
import httpx
class PreservingQueueHandler(logging.handlers.QueueHandler):
def prepare(self, record: logging.LogRecord) -> logging.LogRecord:
copied = copy.copy(record)
copied.message = copied.getMessage()
copied.msg = copied.message
copied.args = None
if copied.exc_info is not None and copied.exc_text is None:
copied.exc_text = logging.Formatter().formatException(copied.exc_info)
copied.exc_info = None
return copied
class HttpxJsonLogHandler(logging.Handler):
def __init__(self, collector_url: str, token: str | None = None) -> None:
super().__init__()
headers = {"content-type": "application/json"}
if token is not None:
headers["authorization"] = f"Bearer {token}"
timeout = httpx.Timeout(2.0, connect=1.0, write=2.0, pool=1.0)
self.client = httpx.Client(base_url=collector_url, headers=headers, timeout=timeout)
def emit(self, record: logging.LogRecord) -> None:
try:
response = self.client.post("/logs", json=self._payload_from_record(record))
response.raise_for_status()
except httpx.HTTPError:
self.handleError(record)
def close(self) -> None:
self.client.close()
super().close()
def _payload_from_record(self, record: logging.LogRecord) -> dict[str, object]:
payload: dict[str, object] = {
"name": record.name,
"levelno": record.levelno,
"levelname": record.levelname,
"pathname": record.pathname,
"lineno": record.lineno,
"funcName": record.funcName,
"created": record.created,
"process": record.process,
"processName": record.processName,
"threadName": record.threadName,
"msg": record.getMessage(),
"args": None,
}
if record.exc_text is not None:
payload["exc_text"] = record.exc_text
return payload
def configure_logging(
collector_url: str = "http://127.0.0.1:9021",
token: str | None = None,
) -> logging.handlers.QueueListener:
log_queue: queue.Queue[logging.LogRecord] = queue.Queue(maxsize=1000)
queue_handler = PreservingQueueHandler(log_queue)
http_handler = HttpxJsonLogHandler(collector_url, token)
listener = logging.handlers.QueueListener(log_queue, http_handler, respect_handler_level=True)
root = logging.getLogger()
root.setLevel(logging.INFO)
root.handlers[:] = [queue_handler]
listener.start()
return listener
```
```python title="app.py"
import logging
```python title="main.py"
from feature import sync_customer
from logging_config import configure_logging
logger = logging.getLogger(__name__)
def main() -> None:
listener = configure_logging()
try:
logger.info("Service started")
logger.warning("Example warning from the HTTPX client")
finally:
listener.stop()
if __name__ == "__main__":
main()
configure_logging()
sync_customer("C-101")
```
Start `log_http_receiver.py` first, then run `app.py`. The receiver should print the records using its own formatter.
## Collector-Side Configuration (Declarative)
## HTTPX Mechanics
Whether you use an internal HTTP endpoint or a managed collector, keep receiver-side formatting and routing declared on the receiver side, not in application modules.
- `httpx.Client` keeps a connection pool. That matters for log handlers because repeated top-level `httpx.post(...)` calls would create new connections instead of reusing them.
- `base_url` makes the handler explicit about the collector host while keeping the endpoint path short.
- `timeout` is explicit. HTTPX has default timeouts, but logging code should state its tolerance for connect, write, read, and pool waits.
- `response.raise_for_status()` turns non-2xx responses into `HTTPStatusError`, which then goes through the logging handler's normal error path.
- `close()` closes the HTTPX connection pool. Pair this with `listener.stop()` during application shutdown so queued records are sent and resources are released.
## Why This Pattern
## Logging Mechanics
- Logging wiring is declared once and applied once.
- Runtime behavior changes by editing config fields, not scattered root mutations.
- Feature modules stay independent from transport details.
- HTTP connection details remain encapsulated in one handler type.
- The application still configures logging once. Feature modules only call named loggers.
- The HTTP handler sits behind `QueueHandler` and `QueueListener` because HTTP requests can block on DNS, connection pooling, TLS, request writes, response reads, and collector back pressure.
- `PreservingQueueHandler` copies the record and formats exception text before clearing `exc_info`, so the queued record is safe to serialize and still carries useful traceback information.
- The handler serializes a deliberate subset of `LogRecord` fields. Sending `record.__dict__` wholesale is easy, but it can include unserializable objects, accidental high-cardinality fields, or data the collector should not receive.
- Authentication is represented as an optional bearer token header. In real applications, read tokens from a secret manager or runtime configuration, not from source code.
## Review Checklist
## Rationale Behind The Pattern
### Prefer HTTP When The Receiver Is Already An HTTP API
HTTP is a good fit when logs go to a collector, gateway, ingestion service, or internal API that already expects JSON over HTTPS. It also gives you familiar deployment controls: TLS termination, authentication, reverse proxies, rate limiting, request size limits, and conventional status codes.
### Keep HTTP Off The Caller Path
Even a fast collector can become slow during deploys, network incidents, or downstream outages. Queueing makes that failure mode a logging concern instead of a request-latency concern.
### Use A Bounded Queue For Honest Back Pressure
The example uses `queue.Queue(maxsize=1000)` so overload becomes visible. The default `QueueHandler.enqueue()` uses `put_nowait()`, so a full queue calls `handleError()`. For production, decide whether to drop logs, block briefly, spill to disk, or switch to a platform collector.
### Filter Before Sending
The collector can reject records, but rejected records already consumed CPU, queue capacity, and network bandwidth. Use client-side logger and handler levels to avoid sending noisy records unless the deployment explicitly needs them.
## Production Checklist
Before using this beyond a local demo:
1. Use HTTPS and authenticate clients. Treat the collector endpoint as an ingestion boundary, not a public anonymous API.
2. Set request size limits and reject malformed payloads early.
3. Decide the outage policy for collector failures and full queues.
4. Add service, environment, instance, request, trace, or tenant identifiers as explicit serialized fields when operators need correlation.
5. Redact or avoid secrets before records leave the process.
6. Normalize untrusted newline-containing values if the final destination is line-oriented.
7. Tune timeouts and queue size under load, not only with a happy-path local receiver.
8. Prefer a managed collector, OpenTelemetry pipeline, or platform-native logging when one already exists.
## Review Questions
Use these questions when reviewing HTTPX logging code:
- Does application code only call named loggers, without direct HTTP calls from feature modules?
- Is the HTTP handler behind a queue for web, async, worker, or high-throughput paths?
- Does the handler use a reusable `httpx.Client` rather than top-level request functions?
- Are timeouts explicit and short enough for a logging path?
- Are collector failures, non-2xx responses, and full queues handled deliberately?
- Does shutdown stop the listener and close the HTTPX client?
- Are authentication, TLS, secrets, request size, and high-cardinality context handled deliberately?
1. Is there one `LOGGING` dict for the application process?
2. Is `dictConfig` called once at startup?
3. Are module loggers created via `logging.getLogger(__name__)`?
4. Are HTTP endpoint, timeout, and auth token inputs declared in handler config?
5. Are final routing/retention decisions handled by the collector side?
docs/skills/python-logging/references/network-logging-minimal-example.md
+122 -200
View File
@@ -1,28 +1,64 @@
# Network Logging Minimal Example
Use this reference when an application needs to send Python logs across a network to a receiver process. The example is intentionally small, but it keeps the important production-shaped boundaries: application modules use normal named loggers, startup code configures routing once, network I/O happens away from the caller path, and the receiver owns final formatting and destinations.
Use this reference when an application should send logs over TCP to a local receiver and you want a complete, working baseline.
This page shows how the pieces fit together end to end:
- application code logs with named loggers
- startup applies one declarative `LOGGING` config
- `SocketHandler` sends records to a receiver
- receiver uses `socketserver` and local logging config for final routing
Source docs to keep nearby:
- [Sending and receiving logging events across a network](https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network) for the standard socket-listener recipe.
- [Dealing with handlers that block](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block) for why `QueueHandler` and `QueueListener` belong in front of slow handlers.
- [`SocketHandler`](https://docs.python.org/3/library/logging.handlers.html#sockethandler) for the built-in network handler and its pickle-based default wire format.
- [`QueueHandler`](https://docs.python.org/3/library/logging.handlers.html#queuehandler) and [`QueueListener`](https://docs.python.org/3/library/logging.handlers.html#queuelistener) for queue-backed logging mechanics.
- [`logging.makeLogRecord`](https://docs.python.org/3/library/logging.html#logging.makeLogRecord) for recreating a `LogRecord` from serialized fields.
- [`socketserver`](https://docs.python.org/3/library/socketserver.html) for a tiny TCP receiver.
- [logging configuration security considerations](https://docs.python.org/3/library/logging.config.html#security-considerations) for treating remote logging configuration and importable config objects as trusted inputs only.
- [Sending and receiving logging events across a network](https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network)
- [`SocketHandler`](https://docs.python.org/3/library/logging.handlers.html#sockethandler)
- [`socketserver`](https://docs.python.org/3/library/socketserver.html)
- [`logging.makeLogRecord`](https://docs.python.org/3/library/logging.html#logging.makeLogRecord)
## Minimal Topology
Run one receiver process near the final logging destination. Application processes send serialized records to it, and the receiver decides how those records are formatted, filtered, written, rotated, or forwarded.
```text
application code -> named logger -> QueueHandler -> QueueListener -> JSON TCP handler -> receiver -> final handlers
app module -> logger -> SocketHandler -> TCP receiver -> local handlers
```
This shape is useful because network handlers can block. Even a socket handler can pause on DNS, connection setup, back pressure, or a slow collector. The queue keeps normal request, worker, or CLI code from doing that work directly.
## 1) Client Logging Config (Declarative)
It also keeps application code boring in the best way:
```python title="logging_config.py"
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"console": {
"format": "%(asctime)s %(levelname)s %(name)s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"network": {
"class": "logging.handlers.SocketHandler",
"host": "127.0.0.1",
"port": 9020,
},
"console": {
"class": "logging.StreamHandler",
"formatter": "console",
"level": "INFO",
"stream": "ext://sys.stdout",
},
},
"root": {
"level": "INFO",
"handlers": ["network", "console"],
},
}
def configure_logging() -> None:
logging.config.dictConfig(LOGGING)
```
```python title="feature.py"
import logging
@@ -34,17 +70,24 @@ def process_order(order_id: str) -> None:
logger.info("Processing order %s", order_id)
```
The feature module does not know whether logs go to a terminal, a file, a socket, or a collector. That decision belongs to application startup.
```python title="main.py"
from feature import process_order
from logging_config import configure_logging
## Receiver
The receiver accepts newline-delimited JSON records, recreates `LogRecord` objects, and routes them through local logging configuration.
def main() -> None:
configure_logging()
process_order("A-42")
```python title="log_receiver.py"
import json
import logging
if __name__ == "__main__":
main()
```
## 2) Receiver Logging Config (Declarative)
```python title="receiver_logging_config.py"
import logging.config
import socketserver
LOGGING = {
"version": 1,
@@ -66,29 +109,63 @@ LOGGING = {
}
class LogRecordHandler(socketserver.StreamRequestHandler):
def configure_receiver_logging() -> None:
logging.config.dictConfig(LOGGING)
```
## 3) Cookbook Receiver (`socketserver`) Implementation
This receiver follows the same structure as the Python logging cookbook example.
`SocketHandler` sends:
- a 4-byte big-endian length prefix
- a pickle payload containing a `LogRecord` dictionary
```python title="log_receiver.py"
import logging
import pickle
import socketserver
import struct
from receiver_logging_config import configure_receiver_logging
class LogRecordStreamHandler(socketserver.StreamRequestHandler):
def handle(self) -> None:
for line in self.rfile:
while True:
chunk = self.connection.recv(4)
if len(chunk) < 4:
break
payload_len = struct.unpack(">L", chunk)[0]
payload = self.connection.recv(payload_len)
while len(payload) < payload_len:
payload = payload + self.connection.recv(payload_len - len(payload))
try:
payload = json.loads(line.decode("utf-8"))
record = logging.makeLogRecord(payload)
record_dict = pickle.loads(payload)
record = logging.makeLogRecord(record_dict)
except Exception:
logging.getLogger(__name__).exception("Dropped malformed log record")
continue
self.handle_log_record(record)
def handle_log_record(self, record: logging.LogRecord) -> None:
logger = logging.getLogger(record.name)
if logger.isEnabledFor(record.levelno):
logger.handle(record)
class LogRecordServer(socketserver.ThreadingTCPServer):
class LogRecordSocketReceiver(socketserver.ThreadingTCPServer):
allow_reuse_address = True
def main() -> None:
logging.config.dictConfig(LOGGING)
with LogRecordServer(("127.0.0.1", 9020), LogRecordHandler) as server:
logging.getLogger(__name__).info("Listening for log records")
configure_receiver_logging()
with LogRecordSocketReceiver(("127.0.0.1", 9020), LogRecordStreamHandler) as server:
logging.getLogger(__name__).info("Receiver listening on 127.0.0.1:9020")
server.serve_forever()
@@ -96,182 +173,27 @@ if __name__ == "__main__":
main()
```
### Receiver Mechanics
## 4) How It Fits Together In Practice
- `dictConfig` is local to the receiver. Client processes do not decide the final formatter, file handler, rotation policy, or downstream sink.
- `makeLogRecord` rebuilds a logging record from plain fields. This is the same reconstruction step used by the cookbook socket receiver, but this example uses JSON instead of unpickling bytes from the network.
- The receiver looks up `logging.getLogger(record.name)` so package-level logger names still route through the normal logging hierarchy.
- The `isEnabledFor` check lets receiver-side logger levels suppress records before handlers run. Client-side filtering is still preferred when possible because it avoids wasted network traffic.
- Binding to `127.0.0.1` makes the demo local-only. Binding to `0.0.0.0` changes the trust boundary and should be paired with network controls, authentication, or a real collector protocol.
1. Start `log_receiver.py`.
2. Start `main.py` from the client app.
3. Client logs go to console and TCP.
4. Receiver reconstructs records and emits them through its own handlers.
## Client Configuration
This split keeps app emission and receiver routing independent while still being fully runnable.
Configure logging once at application startup. The root logger writes to a queue, and a listener thread sends records over TCP.
## Important Security Note
```python title="logging_config.py"
import copy
import json
import logging
import logging.handlers
import queue
import socket
`SocketHandler` uses pickle serialization. Treat this as trusted-network-only transport.
- Bind receiver to localhost or a trusted private network.
- Do not expose this receiver to untrusted clients.
- For hostile boundaries, use JSON/TLS with authenticated ingestion instead of raw pickle.
class PreservingQueueHandler(logging.handlers.QueueHandler):
def prepare(self, record: logging.LogRecord) -> logging.LogRecord:
copied = copy.copy(record)
copied.message = copied.getMessage()
copied.msg = copied.message
copied.args = None
## Review Checklist
if copied.exc_info is not None and copied.exc_text is None:
copied.exc_text = logging.Formatter().formatException(copied.exc_info)
copied.exc_info = None
return copied
class JsonTcpHandler(logging.Handler):
def __init__(self, host: str, port: int, timeout: float = 2.0) -> None:
super().__init__()
self.host = host
self.port = port
self.timeout = timeout
self._socket: socket.socket | None = None
def emit(self, record: logging.LogRecord) -> None:
try:
payload = self._payload_from_record(record)
message = json.dumps(payload, separators=(",", ":")).encode("utf-8")
self._send(message + b"\n")
except Exception:
self._close_socket()
self.handleError(record)
def close(self) -> None:
self._close_socket()
super().close()
def _close_socket(self) -> None:
if self._socket is not None:
self._socket.close()
self._socket = None
def _send(self, message: bytes) -> None:
if self._socket is None:
self._socket = socket.create_connection((self.host, self.port), self.timeout)
self._socket.sendall(message)
def _payload_from_record(self, record: logging.LogRecord) -> dict[str, object]:
payload: dict[str, object] = {
"name": record.name,
"levelno": record.levelno,
"levelname": record.levelname,
"pathname": record.pathname,
"lineno": record.lineno,
"funcName": record.funcName,
"created": record.created,
"process": record.process,
"processName": record.processName,
"threadName": record.threadName,
"msg": record.getMessage(),
"args": None,
}
if record.exc_text is not None:
payload["exc_text"] = record.exc_text
return payload
def configure_logging(host: str = "127.0.0.1", port: int = 9020) -> logging.handlers.QueueListener:
log_queue: queue.Queue[logging.LogRecord] = queue.Queue(maxsize=1000)
queue_handler = PreservingQueueHandler(log_queue)
network_handler = JsonTcpHandler(host, port)
listener = logging.handlers.QueueListener(log_queue, network_handler, respect_handler_level=True)
root = logging.getLogger()
root.setLevel(logging.INFO)
root.handlers[:] = [queue_handler]
listener.start()
return listener
```
```python title="app.py"
import logging
from logging_config import configure_logging
logger = logging.getLogger(__name__)
def main() -> None:
listener = configure_logging()
try:
logger.info("Service started")
logger.warning("Example warning from the client")
finally:
listener.stop()
if __name__ == "__main__":
main()
```
Start `log_receiver.py` first, then run `app.py`. The receiver should print the records using its own formatter.
### Client Mechanics
- `PreservingQueueHandler` copies the record before mutating it for queue transfer. The standard `QueueHandler.prepare()` intentionally formats and strips unpickleable fields; overriding it is the documented escape hatch when the listener side needs custom serialization or exception text.
- `queue.Queue(maxsize=1000)` makes back pressure visible. An unbounded queue is simpler, but a bounded queue forces a production decision about whether to drop, block, buffer elsewhere, or fail when the collector cannot keep up.
- `QueueListener` owns the slow handler thread. It should be stopped during application shutdown so queued records are processed before exit.
- `JsonTcpHandler` keeps a TCP connection open after the first event. That avoids a connection handshake per log record while keeping the example small enough to inspect.
- `handleError()` preserves standard logging error behavior. In production, set an explicit policy for dropped records and collector outages rather than assuming logs always arrive.
## Rationale Behind The Pattern
### Use JSON Instead Of The Default Pickle Payload
The built-in `SocketHandler` sends a pickled record dictionary. That is convenient on a trusted local path, but unpickling network input is a poor default at a trust boundary. JSON is not a complete security boundary by itself, but it is inspectable, language-neutral, and avoids executing pickle payloads.
If you use `SocketHandler` anyway, override `makePickle()` with a safer encoding, sign payloads with a scheme such as HMAC, or keep the listener strictly inside a trusted local network.
### Put The Network Handler Behind A Queue
The cookbook explicitly calls out network handlers as potentially blocking. A queue is the smallest standard-library pattern that separates business code from slow handler work. This matters for web requests, async event loops, worker hot paths, and CLIs where the user should not wait on a collector timeout.
### Centralize Final Destinations In The Receiver
Multiple processes writing one file directly is a common source of garbled output, failed rotation, and confusing retention behavior. A receiver process serializes that responsibility: clients emit events, and one process writes or forwards them according to one logging configuration.
### Keep Logger Names Stable
Use module loggers such as `logging.getLogger(__name__)`. Do not create a logger per request, user, socket, tenant, or connection. Put those values in structured fields or formatted messages instead. Logger objects are singletons and are not freed during normal execution, so unbounded logger names become an avoidable memory and routing problem.
### Filter Early When Volume Matters
The receiver can filter records, but records already crossed the network by then. Set client-side logger or handler levels so routine `DEBUG` records are not serialized and transmitted unless the deployment is intentionally collecting them.
## Production Checklist
Before using this beyond a local demo:
1. Protect the receiver with a trusted network boundary, TLS, a VPN, mutual authentication, or a real log collector. Do not expose an unauthenticated logging port to untrusted clients.
2. Decide the outage policy: drop records, block briefly, buffer locally, retry with backoff, or fail startup when the receiver is unavailable.
3. Size the queue and choose the overflow behavior deliberately. The default `QueueHandler.enqueue()` uses `put_nowait()`, so a full bounded queue goes through `handleError()`.
4. Include service, environment, instance, request, trace, or tenant identifiers when operators need cross-service correlation.
5. Redact or avoid secrets before records leave the process.
6. Escape or normalize untrusted newline-containing values if the final destination is line-oriented and vulnerable to log injection confusion.
7. Load-test the receiver and validate shutdown behavior before relying on the logs during incidents.
8. Prefer a managed collector, OpenTelemetry pipeline, syslog, container stdout collection, or platform-native logging when the deployment environment already provides one.
## Review Questions
Use these questions when reviewing network logging code:
- Does normal application code only call named loggers, without attaching handlers in feature modules?
- Is network or file I/O behind a queue for web, async, worker, or high-throughput paths?
- Is the wire format safe for the trust boundary, or does it rely on unpickling unauthenticated input?
- Are logger and handler levels set so noisy records are filtered before crossing the network?
- Is collector failure behavior explicit and tested?
- Does shutdown stop the listener and flush the queue?
- Are secrets, user-controlled newlines, and high-cardinality context handled deliberately?
1. Is there one `LOGGING` dict per process role (client and receiver)?
2. Is `dictConfig` called once at each process startup?
3. Does the receiver decode length-prefixed payloads correctly?
4. Do modules only use `logging.getLogger(__name__)`?
5. Is the receiver endpoint protected by trust boundaries?