Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9b02007216 | |||
| 22e5357ffb |
@@ -0,0 +1,240 @@
|
||||
---
|
||||
name: fastapi-async-sqlalchemy-modernization
|
||||
description: 'Create a step-by-step modernization plan for an existing FastAPI app using SQLAlchemy async patterns, context managers, and AsyncExitStack. Use when: planning migration from legacy DB setup, standardizing async engine/session lifecycles, defining transaction boundaries, and aligning with SQLAlchemy 2.x best practices.'
|
||||
argument-hint: 'What is your current FastAPI + SQLAlchemy setup (sync/async driver, session pattern, lifespan usage, and deployment model)?'
|
||||
---
|
||||
|
||||
# FastAPI Async SQLAlchemy Modernization Plan
|
||||
|
||||
Create an implementation-ready plan that brings an existing FastAPI application in line with modern async SQLAlchemy practices, with explicit resource lifecycles and deterministic cleanup using async context managers and AsyncExitStack.
|
||||
|
||||
Primary targets: PostgreSQL with asyncpg and SQLite with aiosqlite.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Existing FastAPI app has ad hoc database setup or mixed sync/async access.
|
||||
- Session management is inconsistent across routes/services.
|
||||
- Lifespan startup and shutdown work is spread across globals and side effects.
|
||||
- Team needs a migration plan first, not immediate large-scale rewrites.
|
||||
|
||||
## Outcome
|
||||
|
||||
Produce a practical modernization plan with:
|
||||
|
||||
- Current-state gap assessment.
|
||||
- Target architecture for engine/session/transaction lifecycle.
|
||||
- Branch-based migration path (low-risk staged rollout).
|
||||
- Quality gates and completion checks.
|
||||
- Risks, rollback strategy, and test plan.
|
||||
|
||||
## Top-Level Concepts
|
||||
|
||||
Use these concepts as the planning backbone:
|
||||
|
||||
1. Engine lifecycle and ownership:
|
||||
One AsyncEngine per process for each DB URL, created once and disposed explicitly when the app lifecycle ends.
|
||||
See: [references/engine.md](references/engine.md)
|
||||
2. Session factory and scope:
|
||||
Use async_sessionmaker for configuration; create one AsyncSession per request or unit-of-work, never shared across concurrent tasks.
|
||||
See: [references/session.md](references/session.md)
|
||||
3. Transaction boundaries:
|
||||
Prefer context-managed begin blocks for write units and explicit read-only sessions for queries.
|
||||
See: [references/transactions.md](references/transactions.md)
|
||||
4. Lifespan composition:
|
||||
Compose startup/shutdown resources with AsyncExitStack so cleanup is deterministic and ordered.
|
||||
See: [references/engine.md](references/engine.md)
|
||||
5. Dependency injection:
|
||||
Provide sessions via FastAPI dependencies with async generators/context managers, not globals.
|
||||
See: [references/session.md](references/session.md)
|
||||
6. Implicit I/O control in ORM:
|
||||
Avoid accidental lazy loads; use explicit eager-loading/refresh strategies for asyncio safety.
|
||||
See: [references/implicit_io.md](references/implicit_io.md)
|
||||
7. Observability and resilience:
|
||||
Add pool/connection settings, logging, timeout, and health checks as first-class plan items.
|
||||
See: [references/observability.md](references/observability.md)
|
||||
|
||||
### Concept Reference Map
|
||||
|
||||
| Concept | Reference |
|
||||
|---|---|
|
||||
| Engine lifecycle and ownership | [references/engine.md](references/engine.md) |
|
||||
| Session factory and scope | [references/session.md](references/session.md) |
|
||||
| Transaction boundaries | [references/transactions.md](references/transactions.md) |
|
||||
| Lifespan composition | [references/engine.md](references/engine.md) |
|
||||
| Dependency injection | [references/session.md](references/session.md) |
|
||||
| Implicit I/O control in ORM | [references/implicit_io.md](references/implicit_io.md) |
|
||||
| Observability and resilience | [references/observability.md](references/observability.md) |
|
||||
|
||||
## Decision Points
|
||||
|
||||
Use these branching decisions before proposing migration steps.
|
||||
|
||||
| Decision | Branch A | Branch B |
|
||||
|---|---|---|
|
||||
| DB driver | Already async driver (e.g. asyncpg, aiosqlite): modernize in place | Sync driver: plan driver migration first |
|
||||
| ORM usage | Already ORM 2.x style (`select`, `session.execute`) | Legacy Query API: add compatibility stage and refactor incrementally |
|
||||
| Session scope | Request-scoped already | Global/shared sessions found: prioritize session-scope fix first |
|
||||
| Lifespan | Existing FastAPI lifespan hook | No lifespan hook: introduce lifespan before broader DB changes |
|
||||
| Concurrency | Background jobs/tasks use DB | No background DB use |
|
||||
| Transaction style | Explicit context-managed transactions | Implicit/autobegin side effects |
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 0: Audit Current State
|
||||
|
||||
Inventory the app and write a concise gap list.
|
||||
|
||||
- Engine creation location(s) and count.
|
||||
- Driver URL(s) and async compatibility.
|
||||
- Session creation patterns in routes/services/background tasks.
|
||||
- Transaction handling style (explicit begin/commit/rollback vs implicit).
|
||||
- Lifespan startup/shutdown and cleanup behavior.
|
||||
- ORM loading patterns that may trigger implicit I/O.
|
||||
|
||||
Completion check: every DB touchpoint is mapped to its engine, session, and transaction source.
|
||||
|
||||
### Step 1: Define the Target Runtime Model
|
||||
|
||||
Define one canonical model to migrate toward.
|
||||
|
||||
- Create AsyncEngine once per process.
|
||||
- Configure async_sessionmaker once.
|
||||
- Use per-request AsyncSession dependency.
|
||||
- Keep one AsyncSession per concurrent task.
|
||||
- Use context-managed transactions for writes.
|
||||
|
||||
Completion check: architecture diagram can explain where engine/session are created, used, and closed.
|
||||
|
||||
### Step 2: Plan Engine Modernization
|
||||
|
||||
Plan engine creation and pool behavior.
|
||||
|
||||
- Use `create_async_engine()` with async dialect URL.
|
||||
- Standardize pool settings and pre-ping strategy where relevant.
|
||||
- Decide isolation level strategy at engine level (avoid ad hoc per-operation switching unless justified).
|
||||
- Define explicit disposal policy for short-lived scopes and tests.
|
||||
|
||||
Completion check: engine configuration is centralized and no per-request engine creation remains.
|
||||
|
||||
### Step 3: Plan Session Lifecycle Modernization
|
||||
|
||||
Define session factory and request dependency pattern.
|
||||
|
||||
- Build `async_sessionmaker(engine, expire_on_commit=False)` unless a strict reason says otherwise.
|
||||
- Provide session via dependency that yields exactly one AsyncSession.
|
||||
- Explicitly prohibit sharing a single AsyncSession across concurrent tasks.
|
||||
- Prefer direct dependency passing over async_scoped_session for new designs.
|
||||
|
||||
Completion check: all route/service entry points receive a session from one canonical dependency.
|
||||
|
||||
### Step 4: Plan Transaction Demarcation
|
||||
|
||||
Establish consistent write and read behavior.
|
||||
|
||||
- Writes: `async with session.begin(): ...` for atomic units.
|
||||
- Reads: execute in managed session context with explicit loader options.
|
||||
- Nested/SAVEPOINT use only where required; call out backend caveats.
|
||||
- Define rollback behavior for service-layer exceptions.
|
||||
|
||||
Completion check: every mutating use case has a declared transaction boundary.
|
||||
|
||||
### Step 5: Compose Lifespan with AsyncExitStack
|
||||
|
||||
Use async context composition as the preferred orchestration pattern.
|
||||
|
||||
```python
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
async with AsyncExitStack() as stack:
|
||||
# Compose resources in acquisition order; cleanup is automatic in reverse order.
|
||||
engine = create_async_engine(settings.database_url)
|
||||
stack.push_async_callback(engine.dispose)
|
||||
|
||||
session_factory = async_sessionmaker(engine, expire_on_commit=False)
|
||||
app.state.session_factory = session_factory
|
||||
|
||||
# Add other async resources with stack.enter_async_context(...) as needed.
|
||||
yield
|
||||
```
|
||||
|
||||
Planning rules:
|
||||
|
||||
- Register every acquired resource with AsyncExitStack at acquisition time.
|
||||
- Prefer `enter_async_context()` for resources that already expose async context managers.
|
||||
- Prefer `push_async_callback()` for async cleanup callables.
|
||||
- Keep resource ownership in lifespan, not in route handlers.
|
||||
|
||||
Completion check: startup/shutdown ordering is explicit and deterministic.
|
||||
|
||||
### Step 6: Prevent Implicit ORM I/O Under Asyncio (Advisory Mode)
|
||||
|
||||
Plan for explicit loading behavior, but treat this as progressive guidance rather than a hard gate.
|
||||
|
||||
- Recommend eager-loading strategies (for example selectin-style loading) where relationship access is required.
|
||||
- For lazy/deferred attributes, define explicit awaitable or refresh paths on high-risk and high-traffic paths first.
|
||||
- Document model-level defaults and known exceptions so teams can migrate incrementally.
|
||||
|
||||
Completion check: critical request paths have explicit loading plans; non-critical paths have tracked follow-up items.
|
||||
|
||||
### Step 7: Testing and Verification Plan
|
||||
|
||||
Create modernization quality gates.
|
||||
|
||||
- Unit tests for session dependency and transaction behavior.
|
||||
- Integration tests for commit/rollback semantics.
|
||||
- Concurrency tests confirming one-session-per-task behavior.
|
||||
- Lifespan tests verifying cleanup calls and ordering.
|
||||
- Health/readiness tests including DB connectivity checks.
|
||||
|
||||
Completion check: all quality gates pass under the target async configuration.
|
||||
|
||||
### Step 8: Rollout Strategy
|
||||
|
||||
Plan low-risk migration phases.
|
||||
|
||||
1. Introduce centralized engine/session factory and lifespan orchestration.
|
||||
2. Migrate read paths to new session dependency.
|
||||
3. Migrate write paths to explicit transaction blocks.
|
||||
4. Remove legacy globals/helpers and dead code.
|
||||
5. Enable stricter linting/review checks for forbidden patterns.
|
||||
|
||||
Completion check: no legacy session/engine creation path remains in production code.
|
||||
|
||||
## Quality Criteria
|
||||
|
||||
A plan is complete only when it includes:
|
||||
|
||||
- Clear current vs target architecture.
|
||||
- Branch decisions with rationale.
|
||||
- Explicit context-manager patterns for resource ownership.
|
||||
- AsyncExitStack composition strategy.
|
||||
- Transaction policy and exception behavior.
|
||||
- Concrete tests and rollout checkpoints.
|
||||
- A documented advisory backlog for non-critical implicit I/O improvements.
|
||||
|
||||
## Anti-Patterns to Flag
|
||||
|
||||
- Creating engines inside request handlers.
|
||||
- Sharing one AsyncSession across concurrent tasks.
|
||||
- Implicit commit/rollback behavior with unclear ownership.
|
||||
- Global mutable session state.
|
||||
- Lifespan cleanup that depends on implicit garbage collection.
|
||||
|
||||
## Output Contract
|
||||
|
||||
Return the plan as:
|
||||
|
||||
1. Current-state gap summary.
|
||||
2. Target architecture summary.
|
||||
3. Phased migration checklist with branch notes.
|
||||
4. Risk register and rollback approach.
|
||||
5. Verification matrix (tests + operational checks).
|
||||
|
||||
## References
|
||||
|
||||
- SQLAlchemy engine/connections: https://docs.sqlalchemy.org/en/21/core/connections.html
|
||||
- SQLAlchemy asyncio extension: https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html
|
||||
- Python async context managers and AsyncExitStack: https://docs.python.org/3/library/contextlib.html
|
||||
@@ -0,0 +1,133 @@
|
||||
# Async SQLAlchemy Engine
|
||||
|
||||
Source:
|
||||
- https://docs.sqlalchemy.org/en/21/core/connections.html
|
||||
- https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html
|
||||
- https://docs.sqlalchemy.org/en/21/core/pooling.html#pooling-multiprocessing
|
||||
- https://fastapi.tiangolo.com/advanced/events/
|
||||
|
||||
---
|
||||
|
||||
## Engine Ownership Model
|
||||
|
||||
Create one async engine per process per database URL and keep it for the app lifetime.
|
||||
|
||||
- SQLAlchemy guidance: the engine is intended as a long-lived, concurrent registry over pooled DB connections, not a per-request object.
|
||||
- In FastAPI, app startup and shutdown ownership belongs in lifespan.
|
||||
- Use `FastAPI(lifespan=...)` (not startup/shutdown events) for modern lifecycle wiring.
|
||||
|
||||
Practical rule:
|
||||
- Exactly one `create_async_engine(...)` call in app bootstrap code.
|
||||
- Zero `create_async_engine(...)` calls in request handlers.
|
||||
|
||||
---
|
||||
|
||||
## Canonical Lifespan Pattern (AsyncExitStack)
|
||||
|
||||
Use `@asynccontextmanager` + `AsyncExitStack` to make teardown deterministic and composable.
|
||||
|
||||
```python
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
async with AsyncExitStack() as stack:
|
||||
engine: AsyncEngine = create_async_engine(
|
||||
app.state.settings.database_url,
|
||||
pool_pre_ping=True,
|
||||
# Optional examples:
|
||||
# echo=app.state.settings.sql_echo,
|
||||
# pool_size=10,
|
||||
# max_overflow=20,
|
||||
)
|
||||
app.state.engine = engine
|
||||
|
||||
# Ensure engine disposal always runs at shutdown.
|
||||
stack.push_async_callback(engine.dispose)
|
||||
|
||||
yield
|
||||
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
```
|
||||
|
||||
Why this pattern:
|
||||
- FastAPI executes code before `yield` at startup and after `yield` at shutdown.
|
||||
- `AsyncExitStack` lets you register multiple async cleanups in one place while preserving order.
|
||||
- Explicit disposal (directly awaited or via `AsyncExitStack` callback) avoids event-loop-closed warnings when objects fall out of scope.
|
||||
|
||||
---
|
||||
|
||||
## Driver URLs (Project Requirement: asyncpg + aiosqlite)
|
||||
|
||||
Use SQLAlchemy async driver URLs:
|
||||
|
||||
- PostgreSQL: `postgresql+asyncpg://user:pass@host:5432/dbname`
|
||||
- SQLite: `sqlite+aiosqlite:///./app.db`
|
||||
|
||||
Notes:
|
||||
- Do not mix sync drivers (for example `psycopg2`) with `create_async_engine()`.
|
||||
- Keep URL construction centralized in settings/config, not in feature modules.
|
||||
|
||||
---
|
||||
|
||||
## Pooling Defaults and Tuning
|
||||
|
||||
Default behavior is usually correct first:
|
||||
|
||||
- Async engines use async-compatible pooling (`AsyncAdaptedQueuePool`) by default.
|
||||
- Start with defaults, then tune from observed load (`pool_size`, `max_overflow`, `pool_timeout`, `pool_recycle`).
|
||||
- Enable `pool_pre_ping=True` for safer stale-connection handling in long-running services.
|
||||
|
||||
When to switch pool strategy:
|
||||
|
||||
- `NullPool` if you explicitly need no pooling (special environments, some tests, or strict cross-loop constraints).
|
||||
- Keep in mind this increases connect/disconnect churn.
|
||||
|
||||
---
|
||||
|
||||
## Disposal Semantics
|
||||
|
||||
`engine.dispose()` replaces/disposes the pool, but only checked-in connections are immediately closed.
|
||||
|
||||
Rules:
|
||||
- Dispose when the app is shutting down.
|
||||
- Dispose before reusing an engine across event loops.
|
||||
- In forked child-process initialization, use `engine.dispose(close=False)` (sync API guidance) so child processes do not touch parent-held connections.
|
||||
|
||||
Avoid relying on garbage collection for engine cleanup in async code.
|
||||
|
||||
---
|
||||
|
||||
## Event Loop and Process Boundaries
|
||||
|
||||
Do not share pooled connections across boundaries:
|
||||
|
||||
- Multiple event loops: do not reuse the same pooled async engine across loops unless you intentionally disable pooling (`NullPool`) or dispose before handoff.
|
||||
- Multiprocessing/fork: pooled connections must not be inherited for active use across process boundaries.
|
||||
|
||||
This prevents broken socket state and cross-process connection corruption.
|
||||
|
||||
---
|
||||
|
||||
## What Not to Do
|
||||
|
||||
- Create an engine inside every request dependency.
|
||||
- Create/dispose engines inside repository methods.
|
||||
- Keep engine creation as a hidden side effect of import-time module globals.
|
||||
- Use deprecated FastAPI startup/shutdown events together with lifespan.
|
||||
|
||||
---
|
||||
|
||||
## Engine Design Checklist
|
||||
|
||||
- One engine per process per DB URL.
|
||||
- Engine created in lifespan startup.
|
||||
- Engine disposed in lifespan shutdown.
|
||||
- Async driver URL matches backend (`asyncpg` or `aiosqlite`).
|
||||
- Pooling strategy is explicit for non-default needs.
|
||||
- No request-path engine creation.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Preventing Implicit ORM I/O (Asyncio)
|
||||
|
||||
Source:
|
||||
- https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html#preventing-implicit-io-when-using-asyncsession
|
||||
- https://docs.sqlalchemy.org/en/21/orm/queryguide/relationships.html
|
||||
|
||||
Status: adopted
|
||||
Decision level: advisory
|
||||
Applies to: api-runtime, workers, tests
|
||||
Last reviewed: 2026-06-17
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Minimize unexpected database round-trips caused by attribute access in async ORM code.
|
||||
|
||||
In asyncio applications, hidden lazy loads are easy to miss and can produce runtime surprises. This guide defines explicit-loading defaults and progressive enforcement practices.
|
||||
|
||||
---
|
||||
|
||||
## Scope and Non-Goals
|
||||
|
||||
- In scope: relationship loading strategy, post-commit attribute access, explicit refresh/awaitable access patterns.
|
||||
- Out of scope: full ORM performance tuning and domain-specific query architecture.
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- Prefer explicit eager loading for data required by endpoint/service outputs.
|
||||
- Avoid relying on implicit lazy-load behavior in request critical paths.
|
||||
- Keep `expire_on_commit=False` unless strict expiration behavior is intentionally required.
|
||||
- Use explicit refresh or awaitable-attribute access when loading deferred state is necessary.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Patterns
|
||||
|
||||
### Pattern A: Eager-load what you need
|
||||
|
||||
```python
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
stmt = select(User).options(selectinload(User.roles))
|
||||
users = (await session.scalars(stmt)).all()
|
||||
```
|
||||
|
||||
### Pattern B: Explicit refresh of named attributes
|
||||
|
||||
```python
|
||||
user = await session.get(User, user_id)
|
||||
await session.refresh(user, ["roles"])
|
||||
```
|
||||
|
||||
### Pattern C: Awaitable attribute access where needed
|
||||
|
||||
```python
|
||||
# Requires AsyncAttrs mixin on mapped base or class.
|
||||
roles = await user.awaitable_attrs.roles
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Practical Enforcement Model
|
||||
|
||||
Use phased enforcement:
|
||||
|
||||
1. High-traffic and latency-sensitive routes: enforce explicit eager loading.
|
||||
2. Background tasks and less critical paths: track and progressively tighten.
|
||||
3. Add review checks to prevent newly introduced implicit-load hotspots.
|
||||
|
||||
This keeps modernization pragmatic while reducing hidden I/O over time.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- Returning ORM objects from handlers and triggering lazy loads during serialization.
|
||||
- Assuming post-commit attribute access will always be loaded without explicit strategy.
|
||||
- Relying on broad expiration + implicit reload behavior in async request flows.
|
||||
- Enabling relationship patterns that hide SQL behavior in critical code paths.
|
||||
|
||||
---
|
||||
|
||||
## Operational Checks
|
||||
|
||||
- Endpoint query blocks define loader options for returned related data.
|
||||
- Critical handlers do not depend on incidental lazy loads.
|
||||
- Known exceptions are documented with rationale and follow-up items.
|
||||
|
||||
---
|
||||
|
||||
## Testing Checks
|
||||
|
||||
- Integration tests cover endpoints that return related objects.
|
||||
- Tests verify expected data is present without hidden secondary query surprises.
|
||||
- Regression tests exist for routes previously affected by implicit-load failures.
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- Start advisory: target high-risk paths first.
|
||||
- As coverage improves, elevate selected rules to mandatory in code review policy.
|
||||
@@ -0,0 +1,32 @@
|
||||
# FastAPI Async SQLAlchemy References Index
|
||||
|
||||
Purpose: concept registry for modernization guidance used by this skill.
|
||||
|
||||
---
|
||||
|
||||
## Concepts
|
||||
|
||||
| Concept | File | Status | Decision Level | Owner | Last Reviewed |
|
||||
|---|---|---|---|---|---|
|
||||
| Engine lifecycle and ownership | [engine.md](engine.md) | adopted | mandatory | platform/backend | 2026-06-17 |
|
||||
| Session factory and scope | [session.md](session.md) | adopted | mandatory | platform/backend | 2026-06-17 |
|
||||
| Transaction boundaries | [transactions.md](transactions.md) | adopted | mandatory | platform/backend | 2026-06-17 |
|
||||
| Implicit ORM I/O under asyncio | [implicit_io.md](implicit_io.md) | adopted | advisory | platform/backend | 2026-06-17 |
|
||||
| Observability and resilience | [observability.md](observability.md) | adopted | mandatory | platform/backend | 2026-06-17 |
|
||||
|
||||
---
|
||||
|
||||
## How to Use This Folder
|
||||
|
||||
- `SKILL.md` defines the planning workflow and migration procedure.
|
||||
- Each concept doc defines policy-level guidance for one concern.
|
||||
- Use the template in [template.md](template.md) for new concept docs.
|
||||
- Keep references source-linked and implementation snippets minimal.
|
||||
|
||||
---
|
||||
|
||||
## Update Rules
|
||||
|
||||
- If a PR changes database lifecycle/session/ORM loading behavior, update the relevant concept file.
|
||||
- Keep `Status`, `Decision Level`, and `Last Reviewed` current.
|
||||
- Use `advisory` only when incremental rollout is intended; use `mandatory` for required runtime policy.
|
||||
@@ -0,0 +1,113 @@
|
||||
# DB Observability and Resilience
|
||||
|
||||
Source:
|
||||
- https://docs.sqlalchemy.org/en/21/core/pooling.html
|
||||
- https://docs.sqlalchemy.org/en/21/core/engines.html
|
||||
- https://docs.sqlalchemy.org/en/21/core/events.html
|
||||
- https://fastapi.tiangolo.com/advanced/events/
|
||||
|
||||
Status: adopted
|
||||
Decision level: mandatory
|
||||
Applies to: api-runtime, workers, tests
|
||||
Last reviewed: 2026-06-17
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Define baseline observability and resilience practices for DB connectivity in async FastAPI + SQLAlchemy apps.
|
||||
|
||||
Goals:
|
||||
|
||||
- detect and recover from stale/disconnected connections,
|
||||
- expose useful diagnostics for pool/engine behavior,
|
||||
- make readiness/liveness signals meaningful.
|
||||
|
||||
---
|
||||
|
||||
## Scope and Non-Goals
|
||||
|
||||
- In scope: pool health, connection liveness, SQL/pool logging hygiene, readiness checks, failure handling.
|
||||
- Out of scope: full APM stack design and vendor-specific monitoring platform setup.
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- Enable connection liveness strategy (`pool_pre_ping=True`) for long-running services.
|
||||
- Keep DB health checks out of liveness; include dependency checks in readiness.
|
||||
- Centralize engine options and logging configuration.
|
||||
- Avoid noisy SQL debug logging in production defaults.
|
||||
- Treat disconnect handling as a first-class test scenario.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Baseline
|
||||
|
||||
```python
|
||||
engine = create_async_engine(
|
||||
settings.database_url,
|
||||
pool_pre_ping=True,
|
||||
# Tune only from measured behavior:
|
||||
# pool_size=10,
|
||||
# max_overflow=20,
|
||||
# pool_timeout=30,
|
||||
# pool_recycle=1800,
|
||||
)
|
||||
```
|
||||
|
||||
Operational guidance:
|
||||
|
||||
- `pool_pre_ping=True` for stale-connection resilience.
|
||||
- Introduce `pool_recycle` where backend/network idle timeout behavior warrants it.
|
||||
- Use structured app logs with request correlation and error context.
|
||||
|
||||
---
|
||||
|
||||
## Health Endpoint Policy
|
||||
|
||||
- `/healthz`: process is alive; no DB call required.
|
||||
- `/readyz`: application can currently serve traffic; include DB connectivity verification.
|
||||
|
||||
Readiness checks should be lightweight and bounded (timeouts), not heavy diagnostic queries.
|
||||
|
||||
---
|
||||
|
||||
## Failure Handling Guidance
|
||||
|
||||
- Handle transient disconnects with pool invalidation/reconnect semantics.
|
||||
- Keep one failed request from cascading into broad app instability.
|
||||
- Capture and log contextual DB errors with enough metadata for debugging.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- No readiness check for DB-dependent services.
|
||||
- Permanent debug SQL echo in production.
|
||||
- Per-handler ad hoc pool settings.
|
||||
- Assuming disconnect events are too rare to test.
|
||||
|
||||
---
|
||||
|
||||
## Operational Checks
|
||||
|
||||
- Engine creation is centralized and configured once.
|
||||
- Liveness/readiness behavior is documented and validated.
|
||||
- Pool settings are explicit, versioned, and reviewed.
|
||||
- DB-related errors produce actionable logs.
|
||||
|
||||
---
|
||||
|
||||
## Testing Checks
|
||||
|
||||
- Readiness endpoint test covers healthy and unhealthy DB states.
|
||||
- Integration test simulates disconnect/reconnect behavior.
|
||||
- Load/concurrency tests validate pool behavior under stress.
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- Start with resilient defaults (`pool_pre_ping`) and simple health policy.
|
||||
- Add deeper metrics/event hooks incrementally once baseline reliability is in place.
|
||||
@@ -0,0 +1,140 @@
|
||||
# Async SQLAlchemy Session Management
|
||||
|
||||
Source:
|
||||
- https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html
|
||||
- https://docs.sqlalchemy.org/en/21/orm/session_basics.html
|
||||
- https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/
|
||||
|
||||
Status: adopted
|
||||
Decision level: mandatory
|
||||
Applies to: api-runtime, workers, tests
|
||||
Last reviewed: 2026-06-17
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Define one canonical session model for FastAPI + SQLAlchemy asyncio:
|
||||
|
||||
- configure one shared session factory,
|
||||
- create one AsyncSession per request or per unit-of-work,
|
||||
- never share one AsyncSession across concurrent tasks.
|
||||
|
||||
---
|
||||
|
||||
## Scope and Non-Goals
|
||||
|
||||
- In scope: session factory creation, FastAPI dependency wiring, request/task scoping, transaction demarcation.
|
||||
- Out of scope: ORM model design, query optimization strategy, schema migration tooling.
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- Create `async_sessionmaker` once from app-owned AsyncEngine.
|
||||
- Use a fresh AsyncSession for each request or explicit unit-of-work.
|
||||
- Do not share AsyncSession across `asyncio.gather()` or parallel tasks.
|
||||
- Prefer direct dependency injection over global scoped-session patterns in new code.
|
||||
- Use explicit transaction boundaries (`async with session.begin():`) for writes.
|
||||
|
||||
---
|
||||
|
||||
## Canonical FastAPI Dependency Pattern
|
||||
|
||||
```python
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from fastapi import Depends, Request
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
|
||||
def get_session_factory(request: Request) -> async_sessionmaker[AsyncSession]:
|
||||
return request.app.state.session_factory
|
||||
|
||||
|
||||
async def get_db_session(
|
||||
session_factory: async_sessionmaker[AsyncSession] = Depends(get_session_factory),
|
||||
) -> AsyncIterator[AsyncSession]:
|
||||
async with session_factory() as session:
|
||||
yield session
|
||||
```
|
||||
|
||||
Route usage:
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.post("/items")
|
||||
async def create_item(session: AsyncSession = Depends(get_db_session)) -> dict:
|
||||
async with session.begin():
|
||||
# write operations here
|
||||
...
|
||||
return {"status": "ok"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Guidance
|
||||
|
||||
Typical session factory setup:
|
||||
|
||||
```python
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
session_factory = async_sessionmaker(
|
||||
engine,
|
||||
class_=AsyncSession,
|
||||
expire_on_commit=False,
|
||||
)
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `expire_on_commit=False` is commonly preferred in asyncio applications to reduce accidental post-commit reload behavior.
|
||||
- `AsyncSession.refresh()` is preferred over broad expiration patterns when state refresh is needed.
|
||||
|
||||
---
|
||||
|
||||
## Concurrency Rules
|
||||
|
||||
- One session per concurrent task.
|
||||
- If work fans out into parallel tasks, each task receives its own AsyncSession.
|
||||
- Pass sessions explicitly to service functions; avoid mutable global session state.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- A singleton/global AsyncSession reused across requests.
|
||||
- Sharing one AsyncSession across parallel tasks.
|
||||
- Hidden session creation in lower repository helpers with no caller control.
|
||||
- Mixing commit/rollback ownership across layers without a declared boundary.
|
||||
|
||||
---
|
||||
|
||||
## Operational Checks
|
||||
|
||||
- Exactly one `async_sessionmaker` is registered in app lifecycle.
|
||||
- Request handlers receive sessions from one canonical dependency.
|
||||
- No code path creates AsyncSession in module import side effects.
|
||||
- Background jobs and API handlers each create task-local sessions.
|
||||
|
||||
---
|
||||
|
||||
## Testing Checks
|
||||
|
||||
- Dependency override exists for test session factory.
|
||||
- Rollback behavior is verified for failed write units.
|
||||
- Parallel-task tests verify no shared AsyncSession instances.
|
||||
- Lifespan tests confirm session factory is initialized and teardown-safe.
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- If current code uses global/shared sessions, fix scope first before refactoring query style.
|
||||
- If legacy sync patterns are present, keep session boundary rules stable while migrating incrementally.
|
||||
@@ -0,0 +1,63 @@
|
||||
# <Concept Title>
|
||||
|
||||
Source:
|
||||
- <primary source url>
|
||||
- <secondary source url>
|
||||
|
||||
Status: draft|adopted|deprecated
|
||||
Decision level: advisory|mandatory
|
||||
Applies to: api-runtime|workers|tests
|
||||
Last reviewed: YYYY-MM-DD
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Describe what this concept governs and why it exists.
|
||||
|
||||
## Scope and Non-Goals
|
||||
|
||||
- In scope:
|
||||
- Out of scope:
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- Rule 1
|
||||
- Rule 2
|
||||
|
||||
---
|
||||
|
||||
## Recommended Pattern
|
||||
|
||||
```python
|
||||
# minimal example
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- Anti-pattern 1
|
||||
- Anti-pattern 2
|
||||
|
||||
---
|
||||
|
||||
## Operational Checks
|
||||
|
||||
- Check 1
|
||||
- Check 2
|
||||
|
||||
---
|
||||
|
||||
## Testing Checks
|
||||
|
||||
- Test 1
|
||||
- Test 2
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- Staged rollout notes and compatibility caveats.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Async Transaction Boundaries
|
||||
|
||||
Source:
|
||||
- https://docs.sqlalchemy.org/en/21/orm/session_transaction.html
|
||||
- https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html
|
||||
- https://docs.sqlalchemy.org/en/21/core/connections.html
|
||||
|
||||
Status: adopted
|
||||
Decision level: mandatory
|
||||
Applies to: api-runtime, workers, tests
|
||||
Last reviewed: 2026-06-17
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Define consistent transaction demarcation for async SQLAlchemy so write behavior is predictable, rollback semantics are clear, and concurrent request flows remain safe.
|
||||
|
||||
---
|
||||
|
||||
## Scope and Non-Goals
|
||||
|
||||
- In scope: transaction ownership, write/read policy, exception and rollback behavior, nested transaction guidance.
|
||||
- Out of scope: business-domain validation rules and cross-service distributed transactions.
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- Every mutating use case must run inside an explicit transaction boundary.
|
||||
- Prefer `async with session.begin():` for write units.
|
||||
- Keep transaction ownership at service/use-case boundary, not deep in helper internals.
|
||||
- Read paths should not auto-upgrade into hidden write behavior.
|
||||
- On exception in a transaction block, rely on rollback semantics and propagate or map exceptions intentionally.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Patterns
|
||||
|
||||
### Pattern A: Single write unit
|
||||
|
||||
```python
|
||||
async def create_order(session: AsyncSession, payload: OrderIn) -> Order:
|
||||
async with session.begin():
|
||||
order = Order(...)
|
||||
session.add(order)
|
||||
# additional writes...
|
||||
return order
|
||||
```
|
||||
|
||||
### Pattern B: Explicit read flow
|
||||
|
||||
```python
|
||||
async def get_order(session: AsyncSession, order_id: UUID) -> Order | None:
|
||||
stmt = select(Order).where(Order.id == order_id)
|
||||
return await session.scalar(stmt)
|
||||
```
|
||||
|
||||
### Pattern C: Nested transaction (only when required)
|
||||
|
||||
```python
|
||||
async with session.begin():
|
||||
# outer transaction
|
||||
async with session.begin_nested():
|
||||
# savepoint-scoped operation
|
||||
...
|
||||
```
|
||||
|
||||
Use nested transactions only when partial failure semantics are explicitly required.
|
||||
|
||||
---
|
||||
|
||||
## Exception and Rollback Policy
|
||||
|
||||
- Write block fails: transaction context rolls back.
|
||||
- Caller decides whether to translate exception (for example to domain/API errors).
|
||||
- Do not swallow DB exceptions silently; map or re-raise intentionally.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- Multiple commits scattered across one logical use case.
|
||||
- Helper functions that commit/rollback without caller awareness.
|
||||
- Mixing implicit and explicit transaction styles in confusing ways.
|
||||
- Using savepoints as a default pattern rather than a targeted tool.
|
||||
|
||||
---
|
||||
|
||||
## Operational Checks
|
||||
|
||||
- All mutating service functions declare one clear transaction boundary.
|
||||
- No repository/helper performs hidden commit calls.
|
||||
- Transaction style is consistent across handlers and workers.
|
||||
|
||||
---
|
||||
|
||||
## Testing Checks
|
||||
|
||||
- Success path test verifies expected durable writes.
|
||||
- Failure path test verifies rollback behavior.
|
||||
- Tests cover concurrency-sensitive write flows.
|
||||
- Savepoint usage (if present) has dedicated behavior tests.
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- First stabilize session scope, then normalize transaction ownership.
|
||||
- Replace ad hoc commit patterns incrementally with bounded write units.
|
||||
Reference in New Issue
Block a user