John Lancaster
134 lines
4.4 KiB
Markdown
134 lines
4.4 KiB
Markdown
# Async SQLAlchemy Engine
|
|
|
|
Source:
|
|
- https://docs.sqlalchemy.org/en/21/core/connections.html
|
|
- https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html
|
|
- https://docs.sqlalchemy.org/en/21/core/pooling.html#pooling-multiprocessing
|
|
- https://fastapi.tiangolo.com/advanced/events/
|
|
|
|
---
|
|
|
|
## Engine Ownership Model
|
|
|
|
Create one async engine per process per database URL and keep it for the app lifetime.
|
|
|
|
- SQLAlchemy guidance: the engine is intended as a long-lived, concurrent registry over pooled DB connections, not a per-request object.
|
|
- In FastAPI, app startup and shutdown ownership belongs in lifespan.
|
|
- Use `FastAPI(lifespan=...)` (not startup/shutdown events) for modern lifecycle wiring.
|
|
|
|
Practical rule:
|
|
- Exactly one `create_async_engine(...)` call in app bootstrap code.
|
|
- Zero `create_async_engine(...)` calls in request handlers.
|
|
|
|
---
|
|
|
|
## Canonical Lifespan Pattern (AsyncExitStack)
|
|
|
|
Use `@asynccontextmanager` + `AsyncExitStack` to make teardown deterministic and composable.
|
|
|
|
```python
|
|
from contextlib import AsyncExitStack, asynccontextmanager
|
|
|
|
from fastapi import FastAPI
|
|
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
|
|
|
|
|
@asynccontextmanager
|
|
async def lifespan(app: FastAPI):
|
|
async with AsyncExitStack() as stack:
|
|
engine: AsyncEngine = create_async_engine(
|
|
app.state.settings.database_url,
|
|
pool_pre_ping=True,
|
|
# Optional examples:
|
|
# echo=app.state.settings.sql_echo,
|
|
# pool_size=10,
|
|
# max_overflow=20,
|
|
)
|
|
app.state.engine = engine
|
|
|
|
# Ensure engine disposal always runs at shutdown.
|
|
stack.push_async_callback(engine.dispose)
|
|
|
|
yield
|
|
|
|
|
|
app = FastAPI(lifespan=lifespan)
|
|
```
|
|
|
|
Why this pattern:
|
|
- FastAPI executes code before `yield` at startup and after `yield` at shutdown.
|
|
- `AsyncExitStack` lets you register multiple async cleanups in one place while preserving order.
|
|
- Explicit disposal (directly awaited or via `AsyncExitStack` callback) avoids event-loop-closed warnings when objects fall out of scope.
|
|
|
|
---
|
|
|
|
## Driver URLs (Project Requirement: asyncpg + aiosqlite)
|
|
|
|
Use SQLAlchemy async driver URLs:
|
|
|
|
- PostgreSQL: `postgresql+asyncpg://user:pass@host:5432/dbname`
|
|
- SQLite: `sqlite+aiosqlite:///./app.db`
|
|
|
|
Notes:
|
|
- Do not mix sync drivers (for example `psycopg2`) with `create_async_engine()`.
|
|
- Keep URL construction centralized in settings/config, not in feature modules.
|
|
|
|
---
|
|
|
|
## Pooling Defaults and Tuning
|
|
|
|
Default behavior is usually correct first:
|
|
|
|
- Async engines use async-compatible pooling (`AsyncAdaptedQueuePool`) by default.
|
|
- Start with defaults, then tune from observed load (`pool_size`, `max_overflow`, `pool_timeout`, `pool_recycle`).
|
|
- Enable `pool_pre_ping=True` for safer stale-connection handling in long-running services.
|
|
|
|
When to switch pool strategy:
|
|
|
|
- `NullPool` if you explicitly need no pooling (special environments, some tests, or strict cross-loop constraints).
|
|
- Keep in mind this increases connect/disconnect churn.
|
|
|
|
---
|
|
|
|
## Disposal Semantics
|
|
|
|
`engine.dispose()` replaces/disposes the pool, but only checked-in connections are immediately closed.
|
|
|
|
Rules:
|
|
- Dispose when the app is shutting down.
|
|
- Dispose before reusing an engine across event loops.
|
|
- In forked child-process initialization, use `engine.dispose(close=False)` (sync API guidance) so child processes do not touch parent-held connections.
|
|
|
|
Avoid relying on garbage collection for engine cleanup in async code.
|
|
|
|
---
|
|
|
|
## Event Loop and Process Boundaries
|
|
|
|
Do not share pooled connections across boundaries:
|
|
|
|
- Multiple event loops: do not reuse the same pooled async engine across loops unless you intentionally disable pooling (`NullPool`) or dispose before handoff.
|
|
- Multiprocessing/fork: pooled connections must not be inherited for active use across process boundaries.
|
|
|
|
This prevents broken socket state and cross-process connection corruption.
|
|
|
|
---
|
|
|
|
## What Not to Do
|
|
|
|
- Create an engine inside every request dependency.
|
|
- Create/dispose engines inside repository methods.
|
|
- Keep engine creation as a hidden side effect of import-time module globals.
|
|
- Use deprecated FastAPI startup/shutdown events together with lifespan.
|
|
|
|
---
|
|
|
|
## Engine Design Checklist
|
|
|
|
- One engine per process per DB URL.
|
|
- Engine created in lifespan startup.
|
|
- Engine disposed in lifespan shutdown.
|
|
- Async driver URL matches backend (`asyncpg` or `aiosqlite`).
|
|
- Pooling strategy is explicit for non-default needs.
|
|
- No request-path engine creation.
|