prompts/docs/skills/async-fastapi-sqlmodel/references/engine.md

# Async SQLAlchemy Engine

!!! info "Primary sources"
    - [Python `asynccontextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.asynccontextmanager)
    - [SQLAlchemy connections](https://docs.sqlalchemy.org/en/21/core/connections.html)
    - [SQLAlchemy asyncio extension](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html)
    - [SQLAlchemy pooling and multiprocessing](https://docs.sqlalchemy.org/en/21/core/pooling.html#pooling-multiprocessing)
    - [SQLAlchemy SQLite transaction control](https://docs.sqlalchemy.org/en/21/dialects/sqlite.html#enabling-non-legacy-sqlite-transactional-modes-with-the-sqlite3-or-aiosqlite-driver)
    - [SQLAlchemy SQLite foreign-key support](https://docs.sqlalchemy.org/en/21/dialects/sqlite.html#foreign-key-support)
    - [SQLite PRAGMA reference](https://www.sqlite.org/pragma.html)

---

## Engine Ownership Model

Create one async engine for each application, worker, command, or test lifecycle.

- SQLAlchemy guidance: the engine is intended as a long-lived, concurrent registry over pooled DB connections, not a per-operation object.
- The composition root owns engine creation and disposal.
- Services and repositories receive a session or session factory; they do not resolve an engine.

!!! tip "Practical rule"
    - Exactly one `create_async_engine(...)` call for each application-owned engine lifecycle.
    - Zero `create_async_engine(...)` calls in feature code.
    - Zero engine lookup or disposal calls in repository code.

---

## One Engine Context Manager

Use one [`asynccontextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.asynccontextmanager) to pair engine creation with disposal:

```python
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager

from sqlalchemy.ext.asyncio import AsyncEngine
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.ext.asyncio import create_async_engine
from sqlmodel.ext.asyncio.session import AsyncSession

type SessionFactory = async_sessionmaker[AsyncSession]


@asynccontextmanager
async def database_scope(database_url: str) -> AsyncGenerator[SessionFactory]:
    async with engine_scope(database_url) as engine:
        yield async_sessionmaker(
            bind=engine,
            class_=AsyncSession,
            expire_on_commit=False,
        )


@asynccontextmanager
async def engine_scope(database_url: str) -> AsyncGenerator[AsyncEngine]:
    engine = create_async_engine(database_url, pool_pre_ping=True)
    if engine.dialect.name == "sqlite":
        configure_aiosqlite_engine(engine)

    try:
        yield engine
    finally:
        await engine.dispose()
```

The code that enters `engine_scope()` owns the engine. It keeps that scope open for the complete application, worker, command, or test lifecycle and passes the yielded engine into session-factory construction. Successful and exceptional exits both dispose the pool.

Creating an `AsyncEngine` configures its dialect and pool; the first database operation normally establishes a connection. No cache is required when the application composition root enters this context exactly once. Removing the cache also removes cache-key, refresh, and invalidation behavior that otherwise must remain synchronized with the session factory.

Resolve settings before entering the scope. Do not hide settings lookup or engine creation inside feature code.

Workers, scripts, and other composition roots enter `database_scope()` directly:

```python
async with database_scope(settings.database_url) as session_factory:
    await run_worker(session_factory)
```

For several fixed databases, nest one scope per engine. Use [`AsyncExitStack`](https://docs.python.org/3/library/contextlib.html#contextlib.AsyncExitStack) only when the number of engines is dynamic or conditional.

When directly testing engine construction or lifecycle behavior, enter `engine_scope()` in the test or fixture. Exiting the context disposes the engine even when the test fails; no global cache reset is needed.

See [FastAPI database integration](fastapi.md) for adapting `database_scope()` to application lifespan and dependency injection.

---

## Driver URLs (Project Requirement: asyncpg + aiosqlite)

Use SQLAlchemy async driver URLs:

- PostgreSQL: `postgresql+asyncpg://user:pass@host:5432/dbname`
- SQLite: `sqlite+aiosqlite:///./app.db`

!!! warning "Driver compatibility"
    - Do not mix sync drivers, for example `psycopg2`, with `create_async_engine()`.
    - Keep URL construction centralized in settings/config, not in feature modules.

---

## SQLite Connection and Transaction Policy

SQLite settings do not form one indivisible bundle:

- `PRAGMA foreign_keys=ON` is a correctness requirement when the schema declares foreign keys. SQLite requires it on every connection, including the connection used by `metadata.create_all()`.
- Disabling the driver's implicit `BEGIN` and emitting `BEGIN` from SQLAlchemy provides non-legacy transaction behavior for `aiosqlite`. This makes SELECT, DDL, and SAVEPOINT behavior participate in SQLAlchemy's transaction boundary consistently.
- `PRAGMA busy_timeout` is a per-connection lock-wait policy. Choose the duration from the application's latency and contention requirements.
- `PRAGMA journal_mode=WAL` is an optional file-database concurrency policy. WAL persists in the database file, cannot be enabled for an in-memory database, and is not a substitute for transaction control.

Install instance-level listeners exactly once, immediately after constructing an `aiosqlite` engine and before its first connection:

```python
from sqlalchemy import event
from sqlalchemy.engine import Connection
from sqlalchemy.engine.interfaces import DBAPIConnection
from sqlalchemy.ext.asyncio import AsyncEngine


def configure_aiosqlite_engine(
    engine: AsyncEngine,
    *,
    busy_timeout_ms: int | None = 30_000,
    enable_wal: bool = False,
) -> None:
    if engine.dialect.name != "sqlite" or engine.dialect.driver != "aiosqlite":
        raise ValueError("Expected a sqlite+aiosqlite engine")
    if busy_timeout_ms is not None and busy_timeout_ms < 0:
        raise ValueError("busy_timeout_ms must be non-negative")

    @event.listens_for(engine.sync_engine, "connect")
    def configure_connection(dbapi_connection: DBAPIConnection, _: object) -> None:
        dbapi_connection.isolation_level = None
        cursor = dbapi_connection.cursor()
        try:
            cursor.execute("PRAGMA foreign_keys=ON")
            if busy_timeout_ms is not None:
                cursor.execute(f"PRAGMA busy_timeout={busy_timeout_ms}")
            if enable_wal:
                cursor.execute("PRAGMA journal_mode=WAL")
                journal_mode = cursor.fetchone()
                if journal_mode is None or journal_mode[0].lower() != "wal":
                    raise RuntimeError("SQLite could not enable WAL mode")
        finally:
            cursor.close()

    @event.listens_for(engine.sync_engine, "begin")
    def begin_transaction(connection: Connection) -> None:
        connection.exec_driver_sql("BEGIN")
```

The `connect` listener receives the adapted synchronous DBAPI connection exposed by `engine.sync_engine`; event callbacks themselves are synchronous even though application queries use the async engine. Setting `isolation_level=None` and adding the `begin` listener are one transaction-control strategy and must remain paired. Do not combine this pair with SQLAlchemy's driver-level `AUTOCOMMIT` isolation mode.

The default above enables foreign keys and modern transaction boundaries for file and in-memory databases. Enable WAL only for a file-backed database after deciding that its read/write concurrency model is appropriate. Treat `30_000` as an example policy, not a universal default; `connect_args={"timeout": 30.0}` at engine construction is another way to configure the underlying SQLite lock timeout.

---

## Pooling Defaults and Tuning

Default behavior is usually correct first:

- Async engines use async-compatible pooling (`AsyncAdaptedQueuePool`) by default.
- Start with defaults, then tune from observed load (`pool_size`, `max_overflow`, `pool_timeout`, `pool_recycle`).
- Enable `pool_pre_ping=True` for safer stale-connection handling in long-running services.

When to switch pool strategy:

- `NullPool` if you explicitly need no pooling (special environments, some tests, or strict cross-loop constraints).
- Keep in mind this increases connect/disconnect churn.

### When `StaticPool` Is Appropriate

Use [`StaticPool`](https://docs.sqlalchemy.org/en/21/core/pooling.html#sqlalchemy.pool.StaticPool) only when every checkout must reuse one DBAPI connection and all database access is serialized. Typical cases are:

- A serial test suite using a private in-memory SQLite database. The `sqlite+aiosqlite://` URL already selects `StaticPool` automatically, so specifying `poolclass=StaticPool` is normally redundant.
- A narrowly scoped SQLite engine that must preserve connection-local state, such as temporary tables, across SQLAlchemy connection or session checkouts.

When explicit configuration is required:

```python
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.pool import StaticPool

engine = create_async_engine(
    "sqlite+aiosqlite:///./test.db",
    poolclass=StaticPool,
)
```

`StaticPool` is not a general performance optimization or a way to make SQLite concurrent. All sessions share one underlying connection and its single transaction state, so one session's `COMMIT` or `ROLLBACK` can interfere with another session. Do not use it when several sessions or tasks may access the engine concurrently. For concurrent in-memory work, use a named shared-cache SQLite URL so pooled connections have independent transaction state, or use a temporary file database. See [SQLite test targets](testing.md#sqlite-targets) for those patterns.

---

## Disposal Semantics

`engine.dispose()` replaces/disposes the pool, but only checked-in connections are immediately closed.

Rules:
- Dispose when the app is shutting down.
- Dispose before reusing an engine across event loops.
- In forked child-process initialization, use `engine.dispose(close=False)` (sync API guidance) so child processes do not touch parent-held connections.

Avoid relying on garbage collection for engine cleanup in async code.

---

## Event Loop and Process Boundaries

Do not share pooled connections across boundaries:

- Multiple event loops: do not reuse the same pooled async engine across loops unless you intentionally disable pooling (`NullPool`) or dispose before handoff.
- Multiprocessing/fork: pooled connections must not be inherited for active use across process boundaries.

This prevents broken socket state and cross-process connection corruption.

---

## What Not to Do

- Create an engine inside each operation or unit of work.
- Create/dispose engines inside repository methods.
- Resolve an engine from repositories instead of injecting a session dependency.
- Keep engine creation as a hidden side effect of import-time module globals.
- Keep a session factory alive after its bound engine scope exits.
- Add process-global engine caching when one composition root already owns the lifecycle.
- Install the same SQLite event listeners more than once on one engine.
- Enable WAL blindly for in-memory SQLite or treat a busy timeout as a concurrency guarantee.

---

## Engine Design Checklist

- One engine scope per application-owned database lifecycle.
- Engine creation and disposal paired by one framework-independent context manager.
- The composition root enters the database scope once and keeps it open until shutdown.
- Session factory created inside, and never outlives, its engine scope.
- Async driver URL matches backend (`asyncpg` or `aiosqlite`).
- `aiosqlite` foreign-key and transaction listeners installed once before first use.
- WAL enabled only as an explicit policy for a file-backed SQLite database.
- Pooling strategy is explicit for non-default needs.
- No feature-path engine creation.
- Tests enter the same scope and receive deterministic disposal without global cache cleanup.