prompts/docs/skills/async-fastapi-sqlmodel/references/engine.md

Async SQLAlchemy Engine

!!! info "Primary sources" - Python asynccontextmanager - Python functools.cache - SQLAlchemy connections - SQLAlchemy asyncio extension - SQLAlchemy pooling and multiprocessing - SQLAlchemy SQLite transaction control - SQLAlchemy SQLite foreign-key support - SQLite PRAGMA reference - nicegui-db engine implementation

---

Engine Ownership Model

Resolve one async engine for each database URL within an application, worker, command, or test lifecycle.

• SQLAlchemy guidance: the engine is intended as a long-lived, concurrent registry over pooled DB connections, not a per-operation object.
• get_engine(database_url) owns URL-keyed engine construction and caching.
• The composition root enters engine_scope(database_url) once and therefore owns initialization and disposal.
• Services and repositories receive a session or session factory; they do not resolve an engine.

!!! tip "Practical rule" - Exactly one cached engine for each database URL during an active application-owned lifecycle. - Exactly one active owning engine_scope() for a given URL. - Zero create_async_engine(...) calls in feature code. - Zero engine lookup or disposal calls in repository code.

---

Cached Engine Resolution

functools.cache makes the database URL the engine identity:

from functools import cache

from sqlalchemy.ext.asyncio import AsyncEngine
from sqlalchemy.ext.asyncio import create_async_engine


@cache
def get_engine(database_url: str) -> AsyncEngine:
    engine = create_async_engine(database_url, pool_pre_ping=True)
    if engine.dialect.name == "sqlite":
        configure_aiosqlite_engine(engine)
    return engine

Repeated calls with the same exact URL return the same AsyncEngine; different URLs produce independent cache entries. Construction configures the dialect and pool but normally does not open a database connection until the first operation. SQLite event listeners are installed only when a new cached engine is constructed, before its first connection.

Resolve settings into the final URL before calling get_engine(). Services and repositories should not call it directly: the cache controls construction identity, not ownership.

Owning Engine Scope

Use one asynccontextmanager to pair cached resolution and optional schema initialization with disposal:

from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager

from sqlalchemy.ext.asyncio import AsyncEngine
from sqlmodel import SQLModel


@asynccontextmanager
async def engine_scope(
    database_url: str,
    *,
    initialize: bool = True,
) -> AsyncGenerator[AsyncEngine]:
    engine = get_engine(database_url)
    if initialize:
        await initialize_db(database_url)

    try:
        yield engine
    finally:
        await dispose_engine(database_url)


async def initialize_db(database_url: str) -> None:
    from . import models  # noqa: F401

    engine = get_engine(database_url)
    async with engine.begin() as connection:
        await connection.run_sync(SQLModel.metadata.create_all)


async def dispose_engine(database_url: str) -> None:
    engine = get_engine(database_url)
    try:
        await engine.dispose()
    finally:
        get_engine.cache_clear()

The code that enters engine_scope() owns the engine. It keeps that scope open for the complete application, worker, command, or test lifecycle and passes the yielded engine into session-factory construction. Successful and exceptional exits both dispose the pool and invalidate cached engine resolution.

Initialization imports the model package so every table is registered, then runs SQLModel.metadata.create_all() in engine.begin(). This is suitable for the template and focused tests. Use migrations instead when schema evolution is part of the deployment contract. Pass initialize=False only when another owner provisions the schema or a test is directly exercising construction without schema setup.

dispose_engine() clears the complete function cache, not only the requested URL. This matches the template and is safe under its intended single-database lifecycle. Applications that own several simultaneously active database URLs need per-key lifecycle management rather than this global invalidation behavior.

Workers, scripts, and other composition roots enter database_scope() directly:

async with database_scope(settings.database_url) as session_factory:
    await run_worker(session_factory)

database_scope() is defined in session management. It enters engine_scope() and creates the factory bound to the yielded engine.

Do not overlap two owning scopes for the same URL. Both resolve the same cached engine, and the first scope to exit disposes it and clears the cache while the other still refers to it. For several fixed databases, use one non-overlapping owner per URL and account for global cache invalidation; use AsyncExitStack only after adopting lifecycle semantics that support several simultaneous owners.

When directly testing engine construction or lifecycle behavior, enter engine_scope() in the test or fixture. Exiting the context disposes the engine even when the test fails and clears the cache for the next lifecycle.

See FastAPI database integration for adapting database_scope() to application lifespan and dependency injection.

---

Driver URLs (Project Requirement: asyncpg + aiosqlite)

Use SQLAlchemy async driver URLs:

• PostgreSQL: postgresql+asyncpg://user:pass@host:5432/dbname
• SQLite: sqlite+aiosqlite:///./app.db

!!! warning "Driver compatibility" - Do not mix sync drivers, for example psycopg2, with create_async_engine(). - Keep URL construction centralized in settings/config, not in feature modules.

---

SQLite Connection and Transaction Policy

SQLite settings do not form one indivisible bundle:

• PRAGMA foreign_keys=ON is a correctness requirement when the schema declares foreign keys. SQLite requires it on every connection, including the connection used by metadata.create_all().
• Disabling the driver's implicit BEGIN and emitting BEGIN from SQLAlchemy provides non-legacy transaction behavior for aiosqlite. This makes SELECT, DDL, and SAVEPOINT behavior participate in SQLAlchemy's transaction boundary consistently.
• PRAGMA busy_timeout is a per-connection lock-wait policy. Choose the duration from the application's latency and contention requirements.
• PRAGMA journal_mode=WAL is an optional file-database concurrency policy. WAL persists in the database file, cannot be enabled for an in-memory database, and is not a substitute for transaction control.

Install instance-level listeners exactly once, immediately after constructing an aiosqlite engine and before its first connection:

from sqlalchemy import event
from sqlalchemy.engine import Connection
from sqlalchemy.engine.interfaces import DBAPIConnection
from sqlalchemy.ext.asyncio import AsyncEngine


def configure_aiosqlite_engine(
    engine: AsyncEngine,
    *,
    busy_timeout_ms: int | None = 30_000,
    enable_wal: bool = False,
) -> None:
    if engine.dialect.name != "sqlite" or engine.dialect.driver != "aiosqlite":
        raise ValueError("Expected a sqlite+aiosqlite engine")
    if busy_timeout_ms is not None and busy_timeout_ms < 0:
        raise ValueError("busy_timeout_ms must be non-negative")

    @event.listens_for(engine.sync_engine, "connect")
    def configure_connection(dbapi_connection: DBAPIConnection, _: object) -> None:
        dbapi_connection.isolation_level = None
        cursor = dbapi_connection.cursor()
        try:
            cursor.execute("PRAGMA foreign_keys=ON")
            if busy_timeout_ms is not None:
                cursor.execute(f"PRAGMA busy_timeout={busy_timeout_ms}")
            if enable_wal:
                cursor.execute("PRAGMA journal_mode=WAL")
                journal_mode = cursor.fetchone()
                if journal_mode is None or journal_mode[0].lower() != "wal":
                    raise RuntimeError("SQLite could not enable WAL mode")
        finally:
            cursor.close()

    @event.listens_for(engine.sync_engine, "begin")
    def begin_transaction(connection: Connection) -> None:
        connection.exec_driver_sql("BEGIN")

The connect listener receives the adapted synchronous DBAPI connection exposed by engine.sync_engine; event callbacks themselves are synchronous even though application queries use the async engine. Setting isolation_level=None and adding the begin listener are one transaction-control strategy and must remain paired. Do not combine this pair with SQLAlchemy's driver-level AUTOCOMMIT isolation mode.

The default above enables foreign keys and modern transaction boundaries for file and in-memory databases. Enable WAL only for a file-backed database after deciding that its read/write concurrency model is appropriate. Treat 30_000 as an example policy, not a universal default; connect_args={"timeout": 30.0} at engine construction is another way to configure the underlying SQLite lock timeout.

---

Pooling Defaults and Tuning

Default behavior is usually correct first:

• Async engines use async-compatible pooling (AsyncAdaptedQueuePool) by default.
• Start with defaults, then tune from observed load (pool_size, max_overflow, pool_timeout, pool_recycle).
• Enable pool_pre_ping=True for safer stale-connection handling in long-running services.

When to switch pool strategy:

• NullPool if you explicitly need no pooling (special environments, some tests, or strict cross-loop constraints).
• Keep in mind this increases connect/disconnect churn.

When StaticPool Is Appropriate

Use StaticPool only when every checkout must reuse one DBAPI connection and all database access is serialized. Typical cases are:

• A serial test suite using a private in-memory SQLite database. The sqlite+aiosqlite:// URL already selects StaticPool automatically, so specifying poolclass=StaticPool is normally redundant.
• A narrowly scoped SQLite engine that must preserve connection-local state, such as temporary tables, across SQLAlchemy connection or session checkouts.

When explicit configuration is required:

from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.pool import StaticPool

engine = create_async_engine(
    "sqlite+aiosqlite:///./test.db",
    poolclass=StaticPool,
)

StaticPool is not a general performance optimization or a way to make SQLite concurrent. All sessions share one underlying connection and its single transaction state, so one session's COMMIT or ROLLBACK can interfere with another session. Do not use it when several sessions or tasks may access the engine concurrently. For concurrent in-memory work, use a named shared-cache SQLite URL so pooled connections have independent transaction state, or use a temporary file database. See SQLite test targets for those patterns.

---

Disposal Semantics

dispose_engine(database_url) resolves the cached engine, awaits engine.dispose(), and clears the engine cache in a finally block. engine.dispose() replaces/disposes the pool, but only checked-in connections are immediately closed.

Rules:

• Dispose when the app is shutting down.
• Clear cached resolution even when disposal raises, so a later lifecycle cannot receive the failed engine object.
• Dispose before reusing an engine across event loops.
• In forked child-process initialization, use engine.dispose(close=False) (sync API guidance) so child processes do not touch parent-held connections.

Avoid relying on garbage collection for engine cleanup in async code.

---

Event Loop and Process Boundaries

Do not share pooled connections across boundaries:

• Multiple event loops: do not reuse the same pooled async engine across loops unless you intentionally disable pooling (NullPool) or dispose before handoff.
• Multiprocessing/fork: pooled connections must not be inherited for active use across process boundaries.

This prevents broken socket state and cross-process connection corruption.

---

What Not to Do

• Create an engine inside each operation or unit of work.
• Create/dispose engines inside repository methods.
• Resolve an engine from repositories instead of injecting a session dependency.
• Keep engine creation as a hidden side effect of import-time module globals.
• Keep a session factory alive after its bound engine scope exits.
• Enter overlapping engine scopes for the same cached URL.
• Treat cache_clear() as per-URL invalidation when it clears every cached engine.
• Use metadata.create_all() as a substitute for required production migrations.
• Install the same SQLite event listeners more than once on one engine.
• Enable WAL blindly for in-memory SQLite or treat a busy timeout as a concurrency guarantee.

---

Engine Design Checklist

• One cached engine per exact database URL during an active lifecycle.
• One owning engine scope per URL, with no overlapping owners.
• Cached resolution, optional initialization, disposal, and cache invalidation follow one framework-independent lifecycle.
• The composition root enters the database scope once and keeps it open until shutdown.
• Session factory created inside, and never outlives, its engine scope.
• Model registration occurs before metadata.create_all() when initialization is enabled.
• Async driver URL matches backend (asyncpg or aiosqlite).
• aiosqlite foreign-key and transaction listeners installed once before first use.
• WAL enabled only as an explicit policy for a file-backed SQLite database.
• Pooling strategy is explicit for non-default needs.
• No feature-path engine creation.
• Tests enter the same scope and receive deterministic disposal plus cache cleanup.