bbchops/transcription

Fork 0

generated from john/python-template

Files

T

zoltan57 b3d8eb6e97 V4.6 Scope & Implementation Plan

2026-08-17 15:25:52 -05:00

17 KiB

Raw Blame History

V4.6 Scope Boundary

This document defines the frozen boundary for V4.6, a pure remediation release. V4 through V4.5 remain the architecture and behavioral baseline. V4.6 introduces no new user-facing features; it pays down the defects, duplication, and structural drift catalogued in Architecture & Code Review Report.

Every item in scope is traceable to a review finding ID. Any change that cannot be traced to a finding ID is out of scope.

Purpose

Remove dead code, dead configuration, and duplicate implementations that create maintenance drift.
Re-level the database schema from current SQLModel metadata, ending hand-rolled DDL while the schema is still pre-production.
Correct read amplification, missing indexes, and query patterns that scale with table size rather than result size.
Restore the boundaries the project already wrote down in .github/instructions/services.instructions.md and ui.instructions.md.
Make ty usable as a real quality gate.
Preserve every existing behavior, evidence guarantee, and provenance contract established in V4 through V4.5.

Confirmed Operating Context

These answers are frozen for V4.6 and govern every decision below.

Question	Answer
Database	SQLite only. PostgreSQL remains the intended destination but is deferred beyond V4.6. `JSONBCompat` and the Postgres drivers are retained.
Topology	Single user, single process, single worker. A multi-user server is the stated direction, so forward-compatibility work is retained where it is cheap.
Schema evolution	Re-level from current metadata. No Alembic, no migration framework, no `_upgrade_*` chain.
Existing data	The development database is rebuilt from scratch during implementation and migrated from backup as the final step.
Release character	Pure remediation. No new features.
Scope band	Critical through Low, inclusive.

In Scope

1. Dead Code and Dead Configuration Removal

src/transcription/app_state.py is deleted. It has zero importers and contains a guaranteed TypeError ([HIGH-01]).
src/transcription/services/transcription.py is deleted; build_prompt_execution has exactly one import path ([MED-05]).
The legacy compatibility aliases in services/store.py are deleted ([MED-05]).
ServiceBase.queue is deleted; no service allocates an unused asyncio.Queue ([MED-07]).
sqlite_check_same_thread and worker_retry_backoff_seconds are either wired to real behavior or deleted, along with their tests ([MED-02]).
db/operations.py:get_next_queued_job is deleted as a divergent duplicate ([CRIT-01]).
DATABASE_URL is removed from docker-compose.yml, and the real nested DATABASE__* names are documented. The application never silently ignores a database configuration variable ([MED-10]).
document_panzoom is either fixed or deleted; it is exported but referenced by no page ([HIGH-07]).

2. Schema Re-Level

The following changes are schema-affecting and land as one single pass against a database rebuilt from empty.

upgrade_schema and the three _upgrade_* functions (db/operations.py:25-109) are deleted, along with their tests (tests/test_db.py:109-172) ([HIGH-05]).
The schema is generated exclusively from SQLModel metadata via create_all(), gated by the existing Settings.should_bootstrap_schema ([HIGH-05]).
The hand-written CHAR(32) column for preferred_execution_attempt_id ceases to exist; the column type is whatever the model declares ([HIGH-05]).
A composite index on Job.status, Job.date_created is declared in the model, plus index=True on the foreign keys the worker and detail pages filter on ([HIGH-04]).
Source.preferred_execution_attempt_id declares its foreign key with use_alter=True, resolving the source / job_source / execution_attempt cycle so create_all will succeed on PostgreSQL when that cutover is taken ([HIGH-08]).
Relationship loading defaults change from bidirectional lazy="selectin" to lazy="raise", with per-query selectinload() retained or added where a load path genuinely requires it ([CRIT-02]).

No migration script runs against a populated database. No history table, revision directory, or down path is introduced.

3. Data Migration

A one-time script under tools/ migrates the user's backed-up V4.5 data into the re-leveled schema.
The script is authored after the lazy="raise" flip is complete, so that every relationship it traverses carries an explicit eager load.
The script is idempotent, is never invoked automatically at startup, and never runs as part of the test suite.
Uploaded Source files, portraits, and artifact files on disk are preserved unchanged; only database rows are rewritten.
This is the final step of V4.6.

4. Worker and Provider Reliability

read_next_queued_job gains LIMIT 1 and stops materializing the entire queue plus its eager graph on every poll ([CRIT-01]).
The claim becomes an atomic QUEUED → PROCESSING transition. On SQLite this is a bounded single-writer transaction; the FOR UPDATE SKIP LOCKED path is written and dialect-guarded for the multi-user direction but is not exercised in V4.6 ([CRIT-01]).
Eager relationships are loaded in a second query after the claim succeeds, keeping the hot poll a single narrow row ([CRIT-01]).
ServiceBundle and the provider client are hoisted to worker-loop scope so the HTTP connection pool and TLS session survive across jobs ([HIGH-02], [MED-06]).
ServiceBundle gains a from_session_factory constructor, replacing three duplicated instantiation blocks ([MED-06]).
The le=20.0 cap on worker_provider_timeout_seconds is removed, the default is raised, and an explicit httpx.Timeout is passed to the OpenRouter client ([HIGH-03]).
The TranscriptionProvider Protocol is extended to cover aclose and the evidence attributes; the per-call inspect.signature reflection at sources.py:1237 is deleted ([MED-03]).

5. Service Layer Consolidation

A generic RegistryService[ModelT] owns list, summaries, create, read, update, delete, and reference-check for semantic-key registries. DocumentType and PersonRole become thin subclasses declaring their model, error class, reference query, and noun ([MED-11]).
Label normalization, the casefold key, and the registry summary shape are defined once ([MED-11]).
ServiceBase gains _get_or_raise, and all 38 hand-written not-found guards adopt it, including the three in documents.py that already bypass the local helper ([MED-12]).
services/media_storage.py becomes the single implementation of validate → hash → write → wrap-error, replacing store_source_file, store_person_portrait, and the homepage image writer ([MED-13]).
source_mime_type moves out of services/sources.py to a shared module so documents.py no longer imports a sibling service ([MED-14], partial).
The four query inefficiencies in sources.py are corrected: the job_id filter moves into SQL, navigation uses two bounded queries, list_processing_artifacts gains a limit, and artifact re-hashing moves off the event loop ([LOW-08]).

6. Async I/O and Configuration Hygiene

Blocking filesystem and CPU work — media writes, artifact writes, integrity hashing, and Pillow orientation normalization — is wrapped in asyncio.to_thread ([MED-01]).
functools.cache on the engine and session factories is replaced with an explicit URL-keyed registry supporting targeted eviction ([MED-04]).
The object.__setattr__ mutation of a frozen Settings model in normalize_provider_models is replaced with model_copy(update=...) or a computed property ([Pydantic V2 §3]).
models.py timestamp columns that are expected to track modification gain onupdate, so updated_at and date_updated stop being stale on paths that do not set them by hand ([SQLModel §3]).
The exception swallowed to None in an ORM model property is surfaced ([MED-08]).

7. UI Boundary and Duplication

The three ui.instructions.md violations are corrected ([HIGH-07]):
- jobs_page.py no longer imports session_scope or manages transactions; a service or workflow method owns the session.
- sources_page.py no longer imports sqlalchemy.inspect; the service returns a plain transport_body_deferred flag on a read model.
- document_panzoom no longer calls get_settings(); a ready media URL is passed in.
The duplication catalogued in the review's §4 is extracted, highest value first: confirm_delete, media_urls, guards, formatters, upload_panel, and the hand-rolled tables that should use build_table (~500 lines).
The 23KB inline SVG moves to ui/static/ and is loaded through an importlib.resources reader alongside the existing read_css ([MED-09]).
people_page.py:504 routes its error through error_presenter.show_error like every sibling handler ([LOW-07]).
Untyped handler parameters and loosely-typed dict returns are annotated ([LOW-05]).
The auto-refresh timer is cancelled rather than only deactivated, and its interval becomes a named constant ([LOW-06]).

8. Type Checking and Tooling Gate

The codebase standardizes on ty. Remaining suppressions are converted from # pyright: ignore[...] to # ty: ignore[...] ([HIGH-06]).
The lazy="raise" flip in §2 is expected to eliminate most of the ~160 selectinload diagnostics by removing redundant eager loads.
ty check reaches zero diagnostics and is wired into the existing pre-commit setup as a gate ([HIGH-06]).
The two real bugs currently hidden in the diagnostic noise are fixed: tests/ui/test_sources_page.py:25 constructs Source(...) without the required document_id, and tools/run_destructive_tests.py:76,80 uses fcntl, which does not exist on the Windows development platform ([HIGH-06]).
ruff check reaches zero errors ([LOW-01]).
asyncio_default_fixture_loop_scope is configured explicitly so pytest-asyncio behavior does not change on upgrade ([Testing §3]).
The stale path in .github/instructions/services.instructions.md:10 is corrected to src/transcription/db/models.py ([LOW-02]).
list_jobs stops accepting and discarding load_docs ([LOW-03]).
resolve_worker_notifier validates its getattr result ([LOW-04]).

Out of Scope

Any new user-facing feature, page, action, or field.
PostgreSQL enablement, Postgres-backed CI, or a Postgres cutover. The use_alter fix unblocks it; it does not perform it.
Alembic or any migration framework, revision directory, history table, or down path.
Multi-worker or multi-process execution. Forward-compatible code paths are written but not enabled or exercised.
Concurrency limits, backpressure, or parallel job processing. Jobs remain strictly serial.
Splitting SourceService into per-model services and relocating update_job_source_transcription to workflows.py ([MED-14]). Deferred to V4.7. It touches the transcription write path and cannot safely share a release with the schema re-level.
Any change to transcription prompt content, medium markers, quality-warning rules, or the retranscription workflow established in V4.5.
Any change to the evidence, provenance, or immutability contracts established in V4.2 through V4.5.
Deleting, rewriting, or reinterpreting existing ExecutionAttempt or ProcessingArtifact evidence during data migration.
Rewriting the UI table architecture, theme system, or CSS conventions beyond removing duplication.
Performance work not traceable to a review finding.

Locked Design Decisions

A. Remediation Only

Every change traces to a review finding ID. A desirable improvement discovered during implementation that has no finding ID is recorded for a later revision rather than absorbed.

B. Re-Level, Do Not Migrate

The schema is pre-production and the data is disposable and backed up. Deleting the hand-rolled upgrade chain and regenerating from metadata is correct precisely because this window will not exist again. A migration framework is the right answer once the schema stabilizes, and V4.6 deliberately does not pretend that moment has arrived.

C. One Schema Pass

The re-level, the indexes, the use_alter fix, and the lazy="raise" flip all regenerate the same schema. They land together, are verified together, and are reverted together if verification fails. Partial application is not a valid state.

D. Data Migration Is Last

The migration script is written against the final schema and the final loading strategy. Writing it earlier guarantees rework and risks it carrying implicit lazy loads that lazy="raise" will later reject.

E. Forward Compatibility Where It Is Cheap

Single-process operation makes the atomic job claim non-urgent, not wrong. Where the correct multi-user implementation costs little more than the single-user one, V4.6 writes the correct one and guards it by dialect. Where it costs substantially more, V4.6 defers it and documents the assumption.

F. Behavior Is Preserved Exactly

A pure-remediation release that changes observable behavior has failed. The existing test suite is the contract: 264 passing tests must still pass, and any test that must change is treated as evidence that the change is not remediation.

G. The Instruction Files Are the Standard

Most findings are deviations from rules the project already wrote down. V4.6 restores conformance to services.instructions.md and ui.instructions.md rather than inventing new conventions — except where a rule is itself wrong, in which case the rule is corrected explicitly.

Acceptance Criteria

app_state.py, services/transcription.py, the store.py aliases, ServiceBase.queue, and db/operations.py:get_next_queued_job no longer exist, and the full suite passes without them.
upgrade_schema and the three _upgrade_* functions no longer exist; no raw ALTER TABLE or CREATE INDEX string appears in src.
A database created from empty by create_all() contains the composite Job index, indexed hot foreign keys, and a preferred_execution_attempt_id column whose type matches the model declaration.
Compiling the metadata against the PostgreSQL dialect emits no unresolvable-cycle warning.
No Relationship in db/models.py uses lazy="selectin" as a bidirectional default; every load path that requires eager loading declares it per query, and the suite passes under lazy="raise".
read_next_queued_job returns at most one row and issues no eager-load queries; a test asserts the emitted SQL contains LIMIT.
The worker processes two consecutive jobs against a single provider client instance; a test asserts the client is not reconstructed between jobs.
worker_provider_timeout_seconds accepts a value above 20 seconds, and the OpenRouter client receives an explicit httpx.Timeout.
inspect.signature no longer appears in the transcription call path.
DocumentType and PersonRole CRUD is served by one shared implementation; the existing registry tests for both pass unchanged.
ServiceBase._get_or_raise is the only place a NOT_FOUND guard is written for an entity fetched by id.
One media-storage implementation serves Source files, portraits, and homepage images, and its write is off the event loop.
list_sources_detail filters by job_id in SQL; read_source_navigation issues bounded queries; list_processing_artifacts accepts a limit.
No page imports session_scope, sqlalchemy.inspect, or get_settings.
The 23KB SVG literal no longer appears in any .py file.
ruff check reports zero errors.
ty check reports zero diagnostics and runs as a pre-commit gate.
tools/run_destructive_tests.py runs on Windows.
All 264 pre-existing tests still pass. Any test modified during V4.6 is individually justified as a test defect rather than a behavior change.
The migration script restores the backed-up V4.5 data into the re-leveled schema with row counts matching the backup, and no on-disk Source file, portrait, or artifact is modified.
No new user-facing feature, page, action, or field exists in V4.6 that did not exist in V4.5.
Database, integration, and UI verification uses isolated test data and never modifies data/transcription.db.

Scope Freeze Gate

V4.6 is sufficiently frozen to begin implementation:

The operating context — SQLite, single process, disposable data — is confirmed and its consequences for severity are resolved.
The schema strategy is resolved: re-level, no Alembic, one pass, migration last.
The severity band is resolved: Critical through Low, inclusive.
The service-layer consolidation set is resolved, and the SourceService split is explicitly deferred to V4.7.
The release character is resolved: pure remediation, no new features.

Any expansion into PostgreSQL enablement, multi-worker execution, a migration framework, the SourceService split, or any new feature requires an explicit V4.6 scope amendment or a later revision.

17 KiB Raw Blame History