generated from john/python-template
206 lines
17 KiB
Markdown
206 lines
17 KiB
Markdown
# V4.6 Scope Boundary
|
|
|
|
This document defines the frozen boundary for V4.6, a **pure remediation release**. V4 through V4.5 remain the architecture and behavioral baseline. V4.6 introduces **no new user-facing features**; it pays down the defects, duplication, and structural drift catalogued in [Architecture & Code Review Report](../architecture_code_review_2026-08-17.md).
|
|
|
|
Every item in scope is traceable to a review finding ID. Any change that cannot be traced to a finding ID is out of scope.
|
|
|
|
## Purpose
|
|
|
|
- Remove dead code, dead configuration, and duplicate implementations that create maintenance drift.
|
|
- Re-level the database schema from current SQLModel metadata, ending hand-rolled DDL while the schema is still pre-production.
|
|
- Correct read amplification, missing indexes, and query patterns that scale with table size rather than result size.
|
|
- Restore the boundaries the project already wrote down in `.github/instructions/services.instructions.md` and `ui.instructions.md`.
|
|
- Make `ty` usable as a real quality gate.
|
|
- Preserve every existing behavior, evidence guarantee, and provenance contract established in V4 through V4.5.
|
|
|
|
## Confirmed Operating Context
|
|
|
|
These answers are frozen for V4.6 and govern every decision below.
|
|
|
|
| Question | Answer |
|
|
| :--- | :--- |
|
|
| Database | **SQLite only.** PostgreSQL remains the intended destination but is deferred beyond V4.6. `JSONBCompat` and the Postgres drivers are retained. |
|
|
| Topology | **Single user, single process, single worker.** A multi-user server is the stated direction, so forward-compatibility work is retained where it is cheap. |
|
|
| Schema evolution | **Re-level from current metadata.** No Alembic, no migration framework, no `_upgrade_*` chain. |
|
|
| Existing data | The development database is rebuilt from scratch during implementation and migrated from backup as the final step. |
|
|
| Release character | **Pure remediation.** No new features. |
|
|
| Scope band | Critical through Low, inclusive. |
|
|
|
|
## In Scope
|
|
|
|
### 1. Dead Code and Dead Configuration Removal
|
|
|
|
- `src/transcription/app_state.py` is deleted. It has zero importers and contains a guaranteed `TypeError` ([HIGH-01]).
|
|
- `src/transcription/services/transcription.py` is deleted; `build_prompt_execution` has exactly one import path ([MED-05]).
|
|
- The legacy compatibility aliases in `services/store.py` are deleted ([MED-05]).
|
|
- `ServiceBase.queue` is deleted; no service allocates an unused `asyncio.Queue` ([MED-07]).
|
|
- `sqlite_check_same_thread` and `worker_retry_backoff_seconds` are either wired to real behavior or deleted, along with their tests ([MED-02]).
|
|
- `db/operations.py:get_next_queued_job` is deleted as a divergent duplicate ([CRIT-01]).
|
|
- `DATABASE_URL` is removed from `docker-compose.yml`, and the real nested `DATABASE__*` names are documented. The application never silently ignores a database configuration variable ([MED-10]).
|
|
- `document_panzoom` is either fixed or deleted; it is exported but referenced by no page ([HIGH-07]).
|
|
|
|
### 2. Schema Re-Level
|
|
|
|
The following changes are schema-affecting and land as **one single pass** against a database rebuilt from empty.
|
|
|
|
- `upgrade_schema` and the three `_upgrade_*` functions (`db/operations.py:25-109`) are deleted, along with their tests (`tests/test_db.py:109-172`) ([HIGH-05]).
|
|
- The schema is generated exclusively from SQLModel metadata via `create_all()`, gated by the existing `Settings.should_bootstrap_schema` ([HIGH-05]).
|
|
- The hand-written `CHAR(32)` column for `preferred_execution_attempt_id` ceases to exist; the column type is whatever the model declares ([HIGH-05]).
|
|
- A composite index on `Job.status, Job.date_created` is declared in the model, plus `index=True` on the foreign keys the worker and detail pages filter on ([HIGH-04]).
|
|
- `Source.preferred_execution_attempt_id` declares its foreign key with `use_alter=True`, resolving the `source` / `job_source` / `execution_attempt` cycle so `create_all` will succeed on PostgreSQL when that cutover is taken ([HIGH-08]).
|
|
- Relationship loading defaults change from bidirectional `lazy="selectin"` to `lazy="raise"`, with per-query `selectinload()` retained or added where a load path genuinely requires it ([CRIT-02]).
|
|
|
|
No migration script runs against a populated database. No history table, revision directory, or down path is introduced.
|
|
|
|
### 3. Data Migration
|
|
|
|
- A one-time script under `tools/` migrates the user's backed-up V4.5 data into the re-leveled schema.
|
|
- The script is authored **after** the `lazy="raise"` flip is complete, so that every relationship it traverses carries an explicit eager load.
|
|
- The script is idempotent, is never invoked automatically at startup, and never runs as part of the test suite.
|
|
- Uploaded Source files, portraits, and artifact files on disk are preserved unchanged; only database rows are rewritten.
|
|
- This is the **final** step of V4.6.
|
|
|
|
### 4. Worker and Provider Reliability
|
|
|
|
- `read_next_queued_job` gains `LIMIT 1` and stops materializing the entire queue plus its eager graph on every poll ([CRIT-01]).
|
|
- The claim becomes an atomic `QUEUED` → `PROCESSING` transition. On SQLite this is a bounded single-writer transaction; the `FOR UPDATE SKIP LOCKED` path is written and dialect-guarded for the multi-user direction but is not exercised in V4.6 ([CRIT-01]).
|
|
- Eager relationships are loaded in a second query after the claim succeeds, keeping the hot poll a single narrow row ([CRIT-01]).
|
|
- `ServiceBundle` and the provider client are hoisted to worker-loop scope so the HTTP connection pool and TLS session survive across jobs ([HIGH-02], [MED-06]).
|
|
- `ServiceBundle` gains a `from_session_factory` constructor, replacing three duplicated instantiation blocks ([MED-06]).
|
|
- The `le=20.0` cap on `worker_provider_timeout_seconds` is removed, the default is raised, and an explicit `httpx.Timeout` is passed to the OpenRouter client ([HIGH-03]).
|
|
- The `TranscriptionProvider` Protocol is extended to cover `aclose` and the evidence attributes; the per-call `inspect.signature` reflection at `sources.py:1237` is deleted ([MED-03]).
|
|
|
|
### 5. Service Layer Consolidation
|
|
|
|
- A generic `RegistryService[ModelT]` owns list, summaries, create, read, update, delete, and reference-check for semantic-key registries. `DocumentType` and `PersonRole` become thin subclasses declaring their model, error class, reference query, and noun ([MED-11]).
|
|
- Label normalization, the casefold key, and the registry summary shape are defined once ([MED-11]).
|
|
- `ServiceBase` gains `_get_or_raise`, and all 38 hand-written not-found guards adopt it, including the three in `documents.py` that already bypass the local helper ([MED-12]).
|
|
- `services/media_storage.py` becomes the single implementation of validate → hash → write → wrap-error, replacing `store_source_file`, `store_person_portrait`, and the homepage image writer ([MED-13]).
|
|
- `source_mime_type` moves out of `services/sources.py` to a shared module so `documents.py` no longer imports a sibling service ([MED-14], partial).
|
|
- The four query inefficiencies in `sources.py` are corrected: the `job_id` filter moves into SQL, navigation uses two bounded queries, `list_processing_artifacts` gains a `limit`, and artifact re-hashing moves off the event loop ([LOW-08]).
|
|
|
|
### 6. Async I/O and Configuration Hygiene
|
|
|
|
- Blocking filesystem and CPU work — media writes, artifact writes, integrity hashing, and Pillow orientation normalization — is wrapped in `asyncio.to_thread` ([MED-01]).
|
|
- `functools.cache` on the engine and session factories is replaced with an explicit URL-keyed registry supporting targeted eviction ([MED-04]).
|
|
- The `object.__setattr__` mutation of a frozen `Settings` model in `normalize_provider_models` is replaced with `model_copy(update=...)` or a computed property ([Pydantic V2 §3]).
|
|
- `models.py` timestamp columns that are expected to track modification gain `onupdate`, so `updated_at` and `date_updated` stop being stale on paths that do not set them by hand ([SQLModel §3]).
|
|
- The exception swallowed to `None` in an ORM model property is surfaced ([MED-08]).
|
|
|
|
### 7. UI Boundary and Duplication
|
|
|
|
- The three `ui.instructions.md` violations are corrected ([HIGH-07]):
|
|
- `jobs_page.py` no longer imports `session_scope` or manages transactions; a service or workflow method owns the session.
|
|
- `sources_page.py` no longer imports `sqlalchemy.inspect`; the service returns a plain `transport_body_deferred` flag on a read model.
|
|
- `document_panzoom` no longer calls `get_settings()`; a ready media URL is passed in.
|
|
- The duplication catalogued in the review's §4 is extracted, highest value first: `confirm_delete`, `media_urls`, `guards`, `formatters`, `upload_panel`, and the hand-rolled tables that should use `build_table` (~500 lines).
|
|
- The 23KB inline SVG moves to `ui/static/` and is loaded through an `importlib.resources` reader alongside the existing `read_css` ([MED-09]).
|
|
- `people_page.py:504` routes its error through `error_presenter.show_error` like every sibling handler ([LOW-07]).
|
|
- Untyped handler parameters and loosely-typed dict returns are annotated ([LOW-05]).
|
|
- The auto-refresh timer is cancelled rather than only deactivated, and its interval becomes a named constant ([LOW-06]).
|
|
|
|
### 8. Type Checking and Tooling Gate
|
|
|
|
- The codebase standardizes on `ty`. Remaining suppressions are converted from `# pyright: ignore[...]` to `# ty: ignore[...]` ([HIGH-06]).
|
|
- The `lazy="raise"` flip in §2 is expected to eliminate most of the ~160 `selectinload` diagnostics by removing redundant eager loads.
|
|
- `ty check` reaches zero diagnostics and is wired into the existing pre-commit setup as a gate ([HIGH-06]).
|
|
- The two real bugs currently hidden in the diagnostic noise are fixed: `tests/ui/test_sources_page.py:25` constructs `Source(...)` without the required `document_id`, and `tools/run_destructive_tests.py:76,80` uses `fcntl`, which does not exist on the Windows development platform ([HIGH-06]).
|
|
- `ruff check` reaches zero errors ([LOW-01]).
|
|
- `asyncio_default_fixture_loop_scope` is configured explicitly so pytest-asyncio behavior does not change on upgrade ([Testing §3]).
|
|
- The stale path in `.github/instructions/services.instructions.md:10` is corrected to `src/transcription/db/models.py` ([LOW-02]).
|
|
- `list_jobs` stops accepting and discarding `load_docs` ([LOW-03]).
|
|
- `resolve_worker_notifier` validates its `getattr` result ([LOW-04]).
|
|
|
|
## Out of Scope
|
|
|
|
- Any new user-facing feature, page, action, or field.
|
|
- PostgreSQL enablement, Postgres-backed CI, or a Postgres cutover. The `use_alter` fix unblocks it; it does not perform it.
|
|
- Alembic or any migration framework, revision directory, history table, or down path.
|
|
- Multi-worker or multi-process execution. Forward-compatible code paths are written but not enabled or exercised.
|
|
- Concurrency limits, backpressure, or parallel job processing. Jobs remain strictly serial.
|
|
- **Splitting `SourceService` into per-model services and relocating `update_job_source_transcription` to `workflows.py` ([MED-14]). Deferred to V4.7.** It touches the transcription write path and cannot safely share a release with the schema re-level.
|
|
- Any change to transcription prompt content, medium markers, quality-warning rules, or the retranscription workflow established in V4.5.
|
|
- Any change to the evidence, provenance, or immutability contracts established in V4.2 through V4.5.
|
|
- Deleting, rewriting, or reinterpreting existing `ExecutionAttempt` or `ProcessingArtifact` evidence during data migration.
|
|
- Rewriting the UI table architecture, theme system, or CSS conventions beyond removing duplication.
|
|
- Performance work not traceable to a review finding.
|
|
|
|
## Locked Design Decisions
|
|
|
|
### A. Remediation Only
|
|
|
|
Every change traces to a review finding ID. A desirable improvement discovered during implementation that has no finding ID is recorded for a later revision rather than absorbed.
|
|
|
|
### B. Re-Level, Do Not Migrate
|
|
|
|
The schema is pre-production and the data is disposable and backed up. Deleting the hand-rolled upgrade chain and regenerating from metadata is correct precisely because this window will not exist again. A migration framework is the right answer once the schema stabilizes, and V4.6 deliberately does not pretend that moment has arrived.
|
|
|
|
### C. One Schema Pass
|
|
|
|
The re-level, the indexes, the `use_alter` fix, and the `lazy="raise"` flip all regenerate the same schema. They land together, are verified together, and are reverted together if verification fails. Partial application is not a valid state.
|
|
|
|
### D. Data Migration Is Last
|
|
|
|
The migration script is written against the final schema and the final loading strategy. Writing it earlier guarantees rework and risks it carrying implicit lazy loads that `lazy="raise"` will later reject.
|
|
|
|
### E. Forward Compatibility Where It Is Cheap
|
|
|
|
Single-process operation makes the atomic job claim non-urgent, not wrong. Where the correct multi-user implementation costs little more than the single-user one, V4.6 writes the correct one and guards it by dialect. Where it costs substantially more, V4.6 defers it and documents the assumption.
|
|
|
|
### F. Behavior Is Preserved Exactly
|
|
|
|
A pure-remediation release that changes observable behavior has failed. The existing test suite is the contract: 264 passing tests must still pass, and any test that must change is treated as evidence that the change is not remediation.
|
|
|
|
### G. The Instruction Files Are the Standard
|
|
|
|
Most findings are deviations from rules the project already wrote down. V4.6 restores conformance to `services.instructions.md` and `ui.instructions.md` rather than inventing new conventions — except where a rule is itself wrong, in which case the rule is corrected explicitly.
|
|
|
|
## Acceptance Criteria
|
|
|
|
1. `app_state.py`, `services/transcription.py`, the `store.py` aliases, `ServiceBase.queue`, and `db/operations.py:get_next_queued_job` no longer exist, and the full suite passes without them.
|
|
2. `upgrade_schema` and the three `_upgrade_*` functions no longer exist; no raw `ALTER TABLE` or `CREATE INDEX` string appears in `src`.
|
|
3. A database created from empty by `create_all()` contains the composite `Job` index, indexed hot foreign keys, and a `preferred_execution_attempt_id` column whose type matches the model declaration.
|
|
4. Compiling the metadata against the PostgreSQL dialect emits **no** unresolvable-cycle warning.
|
|
5. No `Relationship` in `db/models.py` uses `lazy="selectin"` as a bidirectional default; every load path that requires eager loading declares it per query, and the suite passes under `lazy="raise"`.
|
|
6. `read_next_queued_job` returns at most one row and issues no eager-load queries; a test asserts the emitted SQL contains `LIMIT`.
|
|
7. The worker processes two consecutive jobs against a single provider client instance; a test asserts the client is not reconstructed between jobs.
|
|
8. `worker_provider_timeout_seconds` accepts a value above 20 seconds, and the OpenRouter client receives an explicit `httpx.Timeout`.
|
|
9. `inspect.signature` no longer appears in the transcription call path.
|
|
10. `DocumentType` and `PersonRole` CRUD is served by one shared implementation; the existing registry tests for both pass unchanged.
|
|
11. `ServiceBase._get_or_raise` is the only place a `NOT_FOUND` guard is written for an entity fetched by id.
|
|
12. One media-storage implementation serves Source files, portraits, and homepage images, and its write is off the event loop.
|
|
13. `list_sources_detail` filters by `job_id` in SQL; `read_source_navigation` issues bounded queries; `list_processing_artifacts` accepts a `limit`.
|
|
14. No page imports `session_scope`, `sqlalchemy.inspect`, or `get_settings`.
|
|
15. The 23KB SVG literal no longer appears in any `.py` file.
|
|
16. `ruff check` reports zero errors.
|
|
17. `ty check` reports zero diagnostics and runs as a pre-commit gate.
|
|
18. `tools/run_destructive_tests.py` runs on Windows.
|
|
19. All 264 pre-existing tests still pass. Any test modified during V4.6 is individually justified as a test defect rather than a behavior change.
|
|
20. The migration script restores the backed-up V4.5 data into the re-leveled schema with row counts matching the backup, and no on-disk Source file, portrait, or artifact is modified.
|
|
21. No new user-facing feature, page, action, or field exists in V4.6 that did not exist in V4.5.
|
|
22. Database, integration, and UI verification uses isolated test data and never modifies `data/transcription.db`.
|
|
|
|
## Scope Freeze Gate
|
|
|
|
V4.6 is sufficiently frozen to begin implementation:
|
|
|
|
- The operating context — SQLite, single process, disposable data — is confirmed and its consequences for severity are resolved.
|
|
- The schema strategy is resolved: re-level, no Alembic, one pass, migration last.
|
|
- The severity band is resolved: Critical through Low, inclusive.
|
|
- The service-layer consolidation set is resolved, and the `SourceService` split is explicitly deferred to V4.7.
|
|
- The release character is resolved: pure remediation, no new features.
|
|
|
|
Any expansion into PostgreSQL enablement, multi-worker execution, a migration framework, the `SourceService` split, or any new feature requires an explicit V4.6 scope amendment or a later revision.
|
|
|
|
## Related Local References
|
|
|
|
- [V4.6 Implementation Plan](implementation_plan_v4_6.md)
|
|
- [Architecture & Code Review Report](../architecture_code_review_2026-08-17.md)
|
|
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
|
|
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
|
|
- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md)
|
|
- [V4 Architecture](../ver4/architecture_v4.md)
|
|
- [V4 Schema](../ver4/schema_v4.md)
|
|
- [Transcription Methodology](../invariant/transcription_methodology.md)
|
|
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
|