V5.1 Update tests and documentation

docs/architecture.md

@@ -1,6 +1,6 @@
-# System Architecture (Version 4)
+# System Architecture (Current Baseline: V5.1)
 
-This document defines the current Version 4 architecture baseline.
+This document defines the current V5.1 architecture baseline.
 
 ## Architecture Objectives
 
@@ -165,5 +165,5 @@ Current architecture rules live in `docs/*`.
 - [System Requirements](requirements.md)
 - [Data Model](schema.md)
 - [Error Handling Policy](error_handling.md)
-- [Error Handling invariant](../invariant/error_handling.md)
+- [Error Handling invariant](./invariant/error_handling.md)
-- [AI evidence invariant](../invariant/ai_evidence_and_provenance.md)
+- [AI evidence invariant](./invariant/ai_evidence_and_provenance.md)

docs/error_handling.md

@@ -1,6 +1,6 @@
-# Error Handling Policy (Version 4)
+# Error Handling Policy (Current Baseline: V5.1)
 
-This policy defines the active Version 4 error taxonomy, translation boundaries, and retry semantics.
+This policy defines the active V5.1 error taxonomy, translation boundaries, and retry semantics.
 
 ## Error Categories
 
@@ -113,6 +113,6 @@ taxonomy to the six canonical categories at the API/UI envelope boundary.
 
 ## Cross-Reference
 
-- [Error Handling invariant](../invariant/error_handling.md)
+- [Error Handling invariant](./invariant/error_handling.md)
 - [System Requirements](requirements.md)
 - [Data Model](schema.md)

docs/index.md

@@ -1,6 +1,6 @@
-# Document Transcription System Overview (Version 4)
+# Document Transcription System Overview (Current Baseline: V5.1)
 
-This directory is the single source of truth for current Version 4 behavior and architecture.
+This directory is the single source of truth for current V5.1 behavior and architecture.
 
 ## Canonical Reading Order
 
@@ -11,13 +11,13 @@ This directory is the single source of truth for current Version 4 behavior and
 
 ## Cross-Version Invariants
 
-- [Historical Document Transcription Design Intent](../invariant/intent.md)
+- [Historical Document Transcription Design Intent](./invariant/intent.md)
-- [Transcription Methodology](../invariant/transcription_methodology.md)
+- [Transcription Methodology](./invariant/transcription_methodology.md)
-- [Error Handling](../invariant/error_handling.md)
+- [Error Handling](./invariant/error_handling.md)
-- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
+- [Digital Evidence and AI Processing Provenance](./invariant/ai_evidence_and_provenance.md)
-- [UI Style Guide](../invariant/ui_style_guide.md)
+- [UI Style Guide](./invariant/ui_style_guide.md)
 
 ## Baseline Statement
 
-The current Version 4 baseline includes behavior delivered through the architectural cleanup phases.
+The current V5.1 baseline includes behavior delivered through the architectural cleanup phases and person-schema redesign.
 Use this `docs/*` canonical set for active design and implementation decisions.

docs/invariant/ui_style_guide.md

@@ -4,14 +4,14 @@
 This guide defines non-negotiable UI styling rules for the transcription application.
 
 The design system is token-first and class-driven:
-1. Theme tokens are defined in [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css).
+1. Theme tokens are defined in [src/transcription/ui/static/theme.css](../../src/transcription/ui/static/theme.css).
 2. Python UI code composes semantic classes instead of inline color values.
 3. Pages and components should share a single visual language across Documents, Jobs, People, and Sources flows.
 
 ## 2. Source of Truth
 Use these files as the style authority:
-1. [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css) for color tokens, semantic utility classes, table styles, and viewer surfaces.
+1. [src/transcription/ui/static/theme.css](../../src/transcription/ui/static/theme.css) for color tokens, semantic utility classes, table styles, and viewer surfaces.
-2. [src/transcription/ui/theme.py](src/transcription/ui/theme.py) for runtime NiceGUI theme bridge and shared UI helpers.
+2. [src/transcription/ui/theme.py](../../src/transcription/ui/theme.py) for runtime NiceGUI theme bridge and shared UI helpers.
 
 If this document conflicts with implementation, update this document to match the code immediately after intentional style changes.
 
@@ -25,7 +25,7 @@ If this document conflicts with implementation, update this document to match th
 ## 4. Token System
 
 ### 4.1 Palette Tokens
-Base palette variables live under :root in [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css):
+Base palette variables live under :root in [src/transcription/ui/static/theme.css](../../src/transcription/ui/static/theme.css):
 1. --palette-carbon-black: #1c2321
 2. --palette-cool-steel: #7d98a1
 3. --palette-blue-slate: #5e6572
@@ -81,7 +81,7 @@ Semantic tokens currently include:
 2. ui-table-header
 3. ui-table-body
 
-Use existing class combinations from [src/transcription/ui/components](src/transcription/ui/components) and [src/transcription/ui/pages](src/transcription/ui/pages) as reference implementations.
+Use existing class combinations from [src/transcription/ui/components](../../src/transcription/ui/components) and [src/transcription/ui/pages](../../src/transcription/ui/pages) as reference implementations.
 
 ## 6. Legacy Class Policy
 Legacy `vibe-` presentation classes are prohibited. Use `ui-` semantic classes from `theme.css`.
@@ -96,7 +96,7 @@ Legacy `vibe-` presentation classes are prohibited. Use `ui-` semantic classes f
 
 ## 8. Implementation Rules For Contributors
 1. Prefer composing existing semantic classes before creating new ones.
-2. If a new class is required, add it to [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css) with a semantic name, then reuse it.
+2. If a new class is required, add it to [src/transcription/ui/static/theme.css](../../src/transcription/ui/static/theme.css) with a semantic name, then reuse it.
 3. Keep behavior ownership in Python and appearance ownership in CSS.
 4. Update UI tests that assert exact text or labels when intentional copy changes are made.
 5. Avoid introducing class churn unrelated to the feature being changed.
@@ -104,7 +104,7 @@ Legacy `vibe-` presentation classes are prohibited. Use `ui-` semantic classes f
 ## 9. Verification Checklist
 Before merging UI changes, verify:
 1. No new inline hex colors were introduced in UI pages/components.
-2. New styles are token-backed and added to [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css).
+2. New styles are token-backed and added to [src/transcription/ui/static/theme.css](../../src/transcription/ui/static/theme.css).
 3. Primary buttons, links, cards, and tables still render with consistent semantics.
 4. Keyboard focus ring visibility is preserved.
 5. Relevant UI and integration tests pass.

docs/requirements.md

@@ -1,14 +1,14 @@
-# System Requirements (Version 4)
+# System Requirements (Current Baseline: V5.1)
 
-These requirements define the active Version 4 contract and align to current implementation.
+These requirements define the active V5.1 contract and align to current implementation.
 
 ## Functional Requirements
 
 ### Domain and Record Management
 
-- **REQ-4-001 Document Registry:** The system must create and update `Document` records with title, type, language, comments, date metadata, and optional location.
+- **REQ-4-001 Document Registry:** The system must create and update `Document` records with title, type, date metadata, optional location, optional archive identifier, and optional notes.
 - **REQ-4-002 Source Registry:** The system must create and update `Source` records linked to exactly one `Document`.
-- **REQ-4-003 People Registry:** The system must create and update `Person` records and support many-to-many links to `Document` with role and confidence.
+- **REQ-4-003 People Registry:** The system must create and update `Person` records, support many-to-many links to `Document` with role, and support many-to-many Person tagging via the shared Tag registry.
 - **REQ-4-004 Registry Semantics:** Document types and person roles must support optional immutable semantic keys and hard-delete only when unreferenced.
 
 ### Job and Workflow Behavior

docs/roadmap_plan.md

@@ -1,239 +1,86 @@
-# Roadmap Review: Recommendations & Version Plan
+# Roadmap Plan (Starting at V6.0)
 
-This is a review/consulting deliverable. It organizes your near-term changes,
+This roadmap starts at **V6.0** and tracks forward-looking work only.
-nice-to-haves, and long-range goals into version buckets, building on
-`./docs/ver4.8/feature_backlog_v4_8.md` (already-scoped image work) and
-folding in the additional items from your original list.
 
-Versioning convention (per your direction): **4.8–4.11** are incremental
+## V6.0 - Hosting Migration
-feature/fix releases; **5.x and 6.x are reserved for major changes** to the
-app (a shared data-model overhaul, and the hosting migration, respectively).
 
-Final deliverable will be saved to `./docs` in the repo per your instruction.
+Objective: move from local-only operation to secure, stable remote hosting.
 
----
+### Scope
+1. Containerize app runtime for production deployment.
+2. Run PostgreSQL in Docker and migrate from SQLite.
+3. Add Cloudflare Tunnel exposure with Access protection.
+4. Add operational safeguards (health checks, restart policies, backups).
 
-## V4.8 — Bug fixes + Image Experience
+### Deliverables
+- Production-ready `docker-compose` deployment for app + database + tunnel.
+- Environment-based configuration for DB, uploads, prompts, and logging.
+- Verified data migration path into PostgreSQL.
+- Runbook updates for deploy, rollback, and backup/restore.
 
-**Bug fixes (do first within this release — data correctness, not features):**
+### Exit Criteria
-1. **Stale Error Detail after resubmit.** Root cause found in `db/models.py`:
+- `/healthz` reports healthy app and worker in deployed environment.
-   `Source.latest_error_detail` sorts all `execution_attempts` on the latest
+- One end-to-end document -> source -> job workflow succeeds remotely.
-   job by `attempt_number` descending and returns the **first attempt with
+- Backup and restore procedure is tested.
-   any `error_detail`**, even if that's an older attempt and the latest
-   attempt succeeded. Fix: only report error detail from the latest attempt,
-   don't fall through to earlier ones.
-2. **"View Jobs" ≠ "View Sources" behavior.** `/documents/{id}/sources`
-   already redirects to the filtered Transcription Pipeline Jobs page;
-   `/documents/{id}/jobs` renders its own bare two-field custom page instead.
-   Fix: make `/documents/{id}/jobs` redirect the same way, filtered by document.
 
-**Image Experience (Track A from the V4.8 backlog doc — already scoped, high
+## V6.1 - Reporting Features
-value/low risk, storage layer already exists):**
-3. **Homepage Image Gallery** (recommended first feature in that doc):
-   `list_homepage_images()` already returns every stored image; only the
-   multi-image carousel UI and optional slideshow rotation are missing.
-4. **Pan and Zoom on Source Detail**: recover `document_panzoom.py` from git
-   history (deleted in V4.6 Phase 5, `6a3ee26`), but vendor the Panzoom
-   library locally instead of the unpkg CDN, and reuse `resolve_media_url`
-   from `media_urls.py` instead of recreating its old helper. Scope to Source
-   Detail only — do not add it to the shared `dark_room_viewer`.
 
-**Note:** the backlog doc's model-performance/telemetry item still applies as
+Objective: improve research value with person-centric outputs.
-stated there — it must wait for V4.7 Phase 4 to land first, since `duration_ms`
-today mixes provider latency with preprocessing/DB commit time.
 
----
+### Scope
+1. Person timeline views using document dates and linked records.
+2. AI-assisted biography/family-history generation from curated sources.
+3. Exportable report views (human-readable, print-oriented).
 
-## V4.9 — Detail Page Parity Pass
+### Deliverables
+- Timeline UI and service queries with clear ordering/filters.
+- Prompted narrative generation workflow using existing evidence-safe patterns.
+- Saved/printable report presentation for review and sharing.
 
-Small, low-risk structural-parity fixes across Person/Job/Document detail pages:
+### Exit Criteria
-- Hide "Maiden Name" on Person detail when empty.
+- Timelines are reproducible from persisted records.
-- Job Record detail "Document Links" box → mirror the Document detail page's
+- Narrative generation is traceable to source records and prompts.
-  "Sources & Pipeline Jobs" box (clickable name, Sources count, single
+- Reports can be reviewed without modifying archival source data.
-  "View Sources" button, no "+Add Job").
-- Transcription Pipeline Jobs page: add a Document Name column; resolve
-  Created vs. Updated by keeping **Updated** as the primary/visible field
-  (more actionable) and de-emphasizing Created rather than deleting it outright.
-- Person detail "Linked Documents" → 3-column table (Document Name, Role,
-  Number of Pages), with clickable document names, dropping the "Open" button.
 
----
+## V6.2 - Access Control and Multi-User Readiness
 
-## V4.10 — Settings Consolidation & Small Enhancements
+Objective: prepare for managed collaboration beyond single-user operation.
 
-- **Settings page → tabs**, not sub-pages (per your confirmation). Convert the
+### Scope
-  existing stacked cards (Document Types, Person Roles, Prompts) into
+1. Introduce application-level authentication.
-  `ui.tabs`/`ui.tab_panels`, and add a new **Home Page Text** tab. One route,
+2. Add role-based authorization (admin/editor/contributor/viewer).
-  no navigation overhead, scales cleanly as more settings are added.
+3. Add audit visibility for user-attributed write actions.
-- **Google Maps links** for Person birth/death locations — cheap, no
-  scraping, a formatted place string becomes a Maps search link.
-- **FamilySearch ID lookup** (scope confirmed as narrow): on the Person
-  create/edit form, add a FamilySearch ID field and an "Auto-fill from
-  FamilySearch" button that fetches only birth date/place, death date/place,
-  and marriage date/spouse for that specific ID — a single-record lookup, not
-  a crawler. This keeps FamilySearch.org as a companion reference rather than
-  a data source the app tries to replace. Because it's ID-driven and
-  single-record, it's a much smaller, safer feature than open-ended scraping —
-  worth doing at this scope, revisit if FamilySearch's page structure changes
-  and breaks the parser.
 
----
+### Deliverables
+- User identity model and login/session flow.
+- Route/page/service authorization enforcement.
+- Audit metadata for sensitive create/update/delete workflows.
 
-## V4.11 — Approved Scope
+### Exit Criteria
+- Unauthorized operations are blocked consistently across UI/API.
+- Role policies are enforced by deterministic tests.
+- User-attributed changes are visible for audit/review.
 
-- **Tags** (supersedes "collections"): many-to-many tagging for Documents with
+## V6.3 - Scalability and Multi-Tenant Direction (Optional)
-  Settings-style management (same pattern as Document Types and Person Roles),
-  autocomplete-capable assignment, and a dedicated **Tags** entry point for
-  browse/filter-by-tag workflows.
-- **Source Detail simplification**: remove the separate **Transcription Text**
-  card; show Source image + Editable Revision + Source/SourceJob metadata in a
-  3-column top layout, then keep Candidate Machine Transcriptions below the
-  image/revision area.
-- **Integrity reconciliation checks in tests**:
-  - document folder count under `UPLOAD_DIR/documents` must equal `document`
-    row count.
-  - source file count under each `UPLOAD_DIR/documents/{document_id}` folder
-    must equal `source` row count for that Document.
-  - failures should include actionable mismatch details (missing row/folder or
-    file/source mapping).
-- **UI table updates**:
-  - Archival Documents: remove **Archive Ref**, add **# Sources**.
-  - Archival Entities: People: remove **Display Name** and **Maiden Name**
-    columns, add **FamilySearch ID**.
-  - Transcription Pipeline Jobs: add **# Sources**.
-- **Create Processing Job page**: Provider and Model must be selectable for new
-  job creation.
 
-Deferred out of this release:
+Objective: keep architecture ready for broader deployment footprints.
 
-- UI theme selection.
+### Scope
-- Settings-based `.env` editing and runtime controls.
+1. Evaluate per-tenant or per-user data partitioning strategy.
-- Person table structural redesign (removing/splitting name fields).
+2. Formalize connection/runtime strategy for tenant-aware DB selection.
+3. Expand operational telemetry for throughput and cost monitoring.
 
----
+### Deliverables
+- Decision document for tenancy model and migration strategy.
+- Prototype-safe runtime boundary for selecting data targets.
+- Monitoring baseline for queue depth, job latency, and provider cost trends.
 
-## V5.0 — Unified Photos Table (major data-model change)
+### Exit Criteria
+- Selected tenancy strategy is documented and testable.
+- Operational metrics support capacity planning.
 
-V5.0 standardizes homepage images and Person portraits into one shared `photo`
+## Planning Notes
-table and one storage layout.
 
-Finalized shape:
+- Keep architecture, schema, and UI contracts synchronized in `docs/` as each version lands.
-- `photo`: `id`, nullable `person_id`, `path`, `description`, `is_primary`,
+- Prefer explicit schema migration over runtime compatibility write paths.
-  timestamps.
+- Preserve evidence/provenance guarantees when adding new AI-powered features.
-- `person_id IS NULL` = homepage photos; non-null = Person photos.
-- `is_primary` is the featured/first photo for that owner (homepage or Person).
-- No separate context enum; ownership is derived from `person_id`.
-- All image files are stored under `UPLOAD_DIR/photos/{photo_id}{suffix}`.
-- `Person.portrait_path` is removed.
-- `HOMEPAGE_DIR` is retired; homepage markdown remains file-backed at
-  `UPLOAD_DIR/homepage.md`.
-
-Migration policy for legacy installs:
-- Export/import rebuild remains the migration mechanism.
-- Legacy `person.portrait_path` values are backfilled into `photo` rows.
-- Legacy homepage images under `UPLOAD_DIR/homepage` are backfilled into
-  homepage `photo` rows.
-- Legacy homepage markdown is relocated to `UPLOAD_DIR/homepage.md`.
-
----
-
-## V5.1 — Person table structural redesign
-
-- Replace `person.full_name` with split required fields:
-  - `last_name`
-  - `given_names` (first + middle)
-- Remove `display_name` and `maiden_name` from active schema/UI.
-- Keep `family_search_id` optional and unique (not required in this version).
-- Add `person_tag` many-to-many links so People use the same Tag registry as Documents.
-- Update Archival Entities: People list columns to:
-  - Last Name
-  - First & Middle
-  - FamilySearch ID
-  - Birth Date
-  - Death Date
-  - # Documents
-- Migration/export-import behavior backfills split names from legacy `full_name` values.
-
----
-
-## V6.0 — Server Hosting Migration
-
-Your stated approach (Postgres in Docker, app in Docker, Cloudflare Tunnel) is
-the standard, low-maintenance way to get secure remote access without exposing
-ports or running your own VPN/reverse-proxy TLS setup. Sequential stages
-(each de-risks the next):
-
-**Stage 0 — Prerequisites**
-- Confirm data access already goes through the service layer (it does, per
-  `.github/instructions/services.instructions.md`) — this is what makes the
-  DB swap and later auth additive rather than a rewrite.
-- Confirm upload/file storage path (`settings.upload_dir`) is Docker-volume-friendly.
-- Confirm DB URL and upload dir are both env-overridable (`.env`/`.env.example`
-  already exist).
-
-**Stage 1 — Containerize** against the *existing* SQLite file first (smallest
-possible change) to validate the container boundary (networking, volumes,
-permissions) before also switching databases. `Dockerfile`/`docker-compose.yml`
-already exist in the repo — confirm current intent vs. extend for production.
-
-**Stage 2 — Migrate to PostgreSQL.** Since the app uses SQLModel/SQLAlchemy,
-this should mostly be a connection string/dialect change plus a data migration
-script. Do this as its own isolated step so a regression is attributable to
-the DB swap alone. Resolve the Postgres connection through a small
-factory/indirection point rather than a single global engine constant — this
-costs nothing now and keeps the door open for per-user databases later (see
-Beyond, below).
-
-**Stage 3 — Expose via Cloudflare Tunnel.** Add a `cloudflared` container
-pointed at the app's internal port (no public port exposure needed). Put the
-tunnel hostname behind Cloudflare Access (free tier, email OTP/SSO) as the
-**first layer of remote-access auth**, even before the app has its own login
-system — gets secure remote access working quickly and buys time to do
-Stage 4 and later user-auth properly instead of rushed.
-
-**Stage 4 — Backups & operational hygiene.** Automated Postgres backups
-(scheduled `pg_dump`) and uploaded-file backups *before* exposing this to the
-internet — this is the point where a data-loss incident would be most
-damaging. Basic container health checks/restart policies in compose.
-
----
-
-## V6.1 — Reporting Features
-
-Independent of hosting — can be built any time, including in parallel with
-V6.0:
-- **Person timelines**: a query/aggregation feature (documents by person,
-  ordered by document date), no hosting dependency.
-- **AI-written biographies/family histories**: likely reuses the existing
-  prompt/provider abstraction (`services/prompts.py`) already built for
-  transcription, rather than needing new infrastructure.
-
----
-
-## Beyond / Pie-in-the-sky (informational only — no version assigned)
-
-Not being built now, but the plan above is checked against these so nothing
-forecloses them later:
-
-- **Individual user logins + roles** (admin/editor/contributor/view-only):
-  Cloudflare Access (V6.0 Stage 3) is a stopgap, not this. When you get here,
-  add a real `User`/`Role` model and auth middleware. Because the codebase
-  already funnels data access through the service layer, adding a
-  "current user + role check" later is additive — this is the reason to keep
-  service-layer discipline intact through all the versions above, so
-  authorization can be bolted on without touching every page.
-- **Per-user databases**: protected for by the Stage 2 connection-factory
-  recommendation above (schema-per-tenant or database-per-tenant becomes
-  much easier if the DB connection is already resolved through an
-  indirection point rather than hardcoded at startup).
-
----
-
-## Current status
-- V5.0 unified photos has been implemented with:
-  - shared `photo` table (`person_id` nullable for homepage ownership),
-  - flat media storage under `UPLOAD_DIR/photos/{photo_id}{suffix}`,
-  - migration backfill from legacy Person portraits and homepage images,
-  - homepage markdown relocated to `UPLOAD_DIR/homepage.md`.
-- Version numbering/grouping is now established by implementation and can
-  proceed to V5.1 planning/execution.

docs/schema.md

@@ -1,6 +1,6 @@
-# Data Model and Persistence Schema (Version 4)
+# Data Model and Persistence Schema (Current Baseline: V5.1)
 
-This document is the field-accurate Version 4 schema contract aligned to `src/transcription/db/models.py`.
+This document is the field-accurate V5.1 schema contract aligned to `src/transcription/db/models.py`.
 
 ## Source of Truth Anchors
 
@@ -291,4 +291,4 @@ Constraint:
 - [System Architecture](architecture.md)
 - [System Requirements](requirements.md)
 - [Error Handling Policy](error_handling.md)
-- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
+- [AI Evidence and Provenance Invariant](./invariant/ai_evidence_and_provenance.md)

docs/ui/README.md

@@ -56,4 +56,4 @@ Each page contract contains:
 
 ## Current Baseline
 
-These contracts describe the current flattened Version 4 baseline.
+These contracts describe the current V5.1 baseline.

docs/ui/pages/people.md

@@ -19,8 +19,8 @@ People manages reusable historical-person records. A Person may appear in many D
 
 - The title is **Archival Entities: People**.
 - **Create new person** opens the create route.
-- The table defaults to Last Name order and supports search and column sorting.
+- The table defaults to Name order (`Last Name, First & Middle`) and supports search and column sorting.
-- Columns are Last Name, First & Middle, Tags, FamilySearch ID, Birth Date, Death Date, and # Documents.
+- Columns are Last Name, First & Middle; Tags; FamilySearch ID; Birth Date; Death Date; and # Documents.
 - Name and Tags are left-aligned; FamilySearch ID, date columns, and # Documents are centered.
 - # Documents reflects how many linked Documents each Person is connected to.
 - Birth and death values independently prefer exact date, then approximate date, then `Unknown`.

docs/ver4.8/feature_backlog_v4_8.md

@@ -1,84 +0,0 @@
-# V4.8 Feature Backlog
-
-**Status: not scoped.** This is a parking document, not a frozen boundary. It records feature work deferred out of V4.6 and V4.7 together with the evidence gathered so far, so that scoping V4.8 does not start from a blank page.
-
-V4.8 is the first release since V4.5 to add **new user-facing behavior**. V4.6 was pure remediation and V4.7 is architectural cleanup; both were held to "no new features." That constraint ends here, which means V4.8 needs a different verification gate: V4.6 and V4.7 could be validated by "the suite still passes unchanged," and V4.8 cannot.
-
-## Dependency on V4.7
-
-**The model-performance rollup below must not begin until V4.7 Phase 4 lands.** `duration_ms` currently measures provider call *plus* image normalization, artifact persistence, and a DB commit, while the timeout governs only the provider call. A rollup built on it would chart preprocessing time mixed with provider latency and look authoritative while quietly misleading. V4.7 Phase 1 removes normalization and artifact persistence from that window, but the commit remains inside it until Phase 4.
-
-## Candidate Features
-
-### 1. Pan and Zoom on Source Detail
-
-**Practicality: high. Effort: S.**
-
-`ui/components/document_panzoom.py` existed and was **deleted in V4.6 Phase 5** (`6a3ee26`) because it was exported but wired to no page. It is 136 lines and recoverable:
-
-```
-git show 6a3ee26^:src/transcription/ui/components/document_panzoom.py
-```
-
-It already handled both images and PDFs (the latter via an iframe).
-
-Two things must change on reintroduction - this is not a straight revert:
-
-- It loaded Panzoom from the **unpkg CDN**. For an archival application the library should be vendored locally, otherwise the viewer breaks offline and depends on a third party staying available.
-- It carried its own `_document_url()` helper. V4.6 Phase 5 extracted exactly that logic into `ui/components/media_urls.py` as `resolve_media_url`. Reintroducing the old helper would recreate the duplication Phase 5 removed.
-
-Scope note: apply it to **Source Detail only**. `dark_room_viewer` (`ui/components/viewers.py`) is shared by four pages - `sources_page.py:268`, `home_page.py:25` and `:88`, `people_page.py:453`, `documents_page.py:524` - so a flag on it would leak pan-zoom into the homepage and document detail, which is not wanted. Add a separate component and use it only at `sources_page.py:268`.
-
-Numbering note: the Phase 5 commit message states pan-zoom would return "in V4.7 alongside the other photo/image work." Moving it to V4.8 preserves that **intent** - it stays grouped with the photo work - and changes only the release number.
-
-### 2. Homepage Image Gallery
-
-**Practicality: high. Effort: S. Recommended first feature.**
-
-The storage layer is already built:
-
-- `ui/homepage_store.py:82` `list_homepage_images()` already returns **every** stored image, sorted by modification time.
-- `store_homepage_image()` already accumulates files rather than overwriting.
-- Today the UI calls only `latest_homepage_image()` and displays one image. `list_homepage_images()` is currently exercised **only by tests**.
-
-So multi-image upload is effectively done; what is missing is presentation. NiceGUI 3.13.0 provides `ui.carousel` for left/right navigation and `ui.timer` for rotation.
-
-Sub-items:
-
-- Multi-image display with left/right navigation - small, mostly wiring.
-- Optional slideshow rotating every ~10 minutes.
-
-**Performance caveat:** `list_homepage_images()` performs a directory scan with a `stat()` per file on every call, and `home_page.py` already performs blocking I/O in the page handler (V4.6 review log [25], which was deliberately left alone). A rotating timer that re-enumerates on every tick would repeat that scan indefinitely. Enumerate once at page load and cache the list.
-
-### 3. Multiple Person Portraits
-
-**Practicality: medium. Effort: M/L. Defer behind item 2.**
-
-`Person.portrait_path` is a **single string column**. Supporting multiple portraits requires a new table, a data migration, and upload UI - a materially larger job than item 2, and a different one.
-
-### 4. Image Descriptions
-
-**Practicality: medium, conditional. Effort: M.**
-
-Homepage images are **filesystem-only with no metadata store**, so a caption has nowhere to live today. This needs either a sidecar JSON file or a real table.
-
-This is cheap **only if** item 3 is being done at the same time, since both need the same metadata layer. Designing that layer twice would be wasteful; design it once or not at all.
-
-## Suggested Grouping
-
-If V4.8 is scoped as one release, the natural split is:
-
-**Track A - image experience:** items 1 and 2. Both are small, both are self-contained UI work, and item 2's storage layer already exists. This is the highest value for the least risk.
-
-**Track B - metadata layer:** items 3 and 4 together, since they share a table. Only worth starting if both are wanted.
-
-**Track C - telemetry:** item 5, gated on V4.7 Phase 4.
-
-Item 6 is not recommended.
-
-## Open Questions for Scoping
-
-- Should Track B happen at all, or is one portrait per person sufficient?
-- Should the slideshow interval be configurable, or fixed?
-- Should vendored Panzoom be committed to the repository, or fetched at build time?
-

tests/api/test_v4_documents.py

@@ -206,6 +206,15 @@ def test_document_person_link_requires_role_id(tmp_path):
         )
 
     assert response.status_code == 422
+    payload = response.json()
+    assert "detail" in payload
+    assert isinstance(payload["detail"], list)
+    missing_role = [
+        item
+        for item in payload["detail"]
+        if isinstance(item, dict) and item.get("loc") == ["body", "role_id"] and item.get("type") == "missing"
+    ]
+    assert missing_role
 
 
 def test_duplicate_document_person_link_returns_conflict_envelope(tmp_path):

tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt

@@ -3,12 +3,11 @@ provider: openrouter
 model: openai/gpt-5.3-codex
 ---
 [document body typewritten]
-
 BY WAY OF INTRODUCTION:~
 
 These few paragraphs of introduction may help you read BOOK 2 which covers a wider range than did BOOK 1 (Pioneer Days).
 
-BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 consisted largely of first generation family history. BOOK 2 throws more light on the second generation. Sidney promises a BOOK 3 and that may begin to do justice to the third generation. We suggest that Sidney get the help of Louis Shinn who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq Plains. Louis has the gift of seeing, recalling and telling. One sentence in his chapter gives a great tribute to the Doumecqers--so far as he knows no one on the Doumecq Plains went on relief during the depression. That in a nutshell shows the sturdy character of the residents of the Doumecq Plains.
+BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 consisted largely of first generation family history. BOOK 2 throws more light on the second generation. Sidney promises a BOOK 3 and that may begin to do justice to the third generation. We suggest that Sidney get the help of Louis Shinn who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq Plains.[sic] Louis has the gift of seeing, recalling and telling. One sentence in his chapter gives a great tribute to the Doumecqers--so far as he knows no one on the Doumecq Plains went on relief during the depression. That in a nutshell shows the sturdy character of the residents of the Doumecq Plains.
 
 We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see by the Table of Contents that the first four chapters have been given to those trips. Those chapters are worth reading and re-reading. Mr. Cochran has eyes to see and a pen to tell. We think the people in Tennessee will read with great pleasure the comments he makes on conditions today.
 

tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt

@@ -24,6 +24,8 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these
 
 We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he didn't and Polly would in my lap and I don't know much about the youngest one of yours so likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
 
+[photograph of a group of people standing on snow]
+
 I expect there were 150 passengers on board and almost or more of the crew and helpers. We had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
 
 I have had jobs nearly all summer but not very much in them. Next week, September 4, school opens. I wish they would wait for a week but you know these school men. Wouldn't make any special difference I suppose for I would just fritter away the time but still one likes to postpone the inevitable.

tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt

@@ -16,18 +16,17 @@ I was at home a
 few nights ago & saw a
 letter from your folks, so
 I decided to write you
-a few lines [in?] regards of
+a few lines myself &
-I am contemplat[ing?] a
+ask contribution a
 trip out west next summer
-& want lots of [places?] to go
+I want lots of old [fellows?] to go
 where I am from.
-
+am getting
-Am getting
+up in years & [somebarries?].
-up in years & [remembering?].
+So you all are the object of
-So you see the object of
 my trip, is to get a wife
-If there is any old maids
+If there is any old maid
 or widows out there I
 want you to hire them
-at [pur?] [find?] me at them
+at our [illegible] me at there
 as soon as I get there.]

tests/test_db.py

@@ -6,7 +6,6 @@ import pytest
 import pytest_asyncio
 import sqlalchemy as sa
 from sqlalchemy import inspect
-from sqlalchemy import text
 from sqlalchemy.dialects import postgresql
 from sqlalchemy.exc import SAWarning
 from sqlmodel import SQLModel
@@ -77,7 +76,9 @@ async def test_get_session_yields_async_session(tmp_path):
 
     try:
         async with session_scope(settings=settings) as session:
-            assert session is not None
+            assert isinstance(session, AsyncSession)
+            value = (await session.exec(select(1))).one()
+            assert value == 1
     finally:
         await dispose_database_runtime()
 

tests/test_storage_reconciliation.py

@@ -16,6 +16,10 @@ from transcription.db.models import Document
 from transcription.db.models import Source
 
 
+def _normalize_identifier(value: str) -> str:
+    return value.replace("-", "").strip().lower()
+
+
 def _document_folder_ids(root: Path) -> set[str]:
     documents_root = root / "documents"
     if not documents_root.exists():
@@ -47,35 +51,51 @@ def _source_file_count_for_document(root: Path, document_id: str) -> int:
 async def assert_storage_reconciliation(*, upload_dir: Path, settings: Settings) -> None:
     folder_ids = _document_folder_ids(upload_dir)
     doc_ids = await _document_ids(settings)
-
+    folder_by_normalized = {_normalize_identifier(folder_id): folder_id for folder_id in folder_ids}
-    missing_in_table = sorted(folder_ids - doc_ids)
+    doc_by_normalized = {_normalize_identifier(doc_id): doc_id for doc_id in doc_ids}
-    missing_in_folders = sorted(doc_ids - folder_ids)
-    if missing_in_table:
-        folder = missing_in_table[0]
-        raise AssertionError(
-            "Document directory count and document.doc_id count do not agree. "
-            f"./data/documents/{folder} does not appear in document table"
-        )
-    if missing_in_folders:
-        doc_id = missing_in_folders[0]
-        raise AssertionError(
-            "Document directory count and document.doc_id count do not agree. "
-            f"document.doc_id {doc_id} has no corresponding folder in ./data/documents"
-        )
-
     source_counts = await _source_counts_by_document(settings)
-    for document_id in sorted(doc_ids):
+    source_counts_by_normalized = {
-        db_count = source_counts.get(document_id, 0)
+        _normalize_identifier(document_id): count for document_id, count in source_counts.items()
-        file_count = _source_file_count_for_document(upload_dir, document_id)
+    }
-        if file_count > db_count:
+    mismatches: list[str] = []
-            raise AssertionError(
+
-                "Source file count and source.source_id count do not agree. "
+    missing_in_table = sorted(set(folder_by_normalized) - set(doc_by_normalized))
-                f"[UPLOAD_DIR]/documents/{document_id}/ contains file(s) with no source row"
+    for folder_key in missing_in_table:
+        folder_name = folder_by_normalized[folder_key]
+        mismatches.append(
+            f"document-folder-without-row: ./data/documents/{folder_name} has no matching document.doc_id"
         )
-        if db_count > file_count:
+
+    missing_in_folders = sorted(set(doc_by_normalized) - set(folder_by_normalized))
+    for doc_key in missing_in_folders:
+        doc_id = doc_by_normalized[doc_key]
+        source_count = source_counts_by_normalized.get(doc_key, 0)
+        mismatches.append(
+            "document-row-without-folder: "
+            f"document.doc_id {doc_id} has no corresponding folder in ./data/documents "
+            f"(source rows: {source_count})"
+        )
+
+    for doc_key in sorted(doc_by_normalized):
+        doc_id = doc_by_normalized[doc_key]
+        folder_name = folder_by_normalized.get(doc_key)
+        db_count = source_counts_by_normalized.get(doc_key, 0)
+        file_count = _source_file_count_for_document(upload_dir, folder_name) if folder_name is not None else 0
+        if db_count != file_count:
+            folder_display = (
+                f"./data/documents/{folder_name}" if folder_name is not None else "./data/documents/<missing-folder>"
+            )
+            mismatches.append(
+                "source-count-mismatch: "
+                f"document.doc_id {doc_id} -> source rows: {db_count}, files in {folder_display}: {file_count}"
+            )
+
+    if mismatches:
+        report = "\n".join(f"- {item}" for item in mismatches)
         raise AssertionError(
-                "Source file count and source.source_id count do not agree. "
+            "Storage reconciliation mismatch(es) detected.\n"
-                f"source.source_id rows exist without files in [UPLOAD_DIR]/documents/{document_id}"
+            f"Reconciling item count: {len(mismatches)}\n"
+            f"{report}"
         )
 
 
@@ -112,5 +132,20 @@ async def test_storage_reconciliation_reports_actionable_mismatch_message(tmp_pa
     orphan_dir = upload_dir / "documents" / str(uuid4())
     orphan_dir.mkdir(parents=True, exist_ok=True)
 
-    with pytest.raises(AssertionError, match="document\\.doc_id count do not agree"):
+    with pytest.raises(AssertionError, match="Reconciling item count:"):
+        await assert_storage_reconciliation(upload_dir=upload_dir, settings=default_settings)
+
+
+@pytest.mark.asyncio
+async def test_storage_reconciliation_reports_missing_document_folder_even_without_sources(
+    tmp_path, default_settings: Settings
+):
+    upload_dir = tmp_path / "uploads"
+    missing_folder_doc_id = uuid4()
+
+    async with session_scope(settings=default_settings) as session:
+        session.add(Document(id=missing_folder_doc_id, name="Needs folder"))
+        await session.commit()
+
+    with pytest.raises(AssertionError, match="document-row-without-folder:"):
         await assert_storage_reconciliation(upload_dir=upload_dir, settings=default_settings)

tests/ui/test_media_urls.py

@@ -2,16 +2,6 @@ from transcription.ui.components.media_urls import public_media_path_label
 from transcription.ui.components.media_urls import resolve_media_url
 
 
-def test_resolve_media_url_maps_managed_absolute_path_to_upload_route(tmp_path):
-    upload_dir = tmp_path / "uploads"
-    canonical_path = upload_dir / "documents" / "abc" / "page.jpg"
-    canonical_path.parent.mkdir(parents=True, exist_ok=True)
-    canonical_path.write_bytes(b"x")
-
-    resolved = resolve_media_url("documents/abc/page.jpg", upload_dir=upload_dir, base_url="http://localhost:8000")
-    assert resolved == "http://localhost:8000/uploads/documents/abc/page.jpg"
-
-
 def test_resolve_media_url_rejects_absolute_filesystem_path(tmp_path):
     upload_dir = tmp_path / "uploads"
     absolute_path = upload_dir / "documents" / "abc" / "page.jpg"

tests/ui/test_print_preview_page.py

@@ -113,3 +113,4 @@ async def test_document_source_media_rejects_cross_document_access(app_client):
 
     response = client.get(f"/api/documents/{other_id}/sources/{source_id}/media")
     assert response.status_code == 404
+    assert response.json()["detail"] == "Source not found for Document"