transcription/docs/roadmap_plan.md

Roadmap Review: Recommendations & Version Plan

This is a review/consulting deliverable. It organizes your near-term changes, nice-to-haves, and long-range goals into version buckets, building on ./docs/ver4.8/feature_backlog_v4_8.md (already-scoped image work) and folding in the additional items from your original list.

Versioning convention (per your direction): 4.8–4.11 are incremental feature/fix releases; 5.x and 6.x are reserved for major changes to the app (a shared data-model overhaul, and the hosting migration, respectively).

Final deliverable will be saved to ./docs in the repo per your instruction.

---

V4.8 — Bug fixes + Image Experience

Bug fixes (do first within this release — data correctness, not features):

1. Stale Error Detail after resubmit. Root cause found in db/models.py: Source.latest_error_detail sorts all execution_attempts on the latest job by attempt_number descending and returns the first attempt with any error_detail, even if that's an older attempt and the latest attempt succeeded. Fix: only report error detail from the latest attempt, don't fall through to earlier ones.
2. "View Jobs" ≠ "View Sources" behavior. /documents/{id}/sources already redirects to the filtered Transcription Pipeline Jobs page; /documents/{id}/jobs renders its own bare two-field custom page instead. Fix: make /documents/{id}/jobs redirect the same way, filtered by document.

Image Experience (Track A from the V4.8 backlog doc — already scoped, high value/low risk, storage layer already exists): 3. Homepage Image Gallery (recommended first feature in that doc): list_homepage_images() already returns every stored image; only the multi-image carousel UI and optional slideshow rotation are missing. 4. Pan and Zoom on Source Detail: recover document_panzoom.py from git history (deleted in V4.6 Phase 5, 6a3ee26), but vendor the Panzoom library locally instead of the unpkg CDN, and reuse resolve_media_url from media_urls.py instead of recreating its old helper. Scope to Source Detail only — do not add it to the shared dark_room_viewer.

Note: the backlog doc's model-performance/telemetry item still applies as stated there — it must wait for V4.7 Phase 4 to land first, since duration_ms today mixes provider latency with preprocessing/DB commit time.

---

V4.9 — Detail Page Parity Pass

Small, low-risk structural-parity fixes across Person/Job/Document detail pages:

• Hide "Maiden Name" on Person detail when empty.
• Job Record detail "Document Links" box → mirror the Document detail page's "Sources & Pipeline Jobs" box (clickable name, Sources count, single "View Sources" button, no "+Add Job").
• Transcription Pipeline Jobs page: add a Document Name column; resolve Created vs. Updated by keeping Updated as the primary/visible field (more actionable) and de-emphasizing Created rather than deleting it outright.
• Person detail "Linked Documents" → 3-column table (Document Name, Role, Number of Pages), with clickable document names, dropping the "Open" button.

---

V4.10 — Settings Consolidation & Small Enhancements

• Settings page → tabs, not sub-pages (per your confirmation). Convert the existing stacked cards (Document Types, Person Roles, Prompts) into ui.tabs/ui.tab_panels, and add a new Home Page Text tab. One route, no navigation overhead, scales cleanly as more settings are added.
• Google Maps links for Person birth/death locations — cheap, no scraping, a formatted place string becomes a Maps search link.
• FamilySearch ID lookup (scope confirmed as narrow): on the Person create/edit form, add a FamilySearch ID field and an "Auto-fill from FamilySearch" button that fetches only birth date/place, death date/place, and marriage date/spouse for that specific ID — a single-record lookup, not a crawler. This keeps FamilySearch.org as a companion reference rather than a data source the app tries to replace. Because it's ID-driven and single-record, it's a much smaller, safer feature than open-ended scraping — worth doing at this scope, revisit if FamilySearch's page structure changes and breaks the parser.

---

V4.11 — Approved Scope

• Tags (supersedes "collections"): many-to-many tagging for Documents with Settings-style management (same pattern as Document Types and Person Roles), autocomplete-capable assignment, and a dedicated Tags entry point for browse/filter-by-tag workflows.
• Source Detail simplification: remove the separate Transcription Text card; show Source image + Editable Revision + Source/SourceJob metadata in a 3-column top layout, then keep Candidate Machine Transcriptions below the image/revision area.
• Integrity reconciliation checks in tests:
  • document folder count under UPLOAD_DIR/documents must equal document row count.
  • source file count under each UPLOAD_DIR/documents/{document_id} folder must equal source row count for that Document.
  • failures should include actionable mismatch details (missing row/folder or file/source mapping).
• UI table updates:
  • Archival Documents: remove Archive Ref, add # Sources.
  • Archival Entities: People: remove Display Name and Maiden Name columns, add FamilySearch ID.
  • Transcription Pipeline Jobs: add # Sources.
• Create Processing Job page: Provider and Model must be selectable for new job creation.

Deferred out of this release:

• UI theme selection.
• Settings-based .env editing and runtime controls.
• Person table structural redesign (removing/splitting name fields).

---

V5.0 — Unified Photos Table (major data-model change)

V5.0 standardizes homepage images and Person portraits into one shared photo table and one storage layout.

Finalized shape:

• photo: id, nullable person_id, path, description, is_primary, timestamps.
• person_id IS NULL = homepage photos; non-null = Person photos.
• is_primary is the featured/first photo for that owner (homepage or Person).
• No separate context enum; ownership is derived from person_id.
• All image files are stored under UPLOAD_DIR/photos/{photo_id}{suffix}.
• Person.portrait_path is removed.
• HOMEPAGE_DIR is retired; homepage markdown remains file-backed at UPLOAD_DIR/homepage.md.

Migration policy for legacy installs:

• Export/import rebuild remains the migration mechanism.
• Legacy person.portrait_path values are backfilled into photo rows.
• Legacy homepage images under UPLOAD_DIR/homepage are backfilled into homepage photo rows.
• Legacy homepage markdown is relocated to UPLOAD_DIR/homepage.md.

---

V5.1 — Person table structural redesign

• Replace person.full_name with split required fields:
  • last_name
  • given_names (first + middle)
• Remove display_name and maiden_name from active schema/UI.
• Keep family_search_id optional and unique (not required in this version).
• Add person_tag many-to-many links so People use the same Tag registry as Documents.
• Update Archival Entities: People list columns to:
  • Last Name
  • First & Middle
  • FamilySearch ID
  • Birth Date
  • Death Date

  • Documents

• Migration/export-import behavior backfills split names from legacy full_name values.

---

V6.0 — Server Hosting Migration

Your stated approach (Postgres in Docker, app in Docker, Cloudflare Tunnel) is the standard, low-maintenance way to get secure remote access without exposing ports or running your own VPN/reverse-proxy TLS setup. Sequential stages (each de-risks the next):

Stage 0 — Prerequisites

• Confirm data access already goes through the service layer (it does, per .github/instructions/services.instructions.md) — this is what makes the DB swap and later auth additive rather than a rewrite.
• Confirm upload/file storage path (settings.upload_dir) is Docker-volume-friendly.
• Confirm DB URL and upload dir are both env-overridable (.env/.env.example already exist).

Stage 1 — Containerize against the existing SQLite file first (smallest possible change) to validate the container boundary (networking, volumes, permissions) before also switching databases. Dockerfile/docker-compose.yml already exist in the repo — confirm current intent vs. extend for production.

Stage 2 — Migrate to PostgreSQL. Since the app uses SQLModel/SQLAlchemy, this should mostly be a connection string/dialect change plus a data migration script. Do this as its own isolated step so a regression is attributable to the DB swap alone. Resolve the Postgres connection through a small factory/indirection point rather than a single global engine constant — this costs nothing now and keeps the door open for per-user databases later (see Beyond, below).

Stage 3 — Expose via Cloudflare Tunnel. Add a cloudflared container pointed at the app's internal port (no public port exposure needed). Put the tunnel hostname behind Cloudflare Access (free tier, email OTP/SSO) as the first layer of remote-access auth, even before the app has its own login system — gets secure remote access working quickly and buys time to do Stage 4 and later user-auth properly instead of rushed.

Stage 4 — Backups & operational hygiene. Automated Postgres backups (scheduled pg_dump) and uploaded-file backups before exposing this to the internet — this is the point where a data-loss incident would be most damaging. Basic container health checks/restart policies in compose.

---

V6.1 — Reporting Features

Independent of hosting — can be built any time, including in parallel with V6.0:

• Person timelines: a query/aggregation feature (documents by person, ordered by document date), no hosting dependency.
• AI-written biographies/family histories: likely reuses the existing prompt/provider abstraction (services/prompts.py) already built for transcription, rather than needing new infrastructure.

---

Beyond / Pie-in-the-sky (informational only — no version assigned)

Not being built now, but the plan above is checked against these so nothing forecloses them later:

• Individual user logins + roles (admin/editor/contributor/view-only): Cloudflare Access (V6.0 Stage 3) is a stopgap, not this. When you get here, add a real User/Role model and auth middleware. Because the codebase already funnels data access through the service layer, adding a "current user + role check" later is additive — this is the reason to keep service-layer discipline intact through all the versions above, so authorization can be bolted on without touching every page.
• Per-user databases: protected for by the Stage 2 connection-factory recommendation above (schema-per-tenant or database-per-tenant becomes much easier if the DB connection is already resolved through an indirection point rather than hardcoded at startup).

---

Current status

• V5.0 unified photos has been implemented with:
  • shared photo table (person_id nullable for homepage ownership),
  • flat media storage under UPLOAD_DIR/photos/{photo_id}{suffix},
  • migration backfill from legacy Person portraits and homepage images,
  • homepage markdown relocated to UPLOAD_DIR/homepage.md.
• Version numbering/grouping is now established by implementation and can proceed to V5.1 planning/execution.