transcription/docs/roadmap_plan.md

Roadmap Review: Recommendations & Version Plan

This is a review/consulting deliverable. It organizes your near-term changes, nice-to-haves, and long-range goals into version buckets, building on ./docs/ver4.8/feature_backlog_v4_8.md (already-scoped image work) and folding in the additional items from your original list.

Versioning convention (per your direction): 4.8–4.11 are incremental feature/fix releases; 5.x and 6.x are reserved for major changes to the app (a shared data-model overhaul, and the hosting migration, respectively).

Final deliverable will be saved to ./docs in the repo per your instruction.

---

V4.8 — Bug fixes + Image Experience

Bug fixes (do first within this release — data correctness, not features):

1. Stale Error Detail after resubmit. Root cause found in db/models.py: Source.latest_error_detail sorts all execution_attempts on the latest job by attempt_number descending and returns the first attempt with any error_detail, even if that's an older attempt and the latest attempt succeeded. Fix: only report error detail from the latest attempt, don't fall through to earlier ones.
2. "View Jobs" ≠ "View Sources" behavior. /documents/{id}/sources already redirects to the filtered Transcription Pipeline Jobs page; /documents/{id}/jobs renders its own bare two-field custom page instead. Fix: make /documents/{id}/jobs redirect the same way, filtered by document.

Image Experience (Track A from the V4.8 backlog doc — already scoped, high value/low risk, storage layer already exists): 3. Homepage Image Gallery (recommended first feature in that doc): list_homepage_images() already returns every stored image; only the multi-image carousel UI and optional slideshow rotation are missing. 4. Pan and Zoom on Source Detail: recover document_panzoom.py from git history (deleted in V4.6 Phase 5, 6a3ee26), but vendor the Panzoom library locally instead of the unpkg CDN, and reuse resolve_media_url from media_urls.py instead of recreating its old helper. Scope to Source Detail only — do not add it to the shared dark_room_viewer.

Note: the backlog doc's model-performance/telemetry item still applies as stated there — it must wait for V4.7 Phase 4 to land first, since duration_ms today mixes provider latency with preprocessing/DB commit time.

---

V4.9 — Detail Page Parity Pass

Small, low-risk structural-parity fixes across Person/Job/Document detail pages:

• Hide "Maiden Name" on Person detail when empty.
• Job Record detail "Document Links" box → mirror the Document detail page's "Sources & Pipeline Jobs" box (clickable name, Sources count, single "View Sources" button, no "+Add Job").
• Transcription Pipeline Jobs page: add a Document Name column; resolve Created vs. Updated by keeping Updated as the primary/visible field (more actionable) and de-emphasizing Created rather than deleting it outright.
• Person detail "Linked Documents" → 3-column table (Document Name, Role, Number of Pages), with clickable document names, dropping the "Open" button.

---

V4.10 — Settings Consolidation & Small Enhancements

• Settings page → tabs, not sub-pages (per your confirmation). Convert the existing stacked cards (Document Types, Person Roles, Prompts) into ui.tabs/ui.tab_panels, and add a new Home Page Text tab. One route, no navigation overhead, scales cleanly as more settings are added.
• Google Maps links for Person birth/death locations — cheap, no scraping, a formatted place string becomes a Maps search link.
• FamilySearch ID lookup (scope confirmed as narrow): on the Person create/edit form, add a FamilySearch ID field and an "Auto-fill from FamilySearch" button that fetches only birth date/place, death date/place, and marriage date/spouse for that specific ID — a single-record lookup, not a crawler. This keeps FamilySearch.org as a companion reference rather than a data source the app tries to replace. Because it's ID-driven and single-record, it's a much smaller, safer feature than open-ended scraping — worth doing at this scope, revisit if FamilySearch's page structure changes and breaks the parser.

---

V4.11 — Approved Scope

• Tags (supersedes "collections"): many-to-many tagging for Documents with Settings-style management (same pattern as Document Types and Person Roles), autocomplete-capable assignment, and a dedicated Tags entry point for browse/filter-by-tag workflows.
• Source Detail simplification: remove the separate Transcription Text card; show Source image + Editable Revision + Source/SourceJob metadata in a 3-column top layout, then keep Candidate Machine Transcriptions below the image/revision area.
• Integrity reconciliation checks in tests:
  • document folder count under UPLOAD_DIR/documents must equal document row count.
  • source file count under each UPLOAD_DIR/documents/{document_id} folder must equal source row count for that Document.
  • failures should include actionable mismatch details (missing row/folder or file/source mapping).
• UI table updates:
  • Archival Documents: remove Archive Ref, add # Sources.
  • Archival Entities: People: remove Display Name and Maiden Name columns, add FamilySearch ID.
  • Transcription Pipeline Jobs: add # Sources.
• Create Processing Job page: Provider and Model must be selectable for new job creation.

Deferred out of this release:

• UI theme selection.
• Settings-based .env editing and runtime controls.
• Person table structural redesign (removing/splitting name fields).

---

V5.0 — Unified Photos Table (major data-model change)

Per your feedback, this replaces the earlier "Track B" idea with a single shared model: a photos table holding an image reference plus a text description field, used by both the homepage gallery and Person portraits — rather than building two separate, one-off metadata layers.

Rough shape (subject to the further discussion you flagged before any implementation):

• photos: id, image path/reference, description, and whatever owner/context linkage is needed (e.g. a polymorphic or nullable person_id plus a context/kind discriminator such as "homepage" vs "portrait" — exact shape is an open design question, not decided here).
• Person gets multiple portraits instead of the current single portrait_path string column — needs a migration to move existing single-portrait data into the new table.
• Homepage images gain descriptions, using the same table instead of a separate sidecar/caption mechanism.
• Because this is a genuine shared-schema decision (one table, two consumers, plus a migration of existing single-portrait data), this is correctly a 5.x "major change" rather than a small add-on — flagged for a follow-up design discussion before implementation starts, specifically on the exact photos table shape and how "which photos belong to which person/context" should be modeled.

---

V6.0 — Server Hosting Migration

Your stated approach (Postgres in Docker, app in Docker, Cloudflare Tunnel) is the standard, low-maintenance way to get secure remote access without exposing ports or running your own VPN/reverse-proxy TLS setup. Sequential stages (each de-risks the next):

Stage 0 — Prerequisites

• Confirm data access already goes through the service layer (it does, per .github/instructions/services.instructions.md) — this is what makes the DB swap and later auth additive rather than a rewrite.
• Confirm upload/file storage path (settings.upload_dir) is Docker-volume-friendly.
• Confirm DB URL and upload dir are both env-overridable (.env/.env.example already exist).

Stage 1 — Containerize against the existing SQLite file first (smallest possible change) to validate the container boundary (networking, volumes, permissions) before also switching databases. Dockerfile/docker-compose.yml already exist in the repo — confirm current intent vs. extend for production.

Stage 2 — Migrate to PostgreSQL. Since the app uses SQLModel/SQLAlchemy, this should mostly be a connection string/dialect change plus a data migration script. Do this as its own isolated step so a regression is attributable to the DB swap alone. Resolve the Postgres connection through a small factory/indirection point rather than a single global engine constant — this costs nothing now and keeps the door open for per-user databases later (see Beyond, below).

Stage 3 — Expose via Cloudflare Tunnel. Add a cloudflared container pointed at the app's internal port (no public port exposure needed). Put the tunnel hostname behind Cloudflare Access (free tier, email OTP/SSO) as the first layer of remote-access auth, even before the app has its own login system — gets secure remote access working quickly and buys time to do Stage 4 and later user-auth properly instead of rushed.

Stage 4 — Backups & operational hygiene. Automated Postgres backups (scheduled pg_dump) and uploaded-file backups before exposing this to the internet — this is the point where a data-loss incident would be most damaging. Basic container health checks/restart policies in compose.

---

V6.1 — Reporting Features

Independent of hosting — can be built any time, including in parallel with V6.0:

• Person timelines: a query/aggregation feature (documents by person, ordered by document date), no hosting dependency.
• AI-written biographies/family histories: likely reuses the existing prompt/provider abstraction (services/prompts.py) already built for transcription, rather than needing new infrastructure.

---

Beyond / Pie-in-the-sky (informational only — no version assigned)

Not being built now, but the plan above is checked against these so nothing forecloses them later:

• Individual user logins + roles (admin/editor/contributor/view-only): Cloudflare Access (V6.0 Stage 3) is a stopgap, not this. When you get here, add a real User/Role model and auth middleware. Because the codebase already funnels data access through the service layer, adding a "current user + role check" later is additive — this is the reason to keep service-layer discipline intact through all the versions above, so authorization can be bolted on without touching every page.
• Per-user databases: protected for by the Stage 2 connection-factory recommendation above (schema-per-tenant or database-per-tenant becomes much easier if the DB connection is already resolved through an indirection point rather than hardcoded at startup).

---

Open items for you

• V5.0 unified photos table: needs the follow-up design discussion you flagged (exact schema, how photos link to homepage vs. person context) before implementation.
• Confirm this version numbering/grouping matches your intent before work starts.