transcription/docs/roadmap_plan.md

# Roadmap Review: Recommendations & Version Plan

This is a review/consulting deliverable. It organizes your near-term changes,
nice-to-haves, and long-range goals into version buckets, building on
`./docs/ver4.8/feature_backlog_v4_8.md` (already-scoped image work) and
folding in the additional items from your original list.

Versioning convention (per your direction): **4.8–4.11** are incremental
feature/fix releases; **5.x and 6.x are reserved for major changes** to the
app (a shared data-model overhaul, and the hosting migration, respectively).

Final deliverable will be saved to `./docs` in the repo per your instruction.

---

## V4.8 — Bug fixes + Image Experience

**Bug fixes (do first within this release — data correctness, not features):**
1. **Stale Error Detail after resubmit.** Root cause found in `db/models.py`:
   `Source.latest_error_detail` sorts all `execution_attempts` on the latest
   job by `attempt_number` descending and returns the **first attempt with
   any `error_detail`**, even if that's an older attempt and the latest
   attempt succeeded. Fix: only report error detail from the latest attempt,
   don't fall through to earlier ones.
2. **"View Jobs" ≠ "View Sources" behavior.** `/documents/{id}/sources`
   already redirects to the filtered Transcription Pipeline Jobs page;
   `/documents/{id}/jobs` renders its own bare two-field custom page instead.
   Fix: make `/documents/{id}/jobs` redirect the same way, filtered by document.

**Image Experience (Track A from the V4.8 backlog doc — already scoped, high
value/low risk, storage layer already exists):**
3. **Homepage Image Gallery** (recommended first feature in that doc):
   `list_homepage_images()` already returns every stored image; only the
   multi-image carousel UI and optional slideshow rotation are missing.
4. **Pan and Zoom on Source Detail**: recover `document_panzoom.py` from git
   history (deleted in V4.6 Phase 5, `6a3ee26`), but vendor the Panzoom
   library locally instead of the unpkg CDN, and reuse `resolve_media_url`
   from `media_urls.py` instead of recreating its old helper. Scope to Source
   Detail only — do not add it to the shared `dark_room_viewer`.

**Note:** the backlog doc's model-performance/telemetry item still applies as
stated there — it must wait for V4.7 Phase 4 to land first, since `duration_ms`
today mixes provider latency with preprocessing/DB commit time.

---

## V4.9 — Detail Page Parity Pass

Small, low-risk structural-parity fixes across Person/Job/Document detail pages:
- Hide "Maiden Name" on Person detail when empty.
- Job Record detail "Document Links" box → mirror the Document detail page's
  "Sources & Pipeline Jobs" box (clickable name, Sources count, single
  "View Sources" button, no "+Add Job").
- Transcription Pipeline Jobs page: add a Document Name column; resolve
  Created vs. Updated by keeping **Updated** as the primary/visible field
  (more actionable) and de-emphasizing Created rather than deleting it outright.
- Person detail "Linked Documents" → 3-column table (Document Name, Role,
  Number of Pages), with clickable document names, dropping the "Open" button.

---

## V4.10 — Settings Consolidation & Small Enhancements

- **Settings page → tabs**, not sub-pages (per your confirmation). Convert the
  existing stacked cards (Document Types, Person Roles, Prompts) into
  `ui.tabs`/`ui.tab_panels`, and add a new **Home Page Text** tab. One route,
  no navigation overhead, scales cleanly as more settings are added.
- **Google Maps links** for Person birth/death locations — cheap, no
  scraping, a formatted place string becomes a Maps search link.
- **FamilySearch ID lookup** (scope confirmed as narrow): on the Person
  create/edit form, add a FamilySearch ID field and an "Auto-fill from
  FamilySearch" button that fetches only birth date/place, death date/place,
  and marriage date/spouse for that specific ID — a single-record lookup, not
  a crawler. This keeps FamilySearch.org as a companion reference rather than
  a data source the app tries to replace. Because it's ID-driven and
  single-record, it's a much smaller, safer feature than open-ended scraping —
  worth doing at this scope, revisit if FamilySearch's page structure changes
  and breaks the parser.

---

## V4.11 — Approved Scope

- **Tags** (supersedes "collections"): many-to-many tagging for Documents with
  Settings-style management (same pattern as Document Types and Person Roles),
  autocomplete-capable assignment, and a dedicated **Tags** entry point for
  browse/filter-by-tag workflows.
- **Source Detail simplification**: remove the separate **Transcription Text**
  card; show Source image + Editable Revision + Source/SourceJob metadata in a
  3-column top layout, then keep Candidate Machine Transcriptions below the
  image/revision area.
- **Integrity reconciliation checks in tests**:
  - document folder count under `UPLOAD_DIR/documents` must equal `document`
    row count.
  - source file count under each `UPLOAD_DIR/documents/{document_id}` folder
    must equal `source` row count for that Document.
  - failures should include actionable mismatch details (missing row/folder or
    file/source mapping).
- **UI table updates**:
  - Archival Documents: remove **Archive Ref**, add **# Sources**.
  - Archival Entities: People: remove **Display Name** and **Maiden Name**
    columns, add **FamilySearch ID**.
  - Transcription Pipeline Jobs: add **# Sources**.
- **Create Processing Job page**: Provider and Model must be selectable for new
  job creation.

Deferred out of this release:

- UI theme selection.
- Settings-based `.env` editing and runtime controls.
- Person table structural redesign (removing/splitting name fields).

---

## V5.0 — Unified Photos Table (major data-model change)

V5.0 standardizes homepage images and Person portraits into one shared `photo`
table and one storage layout.

Finalized shape:
- `photo`: `id`, nullable `person_id`, `path`, `description`, `is_primary`,
  timestamps.
- `person_id IS NULL` = homepage photos; non-null = Person photos.
- `is_primary` is the featured/first photo for that owner (homepage or Person).
- No separate context enum; ownership is derived from `person_id`.
- All image files are stored under `UPLOAD_DIR/photos/{photo_id}{suffix}`.
- `Person.portrait_path` is removed.
- `HOMEPAGE_DIR` is retired; homepage markdown remains file-backed at
  `UPLOAD_DIR/homepage.md`.

Migration policy for legacy installs:
- Export/import rebuild remains the migration mechanism.
- Legacy `person.portrait_path` values are backfilled into `photo` rows.
- Legacy homepage images under `UPLOAD_DIR/homepage` are backfilled into
  homepage `photo` rows.
- Legacy homepage markdown is relocated to `UPLOAD_DIR/homepage.md`.

---

## V5.1 — Person table structural redesign

- Replace `person.full_name` with split required fields:
  - `last_name`
  - `given_names` (first + middle)
- Remove `display_name` and `maiden_name` from active schema/UI.
- Keep `family_search_id` optional and unique (not required in this version).
- Add `person_tag` many-to-many links so People use the same Tag registry as Documents.
- Update Archival Entities: People list columns to:
  - Last Name
  - First & Middle
  - FamilySearch ID
  - Birth Date
  - Death Date
  - # Documents
- Migration/export-import behavior backfills split names from legacy `full_name` values.

---

## V6.0 — Server Hosting Migration

Your stated approach (Postgres in Docker, app in Docker, Cloudflare Tunnel) is
the standard, low-maintenance way to get secure remote access without exposing
ports or running your own VPN/reverse-proxy TLS setup. Sequential stages
(each de-risks the next):

**Stage 0 — Prerequisites**
- Confirm data access already goes through the service layer (it does, per
  `.github/instructions/services.instructions.md`) — this is what makes the
  DB swap and later auth additive rather than a rewrite.
- Confirm upload/file storage path (`settings.upload_dir`) is Docker-volume-friendly.
- Confirm DB URL and upload dir are both env-overridable (`.env`/`.env.example`
  already exist).

**Stage 1 — Containerize** against the *existing* SQLite file first (smallest
possible change) to validate the container boundary (networking, volumes,
permissions) before also switching databases. `Dockerfile`/`docker-compose.yml`
already exist in the repo — confirm current intent vs. extend for production.

**Stage 2 — Migrate to PostgreSQL.** Since the app uses SQLModel/SQLAlchemy,
this should mostly be a connection string/dialect change plus a data migration
script. Do this as its own isolated step so a regression is attributable to
the DB swap alone. Resolve the Postgres connection through a small
factory/indirection point rather than a single global engine constant — this
costs nothing now and keeps the door open for per-user databases later (see
Beyond, below).

**Stage 3 — Expose via Cloudflare Tunnel.** Add a `cloudflared` container
pointed at the app's internal port (no public port exposure needed). Put the
tunnel hostname behind Cloudflare Access (free tier, email OTP/SSO) as the
**first layer of remote-access auth**, even before the app has its own login
system — gets secure remote access working quickly and buys time to do
Stage 4 and later user-auth properly instead of rushed.

**Stage 4 — Backups & operational hygiene.** Automated Postgres backups
(scheduled `pg_dump`) and uploaded-file backups *before* exposing this to the
internet — this is the point where a data-loss incident would be most
damaging. Basic container health checks/restart policies in compose.

---

## V6.1 — Reporting Features

Independent of hosting — can be built any time, including in parallel with
V6.0:
- **Person timelines**: a query/aggregation feature (documents by person,
  ordered by document date), no hosting dependency.
- **AI-written biographies/family histories**: likely reuses the existing
  prompt/provider abstraction (`services/prompts.py`) already built for
  transcription, rather than needing new infrastructure.

---

## Beyond / Pie-in-the-sky (informational only — no version assigned)

Not being built now, but the plan above is checked against these so nothing
forecloses them later:

- **Individual user logins + roles** (admin/editor/contributor/view-only):
  Cloudflare Access (V6.0 Stage 3) is a stopgap, not this. When you get here,
  add a real `User`/`Role` model and auth middleware. Because the codebase
  already funnels data access through the service layer, adding a
  "current user + role check" later is additive — this is the reason to keep
  service-layer discipline intact through all the versions above, so
  authorization can be bolted on without touching every page.
- **Per-user databases**: protected for by the Stage 2 connection-factory
  recommendation above (schema-per-tenant or database-per-tenant becomes
  much easier if the DB connection is already resolved through an
  indirection point rather than hardcoded at startup).

---

## Current status
- V5.0 unified photos has been implemented with:
  - shared `photo` table (`person_id` nullable for homepage ownership),
  - flat media storage under `UPLOAD_DIR/photos/{photo_id}{suffix}`,
  - migration backfill from legacy Person portraits and homepage images,
  - homepage markdown relocated to `UPLOAD_DIR/homepage.md`.
- Version numbering/grouping is now established by implementation and can
  proceed to V5.1 planning/execution.