transcription/docs/roadmap_plan.md

# Roadmap Plan (Starting at V6.0)

This roadmap starts at **V6.0** and tracks forward-looking work only.

## V6.0 - Hosting Migration

Objective: move from local-only operation to secure, stable remote hosting.
Status: **Completed**

Detailed plan: [`v6_0_hosting_migration_plan.md`](v6_0_hosting_migration_plan.md)

### Scope
1. Containerize app runtime for production deployment.
2. Run PostgreSQL in Docker and migrate from SQLite.
3. Add Cloudflare Tunnel exposure with Access protection.
4. Add operational safeguards (health checks, restart policies, backups).

### Deliverables
- Production-ready `docker-compose` deployment for app + database + tunnel.
- Environment-based configuration for DB, uploads, prompts, and logging.
- Verified data migration path into PostgreSQL.
- Runbook updates for deploy, rollback, and backup/restore.

### Exit Criteria
- `/healthz` reports healthy app and worker in deployed environment.
- One end-to-end document -> source -> job workflow succeeds remotely.
- Backup and restore procedure is tested.

### Accomplished
1. Delivered production Docker deployment with split `app`/`worker`, `postgres`, and `cloudflared`.
2. Landed SQLite -> PostgreSQL migration tooling and runbook coverage.
3. Added production health/reliability wiring and operational runbooks for deploy/rollback/recovery.
4. Established host-visible backup workflow and restore path for PostgreSQL plus media/config assets.

## V6.1 - Testing and Refinement

Objective: improve navigation and operational workflows after user feedback.
Status: **Completed**

### Scope
1. Make Document Detail the primary source-page workspace:
   - Use Source-style pan/zoom + previous/next page controls.
   - Move editable revision controls into Document Detail.
   - Move archival/system metadata to dedicated Document Info route.
2. Simplify top navigation:
   - Remove top-level Tags and Sources entries.
   - Retire the Tags page and the global Source Asset Records entry flow.
3. Improve list/detail clarity:
   - Add Document transcription status to Archival Documents list.
   - Add Document Date in People Detail -> Linked Documents table.
4. Add worker-backed Settings maintenance runs:
   - Add `maintenance_run` persistence (`id`, `job_type`, `status`, `started_at`, `finished_at`, `triggered_by`, `summary`, `log_path`, `error_detail`).
   - Add Run Backup and Run Storage Reconciliation actions that enqueue runs and execute in the worker.
   - Add run history with status, duration, summary, and log view/download.
   - Defer daily/weekly scheduling controls to V6.2.

### Accomplished
1. Refactored Document Detail into the primary source-page workspace (pan/zoom viewer, previous/next page navigation, editable revision flow) and moved archival/system metadata to Document Info.
2. Simplified top navigation by removing Tags/Sources entries and retiring the Tags page/global Source Asset Records flow.
3. Improved data clarity with document transcription status in Archival Documents and Document Date in People Detail linked documents.
4. Implemented queue-backed maintenance operations (`maintenance_run` model/service/worker/UI) with run history and log view/download.
5. Hardened runtime settings operations in production:
   - runtime settings writes target mounted `.env.production`,
   - fallback write path for single-file bind mounts,
   - explicit hidden/deployment-key disclosure in Settings UI.
6. Simplified backup configuration and behavior:
   - standardized on `BACKUP_DIR` + `BACKUP_RETENTION_DAYS`,
   - backup script uses `DATABASE__*` persistence keys,
   - compose maps Postgres container init values from `DATABASE__*`,
   - env contract drift tests now guard `.env.production.example`.

## V6.2 - GEDCOM Data Layer

Objective: introduce a genealogical data layer sourced from GEDCOM exports, bridged to
existing `Person` records via FamilySearch ID, without disrupting document-focused Person
workflows.

### Scope
1. Manual `.ged` file upload only. No FamilySearch credentials are stored or used by the
   app; the user runs the third-party `getmyancestors` tool themselves and uploads the
   resulting export.
2. Four new tables: `genealogy_person`, `genealogy_family`, `genealogy_family_child`, and
   `genealogy_citation` (raw GEDCOM `SOUR` citations, reusable in a later version to record
   when a transcribed document itself becomes citation evidence for FamilySearch).
3. Upsert-based import keyed on FamilySearch ID (`fs_id`) so repeat imports update existing
   records in place without breaking existing `Person.family_search_id` links or duplicating
   surrogate keys.
4. Reuse the existing V6.1 worker-backed `maintenance_run` pattern for import runs (run
   history, status, summary, log view/download) rather than new infrastructure.

### Deliverables
- GEDCOM parser/importer producing the four genealogy tables.
- `MaintenanceJobType` entry for GEDCOM import with upsert semantics and a run summary
  (records added/updated).
- Settings UI entry to upload a `.ged` file, trigger an import run, and view history.

### Exit Criteria
- Importing the same `.ged` file twice does not duplicate or orphan data.
- Existing `Person.family_search_id` values continue to resolve to the correct
  `genealogy_person` row after import.
- Import run history is visible with status, duration, and summary, consistent with other
  maintenance runs.

## V6.3 - Reporting and Genealogy-Enriched Features

Objective: improve research value with person-centric outputs, grounded in both archival
documents and the V6.2 genealogical data layer.

This version is broken into five sequential sub-versions because of real dependency
ordering: entity linking must exist before GEDCOM data can be targeted per-person; the
Facts/Events mechanism must exist before timelines or reconciliation have anything
meaningful to consume.

### V6.3.1 - Manual Entity Linking

- Search/browse UI over `genealogy_person` to find and link a candidate match to an
  application `Person`, setting `family_search_id`. Linking is reversible (unlink).
- Once linked, GEDCOM vitals display alongside the `Person` record without requiring any
  schema change to `Person`.

### V6.3.2 - Person Facts and Events

- New fact/event table capturing: person, fact type (birth/death/event/free-form), date
  (+raw), place, free-text description, and a link to the source document as evidence.
- Manual tagging UI while reviewing a transcribed document: select a passage, choose the
  person and fact type, record the date/description.
- One-time migration of existing `Person.birth_date`/`birth_date_raw`/`birth_place`/
  `death_date`/`death_date_raw`/`death_place` values into fact/event rows (tagged as
  legacy/no-document-evidence where no source document is known), followed by retiring those
  six columns from `Person`. Birth/death become Facts/Events like any other locally-known
  fact, for both linked and unlinked people. `Person` permanently keeps `last_name`,
  `given_names`, `biography`, `family_search_id`, `metadata_`, tags, photos, and document
  associations.

### V6.3.3 - Person Timelines

- Timeline query merging GEDCOM milestones (birth, marriage, children's births, death) for
  linked persons with locally recorded Facts/Events.
- Timeline UI on Person Detail with clear ordering/filters; entries link back to their
  originating document or GEDCOM record.

### V6.3.4 - Reconciliation

- Compares Facts/Events (the real, document-evidenced local signal) against corresponding
  `genealogy_person` fields for linked persons.
- Persisted reconciliation record: person, field, local value with evidence-document link,
  GEDCOM value, and status (open / submitted / dismissed).
- Re-evaluated automatically as part of each GEDCOM import maintenance run: opens new
  discrepancies, auto-resolves ones where GEDCOM now matches, leaves others unchanged.
- Reconciliation review UI functions as a manual to-do list for updating FamilySearch; the
  app does not write back to FamilySearch itself.

### V6.3.5 - AI-Assisted Biography Generation

- Prompted narrative generation grounded in GEDCOM facts, Facts/Events, and relevant
  document snippets as structured input, using existing evidence-safe prompting patterns.
- Output cites back to source documents and FamilySearch records.
- Saved/printable report presentation for review; reports do not modify archival source
  data.

### Exit Criteria (applies across V6.3.1-V6.3.5)
- Entity links are reversible and do not alter document associations.
- Timelines are reproducible from persisted records.
- Reconciliation items always carry a link to the document evidence justifying the local
  value, and re-running GEDCOM import correctly opens, resolves, or leaves items unchanged.
- Narrative generation is traceable to source records and prompts.
- Reports can be reviewed without modifying archival source data.

## V6.4 - Access Control and Multi-User Readiness

[ *More thoughts on user accounts:*
* *Create a generic "view only" user that does not have the rights to alter any of the data*
* *Limit user accounts access to data by Tag. I have distant family members that I would want to share the transcribed data with, but they would only be interested in a subset of it. For example my Cochran cousins would have no interest in Lancaster documents, so limit the Cochra Clan cousins to view-only access to documents tagged "cochran clan"* ]

Objective: prepare for managed collaboration beyond single-user operation.

### Scope
1. Introduce application-level authentication.
2. Add role-based authorization (admin/editor/contributor/viewer).
3. Add audit visibility for user-attributed write actions.

### Deliverables
- User identity model and login/session flow.
- Route/page/service authorization enforcement.
- Audit metadata for sensitive create/update/delete workflows.

### Exit Criteria
- Unauthorized operations are blocked consistently across UI/API.
- Role policies are enforced by deterministic tests.
- User-attributed changes are visible for audit/review.

## Deferred / Future Ideas (not committed scope)

Captured for later consideration, not yet scheduled to a version:
* AI-assisted entity disambiguation (kinship co-occurrence, chronological plausibility
  filtering) when linking document mentions to people.
* Kinship-aware `@mention` tagging while transcribing.
* Relationship-calculator badges (e.g., "3rd Great-Grandmother") in the document viewer.
* Interactive migration/geography mapping from GEDCOM and document place mentions.
* AI-suggested document discovery by date/location overlap with known persons.
* Ability to search within a document to find potential people to add to the People table.

## Planning Notes

- Keep architecture, schema, and UI contracts synchronized in `docs/` as each version lands.
- Prefer explicit schema migration over runtime compatibility write paths.
- Preserve evidence/provenance guarantees when adding new AI-powered features.
- GEDCOM/FamilySearch data is external, collaborative, and mutable; treat it as a managed
  cache bridged via `fs_id`, never as a replacement for archival evidence recorded from
  transcribed documents.