transcription/docs/roadmap_plan.md

Roadmap Plan (Starting at V6.0)

This roadmap starts at V6.0 and tracks forward-looking work only.

V6.0 - Hosting Migration

Objective: move from local-only operation to secure, stable remote hosting. Status: Completed

Detailed plan: v6_0_hosting_migration_plan.md

Scope

1. Containerize app runtime for production deployment.
2. Run PostgreSQL in Docker and migrate from SQLite.
3. Add Cloudflare Tunnel exposure with Access protection.
4. Add operational safeguards (health checks, restart policies, backups).

Deliverables

• Production-ready docker-compose deployment for app + database + tunnel.
• Environment-based configuration for DB, uploads, prompts, and logging.
• Verified data migration path into PostgreSQL.
• Runbook updates for deploy, rollback, and backup/restore.

Exit Criteria

• /healthz reports healthy app and worker in deployed environment.
• One end-to-end document -> source -> job workflow succeeds remotely.
• Backup and restore procedure is tested.

Accomplished

1. Delivered production Docker deployment with split app/worker, postgres, and cloudflared.
2. Landed SQLite -> PostgreSQL migration tooling and runbook coverage.
3. Added production health/reliability wiring and operational runbooks for deploy/rollback/recovery.
4. Established host-visible backup workflow and restore path for PostgreSQL plus media/config assets.

V6.1 - Testing and Refinement

Objective: improve navigation and operational workflows after user feedback. Status: Completed

Scope

1. Make Document Detail the primary source-page workspace:
   • Use Source-style pan/zoom + previous/next page controls.
   • Move editable revision controls into Document Detail.
   • Move archival/system metadata to dedicated Document Info route.
2. Simplify top navigation:
   • Remove top-level Tags and Sources entries.
   • Retire the Tags page and the global Source Asset Records entry flow.
3. Improve list/detail clarity:
   • Add Document transcription status to Archival Documents list.
   • Add Document Date in People Detail -> Linked Documents table.
4. Add worker-backed Settings maintenance runs:
   • Add maintenance_run persistence (id, job_type, status, started_at, finished_at, triggered_by, summary, log_path, error_detail).
   • Add Run Backup and Run Storage Reconciliation actions that enqueue runs and execute in the worker.
   • Add run history with status, duration, summary, and log view/download.
   • Defer daily/weekly scheduling controls to V6.2.

Accomplished

1. Refactored Document Detail into the primary source-page workspace (pan/zoom viewer, previous/next page navigation, editable revision flow) and moved archival/system metadata to Document Info.
2. Simplified top navigation by removing Tags/Sources entries and retiring the Tags page/global Source Asset Records flow.
3. Improved data clarity with document transcription status in Archival Documents and Document Date in People Detail linked documents.
4. Implemented queue-backed maintenance operations (maintenance_run model/service/worker/UI) with run history and log view/download.
5. Hardened runtime settings operations in production:
   • runtime settings writes target mounted .env.production,
   • fallback write path for single-file bind mounts,
   • explicit hidden/deployment-key disclosure in Settings UI.
6. Simplified backup configuration and behavior:
   • standardized on BACKUP_DIR + BACKUP_RETENTION_DAYS,
   • backup script uses DATABASE__* persistence keys,
   • compose maps Postgres container init values from DATABASE__*,
   • env contract drift tests now guard .env.production.example.

V6.2 - GEDCOM Data Layer

Objective: introduce a genealogical data layer sourced from GEDCOM exports, bridged to existing Person records via FamilySearch ID, without disrupting document-focused Person workflows.

Scope

1. Manual .ged file upload only. No FamilySearch credentials are stored or used by the app; the user runs the third-party getmyancestors tool themselves and uploads the resulting export.
2. Four new tables: genealogy_person, genealogy_family, genealogy_family_child, and genealogy_citation (raw GEDCOM SOUR citations, reusable in a later version to record when a transcribed document itself becomes citation evidence for FamilySearch).
3. Upsert-based import keyed on FamilySearch ID (fs_id) so repeat imports update existing records in place without breaking existing Person.family_search_id links or duplicating surrogate keys.
4. Reuse the existing V6.1 worker-backed maintenance_run pattern for import runs (run history, status, summary, log view/download) rather than new infrastructure.

Deliverables

• GEDCOM parser/importer producing the four genealogy tables.
• MaintenanceJobType entry for GEDCOM import with upsert semantics and a run summary (records added/updated).
• Settings UI entry to upload a .ged file, trigger an import run, and view history.

Exit Criteria

• Importing the same .ged file twice does not duplicate or orphan data.
• Existing Person.family_search_id values continue to resolve to the correct genealogy_person row after import.
• Import run history is visible with status, duration, and summary, consistent with other maintenance runs.

V6.3 - Reporting and Genealogy-Enriched Features

Objective: improve research value with person-centric outputs, grounded in both archival documents and the V6.2 genealogical data layer.

This version is broken into five sequential sub-versions because of real dependency ordering: entity linking must exist before GEDCOM data can be targeted per-person; the Facts/Events mechanism must exist before timelines or reconciliation have anything meaningful to consume.

V6.3.1 - Manual Entity Linking

• Search/browse UI over genealogy_person to find and link a candidate match to an application Person, setting family_search_id. Linking is reversible (unlink).
• Once linked, GEDCOM vitals display alongside the Person record without requiring any schema change to Person.

V6.3.2 - Person Facts and Events

• New fact/event table capturing: person, fact type (birth/death/event/free-form), date (+raw), place, free-text description, and a link to the source document as evidence.
• Manual tagging UI while reviewing a transcribed document: select a passage, choose the person and fact type, record the date/description.
• One-time migration of existing Person.birth_date/birth_date_raw/birth_place/ death_date/death_date_raw/death_place values into fact/event rows (tagged as legacy/no-document-evidence where no source document is known), followed by retiring those six columns from Person. Birth/death become Facts/Events like any other locally-known fact, for both linked and unlinked people. Person permanently keeps last_name, given_names, biography, family_search_id, metadata_, tags, photos, and document associations.

V6.3.3 - Person Timelines

• Timeline query merging GEDCOM milestones (birth, marriage, children's births, death) for linked persons with locally recorded Facts/Events.
• Timeline UI on Person Detail with clear ordering/filters; entries link back to their originating document or GEDCOM record.

V6.3.4 - Reconciliation

• Compares Facts/Events (the real, document-evidenced local signal) against corresponding genealogy_person fields for linked persons.
• Persisted reconciliation record: person, field, local value with evidence-document link, GEDCOM value, and status (open / submitted / dismissed).
• Re-evaluated automatically as part of each GEDCOM import maintenance run: opens new discrepancies, auto-resolves ones where GEDCOM now matches, leaves others unchanged.
• Reconciliation review UI functions as a manual to-do list for updating FamilySearch; the app does not write back to FamilySearch itself.

V6.3.5 - AI-Assisted Biography Generation

• Prompted narrative generation grounded in GEDCOM facts, Facts/Events, and relevant document snippets as structured input, using existing evidence-safe prompting patterns.
• Output cites back to source documents and FamilySearch records.
• Saved/printable report presentation for review; reports do not modify archival source data.

Exit Criteria (applies across V6.3.1-V6.3.5)

• Entity links are reversible and do not alter document associations.
• Timelines are reproducible from persisted records.
• Reconciliation items always carry a link to the document evidence justifying the local value, and re-running GEDCOM import correctly opens, resolves, or leaves items unchanged.
• Narrative generation is traceable to source records and prompts.
• Reports can be reviewed without modifying archival source data.

V6.4 - Access Control and Multi-User Readiness

[ More thoughts on user accounts:

• Create a generic "view only" user that does not have the rights to alter any of the data
• Limit user accounts access to data by Tag. I have distant family members that I would want to share the transcribed data with, but they would only be interested in a subset of it. For example my Cochran cousins would have no interest in Lancaster documents, so limit the Cochra Clan cousins to view-only access to documents tagged "cochran clan" ]

Objective: prepare for managed collaboration beyond single-user operation.

Scope

1. Introduce application-level authentication.
2. Add role-based authorization (admin/editor/contributor/viewer).
3. Add audit visibility for user-attributed write actions.

Deliverables

• User identity model and login/session flow.
• Route/page/service authorization enforcement.
• Audit metadata for sensitive create/update/delete workflows.

Exit Criteria

• Unauthorized operations are blocked consistently across UI/API.
• Role policies are enforced by deterministic tests.
• User-attributed changes are visible for audit/review.

Deferred / Future Ideas (not committed scope)

Captured for later consideration, not yet scheduled to a version:

• AI-assisted entity disambiguation (kinship co-occurrence, chronological plausibility filtering) when linking document mentions to people.
• Kinship-aware @mention tagging while transcribing.
• Relationship-calculator badges (e.g., "3rd Great-Grandmother") in the document viewer.
• Interactive migration/geography mapping from GEDCOM and document place mentions.
• AI-suggested document discovery by date/location overlap with known persons.
• Ability to search within a document to find potential people to add to the People table.

Planning Notes

• Keep architecture, schema, and UI contracts synchronized in docs/ as each version lands.
• Prefer explicit schema migration over runtime compatibility write paths.
• Preserve evidence/provenance guarantees when adding new AI-powered features.
• GEDCOM/FamilySearch data is external, collaborative, and mutable; treat it as a managed cache bridged via fs_id, never as a replacement for archival evidence recorded from transcribed documents.