generated from john/python-template
This commit is contained in:
Jim Lancaster
+63
-216
@@ -1,239 +1,86 @@
|
||||
# Roadmap Review: Recommendations & Version Plan
|
||||
# Roadmap Plan (Starting at V6.0)
|
||||
|
||||
This is a review/consulting deliverable. It organizes your near-term changes,
|
||||
nice-to-haves, and long-range goals into version buckets, building on
|
||||
`./docs/ver4.8/feature_backlog_v4_8.md` (already-scoped image work) and
|
||||
folding in the additional items from your original list.
|
||||
This roadmap starts at **V6.0** and tracks forward-looking work only.
|
||||
|
||||
Versioning convention (per your direction): **4.8–4.11** are incremental
|
||||
feature/fix releases; **5.x and 6.x are reserved for major changes** to the
|
||||
app (a shared data-model overhaul, and the hosting migration, respectively).
|
||||
## V6.0 - Hosting Migration
|
||||
|
||||
Final deliverable will be saved to `./docs` in the repo per your instruction.
|
||||
Objective: move from local-only operation to secure, stable remote hosting.
|
||||
|
||||
---
|
||||
### Scope
|
||||
1. Containerize app runtime for production deployment.
|
||||
2. Run PostgreSQL in Docker and migrate from SQLite.
|
||||
3. Add Cloudflare Tunnel exposure with Access protection.
|
||||
4. Add operational safeguards (health checks, restart policies, backups).
|
||||
|
||||
## V4.8 — Bug fixes + Image Experience
|
||||
### Deliverables
|
||||
- Production-ready `docker-compose` deployment for app + database + tunnel.
|
||||
- Environment-based configuration for DB, uploads, prompts, and logging.
|
||||
- Verified data migration path into PostgreSQL.
|
||||
- Runbook updates for deploy, rollback, and backup/restore.
|
||||
|
||||
**Bug fixes (do first within this release — data correctness, not features):**
|
||||
1. **Stale Error Detail after resubmit.** Root cause found in `db/models.py`:
|
||||
`Source.latest_error_detail` sorts all `execution_attempts` on the latest
|
||||
job by `attempt_number` descending and returns the **first attempt with
|
||||
any `error_detail`**, even if that's an older attempt and the latest
|
||||
attempt succeeded. Fix: only report error detail from the latest attempt,
|
||||
don't fall through to earlier ones.
|
||||
2. **"View Jobs" ≠ "View Sources" behavior.** `/documents/{id}/sources`
|
||||
already redirects to the filtered Transcription Pipeline Jobs page;
|
||||
`/documents/{id}/jobs` renders its own bare two-field custom page instead.
|
||||
Fix: make `/documents/{id}/jobs` redirect the same way, filtered by document.
|
||||
### Exit Criteria
|
||||
- `/healthz` reports healthy app and worker in deployed environment.
|
||||
- One end-to-end document -> source -> job workflow succeeds remotely.
|
||||
- Backup and restore procedure is tested.
|
||||
|
||||
**Image Experience (Track A from the V4.8 backlog doc — already scoped, high
|
||||
value/low risk, storage layer already exists):**
|
||||
3. **Homepage Image Gallery** (recommended first feature in that doc):
|
||||
`list_homepage_images()` already returns every stored image; only the
|
||||
multi-image carousel UI and optional slideshow rotation are missing.
|
||||
4. **Pan and Zoom on Source Detail**: recover `document_panzoom.py` from git
|
||||
history (deleted in V4.6 Phase 5, `6a3ee26`), but vendor the Panzoom
|
||||
library locally instead of the unpkg CDN, and reuse `resolve_media_url`
|
||||
from `media_urls.py` instead of recreating its old helper. Scope to Source
|
||||
Detail only — do not add it to the shared `dark_room_viewer`.
|
||||
## V6.1 - Reporting Features
|
||||
|
||||
**Note:** the backlog doc's model-performance/telemetry item still applies as
|
||||
stated there — it must wait for V4.7 Phase 4 to land first, since `duration_ms`
|
||||
today mixes provider latency with preprocessing/DB commit time.
|
||||
Objective: improve research value with person-centric outputs.
|
||||
|
||||
---
|
||||
### Scope
|
||||
1. Person timeline views using document dates and linked records.
|
||||
2. AI-assisted biography/family-history generation from curated sources.
|
||||
3. Exportable report views (human-readable, print-oriented).
|
||||
|
||||
## V4.9 — Detail Page Parity Pass
|
||||
### Deliverables
|
||||
- Timeline UI and service queries with clear ordering/filters.
|
||||
- Prompted narrative generation workflow using existing evidence-safe patterns.
|
||||
- Saved/printable report presentation for review and sharing.
|
||||
|
||||
Small, low-risk structural-parity fixes across Person/Job/Document detail pages:
|
||||
- Hide "Maiden Name" on Person detail when empty.
|
||||
- Job Record detail "Document Links" box → mirror the Document detail page's
|
||||
"Sources & Pipeline Jobs" box (clickable name, Sources count, single
|
||||
"View Sources" button, no "+Add Job").
|
||||
- Transcription Pipeline Jobs page: add a Document Name column; resolve
|
||||
Created vs. Updated by keeping **Updated** as the primary/visible field
|
||||
(more actionable) and de-emphasizing Created rather than deleting it outright.
|
||||
- Person detail "Linked Documents" → 3-column table (Document Name, Role,
|
||||
Number of Pages), with clickable document names, dropping the "Open" button.
|
||||
### Exit Criteria
|
||||
- Timelines are reproducible from persisted records.
|
||||
- Narrative generation is traceable to source records and prompts.
|
||||
- Reports can be reviewed without modifying archival source data.
|
||||
|
||||
---
|
||||
## V6.2 - Access Control and Multi-User Readiness
|
||||
|
||||
## V4.10 — Settings Consolidation & Small Enhancements
|
||||
Objective: prepare for managed collaboration beyond single-user operation.
|
||||
|
||||
- **Settings page → tabs**, not sub-pages (per your confirmation). Convert the
|
||||
existing stacked cards (Document Types, Person Roles, Prompts) into
|
||||
`ui.tabs`/`ui.tab_panels`, and add a new **Home Page Text** tab. One route,
|
||||
no navigation overhead, scales cleanly as more settings are added.
|
||||
- **Google Maps links** for Person birth/death locations — cheap, no
|
||||
scraping, a formatted place string becomes a Maps search link.
|
||||
- **FamilySearch ID lookup** (scope confirmed as narrow): on the Person
|
||||
create/edit form, add a FamilySearch ID field and an "Auto-fill from
|
||||
FamilySearch" button that fetches only birth date/place, death date/place,
|
||||
and marriage date/spouse for that specific ID — a single-record lookup, not
|
||||
a crawler. This keeps FamilySearch.org as a companion reference rather than
|
||||
a data source the app tries to replace. Because it's ID-driven and
|
||||
single-record, it's a much smaller, safer feature than open-ended scraping —
|
||||
worth doing at this scope, revisit if FamilySearch's page structure changes
|
||||
and breaks the parser.
|
||||
### Scope
|
||||
1. Introduce application-level authentication.
|
||||
2. Add role-based authorization (admin/editor/contributor/viewer).
|
||||
3. Add audit visibility for user-attributed write actions.
|
||||
|
||||
---
|
||||
### Deliverables
|
||||
- User identity model and login/session flow.
|
||||
- Route/page/service authorization enforcement.
|
||||
- Audit metadata for sensitive create/update/delete workflows.
|
||||
|
||||
## V4.11 — Approved Scope
|
||||
### Exit Criteria
|
||||
- Unauthorized operations are blocked consistently across UI/API.
|
||||
- Role policies are enforced by deterministic tests.
|
||||
- User-attributed changes are visible for audit/review.
|
||||
|
||||
- **Tags** (supersedes "collections"): many-to-many tagging for Documents with
|
||||
Settings-style management (same pattern as Document Types and Person Roles),
|
||||
autocomplete-capable assignment, and a dedicated **Tags** entry point for
|
||||
browse/filter-by-tag workflows.
|
||||
- **Source Detail simplification**: remove the separate **Transcription Text**
|
||||
card; show Source image + Editable Revision + Source/SourceJob metadata in a
|
||||
3-column top layout, then keep Candidate Machine Transcriptions below the
|
||||
image/revision area.
|
||||
- **Integrity reconciliation checks in tests**:
|
||||
- document folder count under `UPLOAD_DIR/documents` must equal `document`
|
||||
row count.
|
||||
- source file count under each `UPLOAD_DIR/documents/{document_id}` folder
|
||||
must equal `source` row count for that Document.
|
||||
- failures should include actionable mismatch details (missing row/folder or
|
||||
file/source mapping).
|
||||
- **UI table updates**:
|
||||
- Archival Documents: remove **Archive Ref**, add **# Sources**.
|
||||
- Archival Entities: People: remove **Display Name** and **Maiden Name**
|
||||
columns, add **FamilySearch ID**.
|
||||
- Transcription Pipeline Jobs: add **# Sources**.
|
||||
- **Create Processing Job page**: Provider and Model must be selectable for new
|
||||
job creation.
|
||||
## V6.3 - Scalability and Multi-Tenant Direction (Optional)
|
||||
|
||||
Deferred out of this release:
|
||||
Objective: keep architecture ready for broader deployment footprints.
|
||||
|
||||
- UI theme selection.
|
||||
- Settings-based `.env` editing and runtime controls.
|
||||
- Person table structural redesign (removing/splitting name fields).
|
||||
### Scope
|
||||
1. Evaluate per-tenant or per-user data partitioning strategy.
|
||||
2. Formalize connection/runtime strategy for tenant-aware DB selection.
|
||||
3. Expand operational telemetry for throughput and cost monitoring.
|
||||
|
||||
---
|
||||
### Deliverables
|
||||
- Decision document for tenancy model and migration strategy.
|
||||
- Prototype-safe runtime boundary for selecting data targets.
|
||||
- Monitoring baseline for queue depth, job latency, and provider cost trends.
|
||||
|
||||
## V5.0 — Unified Photos Table (major data-model change)
|
||||
### Exit Criteria
|
||||
- Selected tenancy strategy is documented and testable.
|
||||
- Operational metrics support capacity planning.
|
||||
|
||||
V5.0 standardizes homepage images and Person portraits into one shared `photo`
|
||||
table and one storage layout.
|
||||
## Planning Notes
|
||||
|
||||
Finalized shape:
|
||||
- `photo`: `id`, nullable `person_id`, `path`, `description`, `is_primary`,
|
||||
timestamps.
|
||||
- `person_id IS NULL` = homepage photos; non-null = Person photos.
|
||||
- `is_primary` is the featured/first photo for that owner (homepage or Person).
|
||||
- No separate context enum; ownership is derived from `person_id`.
|
||||
- All image files are stored under `UPLOAD_DIR/photos/{photo_id}{suffix}`.
|
||||
- `Person.portrait_path` is removed.
|
||||
- `HOMEPAGE_DIR` is retired; homepage markdown remains file-backed at
|
||||
`UPLOAD_DIR/homepage.md`.
|
||||
|
||||
Migration policy for legacy installs:
|
||||
- Export/import rebuild remains the migration mechanism.
|
||||
- Legacy `person.portrait_path` values are backfilled into `photo` rows.
|
||||
- Legacy homepage images under `UPLOAD_DIR/homepage` are backfilled into
|
||||
homepage `photo` rows.
|
||||
- Legacy homepage markdown is relocated to `UPLOAD_DIR/homepage.md`.
|
||||
|
||||
---
|
||||
|
||||
## V5.1 — Person table structural redesign
|
||||
|
||||
- Replace `person.full_name` with split required fields:
|
||||
- `last_name`
|
||||
- `given_names` (first + middle)
|
||||
- Remove `display_name` and `maiden_name` from active schema/UI.
|
||||
- Keep `family_search_id` optional and unique (not required in this version).
|
||||
- Add `person_tag` many-to-many links so People use the same Tag registry as Documents.
|
||||
- Update Archival Entities: People list columns to:
|
||||
- Last Name
|
||||
- First & Middle
|
||||
- FamilySearch ID
|
||||
- Birth Date
|
||||
- Death Date
|
||||
- # Documents
|
||||
- Migration/export-import behavior backfills split names from legacy `full_name` values.
|
||||
|
||||
---
|
||||
|
||||
## V6.0 — Server Hosting Migration
|
||||
|
||||
Your stated approach (Postgres in Docker, app in Docker, Cloudflare Tunnel) is
|
||||
the standard, low-maintenance way to get secure remote access without exposing
|
||||
ports or running your own VPN/reverse-proxy TLS setup. Sequential stages
|
||||
(each de-risks the next):
|
||||
|
||||
**Stage 0 — Prerequisites**
|
||||
- Confirm data access already goes through the service layer (it does, per
|
||||
`.github/instructions/services.instructions.md`) — this is what makes the
|
||||
DB swap and later auth additive rather than a rewrite.
|
||||
- Confirm upload/file storage path (`settings.upload_dir`) is Docker-volume-friendly.
|
||||
- Confirm DB URL and upload dir are both env-overridable (`.env`/`.env.example`
|
||||
already exist).
|
||||
|
||||
**Stage 1 — Containerize** against the *existing* SQLite file first (smallest
|
||||
possible change) to validate the container boundary (networking, volumes,
|
||||
permissions) before also switching databases. `Dockerfile`/`docker-compose.yml`
|
||||
already exist in the repo — confirm current intent vs. extend for production.
|
||||
|
||||
**Stage 2 — Migrate to PostgreSQL.** Since the app uses SQLModel/SQLAlchemy,
|
||||
this should mostly be a connection string/dialect change plus a data migration
|
||||
script. Do this as its own isolated step so a regression is attributable to
|
||||
the DB swap alone. Resolve the Postgres connection through a small
|
||||
factory/indirection point rather than a single global engine constant — this
|
||||
costs nothing now and keeps the door open for per-user databases later (see
|
||||
Beyond, below).
|
||||
|
||||
**Stage 3 — Expose via Cloudflare Tunnel.** Add a `cloudflared` container
|
||||
pointed at the app's internal port (no public port exposure needed). Put the
|
||||
tunnel hostname behind Cloudflare Access (free tier, email OTP/SSO) as the
|
||||
**first layer of remote-access auth**, even before the app has its own login
|
||||
system — gets secure remote access working quickly and buys time to do
|
||||
Stage 4 and later user-auth properly instead of rushed.
|
||||
|
||||
**Stage 4 — Backups & operational hygiene.** Automated Postgres backups
|
||||
(scheduled `pg_dump`) and uploaded-file backups *before* exposing this to the
|
||||
internet — this is the point where a data-loss incident would be most
|
||||
damaging. Basic container health checks/restart policies in compose.
|
||||
|
||||
---
|
||||
|
||||
## V6.1 — Reporting Features
|
||||
|
||||
Independent of hosting — can be built any time, including in parallel with
|
||||
V6.0:
|
||||
- **Person timelines**: a query/aggregation feature (documents by person,
|
||||
ordered by document date), no hosting dependency.
|
||||
- **AI-written biographies/family histories**: likely reuses the existing
|
||||
prompt/provider abstraction (`services/prompts.py`) already built for
|
||||
transcription, rather than needing new infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## Beyond / Pie-in-the-sky (informational only — no version assigned)
|
||||
|
||||
Not being built now, but the plan above is checked against these so nothing
|
||||
forecloses them later:
|
||||
|
||||
- **Individual user logins + roles** (admin/editor/contributor/view-only):
|
||||
Cloudflare Access (V6.0 Stage 3) is a stopgap, not this. When you get here,
|
||||
add a real `User`/`Role` model and auth middleware. Because the codebase
|
||||
already funnels data access through the service layer, adding a
|
||||
"current user + role check" later is additive — this is the reason to keep
|
||||
service-layer discipline intact through all the versions above, so
|
||||
authorization can be bolted on without touching every page.
|
||||
- **Per-user databases**: protected for by the Stage 2 connection-factory
|
||||
recommendation above (schema-per-tenant or database-per-tenant becomes
|
||||
much easier if the DB connection is already resolved through an
|
||||
indirection point rather than hardcoded at startup).
|
||||
|
||||
---
|
||||
|
||||
## Current status
|
||||
- V5.0 unified photos has been implemented with:
|
||||
- shared `photo` table (`person_id` nullable for homepage ownership),
|
||||
- flat media storage under `UPLOAD_DIR/photos/{photo_id}{suffix}`,
|
||||
- migration backfill from legacy Person portraits and homepage images,
|
||||
- homepage markdown relocated to `UPLOAD_DIR/homepage.md`.
|
||||
- Version numbering/grouping is now established by implementation and can
|
||||
proceed to V5.1 planning/execution.
|
||||
- Keep architecture, schema, and UI contracts synchronized in `docs/` as each version lands.
|
||||
- Prefer explicit schema migration over runtime compatibility write paths.
|
||||
- Preserve evidence/provenance guarantees when adding new AI-powered features.
|
||||
|
||||
Reference in New Issue
Block a user