V4.10
Quality Gate / gate (push) Failing after 11s

This commit is contained in:
Jim Lancaster
2026-08-22 10:19:30 -05:00
parent bf2f3ac09c
commit cf49c3c127
35 changed files with 1029 additions and 161 deletions
+212
View File
@@ -0,0 +1,212 @@
# Roadmap Review: Recommendations & Version Plan
This is a review/consulting deliverable. It organizes your near-term changes,
nice-to-haves, and long-range goals into version buckets, building on
`./docs/ver4.8/feature_backlog_v4_8.md` (already-scoped image work) and
folding in the additional items from your original list.
Versioning convention (per your direction): **4.84.11** are incremental
feature/fix releases; **5.x and 6.x are reserved for major changes** to the
app (a shared data-model overhaul, and the hosting migration, respectively).
Final deliverable will be saved to `./docs` in the repo per your instruction.
---
## V4.8 — Bug fixes + Image Experience
**Bug fixes (do first within this release — data correctness, not features):**
1. **Stale Error Detail after resubmit.** Root cause found in `db/models.py`:
`Source.latest_error_detail` sorts all `execution_attempts` on the latest
job by `attempt_number` descending and returns the **first attempt with
any `error_detail`**, even if that's an older attempt and the latest
attempt succeeded. Fix: only report error detail from the latest attempt,
don't fall through to earlier ones.
2. **"View Jobs" ≠ "View Sources" behavior.** `/documents/{id}/sources`
already redirects to the filtered Transcription Pipeline Jobs page;
`/documents/{id}/jobs` renders its own bare two-field custom page instead.
Fix: make `/documents/{id}/jobs` redirect the same way, filtered by document.
**Image Experience (Track A from the V4.8 backlog doc — already scoped, high
value/low risk, storage layer already exists):**
3. **Homepage Image Gallery** (recommended first feature in that doc):
`list_homepage_images()` already returns every stored image; only the
multi-image carousel UI and optional slideshow rotation are missing.
4. **Pan and Zoom on Source Detail**: recover `document_panzoom.py` from git
history (deleted in V4.6 Phase 5, `6a3ee26`), but vendor the Panzoom
library locally instead of the unpkg CDN, and reuse `resolve_media_url`
from `media_urls.py` instead of recreating its old helper. Scope to Source
Detail only — do not add it to the shared `dark_room_viewer`.
**Note:** the backlog doc's model-performance/telemetry item still applies as
stated there — it must wait for V4.7 Phase 4 to land first, since `duration_ms`
today mixes provider latency with preprocessing/DB commit time.
---
## V4.9 — Detail Page Parity Pass
Small, low-risk structural-parity fixes across Person/Job/Document detail pages:
- Hide "Maiden Name" on Person detail when empty.
- Job Record detail "Document Links" box → mirror the Document detail page's
"Sources & Pipeline Jobs" box (clickable name, Sources count, single
"View Sources" button, no "+Add Job").
- Transcription Pipeline Jobs page: add a Document Name column; resolve
Created vs. Updated by keeping **Updated** as the primary/visible field
(more actionable) and de-emphasizing Created rather than deleting it outright.
- Person detail "Linked Documents" → 3-column table (Document Name, Role,
Number of Pages), with clickable document names, dropping the "Open" button.
---
## V4.10 — Settings Consolidation & Small Enhancements
- **Settings page → tabs**, not sub-pages (per your confirmation). Convert the
existing stacked cards (Document Types, Person Roles, Prompts) into
`ui.tabs`/`ui.tab_panels`, and add a new **Home Page Text** tab. One route,
no navigation overhead, scales cleanly as more settings are added.
- **Google Maps links** for Person birth/death locations — cheap, no
scraping, a formatted place string becomes a Maps search link.
- **FamilySearch ID lookup** (scope confirmed as narrow): on the Person
create/edit form, add a FamilySearch ID field and an "Auto-fill from
FamilySearch" button that fetches only birth date/place, death date/place,
and marriage date/spouse for that specific ID — a single-record lookup, not
a crawler. This keeps FamilySearch.org as a companion reference rather than
a data source the app tries to replace. Because it's ID-driven and
single-record, it's a much smaller, safer feature than open-ended scraping —
worth doing at this scope, revisit if FamilySearch's page structure changes
and breaks the parser.
---
## V4.11 — Bigger UI Features
- **Tags** (supersedes "collections"): many-to-many tagging for documents,
reusing the Settings registry pattern (autocomplete against existing tags,
managed like Document Types/Person Roles). Since you want dedicated UI to
browse/filter by tag, include a tag-filter view (e.g. a "Tags" entry point
showing documents grouped/filterable by tag) as part of this version, not
deferred — that's the UI surface that makes tags actually replace collections
day-to-day, not just a data field.
- **UI theme selection**: kept simple, per your confirmation — 24 curated
themes (e.g. Archival/Sepia, Light, Dark, High-contrast), a single stored
preference, swapped via existing CSS variables/Tailwind tokens in
`theme.py`. No open-ended theme builder.
- **Edit Transcription sub-page**: source image + transcription text +
editable revision side-by-side, keeping the existing Save Revision/Reset
buttons. You noted the exact interaction/layout needs to be clarified when
this is scoped — treat this as a design-first item: confirm the two/three
column breakpoint behavior and what happens on narrow windows before
building.
- **Source detail page reflow** (move Candidate Machine Transcriptions to the
bottom of the image column; align Source Metadata's top edge with
Transcription Text): do this *after* the Edit Transcription sub-page above,
since that page will reuse/rearrange the same boxes — avoids reflowing twice.
---
## V5.0 — Unified Photos Table (major data-model change)
Per your feedback, this replaces the earlier "Track B" idea with a single
shared model: a **`photos` table** holding an image reference plus a text
`description` field, used by **both** the homepage gallery and Person
portraits — rather than building two separate, one-off metadata layers.
Rough shape (subject to the further discussion you flagged before any
implementation):
- `photos`: id, image path/reference, description, and whatever
owner/context linkage is needed (e.g. a polymorphic or nullable
`person_id` plus a `context`/`kind` discriminator such as `"homepage"` vs
`"portrait"` — exact shape is an open design question, not decided here).
- Person gets **multiple portraits** instead of the current single
`portrait_path` string column — needs a migration to move existing
single-portrait data into the new table.
- Homepage images gain **descriptions**, using the same table instead of a
separate sidecar/caption mechanism.
- Because this is a genuine shared-schema decision (one table, two
consumers, plus a migration of existing single-portrait data), this is
correctly a 5.x "major change" rather than a small add-on — **flagged for
a follow-up design discussion before implementation starts**, specifically
on the exact `photos` table shape and how "which photos belong to which
person/context" should be modeled.
---
## V6.0 — Server Hosting Migration
Your stated approach (Postgres in Docker, app in Docker, Cloudflare Tunnel) is
the standard, low-maintenance way to get secure remote access without exposing
ports or running your own VPN/reverse-proxy TLS setup. Sequential stages
(each de-risks the next):
**Stage 0 — Prerequisites**
- Confirm data access already goes through the service layer (it does, per
`.github/instructions/services.instructions.md`) — this is what makes the
DB swap and later auth additive rather than a rewrite.
- Confirm upload/file storage path (`settings.upload_dir`) is Docker-volume-friendly.
- Confirm DB URL and upload dir are both env-overridable (`.env`/`.env.example`
already exist).
**Stage 1 — Containerize** against the *existing* SQLite file first (smallest
possible change) to validate the container boundary (networking, volumes,
permissions) before also switching databases. `Dockerfile`/`docker-compose.yml`
already exist in the repo — confirm current intent vs. extend for production.
**Stage 2 — Migrate to PostgreSQL.** Since the app uses SQLModel/SQLAlchemy,
this should mostly be a connection string/dialect change plus a data migration
script. Do this as its own isolated step so a regression is attributable to
the DB swap alone. Resolve the Postgres connection through a small
factory/indirection point rather than a single global engine constant — this
costs nothing now and keeps the door open for per-user databases later (see
Beyond, below).
**Stage 3 — Expose via Cloudflare Tunnel.** Add a `cloudflared` container
pointed at the app's internal port (no public port exposure needed). Put the
tunnel hostname behind Cloudflare Access (free tier, email OTP/SSO) as the
**first layer of remote-access auth**, even before the app has its own login
system — gets secure remote access working quickly and buys time to do
Stage 4 and later user-auth properly instead of rushed.
**Stage 4 — Backups & operational hygiene.** Automated Postgres backups
(scheduled `pg_dump`) and uploaded-file backups *before* exposing this to the
internet — this is the point where a data-loss incident would be most
damaging. Basic container health checks/restart policies in compose.
---
## V6.1 — Reporting Features
Independent of hosting — can be built any time, including in parallel with
V6.0:
- **Person timelines**: a query/aggregation feature (documents by person,
ordered by document date), no hosting dependency.
- **AI-written biographies/family histories**: likely reuses the existing
prompt/provider abstraction (`services/prompts.py`) already built for
transcription, rather than needing new infrastructure.
---
## Beyond / Pie-in-the-sky (informational only — no version assigned)
Not being built now, but the plan above is checked against these so nothing
forecloses them later:
- **Individual user logins + roles** (admin/editor/contributor/view-only):
Cloudflare Access (V6.0 Stage 3) is a stopgap, not this. When you get here,
add a real `User`/`Role` model and auth middleware. Because the codebase
already funnels data access through the service layer, adding a
"current user + role check" later is additive — this is the reason to keep
service-layer discipline intact through all the versions above, so
authorization can be bolted on without touching every page.
- **Per-user databases**: protected for by the Stage 2 connection-factory
recommendation above (schema-per-tenant or database-per-tenant becomes
much easier if the DB connection is already resolved through an
indirection point rather than hardcoded at startup).
---
## Open items for you
- V5.0 unified `photos` table: needs the follow-up design discussion you
flagged (exact schema, how photos link to homepage vs. person context)
before implementation.
- Confirm this version numbering/grouping matches your intent before work starts.
+3
View File
@@ -177,6 +177,9 @@ Index:
Constraint:
- `UniqueConstraint(job_id, source_id)` named `uq_job_source_job_source`
Runtime reconciliation:
- Startup database operations remove retired V4.6 `job_source` evidence columns (`raw_transcription`, `ai_metadata`, `raw_api_response`, `error_detail`, `executed_at`) when present so persisted schema matches this contract.
### `ExecutionAttempt`
| Field | Type | Notes |
+1
View File
@@ -21,6 +21,7 @@ The application root and `/ui` redirect to `/ui/homepage`.
- Missing text displays `No homepage text saved yet.`
- Missing image displays the viewer's empty state.
- **Edit Home Page** opens the edit route.
- The same Home Text content is also editable from **Settings → Home Page Text**.
## Edit Behavior
+4 -3
View File
@@ -19,8 +19,9 @@ Jobs manages transcription processing runs. A Job belongs to one Document, links
- The title is **Transcription Pipeline Jobs**.
- **Create job** opens Job creation and **Refresh** reloads the table.
- Columns are Job ID, Status, Source Filename, Retries, Created, and Updated.
- Search covers Job ID, filename, and status.
- Columns are Job ID, Status, Document Name, Retries, and Updated.
- Updated is the primary date/sort field.
- Search covers Job ID, document name, and status.
- Status is displayed as a semantic status chip.
- Selecting a row opens Job Detail.
- No records displays `No job records found in repository.`
@@ -43,7 +44,7 @@ Jobs manages transcription processing runs. A Job belongs to one Document, links
- The heading shows Job ID and a status badge.
- Execution Logistics shows provider, model, prompt, retry count, and last update.
- Document Links open the parent Document and Job-filtered Sources.
- Document Links show a clickable Document Name, Sources count, and a single **View Sources** action using document filtering.
- Queued and processing Jobs show an auto-refresh notice and reload every four seconds.
- Polling stops when the Job becomes terminal or a refresh fails.
- Queued and processing Jobs expose **Cancel**.
+5 -2
View File
@@ -54,9 +54,12 @@ Rules:
- The header provides **New Document**, **Edit Person**, and **Delete**.
- **New Document** opens Document creation with this Person requested for author preselection.
- The portrait viewer resolves supported relative upload paths and absolute HTTP/data URLs.
- Biographical Record shows names, compact birth/death dates, places, and an **Open in FamilySearch** link when an ID exists.
- Biographical Record shows names, compact birth/death dates, and places.
- Birth and death place values are clickable links to Google Maps when present.
- FamilySearch ID is shown as a metadata value and is clickable to the FamilySearch person details route when present.
- Maiden Name is only shown in Biographical Record when a value exists.
- Biography has an explicit empty value.
- Linked Documents show Document name, relationship role, and an action to open Document Detail.
- Linked Documents render as a table with **Document Name**, **Role**, and **Number of Pages**; selecting a row opens Document Detail.
- No links shows both an empty state and guidance to link from a Document workflow.
- System Logistics shows created and updated timestamps.
+35
View File
@@ -0,0 +1,35 @@
# Settings Page Contract
## Purpose
Settings manages installation-local registries and editable text assets from one route.
## Route
| Route | Purpose |
| --- | --- |
| `/settings` | Manage Document Types, Person Roles, Prompts, and Home Page Text. |
## Behavior
- The page title is **Settings**.
- Configuration surfaces are grouped as tabs:
- **Document Types**
- **Person Roles**
- **Prompts**
- **Home Page Text**
- Document Types and Person Roles support Add/Edit/Delete with existing guardrails.
- Prompts exposes only `transcribe_document.md` for editing and restore-from-backup.
- Home Page Text edits the same Markdown content rendered on `/homepage`.
## Acceptance Checklist
- `/ui/settings` renders all four tabs.
- Registry and prompt workflows keep existing validation and error handling.
- Saving Home Page Text persists content for the homepage view.
## Implementation Anchors
- `src/transcription/ui/pages/settings_page.py`
- `src/transcription/ui/homepage_store.py`
- `tests/ui/test_pages_registration.py`
-25
View File
@@ -64,24 +64,6 @@ Homepage images are **filesystem-only with no metadata store**, so a caption has
This is cheap **only if** item 3 is being done at the same time, since both need the same metadata layer. Designing that layer twice would be wasteful; design it once or not at all.
### 5. Model-Performance Rollup (V4.6 review log [54])
**Practicality: high, but blocked. Effort: M.**
Run-time telemetry is already captured and is per page: `execution_attempt.duration_ms` is a required non-null field written on all three paths in `workflows.py` (success 278, `TimeoutError` 295, general failure 330), with failures using a monotonic clock. Verified against the live database: 80 rows across 80 distinct (job, source, attempt) combinations, one row per page - the largest job has 60 attempts across 60 distinct pages - and zero nulls. Token counts live on the same row in `normalized_metadata.usage`, so tokens-per-second is already derivable without a join.
What is missing is **aggregation**. The figure is visible only for the latest attempt of one source at a time (`sources_page.py:400`), rendered raw as `"27612 ms"`. There is no rollup by model, prompt, or document.
The gap is concrete: calibrating the provider timeout on 2026-08-18 required hand-written SQL against the database, because the application could not answer "which model is slow."
Proposed shape: median / p95 / max duration, tokens per second, and a timeout rate, grouped by model. **Blocked on V4.7 Phase 4.**
### 6. Desaturated Background Wallpaper
**Practicality: low. Recommendation: do not build, or gate behind a setting defaulted off.**
Trivial to implement (`ui.add_css` with a CSS `filter`), but this is a dense archival data application - transcripts, JSON evidence panels, data tables. A background image behind all of that costs contrast and legibility on every page, for aesthetic gain only.
## Suggested Grouping
If V4.8 is scoped as one release, the natural split is:
@@ -98,12 +80,5 @@ Item 6 is not recommended.
- Should Track B happen at all, or is one portrait per person sufficient?
- Should the slideshow interval be configurable, or fixed?
- Should the model-performance rollup be its own page, or a panel on an existing one?
- Should vendored Panzoom be committed to the repository, or fetched at build time?
## Related Local References
- [Architecture & Code Review Report](../architecture_code_review_2026-08-17.md)
- `.github/instructions/ui.instructions.md`
- `src/transcription/ui/homepage_store.py` - existing multi-image storage
- `src/transcription/ui/components/media_urls.py` - canonical URL resolution