From 63373bf24d11c675cf00d49f2b1affbaf1b8e421 Mon Sep 17 00:00:00 2001 From: Jim Lancaster <40281233+zoltan57@users.noreply.github.com> Date: Sat, 15 Aug 2026 14:04:58 -0500 Subject: [PATCH] V4.4 Scope defined (part 2) --- docs/ver4.4/implementation_plan_v4_4.md | 188 +++++++++++++++++++ docs/ver4.4/scope_boundary_v4_4.md | 232 ++++++++++++++++++++++++ 2 files changed, 420 insertions(+) create mode 100644 docs/ver4.4/implementation_plan_v4_4.md create mode 100644 docs/ver4.4/scope_boundary_v4_4.md diff --git a/docs/ver4.4/implementation_plan_v4_4.md b/docs/ver4.4/implementation_plan_v4_4.md new file mode 100644 index 0000000..2b0ce11 --- /dev/null +++ b/docs/ver4.4/implementation_plan_v4_4.md @@ -0,0 +1,188 @@ +# Implementation Plan (Version 4.4) + +## Goal + +Deliver hidden semantic identity for built-in registries, a single atomic Linked People workflow, and safe browser-native printing of archival Documents and their current transcriptions. + +## Planning Constraints + +- V4.3 is the completed implementation baseline. +- V4.4 may replace V4/V4.3 registry and document-person contracts only as specified by the V4.4 scope. +- Semantic keys are internal and immutable; UI and public API contracts use UUIDs and labels. +- Document and link edits must not partially commit. +- Print output must not execute stored text or expose machine-local source paths. +- Source page reordering remains excluded. +- Database, integration, and UI tests must use confirmed isolated data and never modify `data/transcription.db`. +- Potentially destructive tests must run only through `tools/run_destructive_tests.py`. + +## Expected Project Impact + +| Area | Expected impact | +| --- | --- | +| Models and schema bootstrap | Add nullable unique semantic keys, simplify document-person identity, and seed frozen built-ins. | +| Document service | Maintain built-in Document Types, usage summaries, UUID assignment, and atomic Document/link writes. | +| People service | Maintain built-in Person Roles, link summaries, UUID-only role assignment, and one-person-per-document enforcement. | +| V4 document API | Remove role-code selectors and compatibility role fields; enforce UUID-only relationship writes. | +| Settings UI | Use matching table workflows for Document Types and Person Roles. | +| Document Create/Edit | Replace role-specific multiselects with one staged Linked People table and inline editor. | +| Document Detail/printing | Add format selection, print preview, safe Source media rendering, print CSS, and job metadata. | +| Tests and documentation | Replace superseded cardinality/identity assertions and add isolated registry, editor, transaction, and print coverage. | + +## Implementation Phases + +### 1. Align Durable Registry Contracts + +- Add nullable, unique `semantic_key` fields to `DocumentType` and `PersonRole`. +- Keep UUIDs as primary and foreign-key identity. +- Add normalized-label storage and uniqueness to Person Roles using the same trim and case-normalization policy as Document Types. +- Remove the user-created Person Role code contract. +- Define built-in detection as `semantic_key is not null`. +- Centralize the frozen built-in definitions in one domain-owned location. +- Seed six Document Types and three Person Roles idempotently. +- Ensure label edits never change semantic keys. +- Reject deletion of every built-in before checking references. +- Continue blocking deletion of referenced custom entries. +- Return deterministic validation, conflict, dependency, and not-found errors through existing error categories. + +### 2. Establish the Clean Schema + +- Remove legacy `DocumentPerson.role` compatibility storage and the fixed `DocumentPersonRole` enum. +- Make `DocumentPerson.role_id` required. +- Replace role-specific uniqueness with a unique `(document_id, person_id)` constraint. +- Remove obsolete Document Type and Person Role migration paths that exist only for disposable development data. +- Keep fresh schema creation and built-in seeding portable across SQLite and PostgreSQL. +- Make configured development-database recreation a separate operator-confirmed step that displays the resolved target path rather than assuming `app.db` or `data/transcription.db`. +- Never invoke recreation from application startup or test setup. +- Add isolated schema tests for fresh creation, seed idempotence, semantic-key uniqueness, normalized-label uniqueness, required roles, and link uniqueness. + +### 3. Refine Registry Services and API Contracts + +- Add summary queries for Document counts and Person Role link counts without per-row queries. +- Order both registries by normalized label with UUID as a deterministic tie-breaker. +- Expose built-in status as a derived read value where the Settings UI needs it. +- Keep semantic-key lookup behind service methods for application-owned behavior such as resolving authors. +- Ensure create operations always produce custom entries with null semantic keys. +- Ensure update operations accept only label and active state. +- Remove `role_code` request alternatives and compatibility role responses from the V4 document API. +- Require `role_id` for document-person creation and updates. +- Add service and API tests for hidden semantic identity, relabeling, activation, built-in protection, custom deletion, counts, ordering, UUID-only writes, and conflicts. + +### 4. Build Matching Settings Tables + +- Retain the existing Document Types table workflow and add the Built-in column. +- Replace the current per-row Person Role controls with the same selection-based table pattern. +- Render the agreed columns and usage counts. +- Keep labels as the only registry text shown in selectors. +- Add creates custom entries only. +- Edit dialogs expose label and active state only. +- Delete reports protected-built-in and referenced-custom conflicts clearly. +- Avoid direct persistence queries from the Settings page. +- Add component-level UI assertions for columns, actions, label-only selectors, and immutable built-in presentation. + +### 5. Add a Staged Linked People Editor + +- Introduce a small typed UI-state model for staged `(person_id, role_id)` rows rather than storing raw widget values. +- Share the editor component between Create Document and Edit Document. +- Render a multi-selection table with Person and Role labels. +- Add an inline editor whose mode is explicitly Add or Edit. +- Disable already-linked People when adding; retain the edited Person as an option during Edit. +- Require exactly one row for Edit and allow one or more rows for Delete. +- Save and Delete mutate only staged UI state. +- Cancel discards only the active inline edit. +- Preserve inactive-role historical rows in Edit while restricting new assignments and changes to active roles. +- Preserve `person_id` preselection by staging that Person with the active built-in `author` role, with warning behavior for invalid or unavailable selections. +- Preserve the `return_to=jobs_new` success path. +- Keep navigation to Person creation separate; V4.4 does not add an embedded Person editor. +- Add UI tests for staging, duplicate prevention, selection rules, inactive roles, cancel behavior, and both Document forms. + +### 6. Persist Document and Links Atomically + +- Add service commands for Create Document with complete links and Update Document with complete links. +- Validate Document Type, every Person, every Person Role, active assignment rules, and duplicate People before mutation. +- Compute deterministic add, update, and remove deltas for Edit. +- Apply Document and link mutations in one database transaction and commit once. +- Roll back the complete operation on any validation, conflict, or persistence failure. +- Return the persisted Document detail required by the UI after success. +- Reuse these commands from UI orchestration rather than sequencing independent service commits. +- Add failure-injection tests proving no partial Document or link mutation survives. + +### 7. Define a Print Projection + +- Add a read-only service projection containing: + - Document title and selected archival metadata. + - Authors resolved by the `author` semantic key. + - Notes. + - Ordered Sources with application media URLs and current transcription text. + - Ordered Job metadata. +- Load the projection with bounded queries and deterministic ordering. +- Use non-null `revised_text`, including an intentionally empty revision; otherwise fall back to `raw_transcription`. +- Map empty or whitespace-only current text to the explicit unavailable state without falling back past an intentional revision. +- Represent unavailable text and optional metadata explicitly. +- Do not expose semantic keys, direct file paths, full prompts, provider evidence, or raw API responses. +- Keep the projection independent of NiceGUI rendering so formatting tests can use plain typed values. + +### 8. Build Print Preview and Styles + +- Add a Print action to Document Detail. +- Open a dedicated persisted-Document print route with a Facsimile/Text-only format choice. +- Render the exact content order frozen in the scope. +- Render stored Notes and transcription as escaped text. +- For Text-only mode, normalize whitespace by joining single line breaks inside paragraphs while preserving blank-line paragraph boundaries. +- For Facsimile mode, preserve line breaks and use a two-column Source layout. +- Start each Facsimile Source on a new printed sheet with CSS page breaks. +- Allow long transcription content to continue rather than clipping it. +- Fetch images through an application-controlled Source media route. +- Add print-only CSS that hides navigation, controls, and non-document chrome. +- Invoke the browser print dialog only from an explicit user action. +- Add rendering tests for both modes, missing data, long text, special characters, image URLs, and page ordering. + +### 9. Align Documentation and Verification + +- Update V4 architecture, requirements, schema, and Document UI contracts to reflect: + - UUID plus hidden semantic-key registries. + - Built-in protection. + - One Person per Document. + - UUID-only role API writes. + - Atomic Document/link synchronization. + - Browser-native print projection and formats. +- Confirm every database, integration, and UI test target is isolated before execution. +- Run focused registry and service tests first. +- Run schema tests only through the destructive-test wrapper when they are potentially destructive. +- Run Linked People UI and print rendering tests against isolated fixtures. +- Run the broader non-external regression suite after focused coverage passes. +- Verify that `data/transcription.db` was not changed by test execution. + +## Delivery Order + +1. Registry and clean-schema contracts. +2. Registry services, API changes, and Settings tables. +3. Atomic Document/link service commands. +4. Shared staged Linked People editor. +5. Print projection. +6. Print preview and styles. +7. Documentation alignment and regression verification. + +## Done Criteria + +- All V4.4 acceptance criteria are implemented and testable. +- UI and API contracts use UUID identity and never expose semantic keys. +- Built-in registries retain meaning after relabeling and cannot be deleted. +- Custom registries retain reference-aware deletion. +- Both Document forms use one Linked People table. +- One Person cannot be linked twice to the same Document. +- Main Document saves are atomic across fields and relationships. +- Print preview provides both frozen formats and content sections. +- Print output uses current human-preferred text, deterministic ordering, escaped content, and application media URLs. +- Job metadata lists every Job oldest-to-newest and ends with Status. +- Source page reordering and server-generated PDFs are not introduced. +- Verification uses isolated data and does not modify `data/transcription.db`. + +## Related Local References + +- [V4.4 Scope Boundary](scope_boundary_v4_4.md) +- [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md) +- [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md) +- [V4 Architecture](../ver4/architecture_v4.md) +- [V4 Schema](../ver4/schema_v4.md) +- [V4 Requirements](../ver4/requirements_v4.md) +- [V4 Error Handling Policy](../ver4/error_handling_v4.md) diff --git a/docs/ver4.4/scope_boundary_v4_4.md b/docs/ver4.4/scope_boundary_v4_4.md new file mode 100644 index 0000000..3aac488 --- /dev/null +++ b/docs/ver4.4/scope_boundary_v4_4.md @@ -0,0 +1,232 @@ +# V4.4 Scope Boundary + +This document defines the frozen boundary for the semantic-registry, linked-people, and document-printing revision that follows the completed V4.3 Settings work. V4 through V4.3 remain the architecture and behavioral baseline except where this document explicitly replaces a registry or document-person contract. + +## Purpose + +- Keep registry identifiers stable without exposing duplicate machine codes in Settings tables or selectors. +- Replace role-specific person selectors with one coherent Linked People editor. +- Provide an archival print view containing document metadata, source pages, current transcription text, and transcription-job metadata. + +## In Scope + +### 1. Semantic Registry Identity + +- `DocumentType` and `PersonRole` use UUIDs as their canonical record and relationship identity. +- Both registries may carry a nullable, unique, immutable `semantic_key` used only for application-defined built-ins. +- Semantic keys are internal implementation details. Settings tables, selectors, and public API payloads do not display or accept them. +- Labels are trimmed, case-insensitively unique, editable, and used for all user-facing display. +- Active state remains editable. Inactive entries remain valid for historical records and are excluded from default assignment selectors. +- Built-in status is derived from the presence of a semantic key and is displayed as a read-only Yes/No value. +- Built-in entries cannot be deleted or converted to custom entries. +- Custom entries have no semantic key and may be deleted only when unreferenced. +- Users may create custom entries but cannot create, change, or assign semantic keys through the UI or API. + +### 2. Built-In Document Types + +- Seed these built-in semantic keys and initial labels: + +| Semantic key | Initial label | +| --- | --- | +| `book` | Book | +| `letter` | Letter | +| `postcard` | Postcard | +| `photo` | Photo | +| `journal` | Journal | +| `form` | Form | + +- The Document Types Settings table contains Select, Label, Documents, Active, and Built-in columns. +- Document Types are ordered alphabetically by normalized label. +- Add, Edit, and Delete actions operate on table selection. +- Add creates a custom type. Edit changes only label and active state. +- The Documents count is the number of Documents referencing the type. +- Document Type selectors display labels only and submit UUIDs. + +### 3. Built-In Person Roles + +- Seed these built-in semantic keys and initial labels: + +| Semantic key | Initial label | +| --- | --- | +| `author` | Author | +| `recipient` | Recipient | +| `mentioned` | Mentioned | + +- Application behavior that requires authorship resolves the built-in `author` semantic key rather than matching a mutable label. +- The Person Roles Settings table contains Select, Label, Links, Active, and Built-in columns. +- Person Roles are ordered alphabetically by normalized label. +- Add, Edit, and Delete actions operate on table selection. +- Add creates a custom role. Edit changes only label and active state. +- The Links count is the number of document-person relationships referencing the role. +- Person Role selectors display labels only and submit UUIDs. + +### 4. Linked People Editor + +- Replace the separate role-specific person selectors on both Create Document and Edit Document with one Linked People table. +- The table contains Select, Person, and Role columns. +- Add opens an inline editor beneath the table with Person and Person Role selectors. +- Edit requires exactly one selected row and loads it into the inline editor. +- Save stages the inline addition or edit in the table. +- Cancel exits the inline editor without changing the staged link set. +- Delete stages removal of one or more selected rows. +- A Person may be linked to a Document only once, regardless of role. +- Every link has exactly one Person Role. +- Already-linked People are unavailable when adding another row. +- Existing links using inactive roles remain visible and unchanged unless explicitly edited. +- Only active roles are available for new links or role changes. +- Create Document preserves the existing `person_id` preselection workflow by staging that Person with the active built-in `author` role. An invalid Person or unavailable Author role produces a warning rather than an invalid link. +- Create Document preserves the existing `return_to=jobs_new` success path. +- Linked People changes remain staged until the main Create Document or Save Changes action. +- The Document and its complete staged link set are persisted atomically. A conflict or validation failure leaves both unchanged. +- The API and service contracts identify roles by `role_id`; role-code selectors and compatibility role strings are removed. +- Persistence enforces uniqueness on `(document_id, person_id)`. + +### 5. Document Print View + +- Add a Print action to Document Detail. +- The action opens a dedicated print-preview page for the persisted Document. +- The preview offers two formats: + - **Facsimile:** source image on the left and current transcription on the right. Original transcription line breaks are preserved, and each Source begins on a new printed sheet. + - **Text only:** no source images. Single line breaks inside a paragraph are reflowed as spaces, while blank-line paragraph boundaries remain. +- Both formats use browser printing through a dedicated print stylesheet and the browser print dialog. +- Server-generated PDF files are not part of V4.4; users may select the browser's Save as PDF destination. +- Sources are ordered by existing `page_number`, with UUID as a deterministic tie-breaker. +- The current transcription for each Source is the non-null `revised_text`, including an intentionally empty revision, otherwise the latest successful machine-output projection in `raw_transcription`. +- Empty or whitespace-only current text displays the explicit unavailable message rather than falling back past an intentional revision. +- A Source with no current transcription displays an explicit unavailable message. +- Transcription and Notes content is treated as text and escaped; model output is not executed as arbitrary HTML. +- Facsimile images use an application-controlled Source media route. Generated markup does not expose direct machine-local file paths. + +### 6. Printed Content Contract + +The print view contains, in this order: + +1. Document title using the Document name. +2. Archival Metadata table: + - Author, containing People linked through the built-in `author` role. + - Date. + - Location Created. + - Archival Identifier. +3. Notes. +4. Document section containing one numbered section per Source. +5. Transcription Job Metadata table. + +Empty metadata values remain visible as `Not set`. Empty Notes display `No notes recorded`. + +The job metadata table: + +- Lists field names in the first column and adds one column for every Job associated with the Document. +- Orders Job columns from oldest to newest by creation date, then UUID. +- Includes every Job status: `queued`, `processing`, `transcribed`, `completed`, `partial_success`, and `failed`. +- Contains these rows in order: + - Job ID. + - Date, using the Job creation/submission timestamp with timezone. + - Provider. + - Model. + - Prompt, using the frozen prompt filename/name rather than full prompt content. + - Retry Count. + - Status as the final row. +- Displays `Not set` for unavailable optional metadata. + +### 7. Clean Development Schema + +- V4.4 does not require preservation or migration of rows in the operator-configured development database. +- Implementation may recreate the configured development database, including `data/transcription.db` when it is the explicitly selected target, only through a separate operator-confirmed action that identifies the resolved path. Startup and test execution never delete it automatically. +- Fresh schema creation seeds the agreed built-in Document Types and Person Roles idempotently. +- No test may use, modify, replace, or restore `data/transcription.db`. +- Database, integration, and UI tests use confirmed isolated databases. +- Potentially destructive tests run only through `tools/run_destructive_tests.py`. + +## Out of Scope + +- User creation, editing, deletion, or direct display of semantic keys. +- Treating custom registry entries as built-ins. +- Additional built-in Document Types or Person Roles beyond the frozen lists. +- Assigning more than one role to the same Person on the same Document. +- Preserving multiple historical links that violate the new one-person-per-document constraint. +- Source page renumbering or reordering. +- Printing unsaved Create/Edit Document state. +- Print actions on Job Detail or other pages. +- Batch printing multiple Documents. +- Server-side PDF generation or PDF file storage. +- Markdown, DOCX, or evidence-package export through the print feature. +- User-editable print templates, fonts, margins, headers, or footers. +- Full frozen prompt content, prompt hashes, transport evidence, API responses, or execution-attempt details in the print footer. +- Rendering transcription text as unrestricted Markdown or HTML. +- Pixel-identical pagination across browsers and printer drivers. + +## Locked Design Decisions + +### A. UUID Identifies the Row; Semantic Key Identifies Built-In Meaning + +- UUIDs remain the only relationship and API identity. +- A hidden semantic key permits reliable built-in behavior after a label is renamed. +- Mutable labels are never used to infer built-in meaning. + +### B. Built-Ins Are Protected but Mutable in Presentation + +- Built-in labels and active state may change. +- Built-in semantic identity cannot change, and built-ins cannot be deleted. +- Custom entries remain reference-aware and deletable when unreferenced. + +### C. Linked People Is a Single-Role Relationship + +- One `(document_id, person_id)` row represents the complete relationship. +- Changing a role updates that row rather than adding another relationship. +- The main Document save owns one atomic Document-and-links transaction. + +### D. Printing Uses the Current Human-Preferred Text + +- Human-revised text takes precedence over the latest successful machine-output projection. +- Job metadata provides processing context but does not claim that a later human revision is raw output from a listed Job. + +### E. Printing Is Browser-Native + +- A print-specific HTML view and CSS support physical printing and browser Save as PDF. +- Source media is served through application-controlled routes, and all textual content is escaped. + +### F. Source Order Is Read-Only in V4.4 + +- Print order follows existing page numbers. +- Source page reordering remains explicitly excluded. + +## Acceptance Criteria + +1. Registry selectors and Settings forms never display a machine code or semantic key. +2. Document Types and Person Roles use UUID relationship identity and case-insensitively unique labels. +3. The six Document Type and three Person Role built-ins are seeded with immutable internal semantic keys. +4. Built-ins may be relabeled or disabled but cannot be deleted. +5. Unreferenced custom entries may be deleted; referenced custom entries may only be relabeled or disabled. +6. Settings tables show the agreed columns, alphabetical label order, usage counts, and selection-based actions. +7. Create and Edit Document use one Linked People table with inline staged Add/Edit/Save/Cancel and multi-row Delete. +8. The same Person cannot be staged or persisted twice for one Document, even under different roles. +9. Document fields and Linked People changes commit atomically. +10. Historical inactive roles remain displayable, while only active roles are assignable. +11. Document Detail opens a print preview with Facsimile and Text-only formats. +12. Print pages use current revised text when available and deterministic Source ordering. +13. Printed archival metadata resolves authors through the hidden `author` semantic key after any label change. +14. The job table contains one oldest-to-newest column per Job and ends with the Status row. +15. Print output escapes stored text and does not disclose direct local source paths. +16. Source reordering, server PDF generation, and print-template editing are absent. +17. Database, integration, and UI verification uses isolated test data and never modifies `data/transcription.db`. + +## Scope Freeze Gate + +V4.4 is sufficiently frozen to begin implementation: + +- Built-in registry identity, membership, lifecycle, display, and selector behavior are resolved. +- Linked People selection, editing, uniqueness, inactive-role, staging, and transaction behavior are resolved. +- Print entry point, formats, content order, transcription precedence, page order, job metadata, and output mechanism are resolved. +- Clean development-schema and destructive-test boundaries are resolved. +- Source page reordering remains excluded. + +Any expansion of the built-in catalogs, relationship cardinality, print formats, export formats, or print customization requires an explicit V4.4 scope amendment or a later revision. + +## Related Local References + +- [V4.4 Implementation Plan](implementation_plan_v4_4.md) +- [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md) +- [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md) +- [V4 Architecture](../ver4/architecture_v4.md) +- [V4 Schema](../ver4/schema_v4.md) +- [V4 Requirements](../ver4/requirements_v4.md)