bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: bbchops/transcription:a78b58ff40a5e408245d2a52d5fa1c98d3d49496
Branches Tags
bbchops/transcription:traumatized bbchops/transcription:doc_update bbchops/transcription:going-back bbchops/transcription:main
..
compare: bbchops/transcription:63373bf24d11c675cf00d49f2b1affbaf1b8e421
Branches Tags
bbchops/transcription:traumatized bbchops/transcription:doc_update bbchops/transcription:going-back bbchops/transcription:main
2 Commits
a78b58ff40 .. 63373bf24d
Author SHA1 Message Date
Jim Lancaster 63373bf24d V4.4 Scope defined (part 2) 2026-08-15 14:04:58 -05:00
Jim Lancaster 936af9b1d3 V4.4 Scope defined 2026-08-15 14:04:42 -05:00
3 changed files with 422 additions and 1 deletions
Expand all files Collapse all files
docs/ver4.4/implementation_plan_v4_4.md
+188
View File
@@ -0,0 +1,188 @@
# Implementation Plan (Version 4.4)
## Goal
Deliver hidden semantic identity for built-in registries, a single atomic Linked People workflow, and safe browser-native printing of archival Documents and their current transcriptions.
## Planning Constraints
- V4.3 is the completed implementation baseline.
- V4.4 may replace V4/V4.3 registry and document-person contracts only as specified by the V4.4 scope.
- Semantic keys are internal and immutable; UI and public API contracts use UUIDs and labels.
- Document and link edits must not partially commit.
- Print output must not execute stored text or expose machine-local source paths.
- Source page reordering remains excluded.
- Database, integration, and UI tests must use confirmed isolated data and never modify `data/transcription.db`.
- Potentially destructive tests must run only through `tools/run_destructive_tests.py`.
## Expected Project Impact
| Area | Expected impact |
| --- | --- |
| Models and schema bootstrap | Add nullable unique semantic keys, simplify document-person identity, and seed frozen built-ins. |
| Document service | Maintain built-in Document Types, usage summaries, UUID assignment, and atomic Document/link writes. |
| People service | Maintain built-in Person Roles, link summaries, UUID-only role assignment, and one-person-per-document enforcement. |
| V4 document API | Remove role-code selectors and compatibility role fields; enforce UUID-only relationship writes. |
| Settings UI | Use matching table workflows for Document Types and Person Roles. |
| Document Create/Edit | Replace role-specific multiselects with one staged Linked People table and inline editor. |
| Document Detail/printing | Add format selection, print preview, safe Source media rendering, print CSS, and job metadata. |
| Tests and documentation | Replace superseded cardinality/identity assertions and add isolated registry, editor, transaction, and print coverage. |
## Implementation Phases
### 1. Align Durable Registry Contracts
- Add nullable, unique `semantic_key` fields to `DocumentType` and `PersonRole`.
- Keep UUIDs as primary and foreign-key identity.
- Add normalized-label storage and uniqueness to Person Roles using the same trim and case-normalization policy as Document Types.
- Remove the user-created Person Role code contract.
- Define built-in detection as `semantic_key is not null`.
- Centralize the frozen built-in definitions in one domain-owned location.
- Seed six Document Types and three Person Roles idempotently.
- Ensure label edits never change semantic keys.
- Reject deletion of every built-in before checking references.
- Continue blocking deletion of referenced custom entries.
- Return deterministic validation, conflict, dependency, and not-found errors through existing error categories.
### 2. Establish the Clean Schema
- Remove legacy `DocumentPerson.role` compatibility storage and the fixed `DocumentPersonRole` enum.
- Make `DocumentPerson.role_id` required.
- Replace role-specific uniqueness with a unique `(document_id, person_id)` constraint.
- Remove obsolete Document Type and Person Role migration paths that exist only for disposable development data.
- Keep fresh schema creation and built-in seeding portable across SQLite and PostgreSQL.
- Make configured development-database recreation a separate operator-confirmed step that displays the resolved target path rather than assuming `app.db` or `data/transcription.db`.
- Never invoke recreation from application startup or test setup.
- Add isolated schema tests for fresh creation, seed idempotence, semantic-key uniqueness, normalized-label uniqueness, required roles, and link uniqueness.
### 3. Refine Registry Services and API Contracts
- Add summary queries for Document counts and Person Role link counts without per-row queries.
- Order both registries by normalized label with UUID as a deterministic tie-breaker.
- Expose built-in status as a derived read value where the Settings UI needs it.
- Keep semantic-key lookup behind service methods for application-owned behavior such as resolving authors.
- Ensure create operations always produce custom entries with null semantic keys.
- Ensure update operations accept only label and active state.
- Remove `role_code` request alternatives and compatibility role responses from the V4 document API.
- Require `role_id` for document-person creation and updates.
- Add service and API tests for hidden semantic identity, relabeling, activation, built-in protection, custom deletion, counts, ordering, UUID-only writes, and conflicts.
### 4. Build Matching Settings Tables
- Retain the existing Document Types table workflow and add the Built-in column.
- Replace the current per-row Person Role controls with the same selection-based table pattern.
- Render the agreed columns and usage counts.
- Keep labels as the only registry text shown in selectors.
- Add creates custom entries only.
- Edit dialogs expose label and active state only.
- Delete reports protected-built-in and referenced-custom conflicts clearly.
- Avoid direct persistence queries from the Settings page.
- Add component-level UI assertions for columns, actions, label-only selectors, and immutable built-in presentation.
### 5. Add a Staged Linked People Editor
- Introduce a small typed UI-state model for staged `(person_id, role_id)` rows rather than storing raw widget values.
- Share the editor component between Create Document and Edit Document.
- Render a multi-selection table with Person and Role labels.
- Add an inline editor whose mode is explicitly Add or Edit.
- Disable already-linked People when adding; retain the edited Person as an option during Edit.
- Require exactly one row for Edit and allow one or more rows for Delete.
- Save and Delete mutate only staged UI state.
- Cancel discards only the active inline edit.
- Preserve inactive-role historical rows in Edit while restricting new assignments and changes to active roles.
- Preserve `person_id` preselection by staging that Person with the active built-in `author` role, with warning behavior for invalid or unavailable selections.
- Preserve the `return_to=jobs_new` success path.
- Keep navigation to Person creation separate; V4.4 does not add an embedded Person editor.
- Add UI tests for staging, duplicate prevention, selection rules, inactive roles, cancel behavior, and both Document forms.
### 6. Persist Document and Links Atomically
- Add service commands for Create Document with complete links and Update Document with complete links.
- Validate Document Type, every Person, every Person Role, active assignment rules, and duplicate People before mutation.
- Compute deterministic add, update, and remove deltas for Edit.
- Apply Document and link mutations in one database transaction and commit once.
- Roll back the complete operation on any validation, conflict, or persistence failure.
- Return the persisted Document detail required by the UI after success.
- Reuse these commands from UI orchestration rather than sequencing independent service commits.
- Add failure-injection tests proving no partial Document or link mutation survives.
### 7. Define a Print Projection
- Add a read-only service projection containing:
- Document title and selected archival metadata.
- Authors resolved by the `author` semantic key.
- Notes.
- Ordered Sources with application media URLs and current transcription text.
- Ordered Job metadata.
- Load the projection with bounded queries and deterministic ordering.
- Use non-null `revised_text`, including an intentionally empty revision; otherwise fall back to `raw_transcription`.
- Map empty or whitespace-only current text to the explicit unavailable state without falling back past an intentional revision.
- Represent unavailable text and optional metadata explicitly.
- Do not expose semantic keys, direct file paths, full prompts, provider evidence, or raw API responses.
- Keep the projection independent of NiceGUI rendering so formatting tests can use plain typed values.
### 8. Build Print Preview and Styles
- Add a Print action to Document Detail.
- Open a dedicated persisted-Document print route with a Facsimile/Text-only format choice.
- Render the exact content order frozen in the scope.
- Render stored Notes and transcription as escaped text.
- For Text-only mode, normalize whitespace by joining single line breaks inside paragraphs while preserving blank-line paragraph boundaries.
- For Facsimile mode, preserve line breaks and use a two-column Source layout.
- Start each Facsimile Source on a new printed sheet with CSS page breaks.
- Allow long transcription content to continue rather than clipping it.
- Fetch images through an application-controlled Source media route.
- Add print-only CSS that hides navigation, controls, and non-document chrome.
- Invoke the browser print dialog only from an explicit user action.
- Add rendering tests for both modes, missing data, long text, special characters, image URLs, and page ordering.
### 9. Align Documentation and Verification
- Update V4 architecture, requirements, schema, and Document UI contracts to reflect:
- UUID plus hidden semantic-key registries.
- Built-in protection.
- One Person per Document.
- UUID-only role API writes.
- Atomic Document/link synchronization.
- Browser-native print projection and formats.
- Confirm every database, integration, and UI test target is isolated before execution.
- Run focused registry and service tests first.
- Run schema tests only through the destructive-test wrapper when they are potentially destructive.
- Run Linked People UI and print rendering tests against isolated fixtures.
- Run the broader non-external regression suite after focused coverage passes.
- Verify that `data/transcription.db` was not changed by test execution.
## Delivery Order
1. Registry and clean-schema contracts.
2. Registry services, API changes, and Settings tables.
3. Atomic Document/link service commands.
4. Shared staged Linked People editor.
5. Print projection.
6. Print preview and styles.
7. Documentation alignment and regression verification.
## Done Criteria
- All V4.4 acceptance criteria are implemented and testable.
- UI and API contracts use UUID identity and never expose semantic keys.
- Built-in registries retain meaning after relabeling and cannot be deleted.
- Custom registries retain reference-aware deletion.
- Both Document forms use one Linked People table.
- One Person cannot be linked twice to the same Document.
- Main Document saves are atomic across fields and relationships.
- Print preview provides both frozen formats and content sections.
- Print output uses current human-preferred text, deterministic ordering, escaped content, and application media URLs.
- Job metadata lists every Job oldest-to-newest and ends with Status.
- Source page reordering and server-generated PDFs are not introduced.
- Verification uses isolated data and does not modify `data/transcription.db`.
## Related Local References
- [V4.4 Scope Boundary](scope_boundary_v4_4.md)
- [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md)
- [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
- [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
docs/ver4.4/scope_boundary_v4_4.md
+232
View File
@@ -0,0 +1,232 @@
# V4.4 Scope Boundary
This document defines the frozen boundary for the semantic-registry, linked-people, and document-printing revision that follows the completed V4.3 Settings work. V4 through V4.3 remain the architecture and behavioral baseline except where this document explicitly replaces a registry or document-person contract.
## Purpose
- Keep registry identifiers stable without exposing duplicate machine codes in Settings tables or selectors.
- Replace role-specific person selectors with one coherent Linked People editor.
- Provide an archival print view containing document metadata, source pages, current transcription text, and transcription-job metadata.
## In Scope
### 1. Semantic Registry Identity
- `DocumentType` and `PersonRole` use UUIDs as their canonical record and relationship identity.
- Both registries may carry a nullable, unique, immutable `semantic_key` used only for application-defined built-ins.
- Semantic keys are internal implementation details. Settings tables, selectors, and public API payloads do not display or accept them.
- Labels are trimmed, case-insensitively unique, editable, and used for all user-facing display.
- Active state remains editable. Inactive entries remain valid for historical records and are excluded from default assignment selectors.
- Built-in status is derived from the presence of a semantic key and is displayed as a read-only Yes/No value.
- Built-in entries cannot be deleted or converted to custom entries.
- Custom entries have no semantic key and may be deleted only when unreferenced.
- Users may create custom entries but cannot create, change, or assign semantic keys through the UI or API.
### 2. Built-In Document Types
- Seed these built-in semantic keys and initial labels:
| Semantic key | Initial label |
| --- | --- |
| `book` | Book |
| `letter` | Letter |
| `postcard` | Postcard |
| `photo` | Photo |
| `journal` | Journal |
| `form` | Form |
- The Document Types Settings table contains Select, Label, Documents, Active, and Built-in columns.
- Document Types are ordered alphabetically by normalized label.
- Add, Edit, and Delete actions operate on table selection.
- Add creates a custom type. Edit changes only label and active state.
- The Documents count is the number of Documents referencing the type.
- Document Type selectors display labels only and submit UUIDs.
### 3. Built-In Person Roles
- Seed these built-in semantic keys and initial labels:
| Semantic key | Initial label |
| --- | --- |
| `author` | Author |
| `recipient` | Recipient |
| `mentioned` | Mentioned |
- Application behavior that requires authorship resolves the built-in `author` semantic key rather than matching a mutable label.
- The Person Roles Settings table contains Select, Label, Links, Active, and Built-in columns.
- Person Roles are ordered alphabetically by normalized label.
- Add, Edit, and Delete actions operate on table selection.
- Add creates a custom role. Edit changes only label and active state.
- The Links count is the number of document-person relationships referencing the role.
- Person Role selectors display labels only and submit UUIDs.
### 4. Linked People Editor
- Replace the separate role-specific person selectors on both Create Document and Edit Document with one Linked People table.
- The table contains Select, Person, and Role columns.
- Add opens an inline editor beneath the table with Person and Person Role selectors.
- Edit requires exactly one selected row and loads it into the inline editor.
- Save stages the inline addition or edit in the table.
- Cancel exits the inline editor without changing the staged link set.
- Delete stages removal of one or more selected rows.
- A Person may be linked to a Document only once, regardless of role.
- Every link has exactly one Person Role.
- Already-linked People are unavailable when adding another row.
- Existing links using inactive roles remain visible and unchanged unless explicitly edited.
- Only active roles are available for new links or role changes.
- Create Document preserves the existing `person_id` preselection workflow by staging that Person with the active built-in `author` role. An invalid Person or unavailable Author role produces a warning rather than an invalid link.
- Create Document preserves the existing `return_to=jobs_new` success path.
- Linked People changes remain staged until the main Create Document or Save Changes action.
- The Document and its complete staged link set are persisted atomically. A conflict or validation failure leaves both unchanged.
- The API and service contracts identify roles by `role_id`; role-code selectors and compatibility role strings are removed.
- Persistence enforces uniqueness on `(document_id, person_id)`.
### 5. Document Print View
- Add a Print action to Document Detail.
- The action opens a dedicated print-preview page for the persisted Document.
- The preview offers two formats:
- **Facsimile:** source image on the left and current transcription on the right. Original transcription line breaks are preserved, and each Source begins on a new printed sheet.
- **Text only:** no source images. Single line breaks inside a paragraph are reflowed as spaces, while blank-line paragraph boundaries remain.
- Both formats use browser printing through a dedicated print stylesheet and the browser print dialog.
- Server-generated PDF files are not part of V4.4; users may select the browser's Save as PDF destination.
- Sources are ordered by existing `page_number`, with UUID as a deterministic tie-breaker.
- The current transcription for each Source is the non-null `revised_text`, including an intentionally empty revision, otherwise the latest successful machine-output projection in `raw_transcription`.
- Empty or whitespace-only current text displays the explicit unavailable message rather than falling back past an intentional revision.
- A Source with no current transcription displays an explicit unavailable message.
- Transcription and Notes content is treated as text and escaped; model output is not executed as arbitrary HTML.
- Facsimile images use an application-controlled Source media route. Generated markup does not expose direct machine-local file paths.
### 6. Printed Content Contract
The print view contains, in this order:
1. Document title using the Document name.
2. Archival Metadata table:
- Author, containing People linked through the built-in `author` role.
- Date.
- Location Created.
- Archival Identifier.
3. Notes.
4. Document section containing one numbered section per Source.
5. Transcription Job Metadata table.
Empty metadata values remain visible as `Not set`. Empty Notes display `No notes recorded`.
The job metadata table:
- Lists field names in the first column and adds one column for every Job associated with the Document.
- Orders Job columns from oldest to newest by creation date, then UUID.
- Includes every Job status: `queued`, `processing`, `transcribed`, `completed`, `partial_success`, and `failed`.
- Contains these rows in order:
- Job ID.
- Date, using the Job creation/submission timestamp with timezone.
- Provider.
- Model.
- Prompt, using the frozen prompt filename/name rather than full prompt content.
- Retry Count.
- Status as the final row.
- Displays `Not set` for unavailable optional metadata.
### 7. Clean Development Schema
- V4.4 does not require preservation or migration of rows in the operator-configured development database.
- Implementation may recreate the configured development database, including `data/transcription.db` when it is the explicitly selected target, only through a separate operator-confirmed action that identifies the resolved path. Startup and test execution never delete it automatically.
- Fresh schema creation seeds the agreed built-in Document Types and Person Roles idempotently.
- No test may use, modify, replace, or restore `data/transcription.db`.
- Database, integration, and UI tests use confirmed isolated databases.
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.
## Out of Scope
- User creation, editing, deletion, or direct display of semantic keys.
- Treating custom registry entries as built-ins.
- Additional built-in Document Types or Person Roles beyond the frozen lists.
- Assigning more than one role to the same Person on the same Document.
- Preserving multiple historical links that violate the new one-person-per-document constraint.
- Source page renumbering or reordering.
- Printing unsaved Create/Edit Document state.
- Print actions on Job Detail or other pages.
- Batch printing multiple Documents.
- Server-side PDF generation or PDF file storage.
- Markdown, DOCX, or evidence-package export through the print feature.
- User-editable print templates, fonts, margins, headers, or footers.
- Full frozen prompt content, prompt hashes, transport evidence, API responses, or execution-attempt details in the print footer.
- Rendering transcription text as unrestricted Markdown or HTML.
- Pixel-identical pagination across browsers and printer drivers.
## Locked Design Decisions
### A. UUID Identifies the Row; Semantic Key Identifies Built-In Meaning
- UUIDs remain the only relationship and API identity.
- A hidden semantic key permits reliable built-in behavior after a label is renamed.
- Mutable labels are never used to infer built-in meaning.
### B. Built-Ins Are Protected but Mutable in Presentation
- Built-in labels and active state may change.
- Built-in semantic identity cannot change, and built-ins cannot be deleted.
- Custom entries remain reference-aware and deletable when unreferenced.
### C. Linked People Is a Single-Role Relationship
- One `(document_id, person_id)` row represents the complete relationship.
- Changing a role updates that row rather than adding another relationship.
- The main Document save owns one atomic Document-and-links transaction.
### D. Printing Uses the Current Human-Preferred Text
- Human-revised text takes precedence over the latest successful machine-output projection.
- Job metadata provides processing context but does not claim that a later human revision is raw output from a listed Job.
### E. Printing Is Browser-Native
- A print-specific HTML view and CSS support physical printing and browser Save as PDF.
- Source media is served through application-controlled routes, and all textual content is escaped.
### F. Source Order Is Read-Only in V4.4
- Print order follows existing page numbers.
- Source page reordering remains explicitly excluded.
## Acceptance Criteria
1. Registry selectors and Settings forms never display a machine code or semantic key.
2. Document Types and Person Roles use UUID relationship identity and case-insensitively unique labels.
3. The six Document Type and three Person Role built-ins are seeded with immutable internal semantic keys.
4. Built-ins may be relabeled or disabled but cannot be deleted.
5. Unreferenced custom entries may be deleted; referenced custom entries may only be relabeled or disabled.
6. Settings tables show the agreed columns, alphabetical label order, usage counts, and selection-based actions.
7. Create and Edit Document use one Linked People table with inline staged Add/Edit/Save/Cancel and multi-row Delete.
8. The same Person cannot be staged or persisted twice for one Document, even under different roles.
9. Document fields and Linked People changes commit atomically.
10. Historical inactive roles remain displayable, while only active roles are assignable.
11. Document Detail opens a print preview with Facsimile and Text-only formats.
12. Print pages use current revised text when available and deterministic Source ordering.
13. Printed archival metadata resolves authors through the hidden `author` semantic key after any label change.
14. The job table contains one oldest-to-newest column per Job and ends with the Status row.
15. Print output escapes stored text and does not disclose direct local source paths.
16. Source reordering, server PDF generation, and print-template editing are absent.
17. Database, integration, and UI verification uses isolated test data and never modifies `data/transcription.db`.
## Scope Freeze Gate
V4.4 is sufficiently frozen to begin implementation:
- Built-in registry identity, membership, lifecycle, display, and selector behavior are resolved.
- Linked People selection, editing, uniqueness, inactive-role, staging, and transaction behavior are resolved.
- Print entry point, formats, content order, transcription precedence, page order, job metadata, and output mechanism are resolved.
- Clean development-schema and destructive-test boundaries are resolved.
- Source page reordering remains excluded.
Any expansion of the built-in catalogs, relationship cardinality, print formats, export formats, or print customization requires an explicit V4.4 scope amendment or a later revision.
## Related Local References
- [V4.4 Implementation Plan](implementation_plan_v4_4.md)
- [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md)
- [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
- [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
docs/ver4/index_v4.md
+2 -1
View File
@@ -26,4 +26,5 @@ Version 4 is the architecture baseline for the personal-scale application used t
- [V4.1 Scope](../ver4.1/scope_boundary_v4_1.md) and [Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) and [Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
- [Draft V4.3 Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Draft Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
- [V4.3 Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
- [V4.4 Semantic Registries, Linked People, and Printing Scope](../ver4.4/scope_boundary_v4_4.md) and [Implementation Plan](../ver4.4/implementation_plan_v4_4.md)