bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updates to docs. Added new transcription_methodology, revised approach to revisions: 1 revision per document (that can be updated)

Browse Source
This commit is contained in:
Jim Lancaster
2026-07-29 13:29:44 -05:00
parent ec61013b47
commit eaf9805121
6 changed files with 77 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Intent.md
+1 -23
View File
@@ -19,27 +19,5 @@ I have several thousand pages of family history told through letters, postcards,
---
## Methodology
### Verbatim vs. Clean Copy
Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text.
### Prompt Curation Policy
Transcription behavior should be implemented with prompt assets that are human-maintainable over time.
1. Each transcription prompt is stored as an individual Markdown file.
2. Prompt files are refined iteratively as document quality and edge cases are discovered.
3. Prompt changes should be scoped to one prompt file at a time whenever possible to keep review history clear.
### Potential Document Issues
| Document Issue | How to Handle It | Example |
| :--- | :--- | :--- |
| **Misspellings & Errors** | Retain original spelling and insert italicized `[sic]` directly after the error. | `The weather was very cold and publick [sic] business delayed.` |
| **Missing Words / Slips** | Insert the missing word inside square brackets to restore basic readability. | `We went [to] the store to buy supplies.` |
| **Uncertain / Guesswork** | Place your best hypothesis followed by a question mark inside square brackets. | `He went to [Boston?] yesterday to meet the governor.` |
| **Completely Illegible** | Use a clear descriptive term like `[illegible]` or specify the reason (e.g., `[torn]`, `[ink blot]`). | `The total cost was [illegible] dollars.` or `The letter ends here [remainder of page torn].` |
| **Crossed-out Text** | Wrap the removed word or phrase in a deleted tag to preserve the author's edits. | `We left at [deleted: noon] one o'clock instead.` |
| **Squeezed-in Text** | Wrap text that was added above the line or in a tight space in an inserted tag. | `The [inserted: red] house on the hill was abandoned.` |
| **Superscripts & Abbreviations** | Bring raised letters down to the main line, or optionally expand them in brackets. | `Change Gen^l to Genl` OR `Change to Gen[era]l depending on project preference.` |
| **Images / Seals / Signs** | Describe the non-textual element using italicized text inside square brackets. | `[wax notary seal attached here]` or `[sketch of a fort layout]` |
| **Marginalia / Notes** | Note the spatial transition clearly before transcribing the note itself. | `[written in left margin:] Do not share this with anyone.` |
| **Line Breaks / Hyphens** | Rejoin words split across a page margin silently, dropping the line-break hyphen. | `Original: "estab- / lishment" becomes "establishment"` |
| **Ambiguous Capitalization** | Default to modern capitalization rules unless an archaic uppercase letter is clearly intentional. | `If a standard noun like 'Farm' looks randomly capitalized, type 'farm'.` |
**Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using spaces/tabs. Do not correct math or sequence errors silently. | `I. Main Topic`<br>`&nbsp;&nbsp;a. Sub-point`<br>`&nbsp;&nbsp;b. Next point`<br>`III. [sic] Third Topic` |
See [transcription_methodology.md](transcription_methodology.md) for details on the transcription methodology.
docs/architecture.md
+5 -5
View File
@@ -20,7 +20,7 @@ Current scope includes:
- content source upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- original transcription review and optional revision history
- original transcription review and optional revision review
- full-text search over accepted transcripts
- export of transcript data
@@ -61,7 +61,7 @@ flowchart LR
Worker --> MG
```
## Runtime Ownership And Startup Policy (V1 Step 1)
## Runtime Ownership And Startup Policy
The current implementation now uses explicit lifespan-owned runtime resources.
@@ -136,7 +136,7 @@ Production transcription flow:
3. The in-process worker de-queues the job and calls the transcription provider.
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, optional revision history, and searchable transcription text.
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
## Data Model Ownership
@@ -152,8 +152,8 @@ System-of-record entities:
- each processing job stores the original immutable provider output (`text`)
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
- revisions are optional user-authored edits linked to a content source
- a revision can be created from either original `job.text` or a previous revision
- many jobs will have zero revisions; revision history is additive and never overwrites original provider output
- a revision can be created from original `job.text`
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
- a document groups one or more content sources (images, PDFs, and future source types)
Storage strategy:
docs/index.md
+7 -5
View File
@@ -4,7 +4,7 @@ This project is a production application for transcribing and preserving histori
## Start Here
Read [architecture.md](architecture.md) first, then review [ver1/ver1.md](ver1/ver1.md).
Read [architecture.md](architecture.md) first.
The architecture page is the primary technical reference and defines:
@@ -23,7 +23,7 @@ Core capabilities:
- asynchronous transcription with visible job status
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
- transcription prompt management with one Markdown file per prompt for human refinement over time
- optional revision history for user-authored edits (from original text or prior revisions)
- optional revisions for user-authored edits of original immutable transcription text
- full-text search over accepted transcripts
- export of transcript data
@@ -41,11 +41,13 @@ This operating model keeps deployment and maintenance simple while preserving cl
## Documentation Map
- Architecture and technical design: [architecture.md](architecture.md)
- Version 1 implementation plan: [ver1/ver1.md](ver1/ver1.md)
- Architecture decision records (ADR index): [adr/README.md](adr/README.md)
- Runtime and deployment requirements: [requirements.md](requirements.md)
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
- Domain context and transcription policy: [Intent.md](Intent.md)
- Domain context and transcription policy: [intent.md](intent.md)
- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md)
- Data model: [schema.md](schema.md)
## Glossary
docs/schema.md
+5 -5
View File
@@ -48,18 +48,18 @@ erDiagram
document ||--o{ source : "has 0 or more"
document ||--o{ job : "has 0 or more"
job ||--o{ source : "processes 0 or more"
source ||--o{ revision : "has 0 or more"
source ||--o{ revision : "has 0 or 1"
```
---
## Table Relationships & Constraints
* A document can consist of 0 or more content sources. A document can have 0 or more jobs.
* A source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source can have 0 or more transcription revisions.
* A source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source may have one optional transcription revision.
* A job can process one or more sources. A job can belong to only one document.
* A revision can belong to only one source. A source can have 0 or more revisions.
* A revision can belong to only one source. A source may have one optional revision.
* `Job.text` stores the original immutable provider transcription.
* Revision rows are optional user-authored edits and may be derived from either the original transcription or a prior revision.
* Revision rows are optional user-authored edits and are derived from the original transcription. Unlike jobs, revision rows can be updated.
---
@@ -70,4 +70,4 @@ erDiagram
* **PDF** - A PDF containing the image of one or more pages of a document.
* **Job** - A processing job ingests one or more sources, sends them to an AI model along with a prompt for transcription, then stores the results. The results are immutable, *including the original transcription*. The user can create a revision of the original transcription, but the user cannot modify the original.
* **Transcription** - The text contained in a content source. A job creates the original immutable transcription. A user can optionally create a revised transcription, or "revision".
* **Revision** - A revision is a user-created modification of an existing transcription. A user can create a revision from either the original transcription or a previous revision, but it is entirely optional. Some original transcriptions will have no revisions.
* **Revision** - A revision is a user-created modification of an existing transcription. It is optional. Some original transcriptions will have no revisions.
docs/transcription_methodology.md
+54
View File
@@ -0,0 +1,54 @@
# Transcription Methodology & Style Guide
## 1. Overview & Core Philosophy
This document defines the formal transcription standard for processing historical manuscripts, letters, diaries, and printed ephemera.
Following the principles established by Mary-Jo Kline in A Guide to Documentary Editing, this project adheres to a Strict Literal Transcription (Verbatim) model as its foundational layer. The primary goal is total textual fidelity—capturing what the author wrote, not what they intended to write—while ensuring the output remains machine-readable and indexable for downstream digital query and search systems.
## 2. Textual Policy
Transcribers (human or AI) must record the exact text of the source document without silent corrections, modernizations, or stylistic smoothing except where explicitly instructed in this guide.
* **Substantives:** Words, letter forms, structural layout, and semantic content must be recorded strictly as presented in the original document.
* **Accidentals:** Punctuation, capitalization, misspellings, and archaic character representations must be preserved unless an explicit rule below allows for standardization.
## 3. Standard Transcription Rules & Markup
The following rules map directly to editorial conventions for handling common manuscript anomalies and physical document features.
### 3.1 Textual Anomalies & Corrections
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Misspellings & Errors** | Retain original spelling verbatim. Insert an italicized [sic] immediately following the error. Do not correct spelling silently. | [sic] | The weather was very cold and publick [sic] business delayed. |
| **Missing Words / Omissions** | Insert necessary words required to restore basic grammatical sense inside square brackets. | [word] | We went [to] the store to buy supplies. |
| **Uncertain / Conjectural** | Place best hypothesis followed by a question mark inside square brackets when handwriting is doubtful. | [word?] | He went to [Boston?] yesterday to meet the governor. |
| **Completely Illegible** | Use [illegible] for unreadable script. Use explicit damage descriptors when physical impairment prevents reading. | [illegible] or [reason] | The total cost was [illegible] dollars. or The letter ends here [remainder of page torn]. |
| **Canceled / Struck-through** | Wrap text removed by the author inside a [deleted: ...] tag to preserve authorial revisions. | [deleted: text] | We left at [deleted: noon] one o'clock instead. |
| **Interlineations / Additions** | Wrap text inserted above, below, or in margins into the narrative flow inside an [inserted: ...] tag. | [inserted: text] | The [inserted: red] house on the hill was abandoned. |
### 3.2 Typography, Characters & Layout
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Superscripts & Abbreviations** | Bring raised letters down to the main line. Optionally expand abbreviations within square brackets based on project configuration. | [expanded] | Gen^l becomes Genl or Gen[era]l. |
| **Line-End Hyphenation** | Rejoin words split across a page or line boundary silently, dropping the soft hyphen. | Silently rejoin | Original: "estab- / lishment" becomes establishment |
| **Capitalization** | Preserve explicit capitalization. Default to modern capitalization rules only when authorial intent is ambiguous or archaic forms confuse sentence structure. | Literal / Contextual | If a standard noun like 'Farm' is clearly capitalized, record 'Farm'. If ambiguous, default to 'farm'. |
| **Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using standard spacing. Do not correct sequence or mathematical errors. | Preserve syntax | I. Main Topic a. Sub-point b. Next pointIII. [sic] Third Topic |
### 3.3 Visual & Spatial Elements
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Non-Textual Artifacts** | Record non-textual elements (seals, stamps, sketches, physical damage) using brief descriptive text inside square brackets. | [description] | [wax notary seal attached here] or [sketch of a fort layout] |
| **Marginalia & Addenda** | Explicitly indicate spatial transitions before transcribing content located in margins or non-standard orientations. | [location:] | [written in left margin:] Do not share this with anyone. |
## 4. Prompt Asset Integration
When executing programmatic transcriptions via LLM APIs or local models, processing instructions must be packaged into single-purpose system prompts aligned with these rules.
1. **Isolation:** Each transcription prompt file exists as an independent Markdown asset in the repository.
2. **Deterministic Output:** Prompts must explicitly instruct models to follow the markup standards in Section 3 without introducing conversational wrappers, extra prose, or structural markdown outside the source document's native layout.
3. **Iterative Scoping:** Rule modifications or edge-case additions must be submitted as isolated delta commits to individual prompt files to maintain clean revision tracking.
docs/ver1/ver1.md
+5 -7
View File
@@ -34,12 +34,12 @@ V1 is complete when all of the following are true:
1. Confirm and document invariants:
- `Job.text` is original immutable transcription output.
- `Revision` is optional and user-authored.
- Revisions may derive from original `Job.text` or a previous revision.
- Revisions are derived from the original `Job.text`.
2. Verify relationship cardinality assumptions:
- `Document` -> many `Source`
- `Document` -> many `Job`
- `Source` -> one `Job`
- `Source` -> many `Revision`
- `Source` -> one `Revision`
3. Ensure field naming consistency (`date_created`, `date_updated`, `date_uploaded`) across code and docs.
4. Freeze V1 status lifecycle to current implementation (`queued`, `processing`, `transcribed`, `failed`).
@@ -83,11 +83,9 @@ V1 is complete when all of the following are true:
### Tasks
1. Update job detail and related UI components:
- Display original immutable transcription from `Job.text`.
- Display optional revisions sourced from `Source.revisions`.
2. Replace legacy terminology in UI copy:
- “transcript history” -> “revision history” where appropriate.
3. Align date fields with new schema naming.
4. Preserve clear user messaging when no revisions exist.
- Display optional revision sourced from `Source.revisions`.
2. Align date fields with new schema naming.
3. Preserve clear user messaging when no revisions exist.
### Deliverables
- Updated jobs page and detail components.