transcription/docs/invariant/transcription_methodology.md

Transcription Methodology & Style Guide

1. Overview & Core Philosophy

This document defines the formal transcription standard for processing historical manuscripts, letters, diaries, and printed ephemera.

Following the principles established by Mary-Jo Kline in A Guide to Documentary Editing, this project adheres to a Strict Literal Transcription (Verbatim) model as its foundational layer. The primary goal is total textual fidelity—capturing what the author wrote, not what they intended to write—while ensuring the output remains machine-readable and indexable for downstream digital query and search systems.

2. Textual Policy

Transcribers (human or AI) must record the exact text of the source document without silent corrections, modernizations, or stylistic smoothing except where explicitly instructed in this guide.

• Substantives: Words, letter forms, structural layout, and semantic content must be recorded strictly as presented in the original document.

• Accidentals: Punctuation, capitalization, misspellings, and archaic character representations must be preserved unless an explicit rule below allows for standardization.

3. Standard Transcription Rules & Markup

The following rules map directly to editorial conventions for handling common manuscript anomalies and physical document features.

3.1 Textual Anomalies & Corrections

Document Feature	Rule	Standard Markup Format	Output Example
Misspellings & Errors	Retain original spelling verbatim. Insert an italicized [sic] immediately following the error. Do not correct spelling silently.	[sic]	The weather was very cold and publick [sic] business delayed.
Missing Words / Omissions	Insert necessary words required to restore basic grammatical sense inside square brackets.	[word]	We went [to] the store to buy supplies.
Uncertain / Conjectural	Place best hypothesis followed by a question mark inside square brackets when handwriting is doubtful.	[word?]	He went to [Boston?] yesterday to meet the governor.
Completely Illegible	Use [illegible] for unreadable script. Use explicit damage descriptors when physical impairment prevents reading.	[illegible] or [reason]	The total cost was [illegible] dollars. or The letter ends here [remainder of page torn].
Canceled / Struck-through	Wrap text removed by the author inside a [deleted: ...] tag to preserve authorial revisions.	[deleted: text]	We left at [deleted: noon] one o'clock instead.
Interlineations / Additions	Wrap text inserted above, below, or in margins into the narrative flow inside an [inserted: ...] tag.	[inserted: text]	The [inserted: red] house on the hill was abandoned.

3.2 Typography, Characters & Layout

Document Feature	Rule	Standard Markup Format	Output Example
Superscripts & Abbreviations	Bring raised letters down to the main line. Optionally expand abbreviations within square brackets based on project configuration.	[expanded]	Gen^l becomes Genl or Gen[era]l.
Line-End Hyphenation	Rejoin words split across a page or line boundary silently, dropping the soft hyphen.	Silently rejoin	Original: "estab- / lishment" becomes establishment
Capitalization	Preserve explicit capitalization. Default to modern capitalization rules only when authorial intent is ambiguous or archaic forms confuse sentence structure.	Literal / Contextual	If a standard noun like 'Farm' is clearly capitalized, record 'Farm'. If ambiguous, default to 'farm'.
Hierarchical Outlines	Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using standard spacing. Do not correct sequence or mathematical errors.	Preserve syntax	I. Main Topic a. Sub-point b. Next pointIII. [sic] Third Topic

3.3 Visual & Spatial Elements

Document Feature	Rule	Standard Markup Format	Output Example
Non-Textual Artifacts	Record non-textual elements (seals, stamps, sketches, physical damage) using brief descriptive text inside square brackets.	[description]	[wax notary seal attached here] or [sketch of a fort layout]
Marginalia & Addenda	Explicitly indicate spatial transitions before transcribing content located in margins or non-standard orientations.	[location:]	[written in left margin:] Do not share this with anyone.

4. Prompt Asset Integration

When executing programmatic transcriptions via LLM APIs or local models, processing instructions must be packaged into single-purpose system prompts aligned with these rules.

1. Isolation: Each transcription prompt file exists as an independent Markdown asset in the repository.
2. Deterministic Output: Prompts must explicitly instruct models to follow the markup standards in Section 3 without introducing conversational wrappers, extra prose, or structural markdown outside the source document's native layout.
3. Iterative Scoping: Rule modifications or edge-case additions must be submitted as isolated delta commits to individual prompt files to maintain clean revision tracking.