bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: bbchops/transcription:doc_update
Branches Tags
bbchops/transcription:main bbchops/transcription:traumatized bbchops/transcription:session-engine bbchops/transcription:doc_update bbchops/transcription:going-back bbchops/transcription:job-detail
..
compare: bbchops/transcription:a818d982a6e647eeddc211a1a442e996d4d9eeb7
Branches Tags
bbchops/transcription:traumatized bbchops/transcription:session-engine bbchops/transcription:doc_update bbchops/transcription:going-back bbchops/transcription:job-detail bbchops/transcription:main
1 Commits
doc_update .. a818d982a6
Author SHA1 Message Date
bbchops a818d982a6 Initial commit 2026-06-21 17:17:01 -05:00
123 changed files with 853 additions and 10107 deletions
Expand all files Collapse all files
.dockerignore
-13
View File
@@ -1,13 +0,0 @@
.git
.gitignore
.vscode
.venv
.pytest_cache
.ruff_cache
__pycache__/
*.py[cod]
*.db
.env
tests/
docs/
uploads/
.env.example
-8
View File
@@ -1,8 +0,0 @@
PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-...
# PROVIDER_MODEL= # optional: OpenRouter adapter supplies default
# OPENROUTER_HTTP_REFERER=https://example.com
# OPENROUTER_APP_TITLE=Historical Transcription MVP
# DATABASE_URL=sqlite:///./transcription.db
# UPLOAD_DIR=./uploads
# PROMPT_DIR=./prompts
.gitattributes
-1
View File
@@ -1 +0,0 @@
* text=auto eol=lf
.github/instructions/services.instructions.md
-77
View File
@@ -1,77 +0,0 @@
---
description: Follow these guidelines when editing the services
applyTo: 'src/transcription/services/*.py'
---
# Services
## Structure
- Project core data models defined in [models](../../src/transcription/models.py)
- 1 service class per data model
- Only services directly interact with the database, and only through async methods
- Services are completely independent of one another. Any operation that needs to use more than a single service, which is most of them, needs to have a separate orchestration function.
## Error Handling
- Service-specific errors defined at the top of the respective module and inherit from `AppError`
- Use a context manager for large `try/except` blocks like in [transcription](../../src/transcription/services/transcription.py)
## Checklist
- [ ] Uses `ServiceBase` for common logic
- [ ] CRUD methods created at the top
- [ ] Session kwarg for `AsyncSession` to pass in a session object to each method
- [ ] Services use `self._session_scope` in their methods to pass the session thru.
- Multiple operations on the same object(s) require sharing a session between all the methods used.
## CRUD Methods
- Create, read, update, and delete, created in that order
- Name format `<operation>_<model >`, for example `create_document` or `update_job`
- All services must define these 4 methods first, and in that order
## Transaction Finalization
When a service method accepts an optional `session` kwarg, write methods must use `self._finalize` to finalize the transaction properly according to whether or not they are sharing a session.
- If `session` is `None`: the method owns the transaction and should `commit()`.
- If `session` is provided: the method must **not** commit; it should `flush()` so IDs and FK values are available to the caller's transaction.
- Use `refresh()` on returned ORM objects when the caller needs DB-populated values (defaults, triggers, merged state).
Recommended helper behavior:
- Inputs: active session object, original `session` arg (or a boolean ownership flag), and an optional list of objects to refresh.
- Logic: `commit` when service-owned session, `flush` when caller-owned session, then refresh requested objects.
This keeps orchestration functions atomic: they can pass one shared session across multiple services and commit exactly once at the workflow boundary.
## Workflow Transaction Boundaries
For multi-step job lifecycles (for example queued transcription jobs), orchestration functions must use explicit transaction phases.
Required boundary model:
- **Transaction A (claim):** transition `JobStatus.QUEUED -> JobStatus.PROCESSING` and commit immediately.
- Perform provider/network work **outside** database transactions.
- **Transaction B (terminal success):** write transcript content and set `JobStatus.TRANSCRIBED` in the same shared-session commit.
- **Transaction B (terminal failure):** write transcript error detail and set `JobStatus.FAILED` in the same shared-session commit.
- **Transaction C (retry path):** write transcript error detail, increment retry count, and set `JobStatus.QUEUED` in one shared-session commit.
Atomicity rules:
- Never commit transcript updates separately from the paired terminal/retry job status change.
- Terminal state (`TRANSCRIBED` or `FAILED`) and transcript row changes must succeed or roll back together.
- Retry persistence (`QUEUED` + retry increment + error detail) must succeed or roll back together.
Separation of concerns:
- Worker modules should stay lightweight and delegate lifecycle transitions to service/workflow orchestration functions.
- In `workflows.py`, `process_queued_job` should own one complete attempt lifecycle: `QUEUED -> PROCESSING -> TRANSCRIBED|FAILED`.
- In `workflows.py`, `advance_job` should coordinate broader status progression around attempts (for example retry scheduling from `FAILED -> QUEUED`).
- Services should expose session-aware write helpers (flush on caller-owned session) so orchestration controls commit boundaries.
- Backoff/sleep behavior must run outside transactional scopes.
# Service Composition
Some operations, like uploading a picutre, require modifications to multiple tables, which can be done by composing methods from the service object into a separate function.
.github/instructions/ui.instructions.md
-6
View File
@@ -1,6 +0,0 @@
---
description: Copilot rules for modifying the UI
applyTo: 'src/transcription/ui/**/*.py'
---
The UI is forbidden from directly touching the database. All interactions should be done using the [services](../../src/transcription/services/)
.gitignore
-19
View File
@@ -1,19 +0,0 @@
# Python-generated files
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
# Virtual environments
.venv
# Environment secrets
.env
# SQLite database
*.db
# Document images
uploads/*
.python-version
-1
View File
@@ -1 +0,0 @@
3.12
.vscode/launch.json
Vendored
-27
View File
@@ -1,27 +0,0 @@
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Debug transcription app",
"type": "debugpy",
"request": "launch",
"module": "debugpy",
"args": [
"-m",
"uvicorn",
"transcription.app:create_app",
"--factory",
"--host",
// "127.0.0.1",
"0.0.0.0",
"--port",
"8080"
],
"justMyCode": true,
"console": "integratedTerminal",
"env": {
"PYTHONPATH": "${workspaceFolder}/src"
}
}
]
}
Dockerfile
-47
View File
@@ -1,47 +0,0 @@
FROM python:3.12-slim AS builder
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
UV_LINK_MODE=copy
WORKDIR /app
COPY --from=ghcr.io/astral-sh/uv:0.5.24 /uv /uvx /bin/
COPY pyproject.toml uv.lock README.md ./
RUN uv sync --frozen --no-dev --no-install-project
COPY src ./src
COPY prompts ./prompts
RUN uv sync --frozen --no-dev
FROM python:3.12-slim AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PATH="/app/.venv/bin:$PATH" \
PYTHONPATH="/app/src" \
UPLOAD_DIR="/app/uploads" \
PROMPT_DIR="/app/prompts"
WORKDIR /app
RUN groupadd --system --gid 1001 appgroup \
&& useradd --system --uid 1001 --gid appgroup --create-home appuser
COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app/src /app/src
COPY --from=builder /app/prompts /app/prompts
RUN mkdir -p /app/uploads /app/data \
&& chown -R appuser:appgroup /app
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s --start-period=3s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"
CMD ["uvicorn", "transcription.app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8000", "--proxy-headers"]
LICENSE
+232
View File
@@ -0,0 +1,232 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for software and other kinds of works.
The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too.
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.
Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions.
Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and modification follow.
TERMS AND CONDITIONS
0. Definitions.
“This License” refers to version 3 of the GNU General Public License.
“Copyright” also means copyright-like laws that apply to other kinds of works, such as semiconductor masks.
“The Program” refers to any copyrightable work licensed under this License. Each licensee is addressed as “you”. “Licensees” and “recipients” may be individuals or organizations.
To “modify” a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a “modified version” of the earlier work or a work “based on” the earlier work.
A “covered work” means either the unmodified Program or a work based on the Program.
To “propagate” a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.
To “convey” a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays “Appropriate Legal Notices” to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion.
1. Source Code.
The “source code” for a work means the preferred form of the work for making modifications to it. “Object code” means any non-source form of a work.
A “Standard Interface” means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.
The “System Libraries” of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A “Major Component”, in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
The “Corresponding Source” for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified it, and giving a relevant date.
b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to “keep intact all notices”.
c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so.
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an “aggregate” if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:
a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b.
d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d.
A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work.
A “User Product” is either (1) a “consumer product”, which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, “normally used” refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product.
“Installation Information” for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying.
7. Additional Terms.
“Additional permissions” are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or authors of the material; or
e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors.
All other non-permissive additional terms are considered “further restrictions” within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License.
An “entity transaction” is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.
11. Patents.
A “contributor” is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's “contributor version”.
A contributor's “essential patent claims” are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, “control” includes the right to grant patent sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version.
In the following three paragraphs, a “patent license” is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To “grant” such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. “Knowingly relying” means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it.
A patent license is “discriminatory” if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License “or any later version” applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the “copyright” line and a pointer to where the full notice is found.
python-template
Copyright (C) 2026 john
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode:
python-template Copyright (C) 2026 john
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an “about box”.
You should also get your employer (if you work as a programmer) or school, if any, to sign a “copyright disclaimer” for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see <https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read <https://www.gnu.org/philosophy/why-not-lgpl.html>.
README.md
+1 -78
View File
@@ -1,78 +1 @@
# Transcription
Historical document transcription system for family-history documents.
The app lets you upload a document image/PDF, queues a background transcription job, and then shows job status and results in a web UI.
## What the app does
- Upload document files (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`)
- Persist document + job records in SQLite
- Process jobs in a background worker (`queued -> processing -> transcribed/failed`)
- Store transcript text (or failure detail)
- Show status and results in the NiceGUI interface
## Quick start
### 1) Install dependencies
```bash
uv sync
```
### 2) Configure environment
Create a `.env` file in the project root (minimum required setting shown):
```env
OPENROUTER_API_KEY=your_openrouter_api_key
```
Optional settings (defaults shown):
```env
DATABASE_URL=sqlite:///./transcription.db
UPLOAD_DIR=./uploads
PROMPT_DIR=./prompts
WORKER_MAX_RETRIES=0
WORKER_RETRY_BACKOFF_SECONDS=0
WORKER_PROVIDER_TIMEOUT_SECONDS=20
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
```
### 3) Run the app
```bash
uv run uvicorn transcription.app:create_app --factory --reload
```
### 4) Open in browser
- GUI: [http://[IP_ADDRESS]:8000/ui](http://[IP_ADDRESS]:8000/ui)
- Health check: [http://[IP_ADDRESS]:8000/healthz](http://[IP_ADDRESS]:8000/healthz)
## How to navigate the GUI
- **Upload page** (`/ui`)
- Select a supported file to upload.
- The app creates a queued transcription job.
- Use the **View jobs** link to inspect progress.
- **Jobs page** (`/ui/jobs`)
- See all jobs and their status.
- Use **Refresh** to reload current states.
- Open a specific job to see details.
- **Job detail page** (`/ui/jobs/{job_id}`)
- Shows job metadata and status.
- Displays transcript text when successful.
- Displays failure detail when transcription fails.
## Prompt artifacts
Prompt files are stored in `prompts/` and loaded from `PROMPT_DIR` (default: `./prompts`).
The canonical MVP prompt is:
- `prompts/transcribe_document.md`
# Python Template
docker-compose.yml
-21
View File
@@ -1,21 +0,0 @@
services:
transcription:
build:
context: .
dockerfile: Dockerfile
container_name: transcription-app
env_file:
- .env
environment:
DATABASE_URL: sqlite:////app/data/transcription.db
UPLOAD_DIR: /app/uploads
PROMPT_DIR: /app/prompts
ports:
- "8002:8000"
volumes:
- ./uploads:/app/uploads
- transcription_data:/app/data
restart: unless-stopped
volumes:
transcription_data:
docs/Intent.md
-24
View File
@@ -1,24 +0,0 @@
# Historical Document Transcription Design Intent
I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text.
---
## Goals
1. Preserve our family history
2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain them).
3. Make the text easily available and searchable by family members, as well as AI (which may have different requirements).
4. Ability create timelines or assemble the historical record of the family or specific individuals from across the complete document archive. Perhaps use AI to create the timelines in a more narrative form.
---
## Source material
1. **letters, cards, diaries** - handwritten; mostly stored in tubs, with little organization
2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications.
3. **newspaper clippings, event programs, invitations, and other ephemera**
---
## Methodology
1. Follow current best practices per "A Guide to Documentary Editing" by Mary-Jo Kline. (See [Transcription Methodology](transcription_methodology.md))
docs/architecture_v2.md
-136
View File
@@ -1,136 +0,0 @@
# System Architecture (Version 2)
This document describes the V2 production architecture of the personal historical-document transcription system.
## Architecture Objectives
* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata.
* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
* Migrate persistence to PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` document storage.
* Standardize all data validation, API parsing, and database models on **Pydantic V2**.
* Support rich historical attribution (multi-author and multi-recipient relationships).
## Runtime Topology
The V2 runtime operates as an asynchronous Python application:
* FastAPI + NiceGUI web application process.
* In-process `asyncio` background task orchestrator for parallel API execution.
* Relational persistence via PostgreSQL (using `asyncpg` or `psycopg3`).
* Pydantic V2 validation layer wrapping API payloads and PostgreSQL `JSONB` schemas.
```mermaid
flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker Engine]
A --> DB[(PostgreSQL Database)]
W --> P[Vision Provider APIs\nOpenAI / Claude]
W --> DB
```
## Lifecycle Ownership
Application lifespan owns runtime setup/teardown:
* Initialize environment logging and Pydantic configuration.
* Manage asynchronous PostgreSQL connection pools (`asyncpg` / `psycopg3`).
* Execute database migrations and index initialization.
* Recover stale processing jobs on startup.
* Manage graceful shutdown of active `asyncio` worker pools.
## Layered Module Structure
### Interface Layer
* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
### Application & Async Worker Layer
* `src/transcription/services/workflows.py`
* `src/transcription/worker.py`
Responsibilities:
* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
* Pydantic schema parsing (`PageAIMetadata`) and validation prior to database storage.
### Domain & Service Layer
* `src/transcription/models/*.py` (Pydantic V2 schemas and entity definitions)
* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
### Infrastructure Layer
* `src/transcription/db/**` (PostgreSQL connection pooling and raw parameterized SQL execution)
* `src/transcription/providers/**` (OpenAI & Anthropic Vision SDK adapters)
## Processing Workflow
1. User uploads a folder or batch of images for a `Document`.
2. System creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
4. Each task calls Vision API for a **single** `Source` image.
5. On task completion:
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, `ai_metadata` (bounding boxes/confidence), and `raw_api_response`.
* Caches active text to `Source.raw_transcription`.
6. On page failure:
* Writes `JobSource` record with `status='failed'` and `error_detail`.
7. Once all page tasks resolve:
* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
## Domain Ownership & Invariants
* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
## Data Model Summary
* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
* `Job` has many `JobSource` execution records.
## Test Strategy
* Unit tests for Pydantic V2 schemas, custom validators, and JSONB serialization.
* Integration tests for async PostgreSQL connection handling and parameterized queries.
* Async workflow tests using mock AI providers to verify `partial_success` and retry logic.
* UI integration tests for multi-page rendering and person management.
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
docs/ddl_v2.sql
-102
View File
@@ -1,102 +0,0 @@
## PostgreSQL DDL Specification (Version 2)
```sql
-- Enable pgcrypto for UUID generation if on PostgreSQL < 13
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
-- 1. PERSON TABLE
CREATE TABLE person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
full_name TEXT NOT NULL,
display_name TEXT,
maiden_name TEXT,
birth_date DATE,
birth_date_raw TEXT,
birth_place TEXT,
death_date DATE,
death_date_raw TEXT,
death_place TEXT,
biography TEXT,
portrait_path TEXT,
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 2. DOCUMENT TABLE
CREATE TABLE document (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
document_type TEXT,
document_date DATE,
document_date_raw TEXT,
location_created TEXT,
notes TEXT,
archive_identifier TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 3. DOCUMENT_PERSON (Junction Table for Multi-Author / Multi-Recipient)
CREATE TABLE document_person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
person_id UUID NOT NULL REFERENCES person(id) ON DELETE CASCADE,
role VARCHAR(20) NOT NULL, -- 'author' or 'recipient'
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_document_person_role UNIQUE (document_id, person_id, role)
);
-- 4. JOB TABLE (Batch-level orchestrator)
CREATE TABLE job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'queued', -- 'queued', 'processing', 'completed', 'partial_success', 'failed'
retry_count INTEGER NOT NULL DEFAULT 0,
provider TEXT NOT NULL, -- e.g., 'openai', 'anthropic'
model TEXT NOT NULL, -- e.g., 'gpt-4o', 'claude-3-5-sonnet'
prompt_name TEXT,
date_created TIMESTAMPTZ NOT NULL DEFAULT now(),
date_updated TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 5. SOURCE TABLE (Physical image files & active state)
CREATE TABLE source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
page_number INTEGER NOT NULL DEFAULT 1,
upload_name TEXT NOT NULL,
filename TEXT NOT NULL,
file_path TEXT NOT NULL,
raw_transcription TEXT, -- Cached active AI text output
revised_text TEXT, -- Active human edited text
date_uploaded TIMESTAMPTZ NOT NULL DEFAULT now(),
date_revised TIMESTAMPTZ
);
-- 6. JOB_SOURCE (Junction Table: Per-Image Execution Record)
CREATE TABLE job_source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
job_id UUID NOT NULL REFERENCES job(id) ON DELETE CASCADE,
source_id UUID NOT NULL REFERENCES source(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'pending', -- 'pending', 'transcribed', 'failed'
raw_transcription TEXT, -- Point-in-time raw AI text output
ai_metadata JSONB, -- Page-level bounding boxes, tokens, confidence
raw_api_response JSONB, -- Complete REST response envelope
error_detail TEXT,
executed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_job_source UNIQUE (job_id, source_id)
);
-- INDEXES FOR FAST LOOKUPS & QUERY PERFORMANCE
CREATE INDEX idx_person_full_name ON person(full_name);
CREATE INDEX idx_document_date ON document(document_date);
CREATE INDEX idx_document_person_doc ON document_person(document_id);
CREATE INDEX idx_document_person_per ON document_person(person_id);
CREATE INDEX idx_source_document ON source(document_id);
CREATE INDEX idx_source_page_order ON source(document_id, page_number);
CREATE INDEX idx_job_document ON job(document_id);
CREATE INDEX idx_job_source_job ON job_source(job_id);
CREATE INDEX idx_job_source_source ON job_source(source_id);
CREATE INDEX idx_job_source_ai_metadata ON job_source USING GIN (ai_metadata);
```
docs/error_handling_v2.md
-88
View File
@@ -1,88 +0,0 @@
# Error Handling Policy (Version 2)
This document defines the canonical error-handling policy for the V2 document transcription system.
## Error Handling Objectives
* Make failures visible in clear, actionable language at both the document and individual page levels.
* Support **isolated failure handling** in multi-image batches so single page errors do not crash an entire batch job.
* Preserve diagnostic detail (Pydantic validation errors, raw provider responses) in PostgreSQL `JSONB` for fast troubleshooting.
* Ensure consistent error envelope structure across API, UI, and async worker boundaries.
## Scope And Authority
Governs error behavior across NiceGUI pages, FastAPI routes, service orchestration, `asyncio` background tasks, PostgreSQL interactions, and AI provider adapters.
## Error Taxonomy
| Category | Definition | Retriable |
| --- | --- | --- |
| `validation_error` | Pydantic payload or parameter schema validation failure | no |
| `user_input_error` | Unacceptable user file (unsupported image type, corrupt file) | no |
| `not_found_error` | Requested resource (`Document`, `Source`, `Person`, `Job`) missing | no |
| `conflict_error` | Operation violates state constraints (e.g., duplicate `document_person` role) | no |
| `external_provider_error` | AI Provider API failure (rate limit, vision execution error) | yes |
| `infrastructure_transient_error` | Temporary DB connection reset or HTTP timeout | yes |
| `infrastructure_persistent_error` | Database down, missing API credentials, misconfiguration | no |
| `internal_unexpected_error` | Uncaught Python exception or logic defect | no |
## Async Batch & Page-Level Error Behavior
In multi-image `asyncio` batch processing:
1. **Page Isolation:** Exceptions caught during individual page calls are caught within the `asyncio` task wrapper.
2. **Page Record Logging:** Page failure detail is written directly to `job_source.error_detail` and `job_source.status = 'failed'`.
3. **Batch Aggregate State:**
* If **all** page tasks succeed -> `job.status = 'completed'`.
* If **some** page tasks fail -> `job.status = 'partial_success'`.
* If **all** page tasks fail -> `job.status = 'failed'`.
4. **Retry Strategy:** The UI exposes a "Retry Failed Pages" option for `partial_success` jobs, which spawns a new targeted `Job` containing *only* the `Source` IDs marked as `failed`.
## API Error Response Contract
API error responses return a structured JSON envelope:
```json
{
"error_id": "err_uuid_12345",
"category": "validation_error",
"message": "The uploaded payload failed schema validation.",
"suggestion": "Check file format and metadata fields, then try again.",
"details": {
"pydantic_errors": [...]
},
"timestamp": "2026-07-31T07:55:00Z"
}
```
HTTP Status Mappings:
* `validation_error`, `user_input_error` -> `400`
* `not_found_error` -> `404`
* `conflict_error` -> `409`
* `external_provider_error` -> `502` / `503`
* `infrastructure_transient_error` -> `503`
* `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- Error Handling Policy (this document)
- [Implementation Plan](implementation_plan_v2.md)
docs/implementation_plan_v2.md
-248
View File
@@ -1,248 +0,0 @@
# Implementation Plan (Version 2)
This plan defines the path from the V1 baseline to **Version 2 complete**, aligned to the updated multi-image and multi-person relational domain model:
* `Document` acts as a logical parent container for physical artifacts, supporting multi-author and multi-recipient relationships via `DocumentPerson`.
* `Source` represents an individual image page within a document, maintaining sequential order (`page_number`), cached active machine output (`raw_transcription`), and inline single user revisions (`revised_text`).
* `Job` acts as an overarching batch orchestrator for multi-page async processing tasks.
* `JobSource` records individual point-in-time API executions per image page, storing Pydantic-validated `ai_metadata` and raw REST envelopes (`raw_api_response`).
* **Pydantic V2** acts as the single source of truth for runtime validation, API payload parsing, and PostgreSQL JSONB serialization.
The objective is to complete the V2 scope with production readiness while keeping non-V2 enhancements out of active delivery.
---
## V2 Completion Definition
V2 is complete when all of the following are true:
1. **Functional complete**
* Multi-image and whole-folder uploads assign sequential page numbers to `Source` records under a single `Document`.
* Batch jobs process pages concurrently using an `asyncio` worker pool with semaphore rate limiting.
* Partial job failures resolve cleanly to `partial_success`, allowing single-page retries without re-running successful pages.
* Multi-author and multi-recipient tagging is supported on `Document`.
2. **Data-model complete**
* SQLite is fully replaced with PostgreSQL (using `asyncpg` or `psycopg3`).
* Pydantic V2 models validate all API payloads, database row mappings, and `JSONB` structures.
3. **Operational complete**
* Concurrency controls, worker pool metrics, and database connections operate safely under batch load.
4. **Documentation complete**
* `schema_v2.md`, `DDL_v2.sql`, Pydantic model contracts are updated and consistent.
---
## Phase 1 — Data Contract Stabilization & Pydantic Baseline
**Goal:** Lock the PostgreSQL schema, DDL, and Pydantic V2 models before refactoring service logic.
### Tasks
1. Finalize DDL for PostgreSQL native types (`UUID`, `TIMESTAMPTZ`, `JSONB`) and junction tables (`document_person`, `job_source`).
2. Build core Pydantic V2 schemas (`Person`, `Document`, `Source`, `Job`, `JobSource`, `PageAIMetadata`).
3. Confirm and document data invariants:
* `source.raw_transcription` and `job_source.raw_transcription` are immutable machine outputs.
* `source.revised_text` holds user edits. UI renders `COALESCE(revised_text, raw_transcription)`.
* Page sequence is strictly ordered by `source.page_number ASC`.
4. Freeze V2 job status values (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page execution status values (`pending`, `transcribed`, `failed`).
### Deliverables
* Canonical `docs/schema_v2.md` and `docs/DDL_v2.sql`.
* Centralized Pydantic validation suite in `models/schemas_v2.py`.
### Exit Criteria
* All database tables, relationships, and JSONB structures have corresponding Pydantic V2 models passing unit validation tests.
---
## Phase 2 — Persistence Layer Transition (SQLite to PostgreSQL)
**Goal:** Replace the SQLite storage layer with an asynchronous PostgreSQL driver (`asyncpg` or `psycopg3`).
### Tasks
1. Configure PostgreSQL database connection pooling and environment configuration.
2. Refactor `services/store.py` / repository layers to execute parameterized async SQL queries (`$1`, `$2`).
3. Implement JSONB serialization and deserialization helpers using Pydantic's `.model_dump_json()` and `.model_validate()`.
4. Implement database bootstrap routines for PostgreSQL table creation and index initialization.
### Deliverables
* PostgreSQL-native database connection and query service modules.
* Integration test suite confirming connection pooling and JSONB CRUD operations.
### Exit Criteria
* All database reads/writes run asynchronously against PostgreSQL with zero remaining SQLite driver dependencies.
---
## Phase 3 — Service Layer & `asyncio` Engine Refactor
**Goal:** Implement batch orchestration and parallel single-image API execution.
### Tasks
1. Refactor upload service to process folder/multi-image input:
* Group files into a single `Document`.
* Create ordered `Source` rows (`page_number = 1..N`).
2. Refactor `services/workflows.py` with `asyncio` worker pools:
* Use `asyncio.Semaphore` to enforce API provider rate limits.
* Issue parallel single-image requests to Vision APIs (OpenAI/Claude).
* Parse API responses directly into Pydantic models (`PageAIMetadata`).
3. Update execution tracking:
* Create a `JobSource` row per page call to record `raw_transcription`, `ai_metadata`, and `raw_api_response`.
* Update active `source.raw_transcription` upon task completion.
* Calculate aggregate batch status (`completed`, `partial_success`, `failed`) on the parent `Job`.
4. Refactor `services/person.py` and `services/documents.py` to handle multi-person roles via `document_person`.
### Deliverables
* Asynchronous batch execution engine in `services/workflows.py`.
* Service routines for multi-person tagging and page-level retries.
### Exit Criteria
* Executing a folder upload of 10+ images processes concurrently, populates page-level `JobSource` entries, and handles partial worker errors without crashing the batch.
---
## Phase 4 — UI & API Contract Alignment
**Goal:** Update API endpoints and frontend/UI views to render multi-page documents and person roles.
### Tasks
1. Update document and job API endpoints to accept batch file arrays and multi-person ID payloads.
2. Update UI document views:
* Render multi-page document transcriptions sequentially by `page_number`.
* Display author and recipient chips/cards linked from `document_person`.
3. Update job detail UI to show page-level execution statuses (`transcribed` vs. `failed`) and provide a "Retry Failed Pages" action for `partial_success` jobs.
4. Align inline page editing controls to update `source.revised_text` and `source.date_revised`.
### Deliverables
* Refactored API routes and UI components supporting multi-page rendering and person management.
### Exit Criteria
* UI successfully displays multi-page document text, allows per-page human revisions, and shows author/recipient metadata.
---
## Phase 5 — Test Suite Realignment & Concurrency Testing
**Goal:** Ensure end-to-end system stability under concurrent async execution and load.
### Tasks
1. Write unit tests for Pydantic models, custom validators, and JSONB conversions.
2. Write integration tests for async database operations:
* CRUD for `Document`, `Person`, `DocumentPerson`, `Source`, `Job`, and `JobSource`.
3. Write mock-backed async workflow tests:
* Verify `asyncio.Semaphore` bounds concurrent tasks properly.
* Validate state transition logic for `completed`, `partial_success`, and `failed` jobs.
* Confirm retry routines process only targeted `JobSource` records marked as `failed`.
4. Re-enable CI quality gates (linting, type checking with Pyright/mypy, pytest).
### Deliverables
* Passing asynchronous test suite covering core workflows, edge cases, and failure recoveries.
### Exit Criteria
* CI pipeline is green with comprehensive coverage across database operations, Pydantic models, and worker queues.
---
## Phase 6 — Reliability, Operations, and Release Readiness
**Goal:** Prepare V2 for production deployment and operator management.
### Tasks
1. Verify structured logging includes `job_id`, `document_id`, `source_id`, and `person_id`.
2. Tune PostgreSQL connection pool limits and `asyncio` concurrency thresholds for production infrastructure.
3. Update operational documentation:
* Review and update `docs/schema_v2.md` as needed.
* Create `docs/runbook_v2.md` detailing PostgreSQL maintenance, JSONB index management, and worker queue monitoring.
* Create `docs/release_checklist_v2.md` for launch sign-off.
### Deliverables
* Updated project documentation and operational runbooks.
* V2 release sign-off checklist.
### Exit Criteria
* All documentation reflects V2 architecture; launch checklist is fully verified.
---
## Requirement Traceability Focus
Maintain evidence against these V2 requirement groups:
* **Batch & Multi-Image Pipeline:** Folder ingestion, page ordering, async worker execution.
* **Database & Persistence:** PostgreSQL, native UUIDs, JSONB execution storage, `asyncpg` pooling.
* **Validation & Schemas:** Pydantic V2 models for DB rows, API requests, and AI vision responses.
* **Attribution & Metadata:** Multi-author and multi-recipient tagging, biographical entity management.
* **Error Recovery:** Partial success states, page-level status flags, isolated retry execution.
---
## Scope Discipline Rule (V2 Focus)
* Only tasks required for V2 scope (PostgreSQL, Pydantic V2, folder/async processing, multi-person roles) enter this plan.
* V3 candidate features (such as side-by-side multi-provider model output comparison) remain strictly in the future backlog.
* Any schema adjustments during implementation require immediate updates to `DDL_v2.sql`, Pydantic models, and `schema_v2.md`.
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- Implementation Plan (this document)
docs/index_v2.md
-47
View File
@@ -1,47 +0,0 @@
# Document Transcription System Overview (Version 2)
This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
## Start Here
Read [architecture_v2.md](architecture_v2.md) first for technical overview and system design.
## Core V2 Capabilities
* **Folder & Multi-Image Ingestion:** Upload whole folders or image batches that map sequentially (`page_number`) under a single `Document`.
* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
* **Robust PostgreSQL Storage:** Relational storage for entities with native `UUID`, `TIMESTAMPTZ`, and `JSONB` for deep AI spatial metadata and raw envelopes.
* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSONB payload validation.
* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
## Technical Stack
* **Application Web Framework:** FastAPI + NiceGUI
* **Persistence Engine:** PostgreSQL 13+
* **Data Validation & Schemas:** Pydantic V2
* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
* **Vision Providers:** OpenAI (GPT-4o) and Anthropic (Claude 3.5 Sonnet) via native SDKs
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Documentation Index
- System Overview (this document)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/requirements_v2.md
-42
View File
@@ -1,42 +0,0 @@
# Document Transcription System Requirements (Version 2)
This document captures the **Version 2 baseline requirements** for the production implementation.
## Requirements Model
| ID | Category | Requirement | Verify Method |
| --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
| REQ-1 | Functional | Allow users to upload folders or multi-image batches as sequential `Source` pages under a `Document`. | test |
| REQ-2 | Functional | Process multi-page jobs asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Persist page-level execution outputs (`raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
| REQ-4 | Functional | Support job states (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page states (`pending`, `transcribed`, `failed`). | inspection |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple authors/recipients to a `Document` via `DocumentPerson`. | test |
| REQ-6 | Functional | Maintain immutable original machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
| REQ-7 | Data Constraint | Store all persistent domain data in PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` columns. | inspection |
| REQ-8 | Data Constraint | Validate all API requests, database rows, and JSONB structures using Pydantic V2 schemas. | test |
| REQ-9 | Interface | Render multi-page transcriptions sequentially by `page_number` in the web UI with author/recipient metadata. | demonstration |
| REQ-10 | Operations | Allow operators to retry only failed pages for jobs in a `partial_success` state. | test |
## Element Satisfaction Mapping
* **UI (NiceGUI):** Satisfies REQ-1, REQ-5, REQ-6, REQ-9, REQ-10.
* **API (FastAPI):** Satisfies REQ-1, REQ-4, REQ-5, REQ-8.
* **WORKER (asyncio):** Satisfies REQ-2, REQ-3, REQ-4, REQ-10.
* **PERSISTENCE (PostgreSQL):** Satisfies REQ-3, REQ-6, REQ-7.
* **MODELS (Pydantic V2):** Satisfies REQ-8.
---
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- System Requirements (this document)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/schema_v2.md
-136
View File
@@ -1,136 +0,0 @@
# Database Schema (Version 2)
This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
## Entity Relationship Diagram
```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT {
UUID id PK
TEXT name
TEXT document_type
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
VARCHAR role "author | recipient"
TIMESTAMPTZ created_at
}
JOB {
UUID id PK
UUID document_id FK
VARCHAR status "queued | processing | completed | partial_success | failed"
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status "pending | transcribed | failed"
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```
## Domain Invariants & Rules
### Page-Level Execution & AI Outputs
* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
### Page Ordering & Revisions
* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
### Async Job Lifecycle & Failure Isolation
* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* Job States:
- queued: Created, awaiting worker execution.
- processing: Concurrent HTTP tasks actively running.
- completed: 100% of linked job_source tasks succeeded (transcribed).
- partial_success: At least one job_source succeeded and at least one failed.
- failed: All linked job_source tasks failed or a job-level runtime error occurred.
### Attribution & Person Roles
* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
---
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- Data model (this document)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/test_block-beta.md
-92
View File
@@ -1,92 +0,0 @@
```mermaid
block-beta
columns 3
%% UI Component Column
block:UI["UI COMPONENTS / WIREFRAME"]:1
columns 1
block:HeaderUI["Header & Nav"]:1
columns 1
h_title["[Text] Document Name & Type"]
h_date["[Text] Date & Origin Location"]
end
block:EditorUI["Page Transcription Editor"]:1
columns 1
ed_img["[Image Viewer] Source Image"]
ed_page["[Badge] Page Number"]
ed_raw["[Read-Only] AI Raw Output"]
ed_rev["[Textarea] Human Revised Text"]
end
block:PeopleUI["Attribution Sidebar"]:1
columns 1
p_author["[List] Authors (Full Name)"]
p_recip["[List] Recipients (Full Name)"]
p_bio["[Card] Person Biography & Dates"]
end
block:JobUI["AI Processing Drawer"]:1
columns 1
j_status["[Badge] Job Status"]
j_model["[Text] Provider & Model"]
j_tokens["[JSON View] AI Token Usage"]
end
end
%% Directional Mapping / Connectors
block:FLOW["MAPPING / FLOW"]:1
columns 1
f1["Reads / Updates -->"]
f2["Renders Active Page -->"]
f3["Joins via Role -->"]
f4["Executes & Logs -->"]
end
%% Postgres Schema Column
block:DB["POSTGRES SQL SCHEMA"]:1
columns 1
block:DocTbl["Table: document"]:1
columns 1
d_id["id : UUID (PK)"]
d_name["name : TEXT"]
d_type["document_type : TEXT"]
d_date["document_date : DATE"]
end
block:SrcTbl["Table: source"]:1
columns 1
s_id["id : UUID (PK)"]
s_page["page_number : INT"]
s_path["file_path : TEXT"]
s_raw["raw_transcription : TEXT"]
s_rev["revised_text : TEXT"]
end
block:PersonTbl["Table: person & document_person"]:1
columns 1
p_id["id : UUID (PK)"]
p_name["full_name : TEXT"]
p_role["role : 'author' | 'recipient'"]
end
block:JobTbl["Table: job & job_source"]:1
columns 1
j_id["id : UUID (PK)"]
j_stat["status : VARCHAR"]
j_prov["provider / model : TEXT"]
j_meta["ai_metadata : JSONB"]
end
end
%% Connections
HeaderUI --> DocTbl
ed_img --> s_path
ed_page --> s_page
ed_raw --> s_raw
ed_rev --> s_rev
PeopleUI --> PersonTbl
JobUI --> JobTbl
```
docs/test_flowchart.md
-53
View File
@@ -1,53 +0,0 @@
```mermaid
flowchart LR
subgraph UI["UI Components / Wireframe"]
direction TB
subgraph HeaderUI["Header & Nav"]
h_title["[Text] Document Name & Type"]
h_date["[Text] Date & Origin Location"]
end
subgraph EditorUI["Page Transcription Editor"]
ed_img["[Image Viewer] Source Image"]
ed_page["[Badge] Page Number"]
ed_raw["[Read-Only] AI Raw Output"]
ed_rev["[Textarea] Human Revised Text"]
end
subgraph PeopleUI["Attribution Sidebar"]
p_author["[List] Authors / Recipients"]
end
subgraph JobUI["AI Processing Drawer"]
j_status["[Badge] Job Status"]
end
end
subgraph DB["Postgres SQL Schema"]
direction TB
subgraph DocTbl["Table: document"]
d_name["name : TEXT"]
d_type["document_type : TEXT"]
end
subgraph SrcTbl["Table: source"]
s_path["file_path : TEXT"]
s_page["page_number : INT"]
s_raw["raw_transcription : TEXT"]
s_rev["revised_text : TEXT"]
end
subgraph PersonTbl["Table: person & document_person"]
p_name["full_name : TEXT"]
p_role["role : author | recipient"]
end
subgraph JobTbl["Table: job & job_source"]
j_stat["status : VARCHAR"]
j_meta["ai_metadata : JSONB"]
end
end
%% Mappings
HeaderUI --> DocTbl
ed_img --> s_path
ed_page --> s_page
ed_raw --> s_raw
ed_rev --> s_rev
PeopleUI --> PersonTbl
JobUI --> JobTbl
```
docs/transcription_methodology.md
-54
View File
@@ -1,54 +0,0 @@
# Transcription Methodology & Style Guide
## 1. Overview & Core Philosophy
This document defines the formal transcription standard for processing historical manuscripts, letters, diaries, and printed ephemera.
Following the principles established by Mary-Jo Kline in A Guide to Documentary Editing, this project adheres to a Strict Literal Transcription (Verbatim) model as its foundational layer. The primary goal is total textual fidelity—capturing what the author wrote, not what they intended to write—while ensuring the output remains machine-readable and indexable for downstream digital query and search systems.
## 2. Textual Policy
Transcribers (human or AI) must record the exact text of the source document without silent corrections, modernizations, or stylistic smoothing except where explicitly instructed in this guide.
* **Substantives:** Words, letter forms, structural layout, and semantic content must be recorded strictly as presented in the original document.
* **Accidentals:** Punctuation, capitalization, misspellings, and archaic character representations must be preserved unless an explicit rule below allows for standardization.
## 3. Standard Transcription Rules & Markup
The following rules map directly to editorial conventions for handling common manuscript anomalies and physical document features.
### 3.1 Textual Anomalies & Corrections
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Misspellings & Errors** | Retain original spelling verbatim. Insert an italicized [sic] immediately following the error. Do not correct spelling silently. | [sic] | The weather was very cold and publick [sic] business delayed. |
| **Missing Words / Omissions** | Insert necessary words required to restore basic grammatical sense inside square brackets. | [word] | We went [to] the store to buy supplies. |
| **Uncertain / Conjectural** | Place best hypothesis followed by a question mark inside square brackets when handwriting is doubtful. | [word?] | He went to [Boston?] yesterday to meet the governor. |
| **Completely Illegible** | Use [illegible] for unreadable script. Use explicit damage descriptors when physical impairment prevents reading. | [illegible] or [reason] | The total cost was [illegible] dollars. or The letter ends here [remainder of page torn]. |
| **Canceled / Struck-through** | Wrap text removed by the author inside a [deleted: ...] tag to preserve authorial revisions. | [deleted: text] | We left at [deleted: noon] one o'clock instead. |
| **Interlineations / Additions** | Wrap text inserted above, below, or in margins into the narrative flow inside an [inserted: ...] tag. | [inserted: text] | The [inserted: red] house on the hill was abandoned. |
### 3.2 Typography, Characters & Layout
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Superscripts & Abbreviations** | Bring raised letters down to the main line. Optionally expand abbreviations within square brackets based on project configuration. | [expanded] | Gen^l becomes Genl or Gen[era]l. |
| **Line-End Hyphenation** | Rejoin words split across a page or line boundary silently, dropping the soft hyphen. | Silently rejoin | Original: "estab- / lishment" becomes establishment |
| **Capitalization** | Preserve explicit capitalization. Default to modern capitalization rules only when authorial intent is ambiguous or archaic forms confuse sentence structure. | Literal / Contextual | If a standard noun like 'Farm' is clearly capitalized, record 'Farm'. If ambiguous, default to 'farm'. |
| **Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using standard spacing. Do not correct sequence or mathematical errors. | Preserve syntax | I. Main Topic a. Sub-point b. Next pointIII. [sic] Third Topic |
### 3.3 Visual & Spatial Elements
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Non-Textual Artifacts** | Record non-textual elements (seals, stamps, sketches, physical damage) using brief descriptive text inside square brackets. | [description] | [wax notary seal attached here] or [sketch of a fort layout] |
| **Marginalia & Addenda** | Explicitly indicate spatial transitions before transcribing content located in margins or non-standard orientations. | [location:] | [written in left margin:] Do not share this with anyone. |
## 4. Prompt Asset Integration
When executing programmatic transcriptions via LLM APIs or local models, processing instructions must be packaged into single-purpose system prompts aligned with these rules.
1. **Isolation:** Each transcription prompt file exists as an independent Markdown asset in the repository.
2. **Deterministic Output:** Prompts must explicitly instruct models to follow the markup standards in Section 3 without introducing conversational wrappers, extra prose, or structural markdown outside the source document's native layout.
3. **Iterative Scoping:** Rule modifications or edge-case additions must be submitted as isolated delta commits to individual prompt files to maintain clean revision tracking.
docs/ver1/architecture_v1.md
-308
View File
@@ -1,308 +0,0 @@
# System Architecture (Version 1)
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
## Architecture Objectives
The production architecture is designed to:
- preserve verbatim family-history source material as searchable text
- keep operational complexity low for a personal deployment
- support asynchronous transcription without requiring distributed infrastructure
- maintain clear module boundaries so extensions can be added incrementally
## Production Scope And Scale
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
Current scope includes:
- content source upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- original transcription review and optional revision review
- full-text search over accepted transcripts
- export of transcript data
## Deployment Topology
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
### Baseline Topology (Two Containers)
- one application container
- one PostgreSQL container
- embedded background worker execution inside the app process
### Expanded Topology (Three Containers)
- application container
- PostgreSQL container
- MongoDB container
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
## Runtime Architecture
```mermaid
flowchart LR
User[Browser User] --> App[FastAPI + NiceGUI Service]
App --> Worker[In-process Background Worker]
App --> PG[(PostgreSQL)]
App --> MG[(MongoDB Document Store)]
Worker --> AI[Transcription Provider]
Worker --> PG
Worker --> MG
```
## Runtime Ownership And Startup Policy
The current implementation now uses explicit lifespan-owned runtime resources.
- application lifespan initializes and disposes database runtime resources
- worker lifecycle is owned by application lifespan startup/shutdown
- worker receives lifespan-owned database engine dependency explicitly
- schema bootstrap policy is environment-aware and explicit:
- development/test default to bootstrap enabled
- production defaults to bootstrap disabled
- explicit override is available via configuration
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
## Layered Module Structure
### Interface Layer
Responsibility:
- HTTP API and UI routes
- request/response validation
- status and result presentation
Out of scope:
- business-rule enforcement
- data-access implementation
### Application Layer
Responsibility:
- upload and job orchestration
- state transitions and retry policy
- coordination across domain and infrastructure ports
Out of scope:
- provider-specific protocol details
- ORM or storage-specific logic
### Domain Layer
Responsibility:
- verbatim transcription policy
- revision and provenance invariants
- confidence and annotation semantics
Out of scope:
- web framework concerns
- database and network I/O
### Infrastructure Layer
Responsibility:
- persistence adapters (PostgreSQL and MongoDB)
- transcription-provider adapter
Out of scope:
- business policy decisions
## Processing Workflow
Production transcription flow:
1. A user uploads one or more content sources through the UI or API.
2. The application validates payloads and creates document, source, and job records.
3. The in-process worker de-queues the job and calls the transcription provider.
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
## Data Model Ownership
System-of-record entities:
- documents and content sources
- transcription jobs, original transcription, and status events
- transcript revisions
- provenance metadata
### Original Transcription And Revision Ownership
- each processing job stores the original immutable provider output (`text`)
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
- revisions are optional user-authored edits linked to a content source
- a revision can be created from original `job.text`
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
- a document groups one or more content sources (images, PDFs, and future source types)
Storage strategy:
- PostgreSQL for relational system-of-record entities
- MongoDB for document-oriented payloads and large transcription artifacts
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
- in-memory execution state treated as ephemeral
## Transcription Prompt Asset Policy
The production system treats transcription prompts as maintainable content assets.
- each transcription prompt is stored in its own Markdown file
- prompt files are designed for direct human editing and iterative refinement
- prompt updates are independent and do not require bundling unrelated prompt changes
- prompt file identity and revision history are tracked through normal repository version control
## Simplicity Guardrails
The production system enforces these constraints to prevent accidental over-engineering:
- PostgreSQL in a container is treated as a lightweight default dependency
- MongoDB in a container is treated as a lightweight optional dependency
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
- no dedicated queue or search cluster is introduced without measured need
- external infrastructure is added only behind existing ports/adapters
## Extension Path
The architecture supports additive growth without changing domain contracts.
### Stage 1: Foundation (Current)
- upload, transcription, review, search, export
- in-process worker execution
- single provider adapter
- app plus PostgreSQL deployment
### Stage 2: Throughput Hardening
- optional MongoDB document-store enablement
- optional external worker/queue process
- stronger retry and dead-letter handling
### Stage 3: Intelligence Features
- entity extraction and cross-document linking
- timeline and narrative assembly
- optional multi-provider routing
Each stage preserves existing module boundaries and keeps migration risk low.
## Test Strategy
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
### Unit Tests
- domain transcription rules and annotation behavior
- revision-history invariants
- job state-transition logic
### Integration Tests
- repository behavior and transaction boundaries
- persistence-adapter and provider adapter contract mapping
- upload-to-persistence roundtrip
### End-to-End Tests
- happy path: upload, transcribe, review, search, export
- failure path: provider error, retry, surfaced failed status
### CI Execution Model
- fast suite on each push
- optional slower provider-sandbox checks on scheduled runs
## Risks And Controls
### Runtime Responsiveness
Risk:
- long jobs can reduce responsiveness in a single-process deployment
Control:
- bounded concurrency and visible job status in the UI
### Database Concurrency Limits
Risk:
- contention can appear under sustained concurrent writes in personal-scale infrastructure
Control:
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
### Provider Output Variance
Risk:
- transcription quality varies by content source type, handwriting legibility, and source quality
Control:
- first-class human review and immutable revision history
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [Docker Compose documentation](https://docs.docker.com/compose/)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [MongoDB documentation](https://www.mongodb.com/docs/)
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
- Background job: Work executed outside the request/response path so the UI remains responsive.
- Boundary: A strict separation between modules with different responsibilities.
- CI (Continuous Integration): Automated test execution for code changes.
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
- Domain layer: The module that contains core business rules and invariants.
- End-to-end test: A test that validates a full user flow across the running system.
- Full-text search: Text indexing and querying optimized for natural-language search.
- In-process worker: A background executor that runs within the same application process.
- Integration test: A test that verifies interactions between real modules and infrastructure components.
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
- Modular monolith: A single deployable application with strongly separated internal modules.
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
- Provenance: Metadata that records where generated data came from and how it was produced.
- Revision history: Optional versioned record of user-authored transcription edits over time.
- System of record: The authoritative persistent store for canonical data.
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
docs/ver1/error_handling_v1.md
-288
View File
@@ -1,288 +0,0 @@
# Error Handling Policy
This document defines the canonical error-handling policy for the document transcription system. It is the single source of truth for how errors are classified, surfaced to users, logged for diagnosis, and handled across UI, API, service, worker, and provider boundaries.
## Error Handling Objectives
The production error-handling model is designed to:
- make failures visible to the user in clear, actionable language
- preserve enough diagnostic detail for fast troubleshooting
- keep module behavior consistent across all boundaries
- distinguish expected domain failures from unexpected defects
- support safe retries for transient failures without hiding persistent faults
## Scope And Authority
This page governs error-handling behavior for:
- UI interactions (NiceGUI pages)
- API endpoints (FastAPI routes)
- application services and orchestration logic
- in-process background worker execution
- external provider adapters and persistence adapters
If implementation behavior conflicts with this document, this document is authoritative and implementation should be updated.
## Core Principles
- **Clarity first:** user-facing messages should explain what failed in plain language.
- **Actionability required:** each surfaced error should include a suggested next step.
- **Safety by default:** internal details are logged; sensitive details are not exposed by default in UI/API.
- **Consistency across boundaries:** category and structure should remain stable from source to surface.
- **Fail explicitly:** silent failure is prohibited.
- **Traceability:** every non-trivial error should be traceable with an error reference ID.
## Error Taxonomy
The system uses stable, implementation-independent categories:
| Category | Definition | Typical Source | Retriable |
| --- | --- | --- | --- |
| `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no |
| `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes |
| `not_found_error` | Requested resource does not exist | missing job/document/source/revision | no |
| `conflict_error` | Requested operation violates current state constraints | invalid state transition | no |
| `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes |
| `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes |
| `infrastructure_persistent_error` | Non-transient environment issue | missing permissions, misconfiguration | no |
| `internal_unexpected_error` | Unhandled defect or unknown failure | uncaught exceptions, logic errors | unknown (default no) |
### Classification Rules
- Classification occurs as close as possible to the origin boundary.
- Provider-specific exceptions must be normalized into taxonomy categories before crossing service boundaries.
- Unknown exceptions are classified as `internal_unexpected_error` and logged with traceback.
- Category names are stable contracts and must not be changed casually.
## User-Facing Error Experience Contract
When an error is shown in the GUI, it must include:
1. **Title** (short context, e.g., “Upload failed”)
2. **Message** (plain-language explanation)
3. **Suggested action** (explicit next step)
4. **Error reference ID** (for support/debug traceability)
5. **Technical details** (optional/collapsible for advanced users)
### UI Message Rules
- Do not expose raw stack traces by default.
- Do not expose secrets, credentials, connection strings, or filesystem internals unless explicitly in debug tooling.
- Prefer domain language over implementation language.
- Use persistent visibility for important failures (dialog/card), not only transient toasts.
### Suggested Action Requirements
Every user-visible error must include a suggested course of action, such as:
- retry the operation
- check file type/size constraints
- refresh the jobs page
- verify environment configuration
- contact operator with error ID and timestamp
## API Error Response Contract
API errors should return a structured envelope with stable fields:
- `error_id`: short unique reference ID
- `category`: taxonomy category
- `message`: safe human-readable summary
- `suggestion`: recommended next step
- `details`: optional, only when safe and appropriate
- `timestamp`: UTC ISO-8601
HTTP status mapping guidance:
- `validation_error`, `user_input_error` -> `400`
- `not_found_error` -> `404`
- `conflict_error` -> `409`
- `external_provider_error` -> `502` or `503` (depending on failure mode)
- `infrastructure_transient_error` -> `503`
- `infrastructure_persistent_error` -> `500`
- `internal_unexpected_error` -> `500`
## Logging And Observability Contract
All logged errors must include, where available:
- `error_id`
- `category`
- `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`)
- `exception_type`
- `job_id`, `document_id`, `source_id` (when relevant)
- UTC timestamp
Rules:
- Use structured logging fields where practical.
- Use full traceback for unexpected errors (`internal_unexpected_error`).
- Log at boundary handoff points to preserve causal trail.
- Avoid duplicate noisy logging for the same exception at every layer.
## Recovery And Retry Policy
### Retriable Conditions
Retriable failures include:
- transient network/provider timeouts
- intermittent provider unavailability
- temporary DB/network interruptions
### Non-Retriable Conditions
Non-retriable failures include:
- invalid file formats
- missing required data
- permission/configuration failures
- deterministic domain conflicts
### Worker Behavior
- The worker must classify and persist failure details consistently.
- Retries should be bounded by configured limits.
- Exhausted retries must end in explicit failed status with recorded reason.
- No infinite retry loops are allowed.
## Boundary-Specific Responsibilities
### UI Layer
Responsibility:
- display user-safe error summaries and suggested actions
- show persistent error visibility for critical failures
- include error reference IDs in visible output
Out of scope:
- low-level exception parsing
- provider-specific protocol interpretation
### API Layer
Responsibility:
- map application exceptions into stable error envelopes and HTTP statuses
- preserve category and error_id continuity
Out of scope:
- domain-specific remediation logic
### Service Layer
Responsibility:
- classify domain and infrastructure exceptions
- convert adapter-specific failures into taxonomy categories
- return deterministic error types to callers
Out of scope:
- presentation formatting for UI
### Worker Layer
Responsibility:
- execute retry policy for retriable failures
- persist terminal failure details for jobs
- emit operational logs with category and identifiers
Out of scope:
- direct UI messaging
### Provider Adapter Layer
Responsibility:
- normalize provider SDK/HTTP failures into domain-neutral exceptions
- preserve raw provider context for logs (safely)
Out of scope:
- choosing user-facing wording
## Error Lifecycle Workflow
Standard lifecycle:
1. Failure occurs at a boundary or operation.
2. Exception is classified into taxonomy category.
3. `error_id` is created (or propagated).
4. Error is logged with required structured fields.
5. User/API receives safe message + suggested action.
6. Persistent job/resource state is updated when applicable.
7. Tests verify contract behavior for the pathway.
## Test Strategy For Error Handling
### Unit Tests
- category classification behavior
- retry eligibility decisions
- exception-to-message mapping safety
### Integration Tests
- UI pathways show clear message + suggested action for known failures
- API returns structured error envelope with expected status/category
- worker persists failed status and failure detail as required
### Regression Tests
- each previously observed production issue should have a guarding test
- contract tests must cover adapter error normalization behavior
## Known Failure Patterns And Prescribed Responses
| Pattern | Category | User Message | Suggested Action |
| --- | --- | --- | --- |
| Upload payload cannot be parsed by UI handler | `internal_unexpected_error` (until narrowed) | Upload failed due to unexpected processing error | Retry once; if repeated, report error ID and check runtime version compatibility |
| Unsupported extension | `user_input_error` | File type is not supported | Upload JPG, PNG, TIFF, or PDF |
| Empty file upload | `validation_error` | Uploaded file is empty | Choose a valid non-empty file and retry |
| Provider timeout | `external_provider_error` or `infrastructure_transient_error` | Transcription provider timed out | Retry from jobs page; if repeated, check provider status |
| Job lookup missing | `not_found_error` | Requested job was not found | Refresh jobs list and open a valid job |
## Governance And Update Process
This document is a living policy artifact.
Update this document when:
- new error categories are introduced
- handling behavior changes at any boundary
- a production incident reveals missing guidance
- API/UI error contracts change
Change requirements:
- update this document and associated tests in the same change set
- preserve taxonomy stability; if changed, document migration impact
- record noteworthy policy changes in project release notes or changelog
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- Error Handling Policy (this document)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Error category: Stable classification used to drive handling, messaging, and status mapping.
- Error envelope: Structured API payload describing a failure.
- Error reference ID: Short identifier used to correlate user-visible failure with logs.
- Retriable error: Failure likely to succeed on a later attempt without code changes.
- Terminal failure: Failure state after retries are exhausted or retry is not allowed.
docs/ver1/implementation_plan_v1.md
-203
View File
@@ -1,203 +0,0 @@
# Version 1 Implementation Plan
This plan defines the path from current implementation to **Version 1 complete**, aligned to the updated domain model:
- `Document` groups one or more content `Source` records
- `Job` owns original immutable provider output (`text`) and processing metadata
- `Revision` stores optional user-authored edits linked to a `Source`
The objective is to complete V1 scope with production readiness while keeping non-V1 enhancements out of active delivery.
---
## V1 Completion Definition
V1 is complete when all of the following are true:
1. **Functional complete**
- Upload, queue, processing, status display, and transcription result inspection work end-to-end.
- Optional revision workflow is implemented (create/view/update single revision).
2. **Data-model complete**
- Runtime behavior, persistence, and tests all align to `Document` / `Source` / `Job` / `Revision`.
3. **Operational complete**
- Error handling, logs, and runbooks support reliable operation.
4. **Documentation complete**
- Architecture, requirements, schema, error handling, and index are consistent and current.
---
## Phase 1 — Data Contract Stabilization (Schema-First)
**Goal:** Lock a single canonical contract before further feature work.
### Tasks
1. Confirm and document invariants:
- `Job.text` is original immutable transcription output.
- `Revision` is optional and user-authored.
- Revisions are derived from the original `Job.text`.
2. Verify relationship cardinality assumptions:
- `Document` -> many `Source`
- `Document` -> many `Job`
- `Source` -> one `Job`
- `Source` -> one `Revision`
3. Ensure field naming consistency (`date_created`, `date_updated`, `date_uploaded`) across code and docs.
4. Freeze V1 status lifecycle to current implementation (`queued`, `processing`, `transcribed`, `failed`).
### Deliverables
- Updated `schema_v1.md` and `requirements.md` traceability alignment.
- Explicit V1 data invariants section in architecture docs.
### Exit Criteria
- No conflicting definitions of ownership/cardinality/status remain in docs.
---
## Phase 2 — Service Layer Refactor To New Model
**Goal:** Remove all obsolete `Transcript` assumptions from service/workflow code.
### Tasks
1. Refactor `services/transcription.py`:
- Replace transcript CRUD assumptions with job-output + revision operations.
2. Refactor `services/jobs.py`:
- Replace old timestamp/relationship accessors with current model fields.
3. Refactor `services/documents.py` and `services/store.py`:
- Ensure upload creates and links `Document`, `Source`, and `Job` correctly.
4. Refactor `services/workflows.py`:
- Persist original provider output to `Job`.
- Persist failure detail to `Job.error_detail`.
- Use `Revision` only for user-authored edits.
### Deliverables
- Service layer fully aligned with new schema.
### Exit Criteria
- No service module imports or persists `Transcript` model artifacts.
---
## Phase 3 — UI Contract Alignment
**Goal:** Align pages/components to source/job/revision semantics.
### Tasks
1. Update job detail and related UI components:
- Display original immutable transcription from `Job.text`.
- Display optional revision sourced from `Source.revision` (0 or 1).
2. Align date fields with new schema naming.
3. Preserve clear user messaging when no revisions exist.
### Deliverables
- Updated jobs page and detail components.
### Exit Criteria
- UI behavior and labels match documentation and domain model.
---
## Phase 4 — Database Bootstrap, Migration, and Safety
**Goal:** Make schema transition safe in dev/test and repeatable for deployment.
### Tasks
1. Update bootstrap compatibility logic in `db/operations.py`:
- Remove obsolete transcript-table assumptions.
- Add forward-compatible patches for current tables only.
2. Define migration/backfill approach for existing local data.
3. Document rollback and recovery steps.
4. Rehearse migration path against representative data.
### Deliverables
- Migration/upgrade runbook.
- Validated bootstrap behavior for dev/test.
### Exit Criteria
- Migration path is documented and tested with no unresolved data-loss risk.
---
## Phase 5 — Test Suite Realignment
**Goal:** Restore full confidence after the schema redesign.
### Tasks
1. Rewrite model tests for:
- `Document`, `Source`, `Job`, `Revision` relationships and invariants.
2. Rewrite service/integration tests:
- Worker success/failure paths using `Job.text` / `Job.error_detail`.
- Optional single-revision creation/update behavior.
3. Update UI tests for new job-detail/revision rendering behavior.
4. Re-enable strict CI quality gates (lint, type, tests).
### Deliverables
- Updated test matrix and passing CI.
### Exit Criteria
- Critical user flows and failure paths are covered and green.
---
## Phase 6 — Reliability, Operations, and Release Readiness
**Goal:** Ensure V1 is operable and launch-safe.
### Tasks
1. Verify error taxonomy behavior across UI/API/service/worker.
2. Confirm structured logging includes relevant identifiers (`job_id`, `document_id`, `source_id` when applicable).
3. Validate retry behavior and terminal failure handling.
4. Finalize release checklist, deployment steps, and rollback procedure.
5. Execute final acceptance run against requirements traceability.
### Deliverables
- V1 release checklist and acceptance evidence.
- `runbook_v1.md` for incident response and operator workflows.
- `release_checklist_v1.md` for release sign-off.
### Exit Criteria
- Stakeholder sign-off and launch readiness achieved.
---
## Requirement Traceability Focus
The plan must keep clear evidence against these requirement groups:
- **Core flow:** REQ-0 to REQ-6
- **Runtime and operations constraints:** REQ-7 to REQ-12
- **Revision workflow:** REQ-13
A lightweight traceability table should be maintained with:
- requirement ID
- implementation status (`not started` / `in progress` / `done`)
- validation evidence (test name, screenshot, or runbook step)
---
## Suggested Execution Rhythm
- **Weekly:** requirement status and risk review
- **Per PR:** contract checks (model names, field names, lifecycle values)
- **Milestone checks:** end of Phases 2, 4, and 6
---
## Scope Discipline Rule (V1 Focus)
- Only work required to satisfy V1 requirements enters this plan.
- Nice-to-have enhancements are captured in a separate backlog document.
- Schema or contract changes after Phase 1 require explicit approval and traceability impact review.
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- Implementation Plan (this document)
docs/ver1/index_v1.md
-58
View File
@@ -1,58 +0,0 @@
## Document Transcription System Overview
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
## Start Here
Read [architecture_v1.md](architecture_v1.md) first.
The architecture page is the primary technical reference and defines:
- deployed topology and infrastructure limits
- module boundaries and dependency flow
- processing life cycle and data ownership
- test strategy, risk controls, and extension path
## What The Application Does
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
### Core capabilities:
- document grouping with one or more content sources and metadata capture
- asynchronous transcription with visible job status
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
- transcription prompt management with one Markdown file per prompt for human refinement over time
- optional revisions for user-authored edits of original immutable transcription text
- full-text search over accepted transcripts
- export of transcript data
## Production Operating Model
The system runs with minimal operational overhead:
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
- no required queue or search-engine containers in the baseline setup
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
---
## Documentation Map
- System Overview (this document)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
- System of record: The authoritative persistent store for canonical data.
docs/ver1/release_checklist_v1.md
-45
View File
@@ -1,45 +0,0 @@
# V1 Release Readiness Checklist
Use this checklist before declaring V1 operationally complete.
## A) Functional Readiness
- [ ] Upload flow works for supported file types.
- [ ] Worker transitions jobs through `queued -> processing -> transcribed|failed`.
- [ ] Job detail displays immutable original transcription from `Job.text`.
- [ ] Revision workflow supports create/update/view/delete for optional single revision.
## B) Reliability and Error Handling
- [ ] Error categories surface with actionable messages in UI/API pathways.
- [ ] Failed jobs persist `error_detail` and terminal state.
- [ ] Stale processing recovery verified on restart.
- [ ] Retry/timeout behavior validated against configured limits.
## C) Operational Readiness
- [ ] `runbook_v1.md` reviewed and current.
- [ ] `migration_v1.md` reviewed and current.
- [ ] Backup and rollback procedures tested at least once.
- [ ] Incident escalation packet template is known to operators.
## D) Quality Gates
- [ ] Lint/type checks pass.
- [ ] `pytest -m "not external" -q` passes.
- [ ] Targeted external/provider checks executed (if credentials available).
- [ ] Release evidence recorded in `release_evidence_v1.md`.
## E) Traceability and Documentation
- [ ] `requirements_v1.md` aligns with implemented V1 behavior.
- [ ] `architecture_v1.md`, `schema_v1.md`, and `error_handling_v1.md` are consistent.
- [ ] `traceability_v1.md` is updated with current implementation and test evidence.
- [ ] `implementation_plan_v1.md` phase status updated with evidence references.
- [ ] REQ traceability evidence links recorded (tests/runbook/checks).
## Release Sign-Off
- [ ] Technical sign-off complete.
- [ ] Operational sign-off complete.
- [ ] V1 completion date recorded.
docs/ver1/release_evidence_v1.md
-39
View File
@@ -1,39 +0,0 @@
# V1 Release Evidence Log
## Step 5 Quality Gates (2026-07-29)
### Lint
- Command: `python -m ruff check .`
- Result: ✅ pass
- Notes: initial findings were auto-fixed (`ruff --fix`) plus small manual line-wrap/annotation adjustments.
### Tests (primary gate)
- Command: `python -m pytest -m "not external" -q`
- Result: ✅ pass (`[100%]`)
### Tests (external smoke)
- Command: `python -m pytest -m external -q`
- Result: ✅ pass (`[100%]`)
### Type Check
- Command: `python -m ty check src tests`
- Result: ⚠️ not passing
- Summary: existing SQLModel/SQLAlchemy typing incompatibilities and test double typing mismatches remain.
Key current blocker families:
1. SQLModel relationship/query attribute typing (`selectinload`, `order_by`, `.any()`)
2. SQLAlchemy join clause typing in `services/transcription.py`
3. Test fake client type mismatch for `OpenRouterTranscriptionProvider(client=...)`
4. `Settings(**defaults)` typed-dict strictness in `tests/test_config.py`
## Current Gate Status
- Lint: pass
- Non-external tests: pass
- External smoke tests: pass
- Type check: **blocked** (requires dedicated typing cleanup pass)
docs/ver1/requirements_v1.md
-98
View File
@@ -1,98 +0,0 @@
## Document Transcription System Requirements
This page captures a SysML v1.6-style requirements baseline for the production system described in [index_v1.md](index_v1.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
## Scope
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
## Requirements Model (Concise Text Form)
### Requirements
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
### Requirement Relationships
- Contains: REQ-0 contains REQ-1 through REQ-13.
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
- Traces: REQ-5 -> REQ-3.
- Refines: REQ-6 -> REQ-2.
### Architecture Elements
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
### Satisfaction Mapping
- UI satisfies REQ-1, REQ-5, REQ-13.
- API satisfies REQ-5.
- GRAPH satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-10, REQ-13.
- DBDOC satisfies REQ-4, REQ-11.
- OPS satisfies REQ-9.
- PROMPTS satisfies REQ-12.
### Verification Mapping
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
## Requirement Notes
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
## Verification Intent
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
- Inspection: verify architecture and startup/runtime policies in code and configuration.
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
- Test: automate behavioral checks through pytest suites and service-level tests.
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- System Requirements (this document)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
- SysML: Systems Modeling Language used to express structured requirements and traceability.
- System of record: The authoritative persistent store for canonical business data.
docs/ver1/runbook_v1.md
-129
View File
@@ -1,129 +0,0 @@
# V1 Operations Runbook
This runbook provides day-2 operational procedures for the V1 baseline.
## Scope
Applies to:
- local/hosted V1 runtime
- SQLite-backed persistence
- in-process worker lifecycle
- OpenRouter provider integration
## Preconditions
- `.env` contains `OPENROUTER_API_KEY`
- app starts successfully
- `uploads/` and `prompts/` are writable
- health endpoint responds at `/healthz`
## Standard Startup Procedure
1. Start the app using the project-standard command.
2. Open `/healthz` and verify `{"status":"ok"}`.
3. Open `/ui/upload` and submit a small valid file.
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
## Standard Shutdown Procedure
1. Stop the application process.
2. Ensure no active process still holds the SQLite file.
3. If maintenance is planned, copy the DB file before edits:
- `transcription.db` (or configured `DATABASE_URL` file path)
## Incident: Jobs Stuck In `processing`
### Symptoms
- Jobs remain `processing` for longer than provider timeout
- New uploads queue but do not complete
- provider usage increases but no terminal job state is visible
### Checks
1. Confirm app process is still running.
2. Confirm worker loop is active (startup logs include worker lifespan start).
3. Inspect recent app logs for:
- `worker.process_job`
- `error_id`
- `category`
- `job_id` / `document_id` / `source_id`
4. Verify provider credentials and provider status.
### Recovery
1. Restart the app to trigger stale-processing recovery.
2. On startup, app re-queues stale processing jobs based on timeout policy.
3. Re-check jobs page and confirm terminal state progression.
4. If persistent, capture logs + error IDs and move to deep investigation.
## Incident: Provider Authentication Failures
### Symptoms
- failures categorized as provider/auth
- jobs fail quickly with authentication guidance
### Recovery
1. Validate `OPENROUTER_API_KEY` value.
2. Restart app after updating env.
3. Re-run a small transcription to confirm recovery.
## Incident: Upload Failures
### Symptoms
- UI reports upload errors
- unsupported extension or empty payload
### Recovery
1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
2. Validate file is not empty.
3. Validate upload directory permissions.
4. Retry upload.
## Incident: Database File/Permission Issues
### Symptoms
- persistence errors during upload/job update
- startup failures around schema/runtime
### Recovery
1. Confirm the configured DB file path exists and is writable.
2. Confirm parent directory permissions.
3. Restore from last known backup copy if corruption is suspected.
4. Restart app and run smoke test.
## Logging Requirements (Operational)
Operational triage should always capture:
- `error_id`
- category
- operation name
- `job_id`, `document_id`, `source_id` when applicable
- UTC timestamp
## Escalation Packet (When opening an issue)
Include:
- exact timestamp window
- one failing `job_id`
- relevant `error_id` values
- latest 100 lines of app logs
- environment summary (`DATABASE_URL` type, app version/commit)
## Post-Incident Validation
After mitigation, verify:
1. Upload works.
2. One job reaches `transcribed`.
3. One induced failure reaches `failed` with error detail.
4. Jobs page and detail page render correctly.
docs/ver1/schema_v1.md
-98
View File
@@ -1,98 +0,0 @@
## Database Schema (V1 Baseline)
This document describes the current relational schema for the transcription system.
All primary and foreign keys in the domain models are UUID-based in V1.
---
## Schema Diagram
```mermaid
erDiagram
DOCUMENT {
UUID id PK
TEXT name
}
JOB {
UUID id PK
UUID document_id FK
TEXT status
INTEGER retry_count
DATETIME date_created
DATETIME date_updated
TEXT provider
TEXT model
TEXT prompt_name
TEXT text
TEXT error_detail
}
SOURCE {
UUID id PK
UUID document_id FK
UUID job_id FK
TEXT upload_name
TEXT filename
TEXT file_path
DATETIME date_uploaded
}
REVISION {
UUID id PK
UUID source_id "FK, UK"
INTEGER revision
TEXT text
DATETIME date_created
}
DOCUMENT ||--o{ SOURCE : has_many
DOCUMENT ||--o{ JOB : has_many
JOB ||--o{ SOURCE : referenced_by
SOURCE ||--o| REVISION : has_optional_one
```
---
## Table Relationships and Constraints
- A `Document` can have zero or more `Source` records.
- A `Document` can have zero or more `Job` records.
- A `Source` belongs to exactly one `Document` and one `Job`.
- A `Source` may have one optional `Revision`.
- Optional `0..1` revision cardinality is enforced by uniqueness on `revision.source_id`.
### Invariants
- `Job.text` stores immutable original provider transcription output.
- `Revision` rows are optional user-authored edits derived from original transcription.
- Revisions do not overwrite original `Job.text`.
- Job status lifecycle values are: `queued`, `processing`, `transcribed`, `failed`.
### Timestamp Fields
- `Job.date_created`
- `Job.date_updated`
- `Source.date_uploaded`
- `Revision.date_created`
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- Data model (this document)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- **Document**: logical grouping for one or more transcribed sources.
- **Source**: uploaded file content (image/PDF) linked to a job.
- **Job**: processing record that stores lifecycle status and original output.
- **Revision**: optional single user-authored edited text linked to a source.
docs/ver1/traceability_v1.md
-40
View File
@@ -1,40 +0,0 @@
# V1 Traceability Matrix
This matrix provides implementation and validation evidence for V1 requirements (`REQ-0` through `REQ-13`).
Status values:
- `done`: implemented and evidence recorded
- `in progress`: partially implemented or evidence incomplete
- `not started`: no implementation/evidence yet
## Requirement Evidence Table
| Requirement | Status | Implementation Evidence | Validation Evidence |
| --- | --- | --- | --- |
| REQ-0 | done | End-to-end upload + worker pipeline in `src/transcription/services/store.py`, `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py` |
| REQ-1 | done | Upload UI/page flow in `src/transcription/ui/pages/upload_page.py`, `src/transcription/ui/components/upload.py` | `tests/ui/test_upload_page.py`, `tests/integration/test_pipeline_flow.py` |
| REQ-2 | done | Async worker execution and provider call orchestration in `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
| REQ-3 | done | Job lifecycle state model + transitions in `src/transcription/models.py`, `src/transcription/services/jobs.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/ui/test_jobs_page.py` |
| REQ-4 | done | Persistence of original output and failure detail in `src/transcription/services/transcription.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
| REQ-5 | done | Status/result inspection via UI pages and API health route in `src/transcription/ui/pages/jobs_page.py`, `src/transcription/api/health.py` | `tests/ui/test_jobs_page.py`, `tests/ui/test_pages_registration.py`, `tests/api/test_health.py` |
| REQ-6 | done | Background processing trigger/worker notifier and non-blocking workflow in `src/transcription/ui/components/upload.py`, `src/transcription/worker.py` | `tests/test_app.py`, `tests/services/test_workflows_reliability.py` |
| REQ-7 | done | Lifespan-owned runtime resources in `src/transcription/app.py`, `src/transcription/db/runtime.py` | `tests/test_app.py`, `tests/test_db.py` |
| REQ-8 | done | Centralized settings/logging initialization in `src/transcription/config.py`, `src/transcription/app.py` | `tests/test_config.py`, `tests/test_app.py` |
| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `release_checklist_v1.md` (Ops checklist), manual demonstration step |
| REQ-10 | done | Explicit schema bootstrap policy + runtime controls in `src/transcription/config.py`, `src/transcription/app.py`, `src/transcription/db/operations.py` | `tests/test_db.py`, `tests/test_config.py` |
| REQ-11 | done | Service/workflow persistence boundaries in `src/transcription/services/*.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/services/test_transcription_service.py` |
| REQ-12 | done | Prompt artifacts in `prompts/` and loading/validation in `src/transcription/services/transcription.py` | `tests/test_prompts.py` |
| REQ-13 | done | Optional single revision create/update/view/delete in `src/transcription/services/transcription.py`, `src/transcription/ui/pages/jobs_page.py` | `tests/services/test_transcription_service.py`, `tests/ui/test_jobs_page.py` |
## Operational Evidence (Step 3 Artifacts)
- Runbook: `runbook_v1.md`
- Migration/backfill/rollback guidance: `migration_v1.md`
- Release readiness checklist: `release_checklist_v1.md`
## Verification Cadence
- Per change: maintain `tests/test_traceability.py` mappings for touched requirements.
- Per milestone: update this table status and evidence links.
- Pre-release: confirm all rows are `done` and non-external suite is green.
project/.gitignore
+176
View File
@@ -0,0 +1,176 @@
# ---> Python
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
#uv.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# Ruff stuff:
.ruff_cache/
# PyPI configuration file
.pypirc
project/README.md.jinja
+1
View File
@@ -0,0 +1 @@
# {{project_name}}
project/copier.yml
+11
View File
@@ -0,0 +1,11 @@
project_name:
type: str
help: What is the project name?
repo_name:
type: str
help: What is the repo name?
module_name:
type: str
help: What is your Python module name?
project/docs/index.md.jinja
+1
View File
@@ -0,0 +1 @@
# {{project_name}}
project/pyproject.toml.jinja
+11
View File
@@ -0,0 +1,11 @@
[project]
name = "{{repo_name}}"
version = "0.1.0"
requires-python = ">=3.12"
[dependency-groups]
dev = [
"pre-commit>=4.6.0",
"ruff>=0.15.15",
"ipykernel>=7.2.0",
]
ruff.toml → project/ruff.toml
+1 -3
View File
@@ -10,11 +10,10 @@ exclude = [
"build",
"site",
"__pycache__",
]
[lint]
preview = true
extend-select = [
"ARG", # https://docs.astral.sh/ruff/rules/#flake8-unused-arguments-arg
"B", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
@@ -49,7 +48,6 @@ ignore = [
"*.ipynb" = [
"F401", # unused imports
"F841", # unused local variable
"F821", # undefined name in exploratory notebook cells
]
[lint.isort]
src/transcription/__init__.py → project/src/{{module_name}}/__init__.py
View File
project/zensical.toml.jinja
+350
View File
@@ -0,0 +1,350 @@
# ============================================================================
#
# The configuration produced by default is meant to highlight the features
# that Zensical provides and to serve as a starting point for your own
# projects.
#
# ============================================================================
[project]
# The site_name is shown in the page header and the browser window title
#
# Read more: https://zensical.org/docs/setup/basics/#site_name
site_name = "{{project_name}} Documentation"
# The site_description is included in the HTML head and should contain a
# meaningful description of the site content for use by search engines.
#
# Read more: https://zensical.org/docs/setup/basics/#site_description
site_description = "Documentation site for {{project_name}}"
# The site_author attribute. This is used in the HTML head element.
#
# Read more: https://zensical.org/docs/setup/basics/#site_author
site_author = "John Lancaster"
# The site_url is the canonical URL for your site. When building online
# documentation you should set this.
# Read more: https://zensical.org/docs/setup/basics/#site_url
#site_url = "https://www.example.com/"
# The copyright notice appears in the page footer and can contain an HTML
# fragment.
#
# Read more: https://zensical.org/docs/setup/basics/#copyright
copyright = """
Copyright &copy; 2026 The authors
"""
# Zensical supports both implicit navigation and explicitly defined navigation.
# If you decide not to define a navigation here then Zensical will simply
# derive the navigation structure from the directory structure of your
# "docs_dir". The definition below demonstrates how a navigation structure
# can be defined using TOML syntax.
#
# Read more: https://zensical.org/docs/setup/navigation/
# nav = [
# { "Get started" = "index.md" },
# { "Markdown in 5min" = "markdown.md" },
# ]
# With the "extra_css" option you can add your own CSS styling to customize
# your Zensical project according to your needs. You can add any number of
# CSS files.
#
# The path provided should be relative to the "docs_dir".
#
# Read more: https://zensical.org/docs/customization/#additional-css
#
#extra_css = ["stylesheets/extra.css"]
# With the `extra_javascript` option you can add your own JavaScript to your
# project to customize the behavior according to your needs.
#
# The path provided should be relative to the "docs_dir".
#
# Read more: https://zensical.org/docs/customization/#additional-javascript
#extra_javascript = ["javascripts/extra.js"]
# ----------------------------------------------------------------------------
# Section for configuring theme options
# ----------------------------------------------------------------------------
[project.theme]
# change this to "classic" to use the traditional Material for MkDocs look.
#variant = "classic"
# Zensical allows you to override specific blocks, partials, or whole
# templates as well as to define your own templates. To do this, uncomment
# the custom_dir setting below and set it to a directory in which you
# keep your template overrides.
#
# Read more:
# - https://zensical.org/docs/customization/#extending-the-theme
#
#custom_dir = "overrides"
# With the "favicon" option you can set your own image to use as the icon
# browsers will use in the browser title bar or tab bar. The path provided
# must be relative to the "docs_dir".
#
# Read more:
# - https://zensical.org/docs/setup/logo-and-icons/#favicon
# - https://developer.mozilla.org/en-US/docs/Glossary/Favicon
#
#favicon = "images/favicon.png"
# Zensical supports more than 60 different languages. This means that the
# labels and tooltips that Zensical's templates produce are translated.
# The "language" option allows you to set the language used. This language
# is also indicated in the HTML head element to help with accessibility
# and guide search engines and translation tools.
#
# The default language is "en" (English). It is possible to create
# sites with multiple languages and configure a language selector. See
# the documentation for details.
#
# Read more:
# - https://zensical.org/docs/setup/language/
#
language = "en"
# Zensical provides a number of feature toggles that change the behavior
# of the documentation site.
features = [
# Zensical includes an announcement bar. This feature allows users to
# dismiss it when they have read the announcement.
# https://zensical.org/docs/setup/header/#announcement-bar
"announce.dismiss",
# If you have a repository configured and turn on this feature, Zensical
# will generate an edit button for the page. This works for common
# repository hosting services.
# https://zensical.org/docs/setup/repository/#content-actions
#"content.action.edit",
# If you have a repository configured and turn on this feature, Zensical
# will generate a button that allows the user to view the Markdown
# code for the current page.
# https://zensical.org/docs/setup/repository/#content-actions
#"content.action.view",
# Code annotations allow you to add an icon with a tooltip to your
# code blocks to provide explanations at crucial points.
# https://zensical.org/docs/authoring/code-blocks/#code-annotations
"content.code.annotate",
# This feature turns on a button in code blocks that allow users to
# copy the content to their clipboard without first selecting it.
# https://zensical.org/docs/authoring/code-blocks/#code-copy-button
"content.code.copy",
# Code blocks can include a button to allow for the selection of line
# ranges by the user.
# https://zensical.org/docs/authoring/code-blocks/#code-selection-button
"content.code.select",
# Zensical can render footnotes as inline tooltips, so the user can read
# the footnote without leaving the context of the document.
# https://zensical.org/docs/authoring/footnotes/#footnote-tooltips
"content.footnote.tooltips",
# If you have many content tabs that have the same titles (e.g., "Python",
# "JavaScript", "Cobol"), this feature causes all of them to switch to
# at the same time when the user chooses their language in one.
# https://zensical.org/docs/authoring/content-tabs/#linked-content-tabs
"content.tabs.link",
# With this feature enabled users can add tooltips to links that will be
# displayed when the mouse pointer hovers the link.
# https://zensical.org/docs/authoring/tooltips/#improved-tooltips
"content.tooltips",
# With this feature enabled, Zensical will automatically hide parts
# of the header when the user scrolls past a certain point.
# https://zensical.org/docs/setup/header/#automatic-hiding
# "header.autohide",
# Turn on this feature to expand all collapsible sections in the
# navigation sidebar by default.
# https://zensical.org/docs/setup/navigation/#navigation-expansion
# "navigation.expand",
# This feature turns on navigation elements in the footer that allow the
# user to navigate to a next or previous page.
# https://zensical.org/docs/setup/footer/#navigation
"navigation.footer",
# When section index pages are enabled, documents can be directly attached
# to sections, which is particularly useful for providing overview pages.
# https://zensical.org/docs/setup/navigation/#section-index-pages
"navigation.indexes",
# When instant navigation is enabled, clicks on all internal links will be
# intercepted and dispatched via XHR without fully reloading the page.
# https://zensical.org/docs/setup/navigation/#instant-navigation
"navigation.instant",
# With instant prefetching, your site will start to fetch a page once the
# user hovers over a link. This will reduce the perceived loading time
# for the user.
# https://zensical.org/docs/setup/navigation/#instant-prefetching
"navigation.instant.prefetch",
# In order to provide a better user experience on slow connections when
# using instant navigation, a progress indicator can be enabled.
# https://zensical.org/docs/setup/navigation/#progress-indicator
#"navigation.instant.progress",
# When navigation paths are activated, a breadcrumb navigation is rendered
# above the title of each page
# https://zensical.org/docs/setup/navigation/#navigation-path
"navigation.path",
# When pruning is enabled, only the visible navigation items are included
# in the rendered HTML, reducing the size of the built site by 33% or more.
# https://zensical.org/docs/setup/navigation/#navigation-pruning
#"navigation.prune",
# When sections are enabled, top-level sections are rendered as groups in
# the sidebar for viewports above 1220px, but remain as-is on mobile.
# https://zensical.org/docs/setup/navigation/#navigation-sections
"navigation.sections",
# When tabs are enabled, top-level sections are rendered in a menu layer
# below the header for viewports above 1220px, but remain as-is on mobile.
# https://zensical.org/docs/setup/navigation/#navigation-tabs
#"navigation.tabs",
# When sticky tabs are enabled, navigation tabs will lock below the header
# and always remain visible when scrolling down.
# https://zensical.org/docs/setup/navigation/#sticky-navigation-tabs
#"navigation.tabs.sticky",
# A back-to-top button can be shown when the user, after scrolling down,
# starts to scroll up again.
# https://zensical.org/docs/setup/navigation/#back-to-top-button
"navigation.top",
# When anchor tracking is enabled, the URL in the address bar is
# automatically updated with the active anchor as highlighted in the table
# of contents.
# https://zensical.org/docs/setup/navigation/#anchor-tracking
"navigation.tracking",
# When search highlighting is enabled and a user clicks on a search result,
# Zensical will highlight all occurrences after following the link.
# https://zensical.org/docs/setup/search/#search-highlighting
"search.highlight",
# When anchor following for the table of contents is enabled, the sidebar
# is automatically scrolled so that the active anchor is always visible.
# https://zensical.org/docs/setup/navigation/#anchor-following
# "toc.follow",
# When navigation integration for the table of contents is enabled, it is
# always rendered as part of the navigation sidebar on the left.
# https://zensical.org/docs/setup/navigation/#navigation-integration
#"toc.integrate",
]
# ----------------------------------------------------------------------------
# You can configure your own logo to be shown in the header using the "logo"
# option in the "theme" subsection. The logo must be a relative path to a file
# in your "docs_dir", e.g., to use `docs/assets/logo.png` you would set:
# ----------------------------------------------------------------------------
#logo = "assets/logo.png"
# ----------------------------------------------------------------------------
# If you don't have a dedicated project logo, you can use a built-in icon from
# the icon sets shipped in Zensical. Please note that the setting lives in a
# different subsection, and that the above take precedence over the icon.
#
# Read more:
# - https://zensical.org/docs/setup/logo-and-icons
# - https://github.com/zensical/ui/tree/master/dist/.icons
# ----------------------------------------------------------------------------
#[project.theme.icon]
#logo = "lucide/smile"
# ----------------------------------------------------------------------------
# In the "font" subsection you can configure the fonts used. By default, fonts
# are loaded from Google Fonts, giving you a wide range of choices from a set
# of suitably licensed fonts. There are options for a normal text font and for
# a monospaced font used in code blocks.
# ----------------------------------------------------------------------------
#[project.theme.font]
#text = "Inter"
#code = "Jetbrains Mono"
# ----------------------------------------------------------------------------
# In the "palette" subsection you can configure options for the color scheme.
# You can configure different color schemes, e.g., to turn on dark mode,
# that the user can switch between. Each color scheme can be further
# customized.
#
# Read more:
# - https://zensical.org/docs/setup/colors/
# ----------------------------------------------------------------------------
[[project.theme.palette]]
scheme = "default"
toggle.icon = "lucide/sun"
toggle.name = "Switch to dark mode"
[[project.theme.palette]]
scheme = "slate"
toggle.icon = "lucide/moon"
toggle.name = "Switch to light mode"
# ----------------------------------------------------------------------------
# The "extra" section contains miscellaneous settings.
# ----------------------------------------------------------------------------
#[[project.extra.social]]
#icon = "fontawesome/brands/github"
#link = "https://github.com/user/repo"
# ----------------------------------------------------------------------------
# In this section you can configure the Markdown extensions that are used when
# rendering your documentation. We enable the most useful extensions by default,
# but you can customize this list to your needs.
#
# Read more:
# - https://zensical.org/docs/setup/extensions/
# ----------------------------------------------------------------------------
[project.markdown_extensions.abbr]
[project.markdown_extensions.admonition]
[project.markdown_extensions.attr_list]
[project.markdown_extensions.def_list]
[project.markdown_extensions.footnotes]
[project.markdown_extensions.md_in_html]
[project.markdown_extensions.toc]
permalink = true
[project.markdown_extensions.pymdownx.arithmatex]
generic = true
[project.markdown_extensions.pymdownx.betterem]
[project.markdown_extensions.pymdownx.caret]
[project.markdown_extensions.pymdownx.details]
[project.markdown_extensions.pymdownx.emoji]
emoji_generator = "zensical.extensions.emoji.to_svg"
emoji_index = "zensical.extensions.emoji.twemoji"
[project.markdown_extensions.pymdownx.highlight]
anchor_linenums = true
line_spans = "__span"
pygments_lang_class = true
[project.markdown_extensions.pymdownx.inlinehilite]
[project.markdown_extensions.pymdownx.keys]
[project.markdown_extensions.pymdownx.magiclink]
[project.markdown_extensions.pymdownx.mark]
[project.markdown_extensions.pymdownx.smartsymbols]
[project.markdown_extensions.pymdownx.snippets]
[project.markdown_extensions.pymdownx.superfences]
custom_fences = [
{ name = "mermaid", class = "mermaid", format = "pymdownx.superfences.fence_code_format" }
]
[project.markdown_extensions.pymdownx.tabbed]
alternate_style = true
combine_header_slug = true
[project.markdown_extensions.pymdownx.tasklist]
custom_checkbox = true
[project.markdown_extensions.pymdownx.tilde]
project/{{_copier_conf.answers_file}}.jinja
+2
View File
@@ -0,0 +1,2 @@
# Changes here will be overwritten by Copier
{{ _copier_answers|to_nice_yaml -}}
prompts/README.md
-13
View File
@@ -1,13 +0,0 @@
# Prompt Artifacts
This directory stores transcription prompts as individual Markdown artifacts.
## Conventions
- Keep one prompt per file.
- Use stable, descriptive snake_case file names.
- Prefer incremental edits to a single prompt per change for clean history.
- Keep prompts human-readable and policy-focused.
- Do not store secrets in prompt files.
## Current Prompt
- `transcribe_document.md`: baseline verbatim transcription policy for historical documents.
prompts/system.txt
-10
View File
@@ -1,10 +0,0 @@
You are an assistant that may call tools.
Tool safety rules:
1) Tool arguments MUST be strict JSON matching the schema exactly.
2) Never place disallowed, sensitive, explicit, or policy-violating text directly into tool arguments.
3) If user content may be unsafe, first produce a brief neutral summary and pass only that summary.
4) Prefer IDs, enums, booleans, and short fields over raw free-form text.
5) Keep all string arguments <= 300 chars unless schema says otherwise.
6) If you cannot safely provide valid tool args, do not call the tool; respond with "NO_TOOL_CALL" and explain briefly.
7) Never include markdown/code fences in tool arguments.
prompts/transcribe_document.md
-72
View File
@@ -1,72 +0,0 @@
# Historical Document Verbatim Transcription Prompt
## Purpose
Transcribe the provided historical document image as a faithful **verbatim** transcript.
Do not summarize. Do not paraphrase. Do not modernize style.
## Output Contract
- Return only the transcription text.
- Preserve original wording, punctuation, and meaningful structure.
- Keep line/section flow readable while preserving intent and document organization.
- Never invent missing content.
## Rules for Ambiguous or Damaged Text
### Misspellings and original errors
- Preserve original spelling.
- Add `[sic]` immediately after an evident original error.
### Missing words or clear omissions
- If a single missing word is obvious from context, insert it in square brackets.
- Example form: `[to]`
### Uncertain readings
- If best-effort interpretation is uncertain, use bracketed guess with question mark.
- Example form: `[Boston?]`
### Completely illegible text
- Use a clear bracketed label.
- Preferred forms: `[illegible]`, `[torn]`, `[ink blot]`, `[remainder of page torn]`
### Crossed-out text
- Preserve it using: `[deleted: ...]`
### Squeezed-in or above-line insertions
- Preserve it using: `[inserted: ...]`
### Superscripts and abbreviations
- Bring superscript letters down to baseline text.
- Expand only when clearly intended; if expanded, place added letters in brackets.
### Non-text visual elements
- Describe briefly in square brackets.
- Example forms: `[wax notary seal attached here]`, `[sketch of a fort layout]`
### Marginalia and side notes
- Signal location before the note text.
- Example form: `[written in left margin: ...]`
### Line-break hyphenation
- Rejoin words split across line breaks when they are clearly one word.
- Remove only line-break hyphens used for wrapping.
### Ambiguous capitalization
- Prefer modern capitalization only when uncertainty is high.
- Preserve clearly intentional archaic capitalization.
### Hierarchical outlines and numbering
- Preserve original numbering characters exactly (including roman numerals and unusual suffixes).
- Preserve indentation levels.
- Do not silently correct sequence mistakes; if clearly erroneous, preserve and use `[sic]` where appropriate.
## Confidence and Integrity Policy
- When uncertain, mark uncertainty explicitly rather than guessing silently.
- If text cannot be read, use a bracketed illegibility label instead of fabrication.
- Do not add commentary outside the transcription.
## Final Self-Check
Before finalizing, ensure:
1. The transcript is verbatim and not summarized.
2. Uncertain/illegible areas are explicitly marked.
3. Crossed-out and inserted text are preserved with required tags.
4. Structure/ordering is preserved as faithfully as possible.
pyproject.toml
+2 -45
View File
@@ -1,50 +1,7 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/transcription"]
[project]
name = "transcription"
name = "python-template"
version = "0.1.0"
description = "Historical document transcription system"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"aiosqlite>=0.21.0",
"asyncpg>=0.31.0",
"fastapi>=0.138.0",
"nicegui==3.13.0",
"openrouter>=0.7.0",
"psycopg2-binary>=2.9.12",
"pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25",
]
[dependency-groups]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.25",
"httpx2>=2.5.0",
"ipykernel>=7.3.0",
"ipywidgets>=8.1.8",
"pre-commit>=4.6.0",
"rich>=15.0.0",
"ruff>=0.15.20",
"ty>=0.0.54",
]
[tool.pytest.ini_options]
addopts = "--strict-markers -q"
asyncio_mode = "strict"
filterwarnings = [
"error:coroutine .* was never awaited:RuntimeWarning",
]
markers = [
"unit: pure logic tests with no external dependencies",
"integration: tests that touch framework or database contracts",
"external: tests that call external services (slow, requires credentials)",
"copier>=9.15.1",
]
src/transcription/api/__init__.py
-1
View File
@@ -1 +0,0 @@
"""API route modules for the transcription app."""
src/transcription/api/errors.py
-56
View File
@@ -1,56 +0,0 @@
"""Centralized API exception handlers."""
from __future__ import annotations
import logging
from fastapi import FastAPI
from fastapi import Request
from fastapi.responses import JSONResponse
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import build_error_envelope
logger = logging.getLogger(__name__)
_STATUS_BY_CATEGORY: dict[ErrorCategory, int] = {
ErrorCategory.VALIDATION: 400,
ErrorCategory.USER_INPUT: 400,
ErrorCategory.NOT_FOUND: 404,
ErrorCategory.CONFLICT: 409,
ErrorCategory.EXTERNAL_PROVIDER: 503,
ErrorCategory.INFRA_TRANSIENT: 503,
ErrorCategory.INFRA_PERSISTENT: 500,
ErrorCategory.INTERNAL_UNEXPECTED: 500,
}
def _status_for(error: AppError) -> int:
return _STATUS_BY_CATEGORY.get(error.category, 500)
def register_error_handlers(app: FastAPI) -> None:
"""Register API exception handlers on the app."""
@app.exception_handler(AppError)
async def app_error_handler(_request: Request, exc: AppError) -> JSONResponse:
envelope = build_error_envelope(exc)
return JSONResponse(status_code=_status_for(exc), content=envelope.__dict__)
@app.exception_handler(Exception)
async def fallback_error_handler(_request: Request, exc: Exception) -> JSONResponse:
normalized = AppError(
"Unexpected error while handling request",
category=ErrorCategory.INTERNAL_UNEXPECTED,
suggestion="Retry once. If it persists, report the error reference id.",
)
logger.exception(
"Unhandled API exception operation=api.request error_id=%s category=%s exception_type=%s",
normalized.error_id,
normalized.category.value,
type(exc).__name__,
)
envelope = build_error_envelope(normalized)
return JSONResponse(status_code=_status_for(normalized), content=envelope.__dict__)
src/transcription/api/health.py
-16
View File
@@ -1,16 +0,0 @@
"""Health endpoint routes."""
from fastapi import APIRouter
router = APIRouter()
def healthz() -> dict[str, str]:
"""Return a simple health status payload."""
return {"status": "ok"}
@router.get("/healthz")
def healthz_route() -> dict[str, str]:
"""Route wrapper for health status payload."""
return healthz()
src/transcription/app.py
-98
View File
@@ -1,98 +0,0 @@
"""Application factory and lifespan wiring for the transcription app."""
from __future__ import annotations
import logging
from contextlib import AsyncExitStack
from contextlib import asynccontextmanager
from datetime import UTC
from datetime import datetime
from datetime import timedelta
from fastapi import FastAPI
from fastapi import status
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from .api.errors import register_error_handlers
from .api.health import router as health_router
from .config import configure_logging
from .config import get_settings
from .db import create_all
from .db import dispose_database_runtime
from .db import initialize_database_runtime
from .services import ServiceBundle
from .services.jobs import JobService
from .ui import register_pages
from .worker import worker_consumer_lifespan
logger = logging.getLogger(__name__)
@asynccontextmanager
async def _lifespan(app: FastAPI):
configure_logging()
settings = getattr(app.state, "settings", None) or get_settings()
app.state.settings = settings
app.state.services = ServiceBundle()
app.state.runtime = initialize_database_runtime(settings=settings)
if settings.should_bootstrap_schema:
await create_all(engine=app.state.runtime.engine)
settings.upload_dir.mkdir(parents=True, exist_ok=True)
settings.prompt_dir.mkdir(parents=True, exist_ok=True)
await _recover_stale_processing_jobs(app)
async with AsyncExitStack() as stack:
stack.push_async_callback(dispose_database_runtime)
stop_event, worker_notifier = await stack.enter_async_context(
worker_consumer_lifespan(
session_factory=app.state.runtime.session_factory,
poll_interval_seconds=1.0,
)
)
app.state.worker_stop_event = stop_event
app.state.worker_notifier = worker_notifier
yield
async def _recover_stale_processing_jobs(app: FastAPI) -> None:
"""Re-queue stale processing jobs at startup.
Any job left in PROCESSING longer than the configured provider timeout is
assumed orphaned and moved back to QUEUED before the worker starts.
"""
settings = app.state.settings
stale_before = datetime.now(UTC) - timedelta(seconds=settings.worker_provider_timeout_seconds)
job_service = JobService(session_factory=app.state.runtime.session_factory)
recovered = await job_service.requeue_stale_processing_jobs(stale_before=stale_before)
if recovered > 0:
logger.warning("Recovered %s stale processing job(s) at startup", recovered)
def create_app() -> FastAPI:
"""Create and configure the FastAPI application."""
app = FastAPI(title="Transcription", lifespan=_lifespan)
settings = get_settings()
app.state.settings = settings
app.mount(
"/uploads",
StaticFiles(directory=settings.upload_dir, check_dir=False),
name="uploads",
)
@app.get("/", include_in_schema=False)
async def root_redirect() -> RedirectResponse:
return RedirectResponse(url="/ui", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
@app.get("/ui", include_in_schema=False)
async def ui_redirect() -> RedirectResponse:
return RedirectResponse(url="/ui/upload", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
register_error_handlers(app)
register_pages(app)
app.include_router(health_router)
return app
src/transcription/app_state.py
-39
View File
@@ -1,39 +0,0 @@
"""Helpers for accessing lifespan-owned application state resources."""
from __future__ import annotations
from fastapi import FastAPI
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.db.runtime import DatabaseRuntime
from transcription.db.runtime import get_session_factory
from transcription.worker import WorkerNotifier
from transcription.worker import resolve_worker_notifier
def resolve_database_runtime(state: object) -> DatabaseRuntime | None:
"""Return database runtime from app-like state objects when available."""
runtime = getattr(state, "runtime", None)
return runtime if isinstance(runtime, DatabaseRuntime) else None
def require_database_runtime(state: object) -> DatabaseRuntime:
"""Return database runtime or raise when app lifespan has not initialized it."""
runtime = resolve_database_runtime(state)
if runtime is None:
raise RuntimeError("Database runtime is not initialized on application state")
return runtime
def resolve_session_factory(state: object) -> async_sessionmaker[AsyncSession]:
"""Return DB session factory from state when available, otherwise shared runtime."""
runtime = resolve_database_runtime(state)
if runtime is not None:
return runtime.session_factory
return get_session_factory()
def get_worker_notifier(app: FastAPI) -> WorkerNotifier:
"""Return app worker notifier, or a no-op fallback when unavailable."""
return resolve_worker_notifier(app.state)
src/transcription/config.py
-111
View File
@@ -1,111 +0,0 @@
"""Centralized application configuration.
All settings are loaded from environment variables (or a .env file)
once at startup. Provider-specific defaults (model names, base URLs)
are resolved by the provider adapters, not here.
"""
import logging.config
from contextvars import ContextVar
from enum import StrEnum
from pathlib import Path
from typing import Literal
from pydantic import Field
from pydantic_settings import BaseSettings
from pydantic_settings import SettingsConfigDict
logger = logging.getLogger(__name__)
class Provider(StrEnum):
OPENROUTER = "openrouter"
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
# --- AI provider ---
provider: Provider = Provider.OPENROUTER
openrouter_api_key: str
provider_model: str | None = None
openrouter_http_referer: str | None = None
openrouter_app_title: str | None = None
# --- runtime environment ---
environment: Literal["development", "test", "production"] = "development"
# --- persistence ---
database_url: str = "sqlite:///./transcription.db"
bootstrap_schema_on_startup: bool | None = None
sqlite_check_same_thread: bool = False
# --- filesystem paths ---
upload_dir: Path = Path("./uploads")
prompt_dir: Path = Path("./prompts")
# --- worker reliability ---
worker_max_retries: int = 0
worker_retry_backoff_seconds: float = 0.0
worker_provider_timeout_seconds: float = Field(default=20.0, gt=0.0, le=20.0)
worker_min_transcription_chars: int = Field(default=0, ge=0)
worker_min_transcription_lines: int = Field(default=0, ge=0)
worker_fail_on_finish_reason_length: bool = False
@property
def should_bootstrap_schema(self) -> bool:
"""Return whether startup should auto-create schema for this environment."""
if self.bootstrap_schema_on_startup is not None:
return self.bootstrap_schema_on_startup
return self.environment in {"development", "test"}
_settings: ContextVar[Settings | None] = ContextVar("settings", default=None)
def get_settings(**kwargs) -> Settings:
settings = _settings.get()
if settings is None:
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
_settings.set(settings)
return settings
LOGGING_CONFIG: dict[str, object] = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": "%(asctime)s %(levelname)-8s | %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "standard",
"stream": "ext://sys.stdout",
}
},
"root": {
"level": "INFO",
"handlers": ["console"],
},
"loggers": {
"transcription": {
"level": "DEBUG",
"handlers": ["console"],
"propagate": False,
}
},
}
def configure_logging() -> None:
"""Configure root logging once at startup."""
logging.config.dictConfig(LOGGING_CONFIG)
logger.debug("Logging configured")
src/transcription/db/__init__.py
-6
View File
@@ -1,6 +0,0 @@
from .operations import create_all
from .runtime import dispose_database_runtime
from .runtime import get_session
from .runtime import initialize_database_runtime
__all__ = ["create_all", "dispose_database_runtime", "get_session", "initialize_database_runtime"]
src/transcription/db/operations.py
-79
View File
@@ -1,79 +0,0 @@
from __future__ import annotations
import logging
from sqlalchemy import inspect
from sqlalchemy import text
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import AsyncEngine
from sqlmodel import SQLModel
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from ..models import Job
from ..models import JobStatus
from .runtime import get_engine
logger = logging.getLogger(__name__)
async def get_next_queued_job(*, session: AsyncSession) -> Job | None:
"""Get the next queued job, if any."""
result = await session.exec(
select(Job)
.where(Job.status == JobStatus.QUEUED)
.order_by(Job.date_created) # pyright: ignore[reportArgumentType]
.limit(1)
) # fmt: skip
return result.first()
async def create_all(*, engine: AsyncEngine | None = None) -> None:
"""Create all tables on the selected engine."""
# Import models so SQLModel metadata is fully registered before bootstrap.
from transcription import models as _models # noqa: F401
active_engine = engine or get_engine()
async with active_engine.begin() as connection:
await connection.run_sync(SQLModel.metadata.create_all)
await connection.run_sync(_ensure_sqlite_compat_columns)
logger.debug("Database schema bootstrap complete for database_url=%s", active_engine.url)
def _ensure_sqlite_compat_columns(connection: Connection) -> None:
"""Apply lightweight dev/test SQLite compatibility column patches.
This keeps local bootstrap resilient when models evolve but no full
migration tooling is in place yet.
"""
if connection.engine.url.get_backend_name() != "sqlite":
return
inspector = inspect(connection)
table_names = set(inspector.get_table_names())
if "job" in table_names:
job_columns = {column["name"] for column in inspector.get_columns("job")}
if "retry_count" not in job_columns:
connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0"))
logger.warning("Applied SQLite compatibility schema patch table=job column=retry_count default=0")
if "revision" in table_names:
revision_columns = {column["name"] for column in inspector.get_columns("revision")}
if "source_id" in revision_columns:
has_unique_source = False
for index in inspector.get_indexes("revision"):
if index.get("unique") and index.get("column_names") == ["source_id"]:
has_unique_source = True
break
if not has_unique_source:
connection.execute(
text(
"CREATE UNIQUE INDEX IF NOT EXISTS "
"ux_revision_source_id ON revision(source_id)"
)
)
logger.warning(
"Applied SQLite compatibility schema patch "
"table=revision unique_index=ux_revision_source_id"
)
src/transcription/db/runtime.py
-103
View File
@@ -1,103 +0,0 @@
import logging
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from functools import partial
from sqlalchemy.ext.asyncio import AsyncEngine
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.ext.asyncio import create_async_engine
from sqlmodel.ext.asyncio.session import AsyncSession
from sqlmodel.pool import StaticPool
from ..config import Settings
from ..config import get_settings
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class DatabaseRuntime:
"""Database runtime resources owned by app lifespan."""
engine: AsyncEngine
session_factory: async_sessionmaker[AsyncSession]
_runtime: ContextVar[DatabaseRuntime | None] = ContextVar("database_runtime", default=None)
async def dispose_database_runtime() -> None:
"""Dispose lifespan-owned async database resources."""
runtime = _runtime.get()
if runtime is None:
return
await runtime.engine.dispose()
_runtime.set(None)
def _to_async_database_url(database_url: str) -> str:
"""Normalize configured database URL to an async SQLAlchemy driver URL."""
if database_url.startswith("sqlite://") and not database_url.startswith("sqlite+aiosqlite://"):
return database_url.replace("sqlite://", "sqlite+aiosqlite://", 1)
if database_url.startswith("postgresql://") and not database_url.startswith("postgresql+asyncpg://"):
return database_url.replace("postgresql://", "postgresql+asyncpg://", 1)
return database_url
def _build_engine(settings: Settings) -> AsyncEngine:
database_url = _to_async_database_url(settings.database_url)
engine_factory = partial(
create_async_engine,
url=database_url,
echo=False,
pool_pre_ping=True,
)
if database_url.startswith("sqlite"):
sqlite_connect_settings = {"check_same_thread": settings.sqlite_check_same_thread}
engine_factory = partial(engine_factory, connect_args=sqlite_connect_settings)
if ":memory:" in database_url:
engine_factory = partial(engine_factory, poolclass=StaticPool)
return engine_factory()
def initialize_database_runtime(*, settings: Settings | None = None) -> DatabaseRuntime:
"""Initialize lifespan-owned async DB resources once per process."""
runtime = _runtime.get()
if runtime is not None:
return runtime
active_settings = settings or get_settings()
engine = _build_engine(active_settings)
session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
runtime = DatabaseRuntime(engine=engine, session_factory=session_factory)
_runtime.set(runtime)
logger.debug("Initialized async database runtime for database_url=%s", engine.url)
return runtime
def get_engine(settings: Settings | None = None) -> AsyncEngine:
"""Return the current async SQLAlchemy engine."""
runtime = _runtime.get() or initialize_database_runtime(settings=settings)
return runtime.engine
def get_session_factory(settings: Settings | None = None) -> async_sessionmaker[AsyncSession]:
"""Return the shared async session factory."""
runtime = _runtime.get() or initialize_database_runtime(settings=settings)
return runtime.session_factory
@asynccontextmanager
async def get_session(
*,
settings: Settings | None = None,
session_factory: async_sessionmaker[AsyncSession] | None = None,
) -> AsyncGenerator[AsyncSession]:
"""Yield a database session and ensure cleanup."""
active_session_factory = session_factory or get_session_factory(settings)
async with active_session_factory() as session:
yield session
src/transcription/errors.py
-85
View File
@@ -1,85 +0,0 @@
"""Shared error taxonomy and helpers for runtime boundaries."""
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC
from datetime import datetime
from enum import StrEnum
from uuid import uuid4
class ErrorCategory(StrEnum):
"""Stable error categories defined by docs/error_handling.md."""
VALIDATION = "validation_error"
USER_INPUT = "user_input_error"
NOT_FOUND = "not_found_error"
CONFLICT = "conflict_error"
EXTERNAL_PROVIDER = "external_provider_error"
PROCESSING = "processing_error"
INFRA_TRANSIENT = "infrastructure_transient_error"
INFRA_PERSISTENT = "infrastructure_persistent_error"
INTERNAL_UNEXPECTED = "internal_unexpected_error"
def new_error_id() -> str:
"""Return a short, user-shareable error reference id."""
return uuid4().hex[:8]
class AppError(RuntimeError):
"""Base application error carrying user-safe handling metadata."""
def __init__(
self,
message: str,
*,
category: ErrorCategory = ErrorCategory.INTERNAL_UNEXPECTED,
suggestion: str = "Retry once. If it persists, review logs and report the error reference id.",
retriable: bool = False,
error_id: str | None = None,
) -> None:
super().__init__(message)
self.message = message
self.category = category
self.suggestion = suggestion
self.retriable = retriable
self.error_id = error_id or new_error_id()
@dataclass(frozen=True)
class ErrorEnvelope:
"""Serializable API/UI error payload."""
error_id: str
category: str
message: str
suggestion: str
timestamp: str
def build_error_envelope(error: AppError) -> ErrorEnvelope:
"""Build an API-safe response envelope from an AppError."""
return ErrorEnvelope(
error_id=error.error_id,
category=error.category.value,
message=error.message,
suggestion=error.suggestion,
timestamp=datetime.now(UTC).isoformat(),
)
def classify_unexpected_error(exc: Exception, *, operation: str) -> AppError:
"""Normalize unknown exceptions into internal_unexpected_error."""
return AppError(
f"Unexpected error during {operation}: {exc}",
category=ErrorCategory.INTERNAL_UNEXPECTED,
suggestion="Retry once. If it persists, review logs and report the error reference id.",
retriable=False,
)
def format_error_detail(error: AppError) -> str:
"""Return a compact persisted failure string for transcript.error_detail."""
return f"[{error.category.value}] {error.message} | suggestion={error.suggestion} | error_id={error.error_id}"
src/transcription/models.py
-110
View File
@@ -1,110 +0,0 @@
"""SQLModel domain models for the transcription system.
Core V1 lifecycle:
Document -> one-to-many -> Source
Document -> one-to-many -> Job
Source -> one-to-one? -> Revision (optional)
"""
from datetime import UTC
from datetime import datetime
from enum import StrEnum
from typing import Optional
from uuid import UUID
from uuid import uuid4
from sqlalchemy import UniqueConstraint
from sqlmodel import Field
from sqlmodel import Relationship
from sqlmodel import SQLModel
class JobStatus(StrEnum):
QUEUED = "queued"
PROCESSING = "processing"
TRANSCRIBED = "transcribed"
FAILED = "failed"
class Document(SQLModel, table=True):
"""An historical document."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
name: str
# Relationships
jobs: list["Job"] = Relationship(back_populates="document")
sources: list["Source"] = Relationship(back_populates="document")
class Source(SQLModel, table=True):
"""A document source (image or PDF)."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
document_id: UUID = Field(foreign_key="document.id")
job_id: UUID = Field(foreign_key="job.id")
upload_name: str
"""The filename of the source that was uploaded for transcription."""
filename: str
"""The system generated unique source name."""
file_path: str
"""The location where the sources are stored on the local filesystem."""
date_uploaded: datetime = Field(default_factory=lambda: datetime.now(UTC))
# Relationships
document: Optional["Document"] = Relationship(back_populates="sources")
job: Optional["Job"] = Relationship(back_populates="sources")
revision: Optional["Revision"] = Relationship(
back_populates="source",
sa_relationship_kwargs={"uselist": False},
)
class Job(SQLModel, table=True):
"""A transcription job tied to a single document."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
document_id: UUID = Field(foreign_key="document.id")
status: JobStatus = Field(default=JobStatus.QUEUED)
retry_count: int = Field(default=0, ge=0)
date_created: datetime = Field(default_factory=lambda: datetime.now(UTC))
date_updated: datetime = Field(default_factory=lambda: datetime.now(UTC))
provider: str | None = None
"""Name of the transcription provider used to generate this transcript."""
model: str | None = None
"""Model identifier used to generate this transcript."""
prompt_name: str | None = None
"""Name of the prompt used to generate this transcript."""
text: str | None = None
"""The transcribed text. This may be None if the job failed or is still in progress."""
error_detail: str | None = None
"""Details of any error that occurred during transcription."""
# Relationships
document: Optional["Document"] = Relationship(back_populates="jobs")
sources: list["Source"] = Relationship(back_populates="job")
@property
def filename(self) -> str:
"""Return the filename of the associated source, when available."""
if not self.sources:
return "unknown"
return self.sources[0].filename
class Revision(SQLModel, table=True):
"""A revision of a transcription text."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
source_id: UUID = Field(foreign_key="source.id")
"""ID for the associated source."""
revision: int = Field(default=1, ge=1)
"""Revision number of this transcription revision, starting at 1."""
text: str
"""The revised text."""
date_created: datetime = Field(default_factory=lambda: datetime.now(UTC))
__table_args__ = (UniqueConstraint("source_id", name="uq_revision_source_id"),)
# Relationships
source: Optional["Source"] = Relationship(back_populates="revision")
src/transcription/providers/__init__.py
-31
View File
@@ -1,31 +0,0 @@
"""Provider exports and factory for transcription adapters."""
from transcription.config import Provider
from transcription.config import Settings
from transcription.config import get_settings
from transcription.providers.base import ProviderAuthError
from transcription.providers.base import ProviderError
from transcription.providers.base import ProviderResponseError
from transcription.providers.base import TranscriptionProvider
from transcription.providers.base import TranscriptionResult
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
def get_transcription_provider(*, settings: Settings | None = None) -> TranscriptionProvider:
"""Return the configured transcription provider adapter."""
runtime_settings = settings or get_settings()
if runtime_settings.provider == Provider.OPENROUTER:
return OpenRouterTranscriptionProvider(settings=runtime_settings)
raise ProviderError(f"Unsupported transcription provider: {runtime_settings.provider}")
__all__ = [
"OpenRouterTranscriptionProvider",
"ProviderAuthError",
"ProviderError",
"ProviderResponseError",
"TranscriptionProvider",
"TranscriptionResult",
"get_transcription_provider",
]
src/transcription/providers/base.py
-38
View File
@@ -1,38 +0,0 @@
"""Provider interfaces and shared types for transcription adapters."""
from dataclasses import dataclass
from typing import Protocol
class ProviderError(RuntimeError):
"""Base error for provider failures."""
class ProviderAuthError(ProviderError):
"""Raised when provider authentication fails."""
class ProviderResponseError(ProviderError):
"""Raised when provider responses are malformed or unusable."""
@dataclass(frozen=True)
class TranscriptionResult:
"""Normalized output returned by any transcription provider."""
text: str
provider: str
prompt_name: str
model: str
finish_reason: str | None = None
usage_input_tokens: int | None = None
usage_output_tokens: int | None = None
usage_total_tokens: int | None = None
class TranscriptionProvider(Protocol):
"""Contract every transcription provider adapter must satisfy."""
async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
"""Transcribe the provided image according to the prompt text."""
...
src/transcription/providers/openrouter.py
-174
View File
@@ -1,174 +0,0 @@
"""OpenRouter transcription provider adapter."""
from __future__ import annotations
import base64
import logging
from dataclasses import dataclass
from typing import Any
from typing import cast
from openrouter import OpenRouter
from openrouter.components.chatmessages import ChatMessagesTypedDict
from transcription.config import Settings
from transcription.config import get_settings
from transcription.providers.base import ProviderAuthError
from transcription.providers.base import ProviderError
from transcription.providers.base import ProviderResponseError
from transcription.providers.base import TranscriptionResult
logger = logging.getLogger(__name__)
DEFAULT_OPENROUTER_MODEL = "google/gemini-2.5-flash"
@dataclass(frozen=True)
class OpenRouterRequest:
"""Normalized request payload fields for OpenRouter calls."""
model: str
messages: list[dict[str, Any]]
http_referer: str | None
x_open_router_title: str | None
class OpenRouterTranscriptionProvider:
"""Adapter that performs image transcription through OpenRouter."""
def __init__(self, *, settings: Settings | None = None, client: OpenRouter | None = None):
self._settings = settings or get_settings()
self._model = self._settings.provider_model or DEFAULT_OPENROUTER_MODEL
self._client = client or OpenRouter(api_key=self._settings.openrouter_api_key)
@property
def model(self) -> str:
"""Return the resolved OpenRouter model slug."""
return self._model
async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
"""Send prompt + image to OpenRouter and return normalized text output."""
request = self._build_request(prompt_text=prompt_text, image_bytes=image_bytes, mime_type=mime_type)
try:
response = await self._client.chat.send_async(
messages=cast(list[ChatMessagesTypedDict], request.messages),
model=request.model,
http_referer=request.http_referer,
x_open_router_title=request.x_open_router_title,
)
except Exception as exc:
message = str(exc).lower()
if "401" in message or "auth" in message or "api key" in message:
raise ProviderAuthError("OpenRouter authentication failed") from exc
raise ProviderError("OpenRouter request failed") from exc
text = self._extract_text(response)
model = self._get_optional_attr(response, "model") or self.model
finish_reason = self._extract_finish_reason(response)
usage_input_tokens, usage_output_tokens, usage_total_tokens = self._extract_usage(response)
logger.info("OpenRouter transcription completed using model=%s", model)
return TranscriptionResult(
text=text,
provider="openrouter",
prompt_name="",
model=model,
finish_reason=finish_reason,
usage_input_tokens=usage_input_tokens,
usage_output_tokens=usage_output_tokens,
usage_total_tokens=usage_total_tokens,
)
def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest:
image_b64 = base64.b64encode(image_bytes).decode("ascii")
data_url = f"data:{mime_type};base64,{image_b64}"
messages: list[dict[str, Any]] = [
{
"role": "user",
"content": [
{"type": "text", "text": prompt_text},
{"type": "image_url", "image_url": {"url": data_url}},
],
}
]
return OpenRouterRequest(
model=self.model,
messages=messages,
http_referer=self._settings.openrouter_http_referer,
x_open_router_title=self._settings.openrouter_app_title,
)
def _extract_text(self, response: Any) -> str:
choices = self._get_optional_attr(response, "choices")
if not choices:
raise ProviderResponseError("OpenRouter response missing choices")
first_choice = choices[0]
message = self._get_optional_attr(first_choice, "message")
if message is None:
raise ProviderResponseError("OpenRouter response missing assistant message")
content = self._get_optional_attr(message, "content")
text = self._normalize_content(content)
if not text:
raise ProviderResponseError("OpenRouter response contained no transcription text")
return text
def _extract_finish_reason(self, response: Any) -> str | None:
choices = self._get_optional_attr(response, "choices")
if not choices:
return None
first_choice = choices[0]
finish_reason = self._get_optional_attr(first_choice, "finish_reason")
if isinstance(finish_reason, str) and finish_reason.strip():
return finish_reason.strip()
return None
def _extract_usage(self, response: Any) -> tuple[int | None, int | None, int | None]:
usage = self._get_optional_attr(response, "usage")
if usage is None:
return None, None, None
input_tokens = self._as_int(self._get_optional_attr(usage, "prompt_tokens"))
output_tokens = self._as_int(self._get_optional_attr(usage, "completion_tokens"))
total_tokens = self._as_int(self._get_optional_attr(usage, "total_tokens"))
if input_tokens is None:
input_tokens = self._as_int(self._get_optional_attr(usage, "input_tokens"))
if output_tokens is None:
output_tokens = self._as_int(self._get_optional_attr(usage, "output_tokens"))
if total_tokens is None:
total_tokens = self._as_int(self._get_optional_attr(usage, "total"))
return input_tokens, output_tokens, total_tokens
def _normalize_content(self, content: Any) -> str:
if isinstance(content, str):
return content.strip()
if isinstance(content, list):
parts: list[str] = []
for item in content:
text_part = None
text_part = item.get("text") if isinstance(item, dict) else self._get_optional_attr(item, "text")
if isinstance(text_part, str) and text_part.strip():
parts.append(text_part.strip())
return "\n".join(parts).strip()
return ""
@staticmethod
def _get_optional_attr(obj: Any, key: str) -> Any:
if obj is None:
return None
if isinstance(obj, dict):
return obj.get(key)
return getattr(obj, key, None)
@staticmethod
def _as_int(value: Any) -> int | None:
if isinstance(value, int):
return value
return None
src/transcription/services/__init__.py
-19
View File
@@ -1,19 +0,0 @@
"""Service layer exports."""
from dataclasses import dataclass
from dataclasses import field
from .documents import DocumentService
from .jobs import JobService
from .transcription import TranscriptionService
__all__ = ["DocumentService", "JobService", "ServiceBundle", "TranscriptionService"]
@dataclass(frozen=True, slots=True)
class ServiceBundle:
"""Container for all service instances."""
documents: DocumentService = field(default_factory=DocumentService)
jobs: JobService = field(default_factory=JobService)
transcriptions: TranscriptionService = field(default_factory=TranscriptionService)
src/transcription/services/base.py
-60
View File
@@ -1,60 +0,0 @@
import asyncio
from abc import ABC
from collections.abc import Sequence
from contextlib import asynccontextmanager
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlmodel.ext.asyncio.session import AsyncSession
from ..config import Settings
from ..config import get_settings
from ..db.runtime import get_session_factory
class ServiceBase(ABC):
"""Thin service class for managing documents in the database."""
settings: Settings
session_factory: async_sessionmaker[AsyncSession]
queue: asyncio.Queue
def __init__(
self,
session_factory: async_sessionmaker[AsyncSession] | None = None,
queue: asyncio.Queue | None = None,
):
self.settings = get_settings()
self.session_factory = session_factory or get_session_factory()
self.queue = queue or asyncio.Queue()
@asynccontextmanager
async def _session_scope(self, session: AsyncSession | None = None):
"""Provide a transactional scope around a series of operations."""
if session is not None:
# Reuse the provided session if one is passed in
yield session
else:
# Otherwise, create a new session for this scope
async with self.session_factory() as new_session:
yield new_session
async def _finalize(
self,
*,
session: AsyncSession,
caller_session: AsyncSession | None,
refresh: Sequence[object] = (),
) -> None:
"""Finalize a write based on transaction ownership.
Service-owned sessions commit immediately. Caller-owned sessions flush so
orchestration code can commit once at a larger transaction boundary.
"""
should_commit = caller_session is None
if should_commit:
await session.commit()
else:
await session.flush()
for obj in refresh:
await session.refresh(obj)
src/transcription/services/documents.py
-130
View File
@@ -1,130 +0,0 @@
import logging
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from uuid import UUID
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import selectinload
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from ..errors import AppError
from ..errors import ErrorCategory
from ..models import Document
from .base import ServiceBase
logger = logging.getLogger(__name__)
class DocumentError(AppError):
"""Raised when document operations fail."""
class MissingSourceError(DocumentError):
"""Raised when a document has no associated sources."""
class UploadError(DocumentError):
"""Raised when uploaded content cannot be persisted safely."""
class DocumentAlreadyExistsError(DocumentError):
"""Raised when a document with the same name already exists in the database."""
@dataclass(frozen=True)
class UploadJobResult:
"""Summary of created upload records."""
document_id: UUID
job_id: UUID
stored_path: Path
original_filename: str
class DocumentService(ServiceBase):
"""Thin service class for managing documents in the database."""
#
# CRUD Operations
#
async def create_document(
self,
document: Document,
*,
session: AsyncSession | None = None,
) -> Document:
"""Create a new document in the database."""
async with self._session_scope(session) as _session:
_session.add(document)
try:
await self._finalize(session=_session, caller_session=session, refresh=(document,))
except IntegrityError as exc:
raise DocumentAlreadyExistsError(
f"Document with id {document.id} already exists",
category=ErrorCategory.VALIDATION,
suggestion="Rename the file and try again.",
) from exc
return document
async def read_document(self, document_id: UUID, *, session: AsyncSession | None = None) -> Document:
"""Read an existing document from the database.
The selectinload option is used to eagerly load related jobs and sources.
"""
async with self._session_scope(session) as _session:
document = await _session.get(
Document,
document_id,
options=(
selectinload(Document.jobs), # pyright: ignore[reportArgumentType]
selectinload(Document.sources), # pyright: ignore[reportArgumentType]
),
)
if document is None:
raise DocumentError(
f"Document with id {document_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Re-upload the source document and retry.",
)
elif not document.sources:
raise MissingSourceError(
f"Document with id {document_id} has no associated source records",
category=ErrorCategory.NOT_FOUND,
suggestion="Upload at least one source for this document and retry.",
)
return document
async def update_document(self, document: Document, *, session: AsyncSession | None = None) -> Document:
"""Update an existing document in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(document)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_document(self, document: Document, *, session: AsyncSession | None = None) -> None:
"""Delete a document from the database."""
async with self._session_scope(session) as _session:
await _session.delete(document)
await self._finalize(session=_session, caller_session=session)
# Query Operations
async def query_documents(
self, *, name: str | None = None, session: AsyncSession | None = None
) -> Sequence[Document]:
"""Query documents from the database based on provided filters."""
async with self._session_scope(session) as _session:
query = select(Document)
if name is not None:
query = query.where(Document.name == name)
result = await _session.exec(query)
return result.all()
async def list_documents(self, *, session: AsyncSession | None = None) -> Sequence[Document]:
"""List all documents in the database."""
async with self._session_scope(session) as _session:
result = await _session.exec(select(Document))
return result.all()
src/transcription/services/jobs.py
-188
View File
@@ -1,188 +0,0 @@
from collections.abc import Sequence
from datetime import UTC
from datetime import datetime
from uuid import UUID
from sqlalchemy.orm import selectinload
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from ..models import Job
from ..models import JobStatus
from ..models import Source
from .base import ServiceBase
class JobService(ServiceBase):
"""Thin service class for managing jobs in the database."""
#
# CRUD Operations
#
async def create_job(self, job: Job, session: AsyncSession | None = None) -> Job:
"""Create a new job in the database."""
async with self._session_scope(session) as _session:
_session.add(job)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def read_job(self, job_id: UUID, session: AsyncSession | None = None) -> Job:
"""Read an existing job from the database.
The related document is always eagerly loaded so callers can safely
access ``job.document`` in async contexts without triggering lazy-load IO.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources).selectinload(Source.revision), # pyright: ignore[reportArgumentType]
)
.where(Job.id == job_id)
.execution_options(populate_existing=True)
)
job = (await _session.exec(query)).first()
if job is None:
raise ValueError(f"Job with id {job_id} not found")
return job
async def update_job(self, job: Job, session: AsyncSession | None = None) -> Job:
"""Update an existing job in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(job)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_job(self, job: Job, session: AsyncSession | None = None) -> None:
"""Delete a job from the database."""
async with self._session_scope(session) as _session:
await _session.delete(job)
await self._finalize(session=_session, caller_session=session)
# Query Operations
async def query_jobs(
self,
*,
status: JobStatus | None = None,
filename: str | None = None,
session: AsyncSession | None = None,
) -> Sequence[Job]:
"""Query jobs from the database based on provided filters."""
async with self._session_scope(session) as _session:
query = select(Job).options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources), # pyright: ignore[reportArgumentType]
)
if status is not None:
query = query.where(Job.status == status)
if filename is not None:
query = query.where(Job.sources.any(Source.filename == filename))
result = await _session.exec(query)
return result.all()
async def list_jobs(
self,
*,
load_docs: bool = False,
session: AsyncSession | None = None,
) -> Sequence[Job]:
"""List all jobs in the database with eagerly loaded documents."""
_ = load_docs
async with self._session_scope(session) as _session:
query = select(Job).options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources), # pyright: ignore[reportArgumentType]
)
result = await _session.exec(query)
return result.all()
# Other Operations
async def mark_job_status(
self,
job_id: UUID,
status: JobStatus,
session: AsyncSession | None = None,
) -> Job:
"""Mark a job with a new status."""
return await self.update_job_state(job_id=job_id, status=status, session=session)
async def update_job_state(
self,
*,
job_id: UUID,
status: JobStatus,
retry_count_increment: int = 0,
session: AsyncSession | None = None,
) -> Job:
"""Update a job's lifecycle fields.
When ``session`` is provided, this method flushes so callers can commit
once at an orchestration boundary.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
.where(Job.id == job_id)
.execution_options(populate_existing=True)
)
job = (await _session.exec(query)).first()
if job is None:
raise ValueError(f"Job with id {job_id} not found")
job.status = status
if retry_count_increment:
job.retry_count += retry_count_increment
job.date_updated = datetime.now(UTC)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def read_next_queued_job(
self,
*,
session: AsyncSession | None = None,
) -> Job | None:
"""Read the next queued job ordered by creation time."""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources).selectinload(Source.revision), # pyright: ignore[reportArgumentType]
)
.where(Job.status == JobStatus.QUEUED)
.order_by(Job.date_created) # pyright: ignore[reportArgumentType]
)
return (await _session.exec(query)).first()
async def requeue_stale_processing_jobs(
self,
*,
stale_before: datetime,
session: AsyncSession | None = None,
) -> int:
"""Move stale processing jobs back to queued state.
Jobs with ``status=PROCESSING`` and ``date_updated`` older than
``stale_before`` are considered stale and re-queued.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.where(Job.status == JobStatus.PROCESSING)
.where(Job.date_updated < stale_before)
)
stale_jobs = (await _session.exec(query)).all()
if not stale_jobs:
return 0
now = datetime.now(UTC)
for job in stale_jobs:
job.status = JobStatus.QUEUED
job.date_updated = now
await self._finalize(session=_session, caller_session=session, refresh=stale_jobs)
return len(stale_jobs)
src/transcription/services/store.py
-156
View File
@@ -1,156 +0,0 @@
from __future__ import annotations
import logging
from pathlib import Path
from uuid import uuid4
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.config import Settings
from transcription.config import get_settings
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from ..models import Document
from ..models import Job
from ..models import Source
from .documents import UploadJobResult
logger = logging.getLogger(__name__)
SUPPORTED_UPLOAD_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tif", ".tiff", ".pdf"}
class UploadError(AppError):
"""Raised when uploaded content cannot be persisted safely."""
async def create_upload_job(
*,
filename: str,
file_bytes: bytes,
session: AsyncSession,
settings: Settings | None = None,
) -> UploadJobResult:
"""Create upload-backed document and queued job records."""
runtime_settings = settings or get_settings()
stored_path = store_file(
filename=filename,
file_bytes=file_bytes,
settings=runtime_settings,
)
try:
document, job = await _create_upload_records(
session=session,
original_filename=filename,
stored_path=stored_path,
)
except Exception as exc:
_best_effort_delete(stored_path)
raise UploadError(
"Failed to create upload database records",
category=ErrorCategory.INFRA_TRANSIENT,
suggestion="Retry upload. If this keeps happening, verify database availability.",
retriable=True,
) from exc
logger.info("Created upload job document_id=%s job_id=%s", document.id, job.id)
return UploadJobResult(
document_id=document.id,
job_id=job.id,
stored_path=stored_path,
original_filename=Path(filename).name,
)
async def _create_upload_records(
*,
session: AsyncSession,
original_filename: str,
stored_path: Path,
) -> tuple[Document, Job]:
document = Document(
name=Path(original_filename).name,
)
session.add(document)
await session.flush()
job = Job(document_id=document.id)
session.add(job)
await session.flush()
source = Source(
document_id=document.id,
job_id=job.id,
upload_name=Path(original_filename).name,
filename=stored_path.name,
file_path=str(stored_path),
)
session.add(source)
await session.commit()
await session.refresh(document)
await session.refresh(job)
return document, job
def _best_effort_delete(path: Path) -> None:
try:
if path.exists():
path.unlink()
except OSError:
logger.warning("Failed to clean up upload file after DB error: %s", path)
def store_file(*, filename: str, file_bytes: bytes, settings: Settings | None = None) -> Path:
"""Persist an uploaded file to the configured upload directory."""
runtime_settings = settings or get_settings()
_validate_upload(filename=filename, file_bytes=file_bytes)
upload_dir = runtime_settings.upload_dir
upload_dir.mkdir(parents=True, exist_ok=True)
stored_name = _build_stored_filename(filename)
stored_path = upload_dir / stored_name
try:
stored_path.write_bytes(file_bytes)
except OSError as exc:
raise UploadError(
"Failed to persist upload file",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Check upload directory permissions and available disk space, then retry.",
) from exc
logger.info("Stored uploaded file: %s", stored_path)
return stored_path
def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
if not file_bytes:
raise UploadError(
"Upload payload is empty",
category=ErrorCategory.VALIDATION,
suggestion="Select a non-empty file and try again.",
)
safe_name = Path(filename).name
if not safe_name:
raise UploadError(
"Upload filename is required",
category=ErrorCategory.VALIDATION,
suggestion="Choose a file with a valid filename and retry.",
)
suffix = Path(safe_name).suffix.lower()
if suffix not in SUPPORTED_UPLOAD_EXTENSIONS:
raise UploadError(
f"Unsupported upload extension: {suffix}",
category=ErrorCategory.USER_INPUT,
suggestion="Upload JPG, JPEG, PNG, TIFF, or PDF files only.",
)
def _build_stored_filename(filename: str) -> str:
safe_name = Path(filename).name
return f"{uuid4()}_{safe_name}"
src/transcription/services/transcription.py
-341
View File
@@ -1,341 +0,0 @@
"""Prompt loading and provider-backed transcription service."""
from __future__ import annotations
import logging
import mimetypes
from collections.abc import Sequence
from contextlib import contextmanager
from datetime import UTC
from datetime import datetime
from pathlib import Path
from uuid import UUID
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.orm import selectinload
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.config import Settings
from transcription.config import get_settings
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.models import Job
from transcription.models import Revision
from transcription.models import Source
from transcription.providers import ProviderAuthError
from transcription.providers import ProviderError
from transcription.providers import ProviderResponseError
from transcription.providers import TranscriptionProvider
from transcription.providers import TranscriptionResult
from transcription.providers import get_transcription_provider
from .base import ServiceBase
logger = logging.getLogger(__name__)
DEFAULT_PROMPT_FILE = "transcribe_document.md"
SUPPORTED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tif", ".tiff", ".pdf"}
class PromptLoadError(AppError):
"""Raised when prompt artifacts cannot be loaded safely."""
class TranscriptionError(AppError):
"""Raised when transcription execution fails."""
class TranscriptionNotFoundError(TranscriptionError):
"""Raised when a transcription-related resource is not found."""
class TranscriptionService(ServiceBase):
"""Service class for job transcription output and optional source revisions."""
provider: TranscriptionProvider
def __init__(self, session_factory: async_sessionmaker[AsyncSession] | None = None):
super().__init__(session_factory=session_factory)
self.provider = get_transcription_provider(settings=self.settings)
async def create_revision(self, revision: Revision, *, session: AsyncSession | None = None) -> Revision:
"""Create a new revision in the database."""
async with self._session_scope(session) as _session:
_session.add(revision)
await self._finalize(session=_session, caller_session=session, refresh=(revision,))
return revision
async def read_revision(self, revision_id: UUID, *, session: AsyncSession | None = None) -> Revision:
"""Read an existing revision from the database."""
async with self._session_scope(session) as _session:
revision = await _session.get(
Revision,
revision_id,
options=(selectinload(Revision.source),), # pyright: ignore[reportArgumentType]
)
if revision is None:
raise TranscriptionNotFoundError(
f"Revision with id {revision_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the revision id and retry.",
)
return revision
async def update_revision(self, revision: Revision, *, session: AsyncSession | None = None) -> Revision:
"""Update an existing revision in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(revision)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_revision(self, revision: Revision, *, session: AsyncSession | None = None) -> None:
"""Delete a revision from the database."""
async with self._session_scope(session) as _session:
await _session.delete(revision)
await self._finalize(session=_session, caller_session=session)
# Temporary compatibility methods for callers still using transcript naming.
async def read_transcript(self, transcript_id: UUID, *, session: AsyncSession | None = None) -> Revision:
"""Backward-compatible alias for read_revision."""
return await self.read_revision(transcript_id, session=session)
async def delete_transcript(self, transcript: Revision, *, session: AsyncSession | None = None) -> None:
"""Backward-compatible alias for delete_revision."""
await self.delete_revision(transcript, session=session)
async def transcribe_document(
self,
image_path: str | Path,
job_id: UUID,
*,
prompt_name: str = DEFAULT_PROMPT_FILE,
session: AsyncSession | None = None,
) -> None:
"""Transcribe a local image using the configured prompt and provider."""
result = await transcribe_document_image(
image_path=image_path,
prompt_name=prompt_name,
settings=self.settings,
provider=self.provider,
)
await self.update_job_transcription(
job_id=job_id,
text=result.text,
error_detail=None,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=session,
)
async def update_job_transcription(
self,
*,
job_id: UUID,
text: str | None,
error_detail: str | None = None,
provider: str | None = None,
model: str | None = None,
prompt_name: str = DEFAULT_PROMPT_FILE,
session: AsyncSession | None = None,
) -> Job:
"""Persist original transcription output fields on a job."""
async with self._session_scope(session) as _session:
job = await _session.get(Job, job_id)
if job is None:
raise TranscriptionNotFoundError(
f"Job with id {job_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the job id and retry.",
)
job.text = text
job.error_detail = error_detail
job.provider = provider or job.provider or self.settings.provider.value
job.model = model or job.model or _resolve_transcript_model(provider=self.provider, settings=self.settings)
job.prompt_name = prompt_name or job.prompt_name or DEFAULT_PROMPT_FILE
job.date_updated = datetime.now(UTC)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def upsert_revision_for_source(
self,
*,
source_id: UUID,
text: str,
session: AsyncSession | None = None,
) -> Revision:
"""Create or replace the single optional revision for a source."""
async with self._session_scope(session) as _session:
source = await _session.get(Source, source_id)
if source is None:
raise TranscriptionNotFoundError(
f"Source with id {source_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the source id and retry.",
)
query = select(Revision).where(Revision.source_id == source_id)
existing = (await _session.exec(query)).one_or_none()
if existing is None:
revision = Revision(source_id=source_id, text=text)
_session.add(revision)
await self._finalize(session=_session, caller_session=session, refresh=(revision,))
return revision
existing.text = text
merged = await _session.merge(existing)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def read_revision_by_source(
self,
source_id: UUID,
*,
session: AsyncSession | None = None,
) -> Revision | None:
"""Read the single optional revision for a source."""
async with self._session_scope(session) as _session:
query = select(Revision).where(Revision.source_id == source_id)
result = await _session.exec(query)
return result.one_or_none()
async def list_revisions_by_job(
self,
job_id: UUID,
*,
session: AsyncSession | None = None,
) -> Sequence[Revision]:
"""List revisions connected to all sources for a job."""
async with self._session_scope(session) as _session:
query = (
select(Revision)
.join(Source, Source.id == Revision.source_id)
.where(Source.job_id == job_id)
.order_by(Revision.date_created) # pyright: ignore[reportArgumentType]
)
result = await _session.exec(query)
return result.all()
def _resolve_transcript_model(*, provider: TranscriptionProvider, settings: Settings) -> str:
provider_model = getattr(provider, "model", None)
if isinstance(provider_model, str) and provider_model.strip():
return provider_model
if settings.provider_model and settings.provider_model.strip():
return settings.provider_model
return "unknown"
async def transcribe_document_image(
image_path: str | Path,
*,
prompt_name: str = DEFAULT_PROMPT_FILE,
settings: Settings | None = None,
provider: TranscriptionProvider | None = None,
) -> TranscriptionResult:
"""Transcribe a local image using the configured prompt and provider."""
runtime_settings = settings or get_settings()
prompt_text = load_prompt_text(prompt_name=prompt_name, settings=runtime_settings)
image_bytes, mime_type = load_image_payload(image_path)
adapter = provider or get_transcription_provider(settings=runtime_settings)
logger.info("Starting transcription for image=%s mime_type=%s", image_path, mime_type)
with handle_transcription_errors():
result = await adapter.transcribe(
prompt_text=prompt_text,
image_bytes=image_bytes,
mime_type=mime_type,
)
logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
return result
def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settings | None = None) -> str:
"""Load and validate prompt text from PROMPT_DIR."""
runtime_settings = settings or get_settings()
prompt_path = runtime_settings.prompt_dir / prompt_name
if not prompt_path.exists() or not prompt_path.is_file():
raise PromptLoadError(
f"Prompt file not found: {prompt_path}",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify PROMPT_DIR and prompt file configuration, then retry.",
)
prompt_text = prompt_path.read_text(encoding="utf-8").strip()
if not prompt_text:
raise PromptLoadError(
f"Prompt file is empty: {prompt_path}",
category=ErrorCategory.VALIDATION,
suggestion="Populate the prompt file with valid instructions and retry.",
)
logger.info("Loaded prompt artifact: %s", prompt_path)
return prompt_text
def load_image_payload(image_path: str | Path) -> tuple[bytes, str]:
"""Read image bytes and detect mime type for supported uploads."""
path = Path(image_path)
if not path.exists() or not path.is_file():
raise TranscriptionError(
f"Image file not found: {path}",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the uploaded file exists and retry from the jobs page.",
)
suffix = path.suffix.lower()
if suffix not in SUPPORTED_EXTENSIONS:
raise TranscriptionError(
f"Unsupported file type: {suffix}",
category=ErrorCategory.USER_INPUT,
suggestion="Use JPG, JPEG, PNG, TIFF, or PDF files.",
)
mime_type, _ = mimetypes.guess_type(path.name)
if suffix in {".tif", ".tiff"}:
mime_type = "image/tiff"
if not mime_type:
raise TranscriptionError(
f"Unable to determine MIME type for: {path}",
category=ErrorCategory.VALIDATION,
suggestion="Re-save the file in a supported format and retry.",
)
return path.read_bytes(), mime_type
@contextmanager
def handle_transcription_errors():
"""Context manager to handle transcription errors."""
try:
yield
except ProviderAuthError as exc:
raise TranscriptionError(
"Provider authentication failed",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify provider API credentials and retry.",
) from exc
except ProviderResponseError as exc:
raise TranscriptionError(
"Provider returned an invalid response",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry once. If it persists, try a different provider model or inspect provider status.",
retriable=True,
) from exc
except ProviderError as exc:
raise TranscriptionError(
"Provider transcription failed",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
retriable=True,
) from exc
src/transcription/services/workflows.py
-345
View File
@@ -1,345 +0,0 @@
import asyncio
import logging
from sqlmodel.ext.asyncio.session import AsyncSession
from ..config import Settings
from ..config import get_settings
from ..errors import AppError
from ..errors import ErrorCategory
from ..errors import classify_unexpected_error
from ..errors import format_error_detail
from ..models import Job
from ..models import JobStatus
from ..models import Source
from ..providers import TranscriptionResult
from . import ServiceBundle
from .transcription import DEFAULT_PROMPT_FILE
from .transcription import transcribe_document_image
logger = logging.getLogger(__name__)
async def advance_job(
job: Job,
services: ServiceBundle,
settings: Settings | None = None,
session: AsyncSession | None = None,
) -> Job | None:
"""Advance a single job by lifecycle status."""
settings = settings or get_settings()
match job.status:
case JobStatus.QUEUED:
return await process_queued_job(job=job, services=services, settings=settings, session=session)
case JobStatus.FAILED:
if job.retry_count < settings.worker_max_retries:
return await services.jobs.update_job_state(
job_id=job.id,
status=JobStatus.QUEUED,
retry_count_increment=1,
session=session,
)
else:
logger.error(f"Job {job.id} has failed and reached max retries.")
return
case _:
return
async def process_queued_job(
*,
job: Job,
services: ServiceBundle,
settings: Settings | None = None,
session: AsyncSession | None = None,
) -> Job | None:
"""Process one complete transcription attempt for a queued job."""
runtime_settings = settings or get_settings()
if job.status != JobStatus.QUEUED:
logger.warning(f"Job {job.id} is not queued. Current status: {job.status}")
return
# Transaction A: claim job for processing.
if session is None:
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING)
else:
# If we're sharing the session, need to make sure setting the Job to PROCESSING is committed before we start
# the transcription, otherwise other workers may see the job as still QUEUED and try to process it.
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
await session.commit()
source_job = await services.jobs.read_job(job_id=job.id, session=session)
source = _resolve_primary_source(source_job)
assert source is not None, f"Job {job.id} has no associated source record."
started_at = asyncio.get_running_loop().time()
try:
result = await asyncio.wait_for(
transcribe_document_image(source.file_path),
timeout=runtime_settings.worker_provider_timeout_seconds,
)
elapsed_seconds = asyncio.get_running_loop().time() - started_at
logger.info(
"Provider response diagnostics operation=worker.provider_response "
"job_id=%s document_id=%s source_id=%s provider=%s model=%s "
"finish_reason=%s usage_input_tokens=%s usage_output_tokens=%s usage_total_tokens=%s "
"latency_seconds=%.3f text_chars=%s text_lines=%s",
job.id,
job.document_id,
source.id,
result.provider,
result.model,
result.finish_reason or "unknown",
result.usage_input_tokens,
result.usage_output_tokens,
result.usage_total_tokens,
elapsed_seconds,
len(result.text),
_line_count(result.text),
)
_validate_transcription_quality(result=result, settings=runtime_settings)
job = await _finalize_transcribed(job=job, services=services, result=result, session=session)
logger.info(
"Job transcribed operation=worker.process_job job_id=%s document_id=%s source_id=%s provider=%s",
job.id,
job.document_id,
source.id,
result.provider,
)
except TimeoutError:
error = AppError(
f"Provider call timed out after {runtime_settings.worker_provider_timeout_seconds:.1f}s",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry the job. If this repeats, verify provider latency and request payload size.",
retriable=True,
)
job = await _finalize_failed(job=job, services=services, error=error, session=session)
logger.error(
"Job failed operation=worker.process_job job_id=%s document_id=%s source_id=%s error_id=%s category=%s",
job.id,
job.document_id,
source.id,
error.error_id,
error.category.value,
)
except Exception as exc: # noqa: BLE001
match exc:
case AppError() as error:
pass
case _:
error = classify_unexpected_error(exc, operation="worker.process_job")
job = await _finalize_failed(job=job, services=services, error=error, session=session)
logger.error(
"Job failed operation=worker.process_job job_id=%s document_id=%s source_id=%s error_id=%s category=%s",
job.id,
job.document_id,
source.id,
error.error_id,
error.category.value,
)
return job
async def process_next_queued_job(
*,
services: ServiceBundle,
settings: Settings | None = None,
session: AsyncSession | None = None,
) -> bool:
"""Process the next queued job if one exists."""
job = await services.jobs.read_next_queued_job(session=session)
if job is None:
return False
await advance_job(job=job, services=services, settings=settings, session=session)
return True
async def _finalize_transcribed(
*,
job: Job,
services: ServiceBundle,
result: TranscriptionResult,
session: AsyncSession | None = None,
) -> Job:
"""Transaction B: job transcription output + TRANSCRIBED in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.update_job_transcription(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=local_session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.TRANSCRIBED,
session=local_session,
)
await local_session.commit()
return updated_job
await services.transcriptions.update_job_transcription(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.TRANSCRIBED,
session=session,
)
await session.commit()
return updated_job
async def _finalize_retry(
*,
job: Job,
services: ServiceBundle,
error: AppError,
settings: Settings,
session: AsyncSession | None = None,
) -> Job:
"""Transaction C: job error detail + QUEUED + retry increment in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.update_job_transcription(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=local_session,
)
updated_job = await services.jobs.update_job_state(
job_id=job.id,
status=JobStatus.QUEUED,
retry_count_increment=1,
session=local_session,
)
await local_session.commit()
else:
await services.transcriptions.update_job_transcription(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=session,
)
updated_job = await services.jobs.update_job_state(
job_id=job.id,
status=JobStatus.QUEUED,
retry_count_increment=1,
session=session,
)
await session.commit()
if settings.worker_retry_backoff_seconds > 0:
await asyncio.sleep(settings.worker_retry_backoff_seconds)
return updated_job
async def _finalize_failed(
*,
job: Job,
services: ServiceBundle,
error: AppError,
session: AsyncSession | None = None,
) -> Job:
"""Transaction B: job error detail + FAILED in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.update_job_transcription(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=local_session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.FAILED,
session=local_session,
)
await local_session.commit()
return updated_job
await services.transcriptions.update_job_transcription(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.FAILED,
session=session,
)
await session.commit()
return updated_job
def _resolve_primary_source(job: Job) -> Source | None:
if not job.sources:
return None
return job.sources[0]
def _validate_transcription_quality(*, result: TranscriptionResult, settings: Settings) -> None:
text_chars = len(result.text)
text_lines = _line_count(result.text)
if settings.worker_fail_on_finish_reason_length and (result.finish_reason or "").lower() == "length":
raise AppError(
"Provider output appears truncated (finish_reason=length)",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion=(
"Retry the job. If this repeats, use a faster model, reduce input complexity, "
"or increase provider output budget."
),
retriable=True,
)
if settings.worker_min_transcription_chars > 0 and text_chars < settings.worker_min_transcription_chars:
raise AppError(
(
"Transcription output below configured minimum character threshold "
f"({text_chars} < {settings.worker_min_transcription_chars})"
),
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion=(
"Retry the job. If this repeats, switch model or raise minimum thresholds based on document type."
),
retriable=True,
)
if settings.worker_min_transcription_lines > 0 and text_lines < settings.worker_min_transcription_lines:
raise AppError(
(
"Transcription output below configured minimum line threshold "
f"({text_lines} < {settings.worker_min_transcription_lines})"
),
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion=(
"Retry the job. If this repeats, switch model or raise minimum thresholds based on document type."
),
retriable=True,
)
def _line_count(text: str) -> int:
stripped = text.strip()
if not stripped:
return 0
return sum(1 for line in stripped.splitlines() if line.strip())
src/transcription/ui/__init__.py
-45
View File
@@ -1,45 +0,0 @@
"""UI page registration exports."""
from pathlib import Path
from fastapi import FastAPI
from nicegui import app as nicegui_app
from nicegui import ui
from transcription.ui.pages.jobs_page import register_page as register_jobs_page
from transcription.ui.pages.upload_page import register_page as register_upload_page
_THEME_REGISTERED_STATE_KEY = "transcription_ui_theme_registered"
_THEME_COLORS: dict[str, str] = {
"primary": "#6f97e8",
"secondary": "#92b5f5",
"accent": "#7fc0de",
"dark": "#22304a",
"dark_page": "#1a2538",
"positive": "#86c8ad",
"negative": "#d98a9a",
"info": "#7ebdda",
"warning": "#e2c083",
}
def _register_global_styles(app: FastAPI) -> None:
if getattr(app.state, _THEME_REGISTERED_STATE_KEY, False):
return
nicegui_app.colors(**_THEME_COLORS)
css_path = Path(__file__).resolve().parent / "static" / "colors.css"
if css_path.exists():
ui.add_css(css_path, shared=True)
setattr(app.state, _THEME_REGISTERED_STATE_KEY, True)
def register_pages(app: FastAPI) -> None:
"""Register all NiceGUI pages and mount them onto the FastAPI app."""
_register_global_styles(app)
register_upload_page()
register_jobs_page()
ui.run_with(app, mount_path="/ui", show_welcome_message=False, dark=True)
src/transcription/ui/components/__init__.py
-7
View File
@@ -1,7 +0,0 @@
"""Reusable UI component exports."""
from transcription.ui.components.app_shell import NAV_ITEMS
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.document_panzoom import render_document_panzoom
__all__ = ["NAV_ITEMS", "render_document_panzoom", "render_navigation_header"]
src/transcription/ui/components/app_shell.py
-59
View File
@@ -1,59 +0,0 @@
"""Reusable app shell primitives for page-level layout."""
from __future__ import annotations
from nicegui import ui
NAV_ITEMS: tuple[tuple[str, str, str], ...] = (
("Upload", "/upload", "upload_file"),
("Jobs", "/jobs", "work_history"),
)
def _is_active_path(*, current_path: str, item_path: str) -> bool:
if item_path == "/jobs":
return current_path == "/jobs" or current_path.startswith("/jobs/")
return current_path == item_path
def _button_props(*, icon: str, is_active: bool) -> str:
if is_active:
return f"icon={icon} no-caps unelevated color=primary text-color=white"
return f"icon={icon} no-caps outline color=secondary text-color=secondary"
def _button_classes(*, is_active: bool) -> str:
base = "w-full sm:w-auto min-h-[40px] px-3 rounded-lg text-body2 text-weight-medium"
if is_active:
return f"{base}"
return f"{base}"
def _render_nav_button(*, label: str, path: str, icon: str, current_path: str) -> None:
is_active = _is_active_path(current_path=current_path, item_path=path)
button = ui.button(
label,
icon=icon,
on_click=lambda _=None, route=path: ui.navigate.to(route),
)
button.props(_button_props(icon=icon, is_active=is_active)).classes(_button_classes(is_active=is_active))
def _normalize_path(current_path: str | None) -> str:
normalized = (current_path or "").strip()
if not normalized:
return "/upload"
return normalized.rstrip("/") or "/"
def render_navigation_header(*, current_path: str | None = None) -> None:
"""Render a shared app header with links for top-level pages."""
normalized_path = _normalize_path(current_path)
with (
ui.header(elevated=True).props("bordered").classes("bg-dark text-white q-px-sm q-py-xs"),
ui.row().classes("w-full items-center justify-end q-gutter-xs"),
ui.element("div").classes("w-full grid grid-cols-2 gap-2 sm:flex sm:justify-start sm:gap-2"),
):
for label, path, icon in NAV_ITEMS:
_render_nav_button(label=label, path=path, icon=icon, current_path=normalized_path)
src/transcription/ui/components/document_panzoom.py
-211
View File
@@ -1,211 +0,0 @@
"""Panzoom-backed document preview component."""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from urllib.parse import quote
from uuid import uuid4
from nicegui import ui
from transcription.config import get_settings
from transcription.models import Source
PANGOZOOM_CDN_URL = "https://unpkg.com/@panzoom/[email protected]/dist/panzoom.min.js"
UPLOADS_URL_PREFIX = "/uploads"
def render_document_panzoom(*, source: Source) -> None:
"""Render a source preview with pan and zoom interactions."""
_register_panzoom_assets()
host_id = f"document-panzoom-{uuid4().hex}"
document_url = _document_url(source)
document_kind = _document_kind(source)
with ui.card().classes("w-full q-pa-md"):
with ui.row().classes("w-full items-center justify-between no-wrap"):
ui.label("Document preview").classes("text-subtitle1 text-weight-medium")
ui.label(source.filename).classes("text-caption text-grey-4 ellipsis").style(
"max-width: 60%; text-align: right;"
)
with (
ui.element("div").classes("w-full document-panzoom-host rounded-borders q-mt-md")
# .style(f"height: {height};")
) as host:
host.props(f"id={host_id}")
with ui.element("div").classes("document-panzoom-surface"):
if document_kind == "pdf":
ui.html(
f'<iframe class="document-panzoom-iframe" '
f'src="{document_url}" title="{source.filename}" '
"data-panzoom-target></iframe>"
)
else:
ui.html(
f'<img class="document-panzoom-media" '
f'src="{document_url}" alt="{source.filename}" '
"data-panzoom-target data-panzoom-media />"
)
_attach_panzoom(host_id)
@lru_cache(maxsize=1)
def _register_panzoom_assets() -> None:
ui.add_head_html(
f'<script src="{PANGOZOOM_CDN_URL}"></script>',
shared=True,
)
ui.add_head_html(
"""
<style>
.document-panzoom-host {
overflow: hidden;
touch-action: none;
}
.document-panzoom-surface {
width: 100%;
height: 100%;
display: flex;
align-items: flex-start;
justify-content: flex-start;
}
.document-panzoom-media {
width: auto;
height: auto;
display: block;
max-width: 100%;
max-height: 100%;
user-select: none;
-webkit-user-drag: none;
}
.document-panzoom-iframe {
width: 100%;
height: 100%;
border: 0;
pointer-events: none;
background: white;
}
</style>
""",
shared=True,
)
def _document_url(source: Source) -> str:
file_path = Path(source.file_path)
upload_dir = get_settings().upload_dir
relative_path: Path
try:
relative_path = file_path.resolve().relative_to(upload_dir.resolve())
except ValueError:
parts = file_path.parts
if "uploads" in parts:
uploads_index = parts.index("uploads")
relative_path = Path(*parts[uploads_index + 1 :])
else:
relative_path = Path(file_path.name)
encoded_relative_path = "/".join(quote(part) for part in relative_path.parts)
return f"{UPLOADS_URL_PREFIX}/{encoded_relative_path}"
def _document_kind(source: Source) -> str:
suffix = Path(source.file_path).suffix.lower()
if suffix == ".pdf":
return "pdf"
return "image"
def _attach_panzoom(host_id: str) -> None:
ui.run_javascript(
f"""
(function() {{
if (!window.Panzoom) return;
window.__transcriptionPanzoom = window.__transcriptionPanzoom || {{}};
const host = document.getElementById({host_id!r});
if (!host) return;
const target = host.querySelector('[data-panzoom-target]');
const media = host.querySelector('[data-panzoom-media]');
if (!target) return;
const cleanup = () => {{
const existing = window.__transcriptionPanzoom[{host_id!r}];
if (existing?.resizeObserver) existing.resizeObserver.disconnect();
if (existing?.wheelHandler) host.removeEventListener('wheel', existing.wheelHandler);
if (existing?.instance) existing.instance.destroy();
}};
const computeFitScale = () => {{
const hostRect = host.getBoundingClientRect();
if (hostRect.width <= 0 || hostRect.height <= 0) return null;
return 1;
}};
const buildInstance = () => {{
cleanup();
const fitScale = computeFitScale();
if (fitScale === null) return false;
const minScale = Math.min(fitScale, 0.01);
const instance = Panzoom(target, {{
startX: 0,
startY: 0,
startScale: fitScale,
minScale: minScale,
maxScale: 256,
step: 0.2,
roundPixels: false,
panOnlyWhenZoomed: true,
overflow: 'hidden',
}});
const wheelHandler = (event) => instance.zoomWithWheel(event);
host.addEventListener('wheel', wheelHandler, {{ passive: false }});
requestAnimationFrame(() => {{
instance.reset({{ animate: false }});
}});
const resizeObserver = new ResizeObserver(() => {{
const nextFitScale = computeFitScale();
if (nextFitScale === null) return;
instance.setOptions({{
startScale: nextFitScale,
minScale: Math.min(nextFitScale, 0.01),
}});
instance.reset({{ animate: false }});
}});
resizeObserver.observe(host);
window.__transcriptionPanzoom[{host_id!r}] = {{
instance,
wheelHandler,
resizeObserver,
}};
return true;
}};
const initWhenReady = (retries = 15) => {{
if (buildInstance()) return;
if (retries <= 0) return;
requestAnimationFrame(() => initWhenReady(retries - 1));
}};
if (media && media.tagName === 'IMG' && !media.complete) {{
media.addEventListener('load', () => initWhenReady(), {{ once: true }});
return;
}}
initWhenReady();
}})();
"""
)
src/transcription/ui/components/error_presenter.py
-42
View File
@@ -1,42 +0,0 @@
"""Shared UI error rendering helpers."""
from __future__ import annotations
from nicegui import ui
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import classify_unexpected_error
def to_app_error(exc: Exception, *, operation: str) -> AppError:
"""Normalize any exception for consistent UI display."""
if isinstance(exc, AppError):
return exc
return classify_unexpected_error(exc, operation=operation)
def show_error(exc: Exception, *, title: str, operation: str) -> None:
"""Display a visible, actionable UI error with trace id."""
error = to_app_error(exc, operation=operation)
ui.notify(
f"{title}: {error.message} (ref: {error.error_id})",
type="negative",
timeout=0,
close_button="Dismiss",
)
with ui.card().classes("bg-red-1 text-red-10 q-mt-md q-pa-md"):
ui.label(title).classes("text-subtitle1")
ui.label(error.message)
ui.label(f"Suggested action: {error.suggestion}").classes("text-weight-medium")
ui.label(f"Error reference: {error.error_id}").classes("text-caption")
ui.label(f"Category: {error.category.value}").classes("text-caption")
def summarize_error(exc: Exception, *, operation: str) -> str:
"""Return short one-line summary for status labels."""
error = to_app_error(exc, operation=operation)
if error.category == ErrorCategory.INTERNAL_UNEXPECTED:
return f"Unexpected error (ref: {error.error_id})"
return f"{error.message} (ref: {error.error_id})"
src/transcription/ui/components/job_detail.py
-91
View File
@@ -1,91 +0,0 @@
"""Reusable job detail rendering helpers."""
from __future__ import annotations
import logging
from nicegui import ui
from transcription.models import Job
from transcription.models import Revision
from transcription.models import Source
from transcription.ui.components.document_panzoom import render_document_panzoom
from transcription.ui.components.transcript import render_original_transcription_card
from transcription.ui.components.transcript import render_revision_row
logger = logging.getLogger(__name__)
def _status_chip_classes(status: str) -> str:
if status == "queued":
return "bg-blue-1 text-blue-10"
if status == "processing":
return "bg-amber-1 text-amber-10"
if status == "transcribed":
return "bg-green-1 text-green-10"
if status == "failed":
return "bg-red-1 text-red-10"
return "bg-grey-2 text-grey-9"
def _metadata_row(label: str, value: str) -> None:
with ui.row().classes("w-full items-start justify-between q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-5 text-uppercase w-28")
ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
def _render_source_section(source: Source) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Source").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Upload name", source.upload_name)
_metadata_row("Stored filename", source.filename)
_metadata_row("File path", source.file_path)
_metadata_row("Uploaded", source.date_uploaded.isoformat())
ui.separator().classes("q-my-md")
render_document_panzoom(source=source)
def _render_revision_section(revision: Revision | None) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Revision").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
if revision is None:
ui.label("No revision exists for this source.").classes("text-body2 text-grey-3")
return
render_revision_row(revision=revision, initially_expanded=True)
def render_job_detail(*, job: Job, source: Source | None, revision: Revision | None) -> None:
"""Render all sections for the job detail page."""
logger.debug("Rendering job detail for job ID %s", job.id)
status_text = job.status.value
with ui.column().classes("w-full max-w-4xl q-gutter-md"):
with ui.card().classes("w-full q-pa-lg"):
with ui.row().classes("w-full items-center justify-between q-gutter-md"):
with ui.column().classes("q-gutter-none"):
ui.label("Job overview").classes("text-h6 text-weight-bold")
ui.label(str(job.id)).classes("text-caption text-grey-5")
status_chip_classes = (
"q-px-sm q-py-xs rounded-borders "
"text-weight-medium text-capitalize "
f"{_status_chip_classes(status_text)}"
)
ui.label(status_text).classes(status_chip_classes)
ui.separator().classes("q-my-md bg-blue-grey-7")
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Created", job.date_created.isoformat())
_metadata_row("Updated", job.date_updated.isoformat())
_metadata_row("Retries", str(job.retry_count))
render_original_transcription_card(job=job)
if source is not None:
_render_source_section(source)
_render_revision_section(revision)
src/transcription/ui/components/table/__init__.py
-4
View File
@@ -1,4 +0,0 @@
from .jobs import JobTableRow
from .jobs import render_jobs_table
__all__ = ["JobTableRow", "render_jobs_table"]
src/transcription/ui/components/table/common.py
-73
View File
@@ -1,73 +0,0 @@
"""Common logic for generating table widgets."""
import logging
from collections.abc import Callable
from typing import Any
from nicegui import events
from nicegui import ui
logger = logging.getLogger(__name__)
def _extract_row_id(args: Any) -> str | None:
if isinstance(args, dict):
if isinstance(args.get("row"), dict):
row_id = args["row"].get("id")
return str(row_id) if row_id is not None else None
row_id = args.get("id")
return str(row_id) if row_id is not None else None
if isinstance(args, list):
for value in args:
if isinstance(value, dict):
row_id = value.get("id")
if row_id is not None:
return str(row_id)
return None
def _bind_row_click_handler(
table: Any,
*,
on_row_click_id: Callable[[str], None],
) -> None:
def handle_row_click(event: events.GenericEventArguments) -> None:
row_id = _extract_row_id(event.args)
if row_id is None:
return
on_row_click_id(row_id)
table.on("rowClick", handle_row_click)
logger.debug("Row click handler bound to table")
def build_table(
rows: list[dict[str, Any]],
columns: list[dict[str, Any]],
*,
default_sort_by: str | None = None,
default_descending: bool = False,
classes: str = "app-table",
on_row_click_id: Callable[[str], None] | None = None,
) -> Any:
pagination: dict[str, Any] = {"rowsPerPage": 25}
if default_sort_by is not None:
pagination["sortBy"] = default_sort_by
pagination["descending"] = default_descending
table = (
ui.table(
rows=rows,
columns=columns,
row_key="id",
pagination=pagination,
)
.classes(classes)
.props('table-style="table-layout: fixed; width: 100%;"')
)
logger.info("Table built with %d rows and %d columns", len(rows), len(columns))
if on_row_click_id is not None:
_bind_row_click_handler(table, on_row_click_id=on_row_click_id)
return table
src/transcription/ui/components/table/jobs.py
-75
View File
@@ -1,75 +0,0 @@
"""Jobs table rendering helpers."""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass
from datetime import UTC
from datetime import datetime
from typing import Any
from uuid import UUID
from nicegui import ui
from .common import build_table
@dataclass(frozen=True, slots=True)
class JobTableRow:
"""Read model consumed by the jobs table component."""
id: UUID
status: str
filename: str
retry_count: int
date_created: str
date_updated: str
def _format_timestamp(value: str) -> str:
"""Return a friendly UTC timestamp for table display."""
try:
parsed = datetime.fromisoformat(value)
except ValueError:
return value
parsed = parsed.replace(tzinfo=UTC) if parsed.tzinfo is None else parsed.astimezone(UTC)
return parsed.astimezone().strftime("%b %d, %I:%M %p")
def _serialize_rows(rows: Sequence[JobTableRow]) -> list[dict[str, Any]]:
return [
{
"id": str(row.id),
"status": row.status,
"filename": row.filename,
"retry_count": row.retry_count,
"date_created": _format_timestamp(row.date_created),
"date_updated": _format_timestamp(row.date_updated),
"created_sort": row.date_created,
"updated_sort": row.date_updated,
}
for row in rows
]
def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
"""Render jobs table and open a detail page when clicking a row."""
if not rows:
ui.label("No jobs yet.")
return
build_table(
rows=_serialize_rows(rows),
columns=[
{"name": "id", "label": "Job ID", "field": "id", "sortable": True},
{"name": "status", "label": "Status", "field": "status", "sortable": True},
{"name": "filename", "label": "Filename", "field": "filename", "sortable": True},
{"name": "retry_count", "label": "Retries", "field": "retry_count", "sortable": True},
{"name": "date_created", "label": "Created", "field": "date_created", "sortable": True},
{"name": "date_updated", "label": "Updated", "field": "date_updated", "sortable": True},
],
default_sort_by="created_sort",
default_descending=True,
classes="app-table w-full",
on_row_click_id=lambda job_id: ui.navigate.to(f"/jobs/{job_id}"),
)
src/transcription/ui/components/transcript.py
-106
View File
@@ -1,106 +0,0 @@
"""Reusable transcript UI components."""
from __future__ import annotations
from collections.abc import Awaitable
from collections.abc import Callable
from datetime import datetime
from typing import Any
from nicegui import ui
from transcription.models import Job
from transcription.models import Revision
type RevisionAction = Callable[[Revision], Awaitable[None] | None]
def render_original_transcription_card(*, job: Job, classes: str = "w-full") -> Any:
"""Render the immutable original job transcription output."""
status_label = "Failed" if job.error_detail else "Transcribed"
header = f"Original Transcription | {status_label}"
provider = job.provider or "unknown"
model = job.model or "unknown"
caption = f"{provider} | {model} | {_format_created_at(job.date_updated)}"
card = ui.card().classes(f"{classes} q-pa-md bg-blue-grey-10")
with card, ui.column().classes("w-full q-gutter-y-sm"):
ui.label(header).classes("text-subtitle1 text-weight-medium")
ui.label(caption).classes("text-caption text-grey-5")
_metadata_row(label="Prompt", value=job.prompt_name or "unknown")
_metadata_row(label="Updated", value=_format_created_at(job.date_updated))
if job.text:
with ui.card().classes("w-full q-pa-sm"):
ui.markdown(job.text)
if job.error_detail:
with ui.card().classes("w-full bg-red-1 text-red-10 q-pa-sm"):
ui.label("Failure detail").classes("text-caption text-uppercase")
ui.label(job.error_detail).classes("text-body2")
return card
def render_revision_row(
*,
revision: Revision,
initially_expanded: bool = False,
classes: str = "w-full",
on_delete: RevisionAction | None = None,
) -> Any:
"""Render a collapsible row for the single optional source revision."""
header = "Revision | User-authored"
caption = _format_created_at(revision.date_created)
expansion = ui.expansion(value=initially_expanded, group="group").classes(
f"{classes} rounded-borders bg-blue-grey-10"
)
with expansion, ui.column().classes("w-full q-gutter-y-sm q-pa-sm"):
with expansion.add_slot("header"), ui.row().classes("w-full items-start justify-between q-gutter-md"):
with ui.column().classes("q-gutter-none"):
ui.label(header).classes("text-subtitle1 text-weight-medium")
ui.label(caption).classes("text-caption text-grey-5")
if on_delete is not None:
with ui.dialog() as delete_dialog, ui.card().classes("q-pa-md"):
ui.label("Delete this transcript revision?").classes("text-body1")
with ui.row().classes("w-full justify-end q-gutter-sm"):
ui.button("Cancel", on_click=lambda: delete_dialog.submit(False)).props("flat")
ui.button("Delete", on_click=lambda: delete_dialog.submit(True)).props(
'unelevated color="negative"'
)
async def delete_current_transcript() -> None:
delete_dialog.open()
confirmed = await delete_dialog
if not confirmed:
return
maybe_awaitable = on_delete(revision)
if isinstance(maybe_awaitable, Awaitable):
await maybe_awaitable
with ui.column(align_items="center").classes("self-center q-gutter-none"):
ui.button(icon="delete", on_click=delete_current_transcript).props(
'flat round dense color="negative"'
)
_metadata_row(label="Created", value=_format_created_at(revision.date_created))
if revision.text:
with ui.card().classes("w-full q-pa-sm"):
ui.markdown(revision.text)
return expansion
def _format_created_at(value: datetime) -> str:
"""Return a compact UTC-like timestamp for row captions."""
return value.strftime("%Y-%m-%d %H:%M:%S %Z")
def _metadata_row(*, label: str, value: str) -> None:
with ui.row().classes("w-md items-start justify-between q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-5 text-uppercase")
ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
src/transcription/ui/components/upload.py
-66
View File
@@ -1,66 +0,0 @@
"""Reusable upload widget for document submission."""
from __future__ import annotations
from collections.abc import Awaitable
from collections.abc import Callable
from nicegui import ui
from nicegui.binding import bindable_dataclass
from nicegui.events import UploadEventArguments
from transcription.errors import AppError
from transcription.services.documents import UploadJobResult
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.error_presenter import summarize_error
from transcription.worker import WorkerNotifier
type UploadSubmitter = Callable[[str, bytes], Awaitable[UploadJobResult]]
@bindable_dataclass
class UploadWidgetState:
"""Simple state container for upload feedback."""
loading: bool = False
message: str = ""
def render_upload_widget(*, submitter: UploadSubmitter, notifier: WorkerNotifier | None = None) -> None:
"""Render upload controls and common status/error handling."""
state = UploadWidgetState()
status_label = ui.label("Upload a document to start transcription.")
status_label.bind_text(state, "message")
async def on_upload(event: UploadEventArguments) -> None:
if state.loading:
ui.notify("Upload already in progress. Please wait.", type="warning")
return
state.loading = True
status_label.text = "Uploading..."
try:
payload = await event.file.read()
result = await submitter(event.file.name, payload)
job_id = result.job_id
state.message = f"Created job {job_id}" if job_id is not None else "Upload complete"
status_label.text = state.message
if notifier is not None:
notifier.notify()
ui.notify(state.message, type="positive")
except AppError as exc:
state.message = summarize_error(exc, operation="upload.submit")
status_label.text = f"Upload failed: {state.message}"
show_error(exc, title="Upload failed", operation="upload.submit")
except Exception as exc: # noqa: BLE001
state.message = summarize_error(exc, operation="upload.submit")
status_label.text = f"Upload failed: {state.message}"
show_error(exc, title="Upload failed", operation="upload.submit")
finally:
state.loading = False
ui.upload(
on_upload=on_upload,
auto_upload=True,
label="Select document file",
).props('accept=".jpg,.jpeg,.png,.tif,.tiff,.pdf"')
src/transcription/ui/pages/__init__.py
View File
src/transcription/ui/pages/jobs_page.py
-162
View File
@@ -1,162 +0,0 @@
"""Jobs list and detail page registration."""
from __future__ import annotations
from uuid import UUID
from fastapi import Request
from nicegui import ui
from transcription.app_state import resolve_session_factory
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Source
from transcription.services.jobs import JobService
from transcription.services.transcription import TranscriptionService
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.table.jobs import render_jobs_table
from ..components.document_panzoom import render_document_panzoom
from ..components.table.jobs import JobTableRow
from ..components.transcript import render_original_transcription_card
from ..components.transcript import render_revision_row
def register_page() -> None: # noqa: PLR0915
"""Register jobs list and detail routes."""
@ui.page("/jobs")
async def jobs_page(request: Request) -> None:
session_factory = resolve_session_factory(request.app.state)
jobs_service = JobService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
@ui.refreshable
async def render_table() -> None:
jobs = [
JobTableRow(
id=job.id,
status=job.status.value,
filename=job.filename,
retry_count=job.retry_count,
date_created=job.date_created.isoformat(),
date_updated=job.date_updated.isoformat(),
)
for job in await jobs_service.list_jobs()
]
render_jobs_table(jobs)
ui.button("Refresh", on_click=render_table.refresh, icon="refresh")
await render_table()
@ui.page("/jobs/{job_id}")
async def job_detail_page(job_id: str, request: Request) -> None: # noqa: PLR0915
session_factory = resolve_session_factory(request.app.state)
jobs_service = JobService(session_factory=session_factory)
transcription_service = TranscriptionService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
try:
parsed_job_id = UUID(job_id)
except ValueError:
ui.label("Invalid job id").classes("text-h6 text-negative")
return
try:
job = await jobs_service.read_job(job_id=parsed_job_id)
except ValueError:
ui.label("Job not found").classes("text-h6 text-negative")
return
source = _resolve_primary_source(job)
with ui.splitter(value=30).classes("w-full h-[calc(100vh-64px)]") as splitter:
with splitter.before, ui.column(align_items="stretch").classes("w-full h-full p-4 gap-3"):
if source is not None:
render_document_panzoom(source=source)
else:
ui.label("No source preview is available for this job.").classes("text-body2 text-grey-3")
with splitter.after, ui.column(align_items="stretch").classes("w-full h-full p-4 gap-3"):
with ui.row():
ui.button(icon="arrow_back", on_click=ui.navigate.back)
with ui.row().classes("w-full items-center justify-between"):
ui.label(f"{job.id}").classes("text-h6 text-weight-bold")
match job.status:
case JobStatus.TRANSCRIBED:
ui.chip(job.status.value.upper(), color="green", text_color="white").props("outline")
case _:
ui.label(f"{job.status.value}").classes("text-subtitle1 text-weight-medium")
render_original_transcription_card(job=job)
async def delete_revision_by_id(revision_id: UUID) -> None:
try:
revision = await transcription_service.read_revision(revision_id=revision_id)
await transcription_service.delete_revision(revision)
except Exception as exc: # noqa: BLE001
show_error(exc, title="Delete failed", operation="jobs.delete_revision")
return
ui.notify("Deleted revision", type="positive")
await render_revision_panel.refresh()
@ui.refreshable
async def render_revision_panel() -> None:
refreshed_job = await jobs_service.read_job(job_id=parsed_job_id)
refreshed_source = _resolve_primary_source(refreshed_job)
if refreshed_source is None:
ui.label("No source is available for revision editing.").classes("text-body2 text-grey-3")
return
current_revision = refreshed_source.revision
default_revision_text = (
current_revision.text if current_revision is not None else (refreshed_job.text or "")
)
ui.label("Revision Editor").classes("text-subtitle1 text-weight-medium")
editor = ui.textarea(label="Revision text", value=default_revision_text).props("autogrow outlined")
editor.classes("w-full")
async def save_revision() -> None:
candidate = (editor.value or "").strip()
if not candidate:
ui.notify("Revision text is required.", type="warning")
return
try:
await transcription_service.upsert_revision_for_source(
source_id=refreshed_source.id,
text=candidate,
)
except Exception as exc: # noqa: BLE001
show_error(exc, title="Save failed", operation="jobs.save_revision")
return
ui.notify("Revision saved", type="positive")
await render_revision_panel.refresh()
with ui.row().classes("w-full justify-end"):
ui.button(
"Create revision" if current_revision is None else "Update revision",
on_click=save_revision,
icon="save",
).props('unelevated color="primary"')
if current_revision is None:
ui.label("No revision exists for this source.").classes("text-body2 text-grey-3")
return
render_revision_row(
revision=current_revision,
initially_expanded=True,
on_delete=lambda _revision, rid=current_revision.id: delete_revision_by_id(rid),
)
await render_revision_panel()
def _resolve_primary_source(job: Job) -> Source | None:
if not job.sources:
return None
return job.sources[0]
src/transcription/ui/pages/upload_page.py
-33
View File
@@ -1,33 +0,0 @@
"""Upload page registration and handlers."""
from __future__ import annotations
from fastapi import Request
from nicegui import ui
from transcription.app_state import resolve_session_factory
from transcription.db import get_session
from transcription.services.store import create_upload_job
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.upload import render_upload_widget
from transcription.worker import resolve_worker_notifier
def register_page() -> None:
"""Register the upload page route."""
@ui.page("/upload", title="Upload Document")
def upload_page(request: Request) -> None:
render_navigation_header(current_path="/upload")
session_factory = resolve_session_factory(request.app.state)
async def submit_upload(filename: str, file_bytes: bytes):
async with get_session(session_factory=session_factory) as session:
return await create_upload_job(
filename=filename,
file_bytes=file_bytes,
session=session,
)
notify_worker = resolve_worker_notifier(request.app.state)
render_upload_widget(submitter=submit_upload, notifier=notify_worker)
src/transcription/ui/static/colors.css
-30
View File
@@ -1,30 +0,0 @@
:root {
/* Soft blue-night palette tokens */
--ctp-rosewater: #f2dde5;
--ctp-flamingo: #edcfd8;
--ctp-pink: #dcc7de;
--ctp-mauve: #a9bde5;
--ctp-red: #d98a9a;
--ctp-maroon: #d39aa5;
--ctp-peach: #d7af8c;
--ctp-yellow: #e2c083;
--ctp-green: #86c8ad;
--ctp-teal: #77bfbe;
--ctp-sky: #7ebdda;
--ctp-sapphire: #74aed0;
--ctp-blue: #92b5f5;
--ctp-lavender: #6f97e8;
--ctp-text: #d8e2f5;
--ctp-subtext1: #bfcae0;
--ctp-subtext0: #a9b6cf;
--ctp-overlay2: #95a3bf;
--ctp-overlay1: #7c8ca9;
--ctp-overlay0: #657490;
--ctp-surface2: #4d5f7c;
--ctp-surface1: #394a65;
--ctp-surface0: #2a3954;
--ctp-base: #1f2b42;
--ctp-mantle: #1a2538;
--ctp-crust: #141e30;
}
src/transcription/worker.py
-184
View File
@@ -1,184 +0,0 @@
"""Background worker for queued transcription jobs."""
from __future__ import annotations
import asyncio
import logging
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from contextlib import contextmanager
from contextlib import suppress
from typing import Protocol
from uuid import UUID
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.db import get_session
from transcription.errors import AppError
from transcription.errors import classify_unexpected_error
from .services import ServiceBundle
from .services.documents import DocumentService
from .services.jobs import JobService
from .services.transcription import TranscriptionService
from .services.workflows import advance_job
from .services.workflows import process_next_queued_job as process_next_queued_job_workflow
logger = logging.getLogger(__name__)
class WorkerNotifier(Protocol):
"""Abstraction for signaling the worker loop about new work."""
def notify(self) -> None:
"""Signal the worker loop that work may be available."""
class EventWorkerNotifier:
"""Worker notifier backed by an asyncio.Event."""
def __init__(self, wake_event: asyncio.Event):
self._wake_event = wake_event
def notify(self) -> None:
self._wake_event.set()
class NoopWorkerNotifier:
"""Fallback notifier used when worker signaling is unavailable."""
def notify(self) -> None:
return
def resolve_worker_notifier(state: object) -> WorkerNotifier:
"""Resolve notifier from app-like state objects with no-op fallback."""
notifier = getattr(state, "worker_notifier", None)
if isinstance(notifier, NoopWorkerNotifier):
return notifier
if notifier is None:
return NoopWorkerNotifier()
return notifier
@asynccontextmanager
async def worker_consumer_lifespan(
*,
session_factory: async_sessionmaker[AsyncSession] | None = None,
poll_interval_seconds: float = 1.0,
) -> AsyncGenerator[tuple[asyncio.Event, WorkerNotifier]]:
"""Start and stop the worker consumer loop for app lifespan."""
stop_event = asyncio.Event()
wake_event = asyncio.Event()
worker_notifier: WorkerNotifier = EventWorkerNotifier(wake_event)
worker_task = asyncio.create_task(
run_worker_loop(
session_factory=session_factory,
stop_event=stop_event,
wake_event=wake_event,
poll_interval_seconds=poll_interval_seconds,
)
)
worker_notifier.notify()
try:
yield stop_event, worker_notifier
finally:
stop_event.set()
worker_notifier.notify()
try:
await asyncio.wait_for(worker_task, timeout=2.0)
except TimeoutError:
worker_task.cancel()
with suppress(asyncio.CancelledError):
await worker_task
async def queue_consumer_loop(queue: asyncio.Queue[UUID], stop_event: asyncio.Event):
"""Main worker loop that consumes jobs from the queue and processes them.
The queue is for Job UUIDs, and the corresponding documents should already have been uploaded.
"""
service = JobService()
while not stop_event.is_set():
with handle_worker_exceptions():
async with _get_queue_item(queue) as job_id:
job = await service.read_job(job_id)
asyncio.create_task(advance_job(job=job, services=ServiceBundle()))
@contextmanager
def handle_worker_exceptions(operation: str = "worker.loop"):
"""Context manager to log and suppress exceptions in the worker loop."""
try:
yield
except Exception as exc:
error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation=operation)
logger.exception(
"Worker loop exception error_id=%s category=%s",
error.error_id,
error.category.value,
)
@asynccontextmanager
async def _get_queue_item(queue: asyncio.Queue[UUID]) -> AsyncGenerator[UUID]:
"""Context manager to enqueue a job and ensure it is marked done."""
yield await queue.get()
queue.task_done()
async def run_worker_loop(
*,
session_factory: async_sessionmaker[AsyncSession] | None = None,
stop_event: asyncio.Event | None = None,
wake_event: asyncio.Event | None = None,
poll_interval_seconds: float = 1.0,
) -> None:
"""Run worker loop until stop_event is set.
If wake_event is provided, signal activity wakes the loop immediately while
timeout-based wakeups preserve current polling behavior.
"""
while True:
if stop_event is not None and stop_event.is_set():
logger.info("Worker stop event received")
return
if wake_event is not None:
with suppress(TimeoutError):
await asyncio.wait_for(wake_event.wait(), timeout=poll_interval_seconds)
wake_event.clear()
processed_any = False
while await process_next_queued_job(session_factory=session_factory):
processed_any = True
if wake_event is None and not processed_any:
await asyncio.sleep(poll_interval_seconds)
async def process_next_queued_job(
*,
session: AsyncSession | None = None,
session_factory: async_sessionmaker[AsyncSession] | None = None,
) -> bool:
"""Process the next queued job and persist terminal outcome.
Returns True when a job was processed, False when no queued job exists.
"""
if session_factory is None:
services = ServiceBundle()
else:
services = ServiceBundle(
documents=DocumentService(session_factory=session_factory),
jobs=JobService(session_factory=session_factory),
transcriptions=TranscriptionService(session_factory=session_factory),
)
if session is None:
async with get_session(session_factory=session_factory) as local_session:
return await process_next_queued_job_workflow(services=services, session=local_session)
return await process_next_queued_job_workflow(services=services, session=session)
tests/__init__.py
View File
tests/api/test_error_responses.py
-57
View File
@@ -1,57 +0,0 @@
"""Tests for API error response envelope handlers."""
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from transcription.api.errors import register_error_handlers
from transcription.errors import AppError
from transcription.errors import ErrorCategory
@pytest.mark.integration
class TestApiErrorResponses:
"""Verify API-level error serialization and status mapping."""
def test_app_error_returns_structured_envelope(self):
"""AppError maps to policy envelope fields and status code."""
app = FastAPI()
register_error_handlers(app)
@app.get("/boom")
def boom() -> dict[str, str]:
raise AppError(
"Bad upload payload",
category=ErrorCategory.VALIDATION,
suggestion="Upload a non-empty file",
error_id="abc12345",
)
client = TestClient(app)
response = client.get("/boom")
assert response.status_code == 400
payload = response.json()
assert payload["error_id"] == "abc12345"
assert payload["category"] == "validation_error"
assert payload["message"] == "Bad upload payload"
assert payload["suggestion"] == "Upload a non-empty file"
assert "timestamp" in payload
def test_unexpected_error_returns_internal_unexpected_envelope(self):
"""Unexpected exceptions map to internal_unexpected_error with 500."""
app = FastAPI()
register_error_handlers(app)
@app.get("/explode")
def explode() -> dict[str, str]:
raise RuntimeError("unexpected failure")
client = TestClient(app, raise_server_exceptions=False)
response = client.get("/explode")
assert response.status_code == 500
payload = response.json()
assert payload["category"] == "internal_unexpected_error"
assert "error_id" in payload
assert payload["suggestion"]
tests/api/test_health.py
-21
View File
@@ -1,21 +0,0 @@
"""Tests for transcription.api.health."""
from fastapi import FastAPI
from fastapi.testclient import TestClient
from transcription.api.health import router
class TestHealthEndpoint:
"""Verify /healthz endpoint behavior."""
def test_healthz_returns_ok_status(self):
"""GET /healthz returns a healthy status payload."""
app = FastAPI()
app.include_router(router)
client = TestClient(app)
response = client.get("/healthz")
assert response.status_code == 200
assert response.json() == {"status": "ok"}
tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt
-42
View File
@@ -1,42 +0,0 @@
source: Book Two - page 02.jpg
provider: openrouter
model: google/gemini-2.5-flash
---
BY WAY OF INTRODUCTION:-
These few paragraphs of introduction may help you read BOOK 2 which covers a
wider range than did BOOK 1 (Pioneer Days).
BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 con-
sisted largely of first generation family history. BOOK 2 throws more light on
the second generation. Sidney promises a BOOK 3 and that may begin to do justice
to the third generation. We suggest that Sidney get the help of Louis Shinn
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq
Plains. Louis has the gift of seeing, recalling and telling. One sentence in
his chapter gives a great tribute to the Doumeccqers--so [sic] far as he knows no one
on the Doumecq Plains went on relief during the depression. That in a nutshell
shows the sturdy character of the residents of the Doumecq Plains.
We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of
John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see
by the Table of Contents that the first four chapters have been given to those
trips. Those chapters are worth reading and re-reading. Mr. Cochran has eyes
to see and a pen to tell. We think the people in Tennessee will read with
great pleasure the comments he makes on conditions today.
Some who get this book will consider the group picture the best thing in the
book. It took a lot of preliminary photographing to reduce some pictures, enlarge
others and bring out the tin types. We wish that instead of 44 faces we could
have given 88. Do not blame Ethel Cochran-Shinn for the selection. She furnished
enough pictures but we had to take only part of them. We think there are great
possibilities in reproducing old pictures. We wish we had a Pickard group. Some
Pickard descendant may wish to make a collection.
We are much impressed with the future possibilities of getting a complete geneol-
ogy of the Pickard family. Mr. Cochran has a fine chapter on the Pickards but
to date we have not had the pleasure of finding all of the family dates. We had
intended to give more family data in this book but it takes time to get the
correct dates. Often times it requires trips to cemeteries to get dates on the
tombstones. Winter is no time to collect dates on tombstones.
-2-
tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt
-114
View File
@@ -1,114 +0,0 @@
source: Omie Writes Home.pdf
provider: openrouter
model: google/gemini-2.5-flash
---
JOHN E. COCHRAN
FAMILY ASSOCIATION
Family Only
Home | Sibling's Stories | 1st Cousins | Ancestor's Stories | Reunion History | Next Reunion | JECMEF
OMIE WRITES HOME
Ed. Note: The following letter was written by Omie Cochran in Nome, Alaska and sent to her
sister, Ethel Shinn, in Canfield, Idaho in 1923. It has been stored away these 63 years in the
original envelope with its 2 cent stamp. The letter has a number of references to the Shinn
children. Peter was Louis; Polly was Edith; and the 'little black rascal' referred to Maurice.
Miss Saville was the nurse at the Nome Hospital that was mentioned in the article by Inez in
the family newsletter two years ago.
Nome Alaska August 26, 1923
My Dear Ethel et al.
I don't know when I did write or when you did
but I am going to write now however and never
the less. But I wish I could talk (I can yet but I
mean to tell you all) instead and see ole Unc Pete
and Polly sit up and listen and that little black
rascal of yours would fairly sparkle with
listening. Can't I see him listening now to all the
yarns we told last summer?
You see, we-Miss Saville and I, took a trip north
on the Buford and it was very interesting. We
went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain
of our craft and some other mighty hunters went out first in kayaks and later in row boats and
shot seven walrus. When they also took a movie man and camera, so you will likely see all
this in the movies before I get to tell you. They came back on board and the ship went up
along the icebergs and the walrus were 'heisted' on board by the use of the crane which loads
tons of freight and the beasts were so huge that they made the pulleys just creak. They were
over 12 feet long, as big around as three cows, had no feet but toenails on their flippers or
flappers, no head but their body just suddenly ended with a hole for a mouth and big bristles
all around it similar to a currycomb in coarseness; no ears but huge tusks of ivory. They are
the most repulsive looking animals imaginable and tho I have always read about them I never
expect such disagreeable looking creatures. They had a rough brown hairy skin and some of
them looked warty. They must have weighed two ton at least. Ere we got them back to Nome
to the natives they were getting extremely odiferous—in fact, you could scarcely stay on the
ship with any degree of comfort unless you had per chance lost your sense of smell.
Then we went north to a few minutes beyond the 70th degree of latitude and thot for awhile
we would go to Wrangell Island where some men from Steffonsons ship were supposed to be
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz in
Siberia. These Eskimo were very primitive. One white squaw man lived there and had for 23
years. He was a Swede--who else could. Their houses were circular and built up with dirt 2 or
3 feet and then skins were stretched over it and weighted down with rocks. Inside, the room
was partitioned off at the sides with skins for sleeping quarters. In the main part they had the
fire on the ground and the fish drying on lines and the skins hanging around and the dogs and
babies and children. They wore skin clothes entirely. The women's were made like bloomers
and were heavily padded for warmth. They wore high mukluks and really looked very
comfortable. The babies were in fur skins with the fur inside and they looked like little Teddy
bears with faces. I guess they had never seen white women, not so many at one time anyway.
We went ashore in 2 life boats and a launch pulling them. On the launch was a six piece band
playing and the rear of the last life boat was the movie man. 'Twas very thrilling.
The other place we stopped was at Whalen, a trading post in Siberia. There these 'towerists'
went wild. They rushed helter-skelter, hither and thither, here and there, trying to find
something to buy. Prices raised right before your eyes. One would but something for $1.00
and the next might have to pay $2.00, $4.00 or $10.00. That made no difference. They had to
have it. One man I was sort of taking care of, tho he had his son along for the purpose,
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billikens, 6 or 8 ivory and silver rings, one
fishing line, hooks, floats, etc. and two bird slings. The slings have rocks at the end and the
little natives throw them at the flocks of geese and ducks which fly close over the village and
the slings entangle their wings and legs, sometimes more than one, and they can't fly. They
come down and the natives capture them. There was more junk brot aboard than baggage, I
do believe. And they say that at the first stop it was worse than here. The red flag was flying
over Whalen and the Russian soldiers were there—a few, one or two or three, I forget the
number.
We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out
until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let
us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is
all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he
didn't and Polly would in my lap and I don't know much about the youngest one of yours so
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
I expect there were 150 passengers on board and almost or more of the crew and helpers. We
had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
I have had jobs nearly all summer but not very much in them. Next week, September 4,
school opens. I wish they would wait for a week but you know these school men. Wouldn't
make any special difference I suppose for I would just fritter away the time but still one likes
to postpone the inevitable.
I just now stopped and re-read your letter and it sure was a bird. I don't especially blame Ray
for reading over your shoulder. It would seem, then that you have bright children. Maybe
they do know something about Geography. But it is ridiculous to speak of Louis finishing the
eighth grade. Why you and I were grown children when we finished and he is only a baby. I
am rather afraid he doesn't know much. I quite remember your little timid Maurice and how
he shifted his affection from his Aunt Om and yelled and howled and screamed steadily for a
week while his mother went to S.S. Ask him is he recalls this little interview. Do you suppose
he does?
Ever hear from Zen? or Inez? They don't seem to be very writing inclined tho Zen has done
well this year. Even sent me a telegram a few weeks ago. Well, if anything else ever happens,
I'll write again. Don't suppose it ever will, tho.
Lots of love to all,
Ome
Reprinted from Cochran Chronicles, Volume 9, Number 1, November 1986
© JECFA 1986
Up
jecochranclan.org ~ Contact webmaster
tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt
-30
View File
@@ -1,30 +0,0 @@
source: Rod Moser Letter - p1.jpg
provider: openrouter
model: google/gemini-2.5-flash
---
JOHN ISBILL
R. T. MOSER
ISBILL & MOSER
DEALERS IN
GENERAL MERCHANDISE
Vonore, Tenn. Jany 27- 1913
Dear Much Aunt Adeline
Was at home a
few nights ago & saw a
letter from your folks. So
I decided to write you
a few lines myself ok
I am contemplateing [sic] a
trip out west next summer
& [inserted: I] want some of them to go
when I am [inserted: a] them.
Am getting
up in years & unmarried
so you see the object of
my trip, is to get a wife
& if there is any old maids
or widows out there, I
want you to kiss them
at my [hand?] me at there
as soon as I get there.
tests/conftest.py
-71
View File
@@ -1,71 +0,0 @@
"""Shared test fixtures.
Every test gets a fresh in-memory SQLite database so tests are
isolated, fast, and leave no artifacts on disk.
"""
import pytest
import pytest_asyncio
from sqlmodel import Session
from sqlmodel import SQLModel
from sqlmodel import create_engine
from sqlmodel.pool import StaticPool
from transcription.config import Settings
from transcription.config import get_settings
from transcription.db.operations import create_all
from transcription.db.runtime import dispose_database_runtime
from transcription.db.runtime import get_engine
from transcription.db.runtime import get_session
from transcription.db.runtime import get_session_factory
from transcription.services.documents import DocumentService
from transcription.services.jobs import JobService
@pytest.fixture
def session():
"""Provide a clean synchronous database session for sync tests."""
engine = create_engine(
"sqlite://",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
SQLModel.metadata.create_all(engine)
with Session(engine) as sync_session:
yield sync_session
@pytest_asyncio.fixture
async def default_settings():
"""Provide default settings for tests."""
settings = get_settings(database_url="sqlite:///:memory:")
await create_all(engine=get_engine(settings=settings))
return settings
@pytest_asyncio.fixture
async def async_session(default_settings: Settings):
"""Provide a clean asynchronous database session for async tests."""
async with get_session(settings=default_settings) as async_session:
yield async_session
await dispose_database_runtime()
@pytest.fixture
def default_session_factory(default_settings: Settings):
"""Provide a base fixture for tests that require database access."""
session_factory = get_session_factory(settings=default_settings)
return session_factory
@pytest.fixture
def job_service(default_session_factory) -> JobService:
"""Provide a JobService instance for testing."""
return JobService(session_factory=default_session_factory)
@pytest.fixture
def document_service(default_session_factory) -> DocumentService:
"""Provide a DocumentService instance for testing."""
return DocumentService(session_factory=default_session_factory)
tests/fixtures/images/invalid/empty.bin
Vendored
View File
tests/fixtures/images/invalid/wrong_extension.txt
Vendored
-1
View File
@@ -1 +0,0 @@
not an image fixture
tests/fixtures/images/real/Book Two - page 02.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

tests/fixtures/images/real/Omie Writes Home.pdf
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/images/real/Rod Moser Letter - p1.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1010 KiB

tests/fixtures/images/valid/single_page_pdf.pdf
Vendored
-2
View File
@@ -1,2 +0,0 @@
%PDF-1.4
%fixture
tests/fixtures/images/valid/small_jpg.jpg
Vendored
-1
View File
@@ -1 +0,0 @@
˙Ř˙ŕfixture
Side by Side

Before

Width:  |  Height:  |  Size: 11 B

tests/fixtures/images/valid/small_png.png
Vendored
-3
View File
@@ -1,3 +0,0 @@
‰PNG

fixture
Side by Side

Before

Width:  |  Height:  |  Size: 15 B

tests/fixtures/images/valid/small_tiff.tiff
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/prompts/invalid/empty_prompt.md
Vendored
View File

Some files were not shown because too many files have changed in this diff Show More