generated from john/python-template
216 lines
13 KiB
Markdown
216 lines
13 KiB
Markdown
---
|
|
description: Follow these guidelines when editing the services
|
|
applyTo: 'src/transcription/services/*.py'
|
|
---
|
|
|
|
# Services
|
|
|
|
## Structure
|
|
|
|
- Project core data models are defined in [models](../../src/transcription/db/models.py)
|
|
- One service class per **aggregate**, not per table. An aggregate is a root model plus
|
|
the models that have no independent lifecycle of their own. `DocumentType` has no
|
|
meaning without `Document`, so it belongs to `DocumentService`; it does not get its
|
|
own service. Splitting per table produces services that must reach across each other
|
|
for every real operation, which is what line 13 forbids.
|
|
- Only services interact with the database, and only through async methods.
|
|
- **A service module must not import another service module.** This is enforced by
|
|
[test_service_boundaries](../../tests/test_service_boundaries.py). Shared types go in a
|
|
neutral module that defines no service class (see [errors](../../src/transcription/services/errors.py)).
|
|
- Not every module in this package is a service. Modules fall into three kinds:
|
|
- **Aggregate services** own models and define a `*Service` class: `documents.py`, `sources.py`,
|
|
`jobs.py`, `people.py`, `photos.py`, `maintenance.py`, and `evidence.py` (read/projection only,
|
|
owns nothing).
|
|
- **Orchestration modules** define no service class and compose writes across aggregates:
|
|
`store.py`, `workflows.py`. They are the sanctioned place to create or delete rows owned by more
|
|
than one service — see [Service Composition](#service-composition).
|
|
- **Shared infrastructure and free-function helpers** are exempt from the service rules below:
|
|
`base.py` (`ServiceBase`), `registry.py` (`RegistryService`, a generic base for lookup tables —
|
|
not an aggregate owner itself), `unit_of_work.py`, `errors.py`, `normalization.py`, `prompts.py`,
|
|
`quality.py`, `media_storage.py`, `source_media.py`. `__init__.py` exposes `ServiceBundle`.
|
|
- Cross-cutting error behavior must follow
|
|
[error-handling instructions](./error-handling.instructions.md).
|
|
|
|
## Model Ownership
|
|
|
|
Every model has exactly one owning service. The owner defines that model's invariants and
|
|
is the only service that may **create or delete** its rows.
|
|
|
|
| Model | Owner |
|
|
| --- | --- |
|
|
| `Document`, `DocumentType`, `DocumentTag` | `DocumentService` |
|
|
| `Source`, `JobSource` | `SourceService` |
|
|
| `Job` | `JobService` |
|
|
| `Person`, `PersonRole`, `DocumentPerson`, `PersonTag` | `PeopleService` |
|
|
| `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, `GenealogyCitation` | `MaintenanceService` |
|
|
| `Photo` | `PhotosService` |
|
|
| `MaintenanceRun` | `MaintenanceService` |
|
|
| `ExecutionAttempt` | `SourceService` |
|
|
| `Tag` | shared — see below |
|
|
|
|
Keep this table complete: every table in `src/transcription/db/models.py` appears exactly once,
|
|
except `Tag`. When you add a model, add its owner here in the same change.
|
|
|
|
### `Tag` is deliberately shared
|
|
|
|
`Tag` is one table reached through two `RegistryService[Tag]` facades that differ only in the
|
|
reference model they count usage through: `TagRegistry` (`documents.py`, via `DocumentTag`) and
|
|
`PersonTagRegistry` (`people.py`, via `PersonTag`). Both create and delete `Tag` rows through the
|
|
generic registry. This is the single sanctioned exception to one-owner-per-model — do not "fix" it by
|
|
assigning `Tag` to one service, because the other facade would then be creating rows it does not own.
|
|
Any change to `Tag` semantics, labels, or normalization must be validated against **both** facades
|
|
and the junction table each one counts.
|
|
|
|
`DocumentTag` and `PersonTag` follow the junction rule below: each is created and deleted only by the
|
|
service on its own side.
|
|
|
|
### Registries
|
|
|
|
`DocumentTypeRegistry`, `TagRegistry`, `PersonRoleRegistry`, and `PersonTagRegistry` are
|
|
`RegistryService` subclasses, not independent services. A registry belongs to the aggregate service
|
|
whose module declares it and shares that service's ownership. Registry CRUD uses `<operation>_entry`
|
|
naming (see [CRUD Methods](#crud-methods)).
|
|
|
|
### Junction tables
|
|
|
|
A junction table is owned by the service that **creates and deletes its rows** — its
|
|
lifecycle owner. The service on the other side may read through the junction (via
|
|
`selectinload`) but must not create rows in it.
|
|
|
|
- `document_person` -> `PeopleService`. Every write is there; `DocumentService` only
|
|
eager-loads through it.
|
|
- `job_source` -> `SourceService`, which creates the row, records each page's outcome,
|
|
and deletes it.
|
|
|
|
Two consequences follow, and both are deliberate:
|
|
|
|
- **Cascade deletion is not a violation.** A service deleting the aggregate root it owns
|
|
may delete rows referencing that root which cannot outlive it
|
|
(`JobService.delete_job_with_guardrails` deletes the job's `job_source` rows).
|
|
- **Evidence deletion is an explicit workflow, not a runtime path.** `JobService.delete_job_and_evidence`
|
|
deletes `ExecutionAttempt` rows owned by `SourceService`. That is sanctioned because it is the
|
|
named retention workflow that `delete_job_with_guardrails` refuses to perform implicitly — that
|
|
method *blocks* deletion when attempts exist. Append-only means runtime code never rewrites or
|
|
removes history to represent a new outcome; it does not forbid a deliberate, operator-invoked
|
|
retention operation. Do not add a second path that deletes attempts.
|
|
- **Ownership governs creation and deletion, not every state transition.** `job_source` is
|
|
both a link and the transcription work queue. `JobService.cancel_job` and
|
|
`resubmit_failed_sources` transition `job_source.status` across a whole job, because that
|
|
transition is a Job lifecycle event, not a per-page outcome. They create and delete
|
|
nothing.
|
|
|
|
`EvidenceService` is read-focused and projection-focused. It may coordinate selection
|
|
flows, but append-only attempt creation remains in `SourceService` write paths.
|
|
|
|
If a new operation cannot be expressed within one owner, it belongs in an orchestration
|
|
module, not in a cross-service import.
|
|
|
|
## Error Handling
|
|
|
|
- Errors used by a single service are defined at the top of that module and inherit from `AppError`.
|
|
- Errors shared by more than one service go in [errors](../../src/transcription/services/errors.py),
|
|
which defines no service class and is therefore importable by any of them.
|
|
- Use a context manager for large `try/except` blocks, like `handle_transcription_errors` in
|
|
[sources](../../src/transcription/services/sources.py).
|
|
- Category mapping, retry behavior, and translation boundaries are defined in
|
|
[error-handling instructions](./error-handling.instructions.md).
|
|
- Service-edge exception translation must be deterministic: map to canonical categories and preserve clear provider->service->API/UI boundaries.
|
|
|
|
## Checklist
|
|
|
|
- [ ] Uses `ServiceBase` for common logic
|
|
- [ ] Session kwarg for `AsyncSession` to pass a session object into each method
|
|
- [ ] Services use `self._session_scope` in their methods to pass the session through
|
|
- Multiple operations on the same object(s) require sharing a session between all the methods used
|
|
- [ ] Every model the module touches is either owned by it or reached read-only
|
|
- [ ] Evidence writes preserve append-only semantics
|
|
|
|
## CRUD Methods
|
|
|
|
- Name format `<operation>_<model>`, for example `create_document` or `update_job`.
|
|
- Where a service exposes create/read/update/delete for its root model, define them at the
|
|
top of the class in that order, before derived reads and workflow helpers.
|
|
- Not every aggregate needs all four. `ExecutionAttempt` is append-only evidence written by
|
|
`SourceService` workflow-facing methods, so `EvidenceService` deliberately exposes reads and
|
|
no create or delete.
|
|
Do not add unused CRUD methods to satisfy symmetry.
|
|
- `RegistryService` is generic across small lookup models and uses `<operation>_entry`
|
|
naming instead.
|
|
|
|
## Transaction Finalization
|
|
|
|
When a service method accepts an optional `session` kwarg, write methods must use `self._finalize` to finalize the transaction properly according to whether or not they are sharing a session.
|
|
|
|
- If `session` is `None`: the method owns the transaction and should `commit()`.
|
|
- If `session` is provided: the method must **not** commit; it should `flush()` so IDs and FK values are available to the caller's transaction.
|
|
- Use `refresh()` on returned ORM objects when the caller needs DB-populated values (defaults, triggers, merged state).
|
|
|
|
## Workflow Transaction Boundaries
|
|
|
|
For multi-step job lifecycles, orchestration functions must use explicit transaction phases.
|
|
|
|
- **Transaction A (claim):** transition `JobStatus.QUEUED -> JobStatus.PROCESSING` and commit immediately.
|
|
- Perform provider/network work **outside** database transactions.
|
|
- **Transaction B (terminal success):** write transcript content and set `JobStatus.TRANSCRIBED` in the same shared-session commit.
|
|
- **Transaction B (terminal failure):** write transcript error detail and set `JobStatus.FAILED` in the same shared-session commit.
|
|
- **Transaction C (retry path):** write transcript error detail, increment retry count, and set `JobStatus.QUEUED` in one shared-session commit.
|
|
|
|
Atomicity rules:
|
|
|
|
- Never commit transcript updates separately from the paired terminal/retry job status change.
|
|
- Terminal state (`TRANSCRIBED` or `FAILED`) and transcript row changes must succeed or roll back together.
|
|
- Retry persistence (`QUEUED` + retry increment + error detail) must succeed or roll back together.
|
|
|
|
### Multi-page batches
|
|
|
|
These two requirements are in tension for multi-page jobs: each page should be durable as
|
|
soon as its provider call returns, but the last page must commit together with the terminal
|
|
status. `process_queued_job` resolves it by committing every page except the last one
|
|
individually, then deferring the final page's write into `_finalize_batch_outcome` so it
|
|
shares the terminal transaction.
|
|
|
|
Both paths are shielded against cancellation, so the final page is no less durable than the
|
|
pages before it. Enforced by `tests/integration/test_pipeline_atomicity.py`; per-page
|
|
durability is separately enforced by
|
|
`tests/services/test_workflows_reliability.py::TestWorkflowReliability::test_transcribed_page_is_committed_before_next_provider_call_finishes`.
|
|
|
|
### Stale-reclaim safety
|
|
|
|
- `WORKER_STALE_JOB_SECONDS` must remain **greater than** `WORKER_PROVIDER_TIMEOUT_SECONDS`; stale
|
|
recovery must not be able to fire before one provider call can legitimately finish.
|
|
- Long-running multi-page orchestration must refresh job liveness explicitly between intermediate
|
|
page commits. Do not rely on incidental row updates or provider metadata writes to keep
|
|
`Job.date_updated` fresh.
|
|
- Enforced by `tests/test_config.py` and
|
|
`tests/services/test_workflows_reliability.py::TestWorkflowReliability::test_intermediate_page_commit_advances_job_liveness_timestamp`.
|
|
|
|
## Contract Alignment
|
|
|
|
- Treat `docs/` as the active architecture and requirements baseline.
|
|
- Legacy revision trees are out of scope for active implementation decisions and must not be referenced as authoritative service guidance.
|
|
- Treat `src/transcription/db/models.py` as runtime schema ground truth and `docs/schema.md` as the field-accurate contract mirror.
|
|
- `Job.status` success path is `TRANSCRIBED`.
|
|
- `JobSource.status` is queue/projection state only (`PENDING`, `TRANSCRIBED`, `FAILED`, `CANCELLED`).
|
|
- Source ingest may normalize media before persistence; persisted bytes/hash are canonical for processing and provenance.
|
|
- `ExecutionAttempt` is append-only evidence history; do not mutate historical attempt rows in runtime code.
|
|
- `Source.raw_transcription` is a projection, not authoritative history.
|
|
- Service/UI read paths that touch relationships must be eager-loaded for `lazy="raise"` compatibility.
|
|
- If model fields, enums, constraints, indexes, or relationship-loading semantics change, update `docs/schema.md` in the same change.
|
|
- If `Settings` fields or defaults change in `src/transcription/config.py`, update `.env.production.example` in the same change so keys/defaults remain synchronized and no stale settings remain documented.
|
|
|
|
## Schema Drift and Legacy Compatibility Policy
|
|
|
|
- Prefer schema migration over startup reconciliation or runtime compatibility paths in service writes.
|
|
- Do not add legacy read/write compatibility code in service workflows by default.
|
|
- If drift is discovered and a migration decision is ambiguous (for example, one-way destructive DDL, uncertain data retention impact, or unknown deployment sequence), pause and ask the user to choose migration vs compatibility before coding.
|
|
- If a temporary compatibility path is explicitly approved, document an expiration/removal plan in the same change.
|
|
|
|
# Service Composition
|
|
|
|
A service method may read across models it does not own, using eager loads from its own
|
|
aggregate root. What it may not do is import another service.
|
|
|
|
Operations that must **write** models owned by more than one service are composed in an orchestration module
|
|
([store](../../src/transcription/services/store.py),
|
|
[workflows](../../src/transcription/services/workflows.py)).
|