Update V1 & V2 core documents and reorganize docs folder

docs/architecture_v2.md

@@ -0,0 +1,136 @@
+# System Architecture (Version 2)
+
+This document describes the V2 production architecture of the personal historical-document transcription system.
+
+## Architecture Objectives
+
+* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata.
+* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
+* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
+* Migrate persistence to PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` document storage.
+* Standardize all data validation, API parsing, and database models on **Pydantic V2**.
+* Support rich historical attribution (multi-author and multi-recipient relationships).
+
+## Runtime Topology
+
+The V2 runtime operates as an asynchronous Python application:
+
+* FastAPI + NiceGUI web application process.
+* In-process `asyncio` background task orchestrator for parallel API execution.
+* Relational persistence via PostgreSQL (using `asyncpg` or `psycopg3`).
+* Pydantic V2 validation layer wrapping API payloads and PostgreSQL `JSONB` schemas.
+
+```mermaid
+flowchart LR
+U[Browser User] --> A[FastAPI + NiceGUI App]
+A --> W[Asyncio Worker Engine]
+A --> DB[(PostgreSQL Database)]
+W --> P[Vision Provider APIs\nOpenAI / Claude]
+W --> DB
+```
+
+## Lifecycle Ownership
+
+Application lifespan owns runtime setup/teardown:
+
+* Initialize environment logging and Pydantic configuration.
+* Manage asynchronous PostgreSQL connection pools (`asyncpg` / `psycopg3`).
+* Execute database migrations and index initialization.
+* Recover stale processing jobs on startup.
+* Manage graceful shutdown of active `asyncio` worker pools.
+
+## Layered Module Structure
+
+### Interface Layer
+
+* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
+* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
+
+### Application & Async Worker Layer
+
+* `src/transcription/services/workflows.py`
+* `src/transcription/worker.py`
+
+Responsibilities:
+
+* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
+* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
+* Pydantic schema parsing (`PageAIMetadata`) and validation prior to database storage.
+
+### Domain & Service Layer
+
+* `src/transcription/models/*.py` (Pydantic V2 schemas and entity definitions)
+* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
+
+### Infrastructure Layer
+
+* `src/transcription/db/**` (PostgreSQL connection pooling and raw parameterized SQL execution)
+* `src/transcription/providers/**` (OpenAI & Anthropic Vision SDK adapters)
+
+## Processing Workflow
+
+1. User uploads a folder or batch of images for a `Document`.
+2. System creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
+3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
+4. Each task calls Vision API for a **single** `Source` image.
+5. On task completion:
+* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, `ai_metadata` (bounding boxes/confidence), and `raw_api_response`.
+* Caches active text to `Source.raw_transcription`.
+
+
+6. On page failure:
+* Writes `JobSource` record with `status='failed'` and `error_detail`.
+
+
+7. Once all page tasks resolve:
+* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
+
+
+
+## Domain Ownership & Invariants
+
+* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
+* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
+* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
+* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
+
+## Data Model Summary
+
+* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
+* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
+* `Job` has many `JobSource` execution records.
+
+## Test Strategy
+
+* Unit tests for Pydantic V2 schemas, custom validators, and JSONB serialization.
+* Integration tests for async PostgreSQL connection handling and parameterized queries.
+* Async workflow tests using mock AI providers to verify `partial_success` and retry logic.
+* UI integration tests for multi-page rendering and person management.
+
+---
+
+## Technology References
+
+- [FastAPI documentation](https://fastapi.tiangolo.com/)
+- [NiceGUI documentation](https://nicegui.io/documentation)
+- [PostgreSQL documentation](https://www.postgresql.org/docs/)
+- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
+- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
+ 
+
+## Related Local References
+
+- [System Overview](index_v1.md)
+- [System Design Intent](intent.md)
+- [Transcription Methodology](transcription_methodology.md)
+- System Architecture (this document)
+- [System Requirements](requirements_v1.md)
+- [Data model](schema_v1.md)
+- [Error Handling Policy](error_handling_v1.md)
+- [Implementation Plan](implementation_plan_v1.md)
+
+
+
+
+