generated from john/python-template
@@ -42,6 +42,7 @@ is the only service that may **create or delete** its rows.
|
|||||||
| `Source`, `JobSource` | `SourceService` |
|
| `Source`, `JobSource` | `SourceService` |
|
||||||
| `Job` | `JobService` |
|
| `Job` | `JobService` |
|
||||||
| `Person`, `PersonRole`, `DocumentPerson`, `PersonTag` | `PeopleService` |
|
| `Person`, `PersonRole`, `DocumentPerson`, `PersonTag` | `PeopleService` |
|
||||||
|
| `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, `GenealogyCitation` | `MaintenanceService` |
|
||||||
| `Photo` | `PhotosService` |
|
| `Photo` | `PhotosService` |
|
||||||
| `MaintenanceRun` | `MaintenanceService` |
|
| `MaintenanceRun` | `MaintenanceService` |
|
||||||
| `ExecutionAttempt` | `SourceService` |
|
| `ExecutionAttempt` | `SourceService` |
|
||||||
|
|||||||
@@ -121,6 +121,8 @@ Responsibilities:
|
|||||||
- `ExecutionAttempt` is append-only evidence for each provider call.
|
- `ExecutionAttempt` is append-only evidence for each provider call.
|
||||||
- `Photo` is person imagery owned by `PhotosService`.
|
- `Photo` is person imagery owned by `PhotosService`.
|
||||||
- `MaintenanceRun` is one queued or executed operational maintenance run.
|
- `MaintenanceRun` is one queued or executed operational maintenance run.
|
||||||
|
- `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, and `GenealogyCitation` store
|
||||||
|
imported GEDCOM genealogy data and citation provenance.
|
||||||
- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
|
- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
|
||||||
- `Tag` is a shared registry reached through both document and person tagging, linked by
|
- `Tag` is a shared registry reached through both document and person tagging, linked by
|
||||||
`DocumentTag` and `PersonTag`.
|
`DocumentTag` and `PersonTag`.
|
||||||
@@ -147,7 +149,7 @@ Responsibilities:
|
|||||||
- **MaintenanceRun statuses:** `queued`, `processing`, `succeeded`, `failed`
|
- **MaintenanceRun statuses:** `queued`, `processing`, `succeeded`, `failed`
|
||||||
- Maintenance uses `succeeded` rather than `transcribed`; the transcription vocabulary does not
|
- Maintenance uses `succeeded` rather than `transcribed`; the transcription vocabulary does not
|
||||||
apply to operational runs.
|
apply to operational runs.
|
||||||
- **Maintenance job types:** `backup`, `storage_reconciliation`
|
- **Maintenance job types:** `backup`, `storage_reconciliation`, `gedcom_import`
|
||||||
|
|
||||||
## Maintenance Execution
|
## Maintenance Execution
|
||||||
|
|
||||||
@@ -157,7 +159,8 @@ lifetime and is recorded:
|
|||||||
1. Settings enqueues a `MaintenanceRun` with `status=queued` and a `triggered_by` marker.
|
1. Settings enqueues a `MaintenanceRun` with `status=queued` and a `triggered_by` marker.
|
||||||
2. The worker claims the oldest queued run with a conditional update, moving it to `processing`.
|
2. The worker claims the oldest queued run with a conditional update, moving it to `processing`.
|
||||||
3. `backup` runs the deploy backup script; `storage_reconciliation` compares stored media against
|
3. `backup` runs the deploy backup script; `storage_reconciliation` compares stored media against
|
||||||
`Document`/`Source` records.
|
`Document`/`Source` records; `gedcom_import` parses the latest uploaded `.ged` file and upserts
|
||||||
|
genealogy records.
|
||||||
4. The run finalizes to `succeeded` or `failed` with summary, timing, log path, and `error_detail`.
|
4. The run finalizes to `succeeded` or `failed` with summary, timing, log path, and `error_detail`.
|
||||||
|
|
||||||
`MaintenanceRun` records operational history and is not evidence in the `ExecutionAttempt` sense;
|
`MaintenanceRun` records operational history and is not evidence in the `ExecutionAttempt` sense;
|
||||||
|
|||||||
@@ -53,10 +53,12 @@ are stable. Never renumber an existing ID; retire it explicitly instead.
|
|||||||
### Operational Maintenance
|
### Operational Maintenance
|
||||||
|
|
||||||
- **REQ-6-010 Queued Maintenance Runs:** Settings-initiated maintenance must persist a `MaintenanceRun` and execute in the worker, not inline in the request that started it.
|
- **REQ-6-010 Queued Maintenance Runs:** Settings-initiated maintenance must persist a `MaintenanceRun` and execute in the worker, not inline in the request that started it.
|
||||||
- **REQ-6-011 Maintenance Run Types:** `MaintenanceRun.job_type` must use one of `backup`, `storage_reconciliation`.
|
- **REQ-6-011 Maintenance Run Types:** `MaintenanceRun.job_type` must use one of `backup`, `storage_reconciliation`, `gedcom_import`.
|
||||||
- **REQ-6-012 Maintenance Status Lifecycle:** `MaintenanceRun.status` must use one of `queued`, `processing`, `succeeded`, `failed`.
|
- **REQ-6-012 Maintenance Status Lifecycle:** `MaintenanceRun.status` must use one of `queued`, `processing`, `succeeded`, `failed`.
|
||||||
- **REQ-6-013 Single Claim:** A queued run must be claimed by at most one worker, using a conditional status update rather than read-then-write.
|
- **REQ-6-013 Single Claim:** A queued run must be claimed by at most one worker, using a conditional status update rather than read-then-write.
|
||||||
- **REQ-6-014 Run History:** Completed runs must retain status, timing, summary, log reference, and error detail, and expose the log for viewing and download.
|
- **REQ-6-014 Run History:** Completed runs must retain status, timing, summary, log reference, and error detail, and expose the log for viewing and download.
|
||||||
|
- **REQ-6-015 GEDCOM Upload Import:** Settings must support manual `.ged` upload and queue-backed import into genealogy tables.
|
||||||
|
- **REQ-6-016 GEDCOM Idempotent Upsert:** GEDCOM import must upsert `GenealogyPerson` and `GenealogyFamily` by FamilySearch IDs and avoid duplicate imported citations on re-run.
|
||||||
|
|
||||||
### Deployment and Runtime Configuration
|
### Deployment and Runtime Configuration
|
||||||
|
|
||||||
|
|||||||
+66
-1
@@ -7,7 +7,8 @@ This document is the field-accurate V6.1 schema contract aligned to `src/transcr
|
|||||||
- `src/transcription/db/models.py` (status and purpose enums, including maintenance lifecycle enums)
|
- `src/transcription/db/models.py` (status and purpose enums, including maintenance lifecycle enums)
|
||||||
- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`)
|
- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`)
|
||||||
- `src/transcription/db/models.py:122-172` (`Tag`, `Document`)
|
- `src/transcription/db/models.py:122-172` (`Tag`, `Document`)
|
||||||
- `src/transcription/db/models.py:175-281` (`Person`, `Photo`, `DocumentPerson`, `DocumentTag`)
|
- `src/transcription/db/models.py` (`Person`, `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, `GenealogyCitation`)
|
||||||
|
- `src/transcription/db/models.py` (`Photo`, `DocumentPerson`, `DocumentTag`)
|
||||||
- `src/transcription/db/models.py:285-347` (`Job`)
|
- `src/transcription/db/models.py:285-347` (`Job`)
|
||||||
- `src/transcription/db/models.py` (`MaintenanceRun`)
|
- `src/transcription/db/models.py` (`MaintenanceRun`)
|
||||||
- `src/transcription/db/models.py:350-462` (`Source`, `JobSource`)
|
- `src/transcription/db/models.py:350-462` (`Source`, `JobSource`)
|
||||||
@@ -25,6 +26,13 @@ erDiagram
|
|||||||
Person ||--o{ DocumentPerson : links
|
Person ||--o{ DocumentPerson : links
|
||||||
Person ||--o{ PersonTag : tagged
|
Person ||--o{ PersonTag : tagged
|
||||||
Person ||--o{ Photo : owns
|
Person ||--o{ Photo : owns
|
||||||
|
GenealogyPerson ||--o{ GenealogyFamily : husband
|
||||||
|
GenealogyPerson ||--o{ GenealogyFamily : wife
|
||||||
|
GenealogyPerson ||--o{ GenealogyFamilyChild : child
|
||||||
|
GenealogyFamily ||--o{ GenealogyFamilyChild : includes
|
||||||
|
GenealogyPerson ||--o{ GenealogyCitation : cited
|
||||||
|
GenealogyFamily ||--o{ GenealogyCitation : cited
|
||||||
|
Document ||--o{ GenealogyCitation : evidence
|
||||||
PersonRole ||--o{ DocumentPerson : labels
|
PersonRole ||--o{ DocumentPerson : labels
|
||||||
Tag ||--o{ DocumentTag : labels
|
Tag ||--o{ DocumentTag : labels
|
||||||
Tag ||--o{ PersonTag : labels
|
Tag ||--o{ PersonTag : labels
|
||||||
@@ -62,6 +70,7 @@ erDiagram
|
|||||||
|
|
||||||
- `backup`
|
- `backup`
|
||||||
- `storage_reconciliation`
|
- `storage_reconciliation`
|
||||||
|
- `gedcom_import`
|
||||||
|
|
||||||
### MaintenanceRunStatus
|
### MaintenanceRunStatus
|
||||||
|
|
||||||
@@ -142,6 +151,62 @@ erDiagram
|
|||||||
| `created_at` | `datetime` | default now |
|
| `created_at` | `datetime` | default now |
|
||||||
| `updated_at` | `datetime` | default now, onupdate |
|
| `updated_at` | `datetime` | default now, onupdate |
|
||||||
|
|
||||||
|
### `GenealogyPerson`
|
||||||
|
|
||||||
|
| Field | Type | Notes |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| `id` | `UUID` | PK |
|
||||||
|
| `fs_id` | `str` | unique, indexed FamilySearch identifier |
|
||||||
|
| `full_name` | `str` | required |
|
||||||
|
| `birth_date` | `date \| None` | optional |
|
||||||
|
| `birth_date_raw` | `str \| None` | optional |
|
||||||
|
| `birth_place` | `str \| None` | optional |
|
||||||
|
| `death_date` | `date \| None` | optional |
|
||||||
|
| `death_date_raw` | `str \| None` | optional |
|
||||||
|
| `death_place` | `str \| None` | optional |
|
||||||
|
| `created_at` | `datetime` | default now |
|
||||||
|
| `updated_at` | `datetime` | default now, onupdate |
|
||||||
|
|
||||||
|
### `GenealogyFamily`
|
||||||
|
|
||||||
|
| Field | Type | Notes |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| `id` | `UUID` | PK |
|
||||||
|
| `fs_family_id` | `str` | unique, indexed FamilySearch family identifier |
|
||||||
|
| `husband_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
|
||||||
|
| `wife_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
|
||||||
|
| `marriage_date` | `date \| None` | optional |
|
||||||
|
| `marriage_date_raw` | `str \| None` | optional |
|
||||||
|
| `marriage_place` | `str \| None` | optional |
|
||||||
|
| `created_at` | `datetime` | default now |
|
||||||
|
| `updated_at` | `datetime` | default now, onupdate |
|
||||||
|
|
||||||
|
### `GenealogyFamilyChild`
|
||||||
|
|
||||||
|
| Field | Type | Notes |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| `id` | `UUID` | PK |
|
||||||
|
| `family_id` | `UUID` | FK -> `genealogy_family.id`, indexed |
|
||||||
|
| `child_id` | `UUID` | FK -> `genealogy_person.id`, indexed |
|
||||||
|
| `relationship_type` | `str \| None` | optional |
|
||||||
|
| `created_at` | `datetime` | default now |
|
||||||
|
|
||||||
|
Constraint:
|
||||||
|
- `UniqueConstraint(family_id, child_id)` named `uq_genealogy_family_child`
|
||||||
|
|
||||||
|
### `GenealogyCitation`
|
||||||
|
|
||||||
|
| Field | Type | Notes |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| `id` | `UUID` | PK |
|
||||||
|
| `genealogy_person_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
|
||||||
|
| `genealogy_family_id` | `UUID \| None` | nullable FK -> `genealogy_family.id`, indexed |
|
||||||
|
| `fact_type` | `GenealogyCitationFactType` | enum: `birth`, `death`, `marriage`, `other` |
|
||||||
|
| `raw_citation_text` | `str` | required raw GEDCOM citation text |
|
||||||
|
| `source_kind` | `GenealogyCitationSourceKind` | enum: `familysearch_imported`, `transcription_evidence` |
|
||||||
|
| `document_id` | `UUID \| None` | nullable FK -> `document.id`, indexed |
|
||||||
|
| `created_at` | `datetime` | default now |
|
||||||
|
|
||||||
### `Photo`
|
### `Photo`
|
||||||
|
|
||||||
| Field | Type | Notes |
|
| Field | Type | Notes |
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ Settings manages installation-local registries, safe runtime .env settings, and
|
|||||||
- Prompts exposes only `transcribe_document.md` for editing and restore-from-backup.
|
- Prompts exposes only `transcribe_document.md` for editing and restore-from-backup.
|
||||||
- Home Page Text edits the same Markdown content rendered on `/homepage`.
|
- Home Page Text edits the same Markdown content rendered on `/homepage`.
|
||||||
- Maintenance provides queue-backed **Run Backup** and **Run Storage Reconciliation** actions.
|
- Maintenance provides queue-backed **Run Backup** and **Run Storage Reconciliation** actions.
|
||||||
|
- Maintenance also provides GEDCOM upload and **Run GEDCOM Import** actions, using the same queue-backed `MaintenanceRun` history/log flow.
|
||||||
- Maintenance run history shows job type, status, started/finished timestamps, duration, summary, and log view/download actions.
|
- Maintenance run history shows job type, status, started/finished timestamps, duration, summary, and log view/download actions.
|
||||||
- Maintenance actions enqueue work and signal the worker; the page itself does not execute shell commands directly.
|
- Maintenance actions enqueue work and signal the worker; the page itself does not execute shell commands directly.
|
||||||
|
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ dependencies = [
|
|||||||
"psycopg2-binary>=2.9.12",
|
"psycopg2-binary>=2.9.12",
|
||||||
"pydantic>=2.13.4",
|
"pydantic>=2.13.4",
|
||||||
"pydantic-settings>=2.9.1",
|
"pydantic-settings>=2.9.1",
|
||||||
|
"python-gedcom>=1.1.0",
|
||||||
"sqlmodel>=0.0.25",
|
"sqlmodel>=0.0.25",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
@@ -39,6 +39,8 @@ EXPORT_TABLE_ORDER = (
|
|||||||
"tag",
|
"tag",
|
||||||
"document",
|
"document",
|
||||||
"person",
|
"person",
|
||||||
|
"genealogy_person",
|
||||||
|
"genealogy_family",
|
||||||
"photo",
|
"photo",
|
||||||
"document_person",
|
"document_person",
|
||||||
"document_tag",
|
"document_tag",
|
||||||
@@ -47,6 +49,8 @@ EXPORT_TABLE_ORDER = (
|
|||||||
"source",
|
"source",
|
||||||
"job_source",
|
"job_source",
|
||||||
"execution_attempt",
|
"execution_attempt",
|
||||||
|
"genealogy_family_child",
|
||||||
|
"genealogy_citation",
|
||||||
)
|
)
|
||||||
|
|
||||||
BYTES_FIELDS = {"transport_body"}
|
BYTES_FIELDS = {"transport_body"}
|
||||||
|
|||||||
@@ -85,6 +85,7 @@ class JobPurpose(StrEnum):
|
|||||||
class MaintenanceJobType(StrEnum):
|
class MaintenanceJobType(StrEnum):
|
||||||
BACKUP = "backup"
|
BACKUP = "backup"
|
||||||
STORAGE_RECONCILIATION = "storage_reconciliation"
|
STORAGE_RECONCILIATION = "storage_reconciliation"
|
||||||
|
GEDCOM_IMPORT = "gedcom_import"
|
||||||
|
|
||||||
|
|
||||||
class MaintenanceRunStatus(StrEnum):
|
class MaintenanceRunStatus(StrEnum):
|
||||||
@@ -94,6 +95,18 @@ class MaintenanceRunStatus(StrEnum):
|
|||||||
FAILED = "failed"
|
FAILED = "failed"
|
||||||
|
|
||||||
|
|
||||||
|
class GenealogyCitationFactType(StrEnum):
|
||||||
|
BIRTH = "birth"
|
||||||
|
DEATH = "death"
|
||||||
|
MARRIAGE = "marriage"
|
||||||
|
OTHER = "other"
|
||||||
|
|
||||||
|
|
||||||
|
class GenealogyCitationSourceKind(StrEnum):
|
||||||
|
FAMILYSEARCH_IMPORTED = "familysearch_imported"
|
||||||
|
TRANSCRIPTION_EVIDENCE = "transcription_evidence"
|
||||||
|
|
||||||
|
|
||||||
class DocumentType(SQLModel, table=True):
|
class DocumentType(SQLModel, table=True):
|
||||||
"""Registry of allowed document types."""
|
"""Registry of allowed document types."""
|
||||||
|
|
||||||
@@ -235,6 +248,146 @@ class Person(SQLModel, table=True):
|
|||||||
return f"{self.given_names} {self.last_name}".strip()
|
return f"{self.given_names} {self.last_name}".strip()
|
||||||
|
|
||||||
|
|
||||||
|
class GenealogyPerson(SQLModel, table=True):
|
||||||
|
"""An individual imported from a GEDCOM export."""
|
||||||
|
|
||||||
|
__tablename__ = "genealogy_person"
|
||||||
|
|
||||||
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
|
fs_id: str = Field(index=True, unique=True)
|
||||||
|
full_name: str
|
||||||
|
birth_date: date | None = None
|
||||||
|
birth_date_raw: str | None = None
|
||||||
|
birth_place: str | None = None
|
||||||
|
death_date: date | None = None
|
||||||
|
death_date_raw: str | None = None
|
||||||
|
death_place: str | None = None
|
||||||
|
created_at: datetime = Field(default_factory=_utc_now_naive)
|
||||||
|
updated_at: datetime = Field(
|
||||||
|
default_factory=_utc_now_naive,
|
||||||
|
sa_column_kwargs={"onupdate": _utc_now_naive},
|
||||||
|
)
|
||||||
|
|
||||||
|
husband_families: list["GenealogyFamily"] = Relationship(
|
||||||
|
back_populates="husband",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.husband_id]"},
|
||||||
|
)
|
||||||
|
wife_families: list["GenealogyFamily"] = Relationship(
|
||||||
|
back_populates="wife",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.wife_id]"},
|
||||||
|
)
|
||||||
|
child_family_memberships: list["GenealogyFamilyChild"] = Relationship(
|
||||||
|
back_populates="child",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
citations: list["GenealogyCitation"] = Relationship(
|
||||||
|
back_populates="genealogy_person",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class GenealogyFamily(SQLModel, table=True):
|
||||||
|
"""A family linking two GenealogyPerson records."""
|
||||||
|
|
||||||
|
__tablename__ = "genealogy_family"
|
||||||
|
|
||||||
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
|
fs_family_id: str = Field(index=True, unique=True)
|
||||||
|
husband_id: UUID | None = Field(default=None, foreign_key="genealogy_person.id", index=True)
|
||||||
|
wife_id: UUID | None = Field(default=None, foreign_key="genealogy_person.id", index=True)
|
||||||
|
marriage_date: date | None = None
|
||||||
|
marriage_date_raw: str | None = None
|
||||||
|
marriage_place: str | None = None
|
||||||
|
created_at: datetime = Field(default_factory=_utc_now_naive)
|
||||||
|
updated_at: datetime = Field(
|
||||||
|
default_factory=_utc_now_naive,
|
||||||
|
sa_column_kwargs={"onupdate": _utc_now_naive},
|
||||||
|
)
|
||||||
|
|
||||||
|
husband: Optional["GenealogyPerson"] = Relationship(
|
||||||
|
back_populates="husband_families",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.husband_id]"},
|
||||||
|
)
|
||||||
|
wife: Optional["GenealogyPerson"] = Relationship(
|
||||||
|
back_populates="wife_families",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.wife_id]"},
|
||||||
|
)
|
||||||
|
children: list["GenealogyFamilyChild"] = Relationship(
|
||||||
|
back_populates="family",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
citations: list["GenealogyCitation"] = Relationship(
|
||||||
|
back_populates="genealogy_family",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class GenealogyFamilyChild(SQLModel, table=True):
|
||||||
|
"""Junction table for child membership within a genealogy family."""
|
||||||
|
|
||||||
|
__tablename__ = "genealogy_family_child"
|
||||||
|
|
||||||
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
|
family_id: UUID = Field(foreign_key="genealogy_family.id", index=True)
|
||||||
|
child_id: UUID = Field(foreign_key="genealogy_person.id", index=True)
|
||||||
|
relationship_type: str | None = None
|
||||||
|
created_at: datetime = Field(default_factory=_utc_now_naive)
|
||||||
|
|
||||||
|
__table_args__ = (UniqueConstraint("family_id", "child_id", name="uq_genealogy_family_child"),)
|
||||||
|
|
||||||
|
family: Optional["GenealogyFamily"] = Relationship(
|
||||||
|
back_populates="children",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
child: Optional["GenealogyPerson"] = Relationship(
|
||||||
|
back_populates="child_family_memberships",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class GenealogyCitation(SQLModel, table=True):
|
||||||
|
"""A source citation attached to a genealogical fact."""
|
||||||
|
|
||||||
|
__tablename__ = "genealogy_citation"
|
||||||
|
|
||||||
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
|
genealogy_person_id: UUID | None = Field(default=None, foreign_key="genealogy_person.id", index=True)
|
||||||
|
genealogy_family_id: UUID | None = Field(default=None, foreign_key="genealogy_family.id", index=True)
|
||||||
|
fact_type: GenealogyCitationFactType = Field(
|
||||||
|
sa_column=Column(
|
||||||
|
SAEnum(
|
||||||
|
GenealogyCitationFactType,
|
||||||
|
values_callable=lambda enum_cls: [item.value for item in enum_cls],
|
||||||
|
native_enum=False,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
raw_citation_text: str
|
||||||
|
source_kind: GenealogyCitationSourceKind = Field(
|
||||||
|
sa_column=Column(
|
||||||
|
SAEnum(
|
||||||
|
GenealogyCitationSourceKind,
|
||||||
|
values_callable=lambda enum_cls: [item.value for item in enum_cls],
|
||||||
|
native_enum=False,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
document_id: UUID | None = Field(default=None, foreign_key="document.id", index=True)
|
||||||
|
created_at: datetime = Field(default_factory=_utc_now_naive)
|
||||||
|
|
||||||
|
genealogy_person: Optional["GenealogyPerson"] = Relationship(
|
||||||
|
back_populates="citations",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
genealogy_family: Optional["GenealogyFamily"] = Relationship(
|
||||||
|
back_populates="citations",
|
||||||
|
sa_relationship_kwargs={"lazy": "raise"},
|
||||||
|
)
|
||||||
|
document: Optional["Document"] = Relationship(sa_relationship_kwargs={"lazy": "raise"})
|
||||||
|
|
||||||
|
|
||||||
class Photo(SQLModel, table=True):
|
class Photo(SQLModel, table=True):
|
||||||
"""A reusable image record for Person and homepage galleries."""
|
"""A reusable image record for Person and homepage galleries."""
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,599 @@
|
|||||||
|
"""GEDCOM parsing and import helpers for maintenance-driven genealogy sync."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import date
|
||||||
|
from pathlib import Path
|
||||||
|
from uuid import UUID
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
from gedcom.element.element import Element
|
||||||
|
from gedcom.parser import Parser
|
||||||
|
from sqlalchemy import or_
|
||||||
|
from sqlmodel import col
|
||||||
|
from sqlmodel import delete
|
||||||
|
from sqlmodel import select
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from transcription.db.models import GenealogyCitation
|
||||||
|
from transcription.db.models import GenealogyCitationFactType
|
||||||
|
from transcription.db.models import GenealogyCitationSourceKind
|
||||||
|
from transcription.db.models import GenealogyFamily
|
||||||
|
from transcription.db.models import GenealogyFamilyChild
|
||||||
|
from transcription.db.models import GenealogyPerson
|
||||||
|
from transcription.errors import AppError
|
||||||
|
from transcription.errors import ErrorCategory
|
||||||
|
|
||||||
|
_MONTHS = {
|
||||||
|
"JAN": 1,
|
||||||
|
"FEB": 2,
|
||||||
|
"MAR": 3,
|
||||||
|
"APR": 4,
|
||||||
|
"MAY": 5,
|
||||||
|
"JUN": 6,
|
||||||
|
"JUL": 7,
|
||||||
|
"AUG": 8,
|
||||||
|
"SEP": 9,
|
||||||
|
"OCT": 10,
|
||||||
|
"NOV": 11,
|
||||||
|
"DEC": 12,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ParsedCitation:
|
||||||
|
fact_type: GenealogyCitationFactType
|
||||||
|
raw_citation_text: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ParsedPerson:
|
||||||
|
pointer: str
|
||||||
|
fs_id: str | None
|
||||||
|
full_name: str
|
||||||
|
birth_date: date | None
|
||||||
|
birth_date_raw: str | None
|
||||||
|
birth_place: str | None
|
||||||
|
death_date: date | None
|
||||||
|
death_date_raw: str | None
|
||||||
|
death_place: str | None
|
||||||
|
citations: tuple[ParsedCitation, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ParsedFamilyChild:
|
||||||
|
child_pointer: str
|
||||||
|
relationship_type: str | None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ParsedFamily:
|
||||||
|
fs_family_id: str | None
|
||||||
|
husband_pointer: str | None
|
||||||
|
wife_pointer: str | None
|
||||||
|
marriage_date: date | None
|
||||||
|
marriage_date_raw: str | None
|
||||||
|
marriage_place: str | None
|
||||||
|
children: tuple[ParsedFamilyChild, ...]
|
||||||
|
citations: tuple[ParsedCitation, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ParsedGedcom:
|
||||||
|
people: tuple[ParsedPerson, ...]
|
||||||
|
families: tuple[ParsedFamily, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class GedcomImportResult:
|
||||||
|
new_people: int
|
||||||
|
updated_people: int
|
||||||
|
skipped_people_without_fs_id: int
|
||||||
|
new_families: int
|
||||||
|
updated_families: int
|
||||||
|
skipped_families_without_fs_id: int
|
||||||
|
family_children: int
|
||||||
|
citations: int
|
||||||
|
|
||||||
|
|
||||||
|
class GedcomImportError(AppError):
|
||||||
|
"""Raised when GEDCOM content cannot be parsed or imported."""
|
||||||
|
|
||||||
|
|
||||||
|
def parse_gedcom(*, file_path: Path) -> ParsedGedcom:
|
||||||
|
parser = Parser()
|
||||||
|
try:
|
||||||
|
parser.parse_file(str(file_path))
|
||||||
|
except Exception as exc:
|
||||||
|
raise GedcomImportError(
|
||||||
|
"GEDCOM file could not be parsed.",
|
||||||
|
category=ErrorCategory.VALIDATION,
|
||||||
|
suggestion="Upload a GEDCOM 5.5.1-compatible export and retry.",
|
||||||
|
detail=f"{type(exc).__name__}: {exc}",
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
people: list[ParsedPerson] = []
|
||||||
|
families: list[ParsedFamily] = []
|
||||||
|
for element in parser.get_root_child_elements():
|
||||||
|
tag = element.get_tag()
|
||||||
|
if tag == "INDI":
|
||||||
|
people.append(_parse_person(element))
|
||||||
|
elif tag == "FAM":
|
||||||
|
families.append(_parse_family(element))
|
||||||
|
return ParsedGedcom(people=tuple(people), families=tuple(families))
|
||||||
|
|
||||||
|
|
||||||
|
async def import_gedcom_file(*, session: AsyncSession, file_path: Path) -> GedcomImportResult:
|
||||||
|
parsed = parse_gedcom(file_path=file_path)
|
||||||
|
pointer_to_fs_id = _pointer_to_fs_id_map(parsed=parsed)
|
||||||
|
people_by_fs_id, new_people, updated_people, skipped_people_without_fs_id = await _upsert_people(
|
||||||
|
session=session,
|
||||||
|
parsed=parsed,
|
||||||
|
)
|
||||||
|
(
|
||||||
|
families_by_fs_id,
|
||||||
|
new_families,
|
||||||
|
updated_families,
|
||||||
|
skipped_families_without_fs_id,
|
||||||
|
family_children,
|
||||||
|
) = await _upsert_families(
|
||||||
|
session=session,
|
||||||
|
parsed=parsed,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people_by_fs_id=people_by_fs_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
citation_rows = _citation_rows(
|
||||||
|
parsed=parsed,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people=people_by_fs_id,
|
||||||
|
families=families_by_fs_id,
|
||||||
|
)
|
||||||
|
await _replace_imported_citations(
|
||||||
|
session=session,
|
||||||
|
person_ids={item.id for item in people_by_fs_id.values()},
|
||||||
|
family_ids={item.id for item in families_by_fs_id.values()},
|
||||||
|
citations=citation_rows,
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
return GedcomImportResult(
|
||||||
|
new_people=new_people,
|
||||||
|
updated_people=updated_people,
|
||||||
|
skipped_people_without_fs_id=skipped_people_without_fs_id,
|
||||||
|
new_families=new_families,
|
||||||
|
updated_families=updated_families,
|
||||||
|
skipped_families_without_fs_id=skipped_families_without_fs_id,
|
||||||
|
family_children=family_children,
|
||||||
|
citations=len(citation_rows),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _load_people_by_fs_id(*, session: AsyncSession, fs_ids: set[str]) -> dict[str, GenealogyPerson]:
|
||||||
|
if not fs_ids:
|
||||||
|
return {}
|
||||||
|
query = select(GenealogyPerson).where(col(GenealogyPerson.fs_id).in_(fs_ids))
|
||||||
|
return {person.fs_id: person for person in (await session.exec(query)).all()}
|
||||||
|
|
||||||
|
|
||||||
|
async def _load_families_by_fs_id(*, session: AsyncSession, fs_family_ids: set[str]) -> dict[str, GenealogyFamily]:
|
||||||
|
if not fs_family_ids:
|
||||||
|
return {}
|
||||||
|
query = select(GenealogyFamily).where(col(GenealogyFamily.fs_family_id).in_(fs_family_ids))
|
||||||
|
return {family.fs_family_id: family for family in (await session.exec(query)).all()}
|
||||||
|
|
||||||
|
|
||||||
|
def _pointer_to_fs_id_map(*, parsed: ParsedGedcom) -> dict[str, str]:
|
||||||
|
return {person.pointer: person.fs_id for person in parsed.people if person.pointer and person.fs_id is not None}
|
||||||
|
|
||||||
|
|
||||||
|
async def _upsert_people(
|
||||||
|
*,
|
||||||
|
session: AsyncSession,
|
||||||
|
parsed: ParsedGedcom,
|
||||||
|
) -> tuple[dict[str, GenealogyPerson], int, int, int]:
|
||||||
|
people_with_fs_id = [person for person in parsed.people if person.fs_id is not None]
|
||||||
|
fs_ids = {person.fs_id for person in people_with_fs_id if person.fs_id is not None}
|
||||||
|
people_by_fs_id = await _load_people_by_fs_id(session=session, fs_ids=fs_ids)
|
||||||
|
new_people = 0
|
||||||
|
updated_people = 0
|
||||||
|
|
||||||
|
for person in people_with_fs_id:
|
||||||
|
assert person.fs_id is not None
|
||||||
|
existing = people_by_fs_id.get(person.fs_id)
|
||||||
|
if existing is None:
|
||||||
|
existing = GenealogyPerson(
|
||||||
|
fs_id=person.fs_id,
|
||||||
|
full_name=person.full_name,
|
||||||
|
birth_date=person.birth_date,
|
||||||
|
birth_date_raw=person.birth_date_raw,
|
||||||
|
birth_place=person.birth_place,
|
||||||
|
death_date=person.death_date,
|
||||||
|
death_date_raw=person.death_date_raw,
|
||||||
|
death_place=person.death_place,
|
||||||
|
)
|
||||||
|
session.add(existing)
|
||||||
|
await session.flush()
|
||||||
|
people_by_fs_id[person.fs_id] = existing
|
||||||
|
new_people += 1
|
||||||
|
continue
|
||||||
|
if _apply_person_updates(existing=existing, person=person):
|
||||||
|
updated_people += 1
|
||||||
|
|
||||||
|
skipped = len(parsed.people) - len(people_with_fs_id)
|
||||||
|
return people_by_fs_id, new_people, updated_people, skipped
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_person_updates(*, existing: GenealogyPerson, person: ParsedPerson) -> bool:
|
||||||
|
changed = False
|
||||||
|
fields = (
|
||||||
|
("full_name", person.full_name),
|
||||||
|
("birth_date", person.birth_date),
|
||||||
|
("birth_date_raw", person.birth_date_raw),
|
||||||
|
("birth_place", person.birth_place),
|
||||||
|
("death_date", person.death_date),
|
||||||
|
("death_date_raw", person.death_date_raw),
|
||||||
|
("death_place", person.death_place),
|
||||||
|
)
|
||||||
|
for name, value in fields:
|
||||||
|
if getattr(existing, name) != value:
|
||||||
|
setattr(existing, name, value)
|
||||||
|
changed = True
|
||||||
|
return changed
|
||||||
|
|
||||||
|
|
||||||
|
async def _upsert_families(
|
||||||
|
*,
|
||||||
|
session: AsyncSession,
|
||||||
|
parsed: ParsedGedcom,
|
||||||
|
pointer_to_fs_id: Mapping[str, str],
|
||||||
|
people_by_fs_id: dict[str, GenealogyPerson],
|
||||||
|
) -> tuple[dict[str, GenealogyFamily], int, int, int, int]:
|
||||||
|
families_with_fs_id = [family for family in parsed.families if family.fs_family_id is not None]
|
||||||
|
fs_family_ids = {family.fs_family_id for family in families_with_fs_id if family.fs_family_id is not None}
|
||||||
|
families_by_fs_id = await _load_families_by_fs_id(session=session, fs_family_ids=fs_family_ids)
|
||||||
|
new_families = 0
|
||||||
|
updated_families = 0
|
||||||
|
family_children = 0
|
||||||
|
|
||||||
|
for family in families_with_fs_id:
|
||||||
|
assert family.fs_family_id is not None
|
||||||
|
husband_id = _person_id_from_pointer(
|
||||||
|
pointer=family.husband_pointer,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people=people_by_fs_id,
|
||||||
|
)
|
||||||
|
wife_id = _person_id_from_pointer(
|
||||||
|
pointer=family.wife_pointer,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people=people_by_fs_id,
|
||||||
|
)
|
||||||
|
target = families_by_fs_id.get(family.fs_family_id)
|
||||||
|
if target is None:
|
||||||
|
target = GenealogyFamily(
|
||||||
|
fs_family_id=family.fs_family_id,
|
||||||
|
husband_id=husband_id,
|
||||||
|
wife_id=wife_id,
|
||||||
|
marriage_date=family.marriage_date,
|
||||||
|
marriage_date_raw=family.marriage_date_raw,
|
||||||
|
marriage_place=family.marriage_place,
|
||||||
|
)
|
||||||
|
session.add(target)
|
||||||
|
await session.flush()
|
||||||
|
families_by_fs_id[family.fs_family_id] = target
|
||||||
|
new_families += 1
|
||||||
|
elif _apply_family_updates(existing=target, family=family, husband_id=husband_id, wife_id=wife_id):
|
||||||
|
updated_families += 1
|
||||||
|
|
||||||
|
family_children += await _replace_family_children(
|
||||||
|
session=session,
|
||||||
|
family=family,
|
||||||
|
family_id=target.id,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people=people_by_fs_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
skipped = len(parsed.families) - len(families_with_fs_id)
|
||||||
|
return families_by_fs_id, new_families, updated_families, skipped, family_children
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_family_updates(
|
||||||
|
*,
|
||||||
|
existing: GenealogyFamily,
|
||||||
|
family: ParsedFamily,
|
||||||
|
husband_id: UUID | None,
|
||||||
|
wife_id: UUID | None,
|
||||||
|
) -> bool:
|
||||||
|
changed = False
|
||||||
|
fields = (
|
||||||
|
("husband_id", husband_id),
|
||||||
|
("wife_id", wife_id),
|
||||||
|
("marriage_date", family.marriage_date),
|
||||||
|
("marriage_date_raw", family.marriage_date_raw),
|
||||||
|
("marriage_place", family.marriage_place),
|
||||||
|
)
|
||||||
|
for name, value in fields:
|
||||||
|
if getattr(existing, name) != value:
|
||||||
|
setattr(existing, name, value)
|
||||||
|
changed = True
|
||||||
|
return changed
|
||||||
|
|
||||||
|
|
||||||
|
async def _replace_family_children(
|
||||||
|
*,
|
||||||
|
session: AsyncSession,
|
||||||
|
family: ParsedFamily,
|
||||||
|
family_id: UUID,
|
||||||
|
pointer_to_fs_id: Mapping[str, str],
|
||||||
|
people: dict[str, GenealogyPerson],
|
||||||
|
) -> int:
|
||||||
|
await session.exec(delete(GenealogyFamilyChild).where(col(GenealogyFamilyChild.family_id) == family_id))
|
||||||
|
child_rows = _family_child_rows(
|
||||||
|
family=family,
|
||||||
|
family_id=family_id,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people=people,
|
||||||
|
)
|
||||||
|
for child_row in child_rows:
|
||||||
|
session.add(child_row)
|
||||||
|
return len(child_rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _person_id_from_pointer(
|
||||||
|
*,
|
||||||
|
pointer: str | None,
|
||||||
|
pointer_to_fs_id: Mapping[str, str],
|
||||||
|
people: dict[str, GenealogyPerson],
|
||||||
|
) -> UUID | None:
|
||||||
|
if pointer is None:
|
||||||
|
return None
|
||||||
|
fs_id = pointer_to_fs_id.get(pointer)
|
||||||
|
if fs_id is None:
|
||||||
|
return None
|
||||||
|
person = people.get(fs_id)
|
||||||
|
return person.id if person is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def _family_child_rows(
|
||||||
|
*,
|
||||||
|
family: ParsedFamily,
|
||||||
|
family_id: UUID,
|
||||||
|
pointer_to_fs_id: Mapping[str, str],
|
||||||
|
people: dict[str, GenealogyPerson],
|
||||||
|
) -> list[GenealogyFamilyChild]:
|
||||||
|
rows: list[GenealogyFamilyChild] = []
|
||||||
|
seen_child_ids: set[UUID] = set()
|
||||||
|
for child in family.children:
|
||||||
|
child_id = _person_id_from_pointer(
|
||||||
|
pointer=child.child_pointer,
|
||||||
|
pointer_to_fs_id=pointer_to_fs_id,
|
||||||
|
people=people,
|
||||||
|
)
|
||||||
|
if child_id is None or child_id in seen_child_ids:
|
||||||
|
continue
|
||||||
|
seen_child_ids.add(child_id)
|
||||||
|
rows.append(
|
||||||
|
GenealogyFamilyChild(
|
||||||
|
id=uuid4(),
|
||||||
|
family_id=family_id,
|
||||||
|
child_id=child_id,
|
||||||
|
relationship_type=child.relationship_type,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _citation_rows(
|
||||||
|
*,
|
||||||
|
parsed: ParsedGedcom,
|
||||||
|
pointer_to_fs_id: Mapping[str, str],
|
||||||
|
people: dict[str, GenealogyPerson],
|
||||||
|
families: dict[str, GenealogyFamily],
|
||||||
|
) -> list[GenealogyCitation]:
|
||||||
|
rows: list[GenealogyCitation] = []
|
||||||
|
seen: set[tuple[UUID | None, UUID | None, str, str]] = set()
|
||||||
|
|
||||||
|
for person in parsed.people:
|
||||||
|
fs_id = pointer_to_fs_id.get(person.pointer)
|
||||||
|
if fs_id is None:
|
||||||
|
continue
|
||||||
|
person_row = people.get(fs_id)
|
||||||
|
if person_row is None:
|
||||||
|
continue
|
||||||
|
for citation in person.citations:
|
||||||
|
key = (person_row.id, None, citation.fact_type.value, citation.raw_citation_text)
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
rows.append(
|
||||||
|
GenealogyCitation(
|
||||||
|
id=uuid4(),
|
||||||
|
genealogy_person_id=person_row.id,
|
||||||
|
genealogy_family_id=None,
|
||||||
|
fact_type=citation.fact_type,
|
||||||
|
raw_citation_text=citation.raw_citation_text,
|
||||||
|
source_kind=GenealogyCitationSourceKind.FAMILYSEARCH_IMPORTED,
|
||||||
|
document_id=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for family in parsed.families:
|
||||||
|
if family.fs_family_id is None:
|
||||||
|
continue
|
||||||
|
family_row = families.get(family.fs_family_id)
|
||||||
|
if family_row is None:
|
||||||
|
continue
|
||||||
|
for citation in family.citations:
|
||||||
|
key = (None, family_row.id, citation.fact_type.value, citation.raw_citation_text)
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
rows.append(
|
||||||
|
GenealogyCitation(
|
||||||
|
id=uuid4(),
|
||||||
|
genealogy_person_id=None,
|
||||||
|
genealogy_family_id=family_row.id,
|
||||||
|
fact_type=citation.fact_type,
|
||||||
|
raw_citation_text=citation.raw_citation_text,
|
||||||
|
source_kind=GenealogyCitationSourceKind.FAMILYSEARCH_IMPORTED,
|
||||||
|
document_id=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
async def _replace_imported_citations(
|
||||||
|
*,
|
||||||
|
session: AsyncSession,
|
||||||
|
person_ids: set[UUID],
|
||||||
|
family_ids: set[UUID],
|
||||||
|
citations: list[GenealogyCitation],
|
||||||
|
) -> None:
|
||||||
|
where_clauses = []
|
||||||
|
if person_ids:
|
||||||
|
where_clauses.append(col(GenealogyCitation.genealogy_person_id).in_(person_ids))
|
||||||
|
if family_ids:
|
||||||
|
where_clauses.append(col(GenealogyCitation.genealogy_family_id).in_(family_ids))
|
||||||
|
if where_clauses:
|
||||||
|
target_scope = where_clauses[0] if len(where_clauses) == 1 else or_(*where_clauses)
|
||||||
|
await session.exec(
|
||||||
|
delete(GenealogyCitation).where(
|
||||||
|
col(GenealogyCitation.source_kind) == GenealogyCitationSourceKind.FAMILYSEARCH_IMPORTED,
|
||||||
|
target_scope,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for citation in citations:
|
||||||
|
session.add(citation)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_person(element: Element) -> ParsedPerson:
|
||||||
|
birth_event = _first_child(element, "BIRT")
|
||||||
|
death_event = _first_child(element, "DEAT")
|
||||||
|
birth_date_raw = _child_value(birth_event, "DATE")
|
||||||
|
death_date_raw = _child_value(death_event, "DATE")
|
||||||
|
return ParsedPerson(
|
||||||
|
pointer=element.get_pointer() or "",
|
||||||
|
fs_id=_extract_fs_identifier(element),
|
||||||
|
full_name=_person_name(element),
|
||||||
|
birth_date=_parse_exact_date(birth_date_raw),
|
||||||
|
birth_date_raw=birth_date_raw,
|
||||||
|
birth_place=_child_value(birth_event, "PLAC"),
|
||||||
|
death_date=_parse_exact_date(death_date_raw),
|
||||||
|
death_date_raw=death_date_raw,
|
||||||
|
death_place=_child_value(death_event, "PLAC"),
|
||||||
|
citations=(
|
||||||
|
*_fact_citations(fact_element=birth_event, fact_type=GenealogyCitationFactType.BIRTH),
|
||||||
|
*_fact_citations(fact_element=death_event, fact_type=GenealogyCitationFactType.DEATH),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_family(element: Element) -> ParsedFamily:
|
||||||
|
marriage_event = _first_child(element, "MARR")
|
||||||
|
marriage_date_raw = _child_value(marriage_event, "DATE")
|
||||||
|
children = tuple(
|
||||||
|
ParsedFamilyChild(
|
||||||
|
child_pointer=(child.get_value() or "").strip(),
|
||||||
|
relationship_type=_child_value(child, "PEDI"),
|
||||||
|
)
|
||||||
|
for child in _children(element, "CHIL")
|
||||||
|
if (child.get_value() or "").strip()
|
||||||
|
)
|
||||||
|
return ParsedFamily(
|
||||||
|
fs_family_id=_extract_fs_identifier(element),
|
||||||
|
husband_pointer=(_child_value(element, "HUSB") or "").strip() or None,
|
||||||
|
wife_pointer=(_child_value(element, "WIFE") or "").strip() or None,
|
||||||
|
marriage_date=_parse_exact_date(marriage_date_raw),
|
||||||
|
marriage_date_raw=marriage_date_raw,
|
||||||
|
marriage_place=_child_value(marriage_event, "PLAC"),
|
||||||
|
children=children,
|
||||||
|
citations=_fact_citations(fact_element=marriage_event, fact_type=GenealogyCitationFactType.MARRIAGE),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _fact_citations(
|
||||||
|
*, fact_element: Element | None, fact_type: GenealogyCitationFactType
|
||||||
|
) -> tuple[ParsedCitation, ...]:
|
||||||
|
if fact_element is None:
|
||||||
|
return ()
|
||||||
|
citations: list[ParsedCitation] = []
|
||||||
|
for source in _children(fact_element, "SOUR"):
|
||||||
|
raw_citation = _flatten_tag_values(source)
|
||||||
|
if raw_citation is None:
|
||||||
|
continue
|
||||||
|
citations.append(ParsedCitation(fact_type=fact_type, raw_citation_text=raw_citation))
|
||||||
|
return tuple(citations)
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_fs_identifier(element: Element) -> str | None:
|
||||||
|
for tag in ("_FSFTID", "FSFTID"):
|
||||||
|
value = _child_value(element, tag)
|
||||||
|
if value:
|
||||||
|
return value
|
||||||
|
for refn in _children(element, "REFN"):
|
||||||
|
refn_value = (refn.get_value() or "").strip()
|
||||||
|
refn_type = (_child_value(refn, "TYPE") or "").strip().casefold()
|
||||||
|
if refn_value and ("fsftid" in refn_type or "familysearch" in refn_type):
|
||||||
|
return refn_value
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _person_name(element: Element) -> str:
|
||||||
|
raw_name = _child_value(element, "NAME")
|
||||||
|
if raw_name is None:
|
||||||
|
return "Unknown"
|
||||||
|
cleaned = raw_name.replace("/", " ").strip()
|
||||||
|
return " ".join(part for part in cleaned.split() if part) or "Unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_exact_date(raw: str | None) -> date | None:
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
tokens = [token for token in raw.strip().upper().split() if token]
|
||||||
|
if len(tokens) != 3:
|
||||||
|
return None
|
||||||
|
day_token, month_token, year_token = tokens
|
||||||
|
if month_token not in _MONTHS:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return date(year=int(year_token), month=_MONTHS[month_token], day=int(day_token))
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _first_child(element: Element | None, tag: str) -> Element | None:
|
||||||
|
if element is None:
|
||||||
|
return None
|
||||||
|
for child in element.get_child_elements():
|
||||||
|
if child.get_tag() == tag:
|
||||||
|
return child
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _children(element: Element | None, tag: str) -> list[Element]:
|
||||||
|
if element is None:
|
||||||
|
return []
|
||||||
|
return [child for child in element.get_child_elements() if child.get_tag() == tag]
|
||||||
|
|
||||||
|
|
||||||
|
def _child_value(element: Element | None, tag: str) -> str | None:
|
||||||
|
child = _first_child(element, tag)
|
||||||
|
if child is None:
|
||||||
|
return None
|
||||||
|
value = (child.get_value() or "").strip()
|
||||||
|
return value or None
|
||||||
|
|
||||||
|
|
||||||
|
def _flatten_tag_values(element: Element) -> str | None:
|
||||||
|
lines: list[str] = []
|
||||||
|
|
||||||
|
def walk(node: Element) -> None:
|
||||||
|
value = (node.get_value() or "").strip()
|
||||||
|
if value:
|
||||||
|
lines.append(f"{node.get_tag()}: {value}")
|
||||||
|
for child in node.get_child_elements():
|
||||||
|
walk(child)
|
||||||
|
|
||||||
|
walk(element)
|
||||||
|
return " | ".join(lines) if lines else None
|
||||||
@@ -8,6 +8,7 @@ from datetime import UTC
|
|||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from uuid import UUID
|
from uuid import UUID
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
from sqlalchemy import update
|
from sqlalchemy import update
|
||||||
from sqlalchemy.exc import SQLAlchemyError
|
from sqlalchemy.exc import SQLAlchemyError
|
||||||
@@ -26,6 +27,9 @@ from transcription.errors import ErrorCategory
|
|||||||
from transcription.errors import classify_unexpected_error
|
from transcription.errors import classify_unexpected_error
|
||||||
|
|
||||||
from .base import ServiceBase
|
from .base import ServiceBase
|
||||||
|
from .gedcom_import import GedcomImportError
|
||||||
|
from .gedcom_import import import_gedcom_file
|
||||||
|
from .media_storage import persist_named_media
|
||||||
|
|
||||||
|
|
||||||
def _utc_now_naive() -> datetime:
|
def _utc_now_naive() -> datetime:
|
||||||
@@ -49,6 +53,38 @@ class MaintenanceError(AppError):
|
|||||||
class MaintenanceService(ServiceBase):
|
class MaintenanceService(ServiceBase):
|
||||||
"""Persist and execute background maintenance runs."""
|
"""Persist and execute background maintenance runs."""
|
||||||
|
|
||||||
|
async def store_gedcom_upload(self, *, filename: str, file_bytes: bytes) -> str:
|
||||||
|
if not file_bytes:
|
||||||
|
raise MaintenanceError(
|
||||||
|
"GEDCOM upload is empty.",
|
||||||
|
category=ErrorCategory.VALIDATION,
|
||||||
|
suggestion="Upload a non-empty .ged file and retry.",
|
||||||
|
)
|
||||||
|
if Path(filename).suffix.casefold() != ".ged":
|
||||||
|
raise MaintenanceError(
|
||||||
|
"Unsupported GEDCOM upload format.",
|
||||||
|
category=ErrorCategory.USER_INPUT,
|
||||||
|
suggestion="Upload a file with a .ged extension.",
|
||||||
|
)
|
||||||
|
stored_path = await persist_named_media(
|
||||||
|
root=self.settings.upload_dir,
|
||||||
|
namespace=Path("genealogy"),
|
||||||
|
filename=filename,
|
||||||
|
file_bytes=file_bytes,
|
||||||
|
filename_stem=str(uuid4()),
|
||||||
|
error=MaintenanceError,
|
||||||
|
failure_message="GEDCOM file could not be persisted.",
|
||||||
|
failure_suggestion="Check upload directory permissions and retry.",
|
||||||
|
log_label="gedcom file",
|
||||||
|
)
|
||||||
|
return str(stored_path.resolve().relative_to(self.settings.upload_dir.resolve()).as_posix())
|
||||||
|
|
||||||
|
def latest_gedcom_upload_path(self) -> str | None:
|
||||||
|
latest = self._latest_gedcom_upload()
|
||||||
|
if latest is None:
|
||||||
|
return None
|
||||||
|
return str(latest.resolve().relative_to(self.settings.upload_dir.resolve()).as_posix())
|
||||||
|
|
||||||
async def list_runs(
|
async def list_runs(
|
||||||
self,
|
self,
|
||||||
*,
|
*,
|
||||||
@@ -174,6 +210,8 @@ class MaintenanceService(ServiceBase):
|
|||||||
return await self._execute_backup()
|
return await self._execute_backup()
|
||||||
if run.job_type == MaintenanceJobType.STORAGE_RECONCILIATION:
|
if run.job_type == MaintenanceJobType.STORAGE_RECONCILIATION:
|
||||||
return await self._execute_storage_reconciliation()
|
return await self._execute_storage_reconciliation()
|
||||||
|
if run.job_type == MaintenanceJobType.GEDCOM_IMPORT:
|
||||||
|
return await self._execute_gedcom_import()
|
||||||
raise MaintenanceError(
|
raise MaintenanceError(
|
||||||
"Unsupported maintenance job type.",
|
"Unsupported maintenance job type.",
|
||||||
category=ErrorCategory.VALIDATION,
|
category=ErrorCategory.VALIDATION,
|
||||||
@@ -247,6 +285,55 @@ class MaintenanceService(ServiceBase):
|
|||||||
output="No storage reconciliation mismatches detected.",
|
output="No storage reconciliation mismatches detected.",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def _execute_gedcom_import(self) -> MaintenanceExecution:
|
||||||
|
latest_upload = self._latest_gedcom_upload()
|
||||||
|
if latest_upload is None:
|
||||||
|
return MaintenanceExecution(
|
||||||
|
status=MaintenanceRunStatus.FAILED,
|
||||||
|
summary="No GEDCOM upload is available.",
|
||||||
|
output="No .ged file found under uploads/genealogy.",
|
||||||
|
error_detail="Missing GEDCOM upload in uploads/genealogy.",
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with self._session_scope() as session:
|
||||||
|
result = await import_gedcom_file(session=session, file_path=latest_upload)
|
||||||
|
except GedcomImportError as exc:
|
||||||
|
return MaintenanceExecution(
|
||||||
|
status=MaintenanceRunStatus.FAILED,
|
||||||
|
summary="GEDCOM import failed.",
|
||||||
|
output=f"GEDCOM import failed for {latest_upload.name}.",
|
||||||
|
error_detail=exc.detail,
|
||||||
|
)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
error = classify_unexpected_error(exc, operation="maintenance.gedcom_import")
|
||||||
|
return MaintenanceExecution(
|
||||||
|
status=MaintenanceRunStatus.FAILED,
|
||||||
|
summary="GEDCOM import failed.",
|
||||||
|
output=f"GEDCOM import failed for {latest_upload.name}.",
|
||||||
|
error_detail=error.detail,
|
||||||
|
)
|
||||||
|
|
||||||
|
summary = (
|
||||||
|
f"Imported {result.new_people + result.updated_people} people "
|
||||||
|
f"({result.new_people} new, {result.updated_people} updated) and "
|
||||||
|
f"{result.new_families + result.updated_families} families "
|
||||||
|
f"({result.new_families} new, {result.updated_families} updated)."
|
||||||
|
)
|
||||||
|
output_lines = [
|
||||||
|
f"GEDCOM file: {latest_upload.as_posix()}",
|
||||||
|
summary,
|
||||||
|
f"Skipped people without FamilySearch ID: {result.skipped_people_without_fs_id}",
|
||||||
|
f"Skipped families without FamilySearch ID: {result.skipped_families_without_fs_id}",
|
||||||
|
f"Family child links written: {result.family_children}",
|
||||||
|
f"Imported citations: {result.citations}",
|
||||||
|
]
|
||||||
|
return MaintenanceExecution(
|
||||||
|
status=MaintenanceRunStatus.SUCCEEDED,
|
||||||
|
summary=summary,
|
||||||
|
output="\n".join(output_lines),
|
||||||
|
)
|
||||||
|
|
||||||
async def _collect_storage_mismatches(self) -> list[str]:
|
async def _collect_storage_mismatches(self) -> list[str]:
|
||||||
upload_root = self.settings.upload_dir
|
upload_root = self.settings.upload_dir
|
||||||
folder_ids = _document_folder_ids(upload_root)
|
folder_ids = _document_folder_ids(upload_root)
|
||||||
@@ -282,6 +369,15 @@ class MaintenanceService(ServiceBase):
|
|||||||
)
|
)
|
||||||
return mismatches
|
return mismatches
|
||||||
|
|
||||||
|
def _latest_gedcom_upload(self) -> Path | None:
|
||||||
|
genealogy_root = self.settings.upload_dir / "genealogy"
|
||||||
|
if not genealogy_root.exists():
|
||||||
|
return None
|
||||||
|
candidates = [path for path in genealogy_root.rglob("*.ged") if path.is_file()]
|
||||||
|
if not candidates:
|
||||||
|
return None
|
||||||
|
return max(candidates, key=lambda path: (path.stat().st_mtime_ns, path.name.casefold()))
|
||||||
|
|
||||||
async def _document_ids(self) -> set[str]:
|
async def _document_ids(self) -> set[str]:
|
||||||
async with self._session_scope() as session:
|
async with self._session_scope() as session:
|
||||||
rows = await session.exec(select(Document.id))
|
rows = await session.exec(select(Document.id))
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ from nicegui import ui
|
|||||||
from transcription.services.source_media import SOURCE_EXTENSIONS
|
from transcription.services.source_media import SOURCE_EXTENSIONS
|
||||||
|
|
||||||
SOURCE_UPLOAD_EXTENSIONS: tuple[str, ...] = tuple(sorted(SOURCE_EXTENSIONS))
|
SOURCE_UPLOAD_EXTENSIONS: tuple[str, ...] = tuple(sorted(SOURCE_EXTENSIONS))
|
||||||
|
GEDCOM_UPLOAD_EXTENSIONS: tuple[str, ...] = (".ged",)
|
||||||
IMAGE_UPLOAD_EXTENSIONS: tuple[str, ...] = (
|
IMAGE_UPLOAD_EXTENSIONS: tuple[str, ...] = (
|
||||||
".bmp",
|
".bmp",
|
||||||
".gif",
|
".gif",
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ from typing import Any
|
|||||||
from uuid import UUID
|
from uuid import UUID
|
||||||
|
|
||||||
from fastapi import Request
|
from fastapi import Request
|
||||||
|
from nicegui import events
|
||||||
from nicegui import ui
|
from nicegui import ui
|
||||||
|
|
||||||
from transcription.config import Settings
|
from transcription.config import Settings
|
||||||
@@ -27,6 +28,8 @@ from transcription.ui.components.primitives import render_empty_state
|
|||||||
from transcription.ui.components.primitives import section_header_row
|
from transcription.ui.components.primitives import section_header_row
|
||||||
from transcription.ui.components.table.common import build_table
|
from transcription.ui.components.table.common import build_table
|
||||||
from transcription.ui.components.table.registry import render_registry_table
|
from transcription.ui.components.table.registry import render_registry_table
|
||||||
|
from transcription.ui.components.upload_panel import GEDCOM_UPLOAD_EXTENSIONS
|
||||||
|
from transcription.ui.components.upload_panel import render_upload_picker
|
||||||
from transcription.ui.homepage_store import read_homepage_markdown
|
from transcription.ui.homepage_store import read_homepage_markdown
|
||||||
from transcription.ui.homepage_store import save_homepage_markdown
|
from transcription.ui.homepage_store import save_homepage_markdown
|
||||||
from transcription.ui.runtime_settings_store import HIDDEN_SETTINGS_CATEGORIES
|
from transcription.ui.runtime_settings_store import HIDDEN_SETTINGS_CATEGORIES
|
||||||
@@ -516,6 +519,11 @@ def register_page(*, settings: Settings) -> None: # noqa: PLR0915
|
|||||||
ui.label(
|
ui.label(
|
||||||
"Queue maintenance tasks for worker execution. Runs are persisted with summary and logs."
|
"Queue maintenance tasks for worker execution. Runs are persisted with summary and logs."
|
||||||
).classes("text-xs ui-text-muted mb-3")
|
).classes("text-xs ui-text-muted mb-3")
|
||||||
|
await _render_gedcom_import_controls(
|
||||||
|
maintenance=maintenance,
|
||||||
|
request=request,
|
||||||
|
refresh=render_maintenance.refresh,
|
||||||
|
)
|
||||||
with ui.row().classes("w-full items-center gap-2"):
|
with ui.row().classes("w-full items-center gap-2"):
|
||||||
ui.button(
|
ui.button(
|
||||||
"Run Backup",
|
"Run Backup",
|
||||||
@@ -526,7 +534,7 @@ def register_page(*, settings: Settings) -> None: # noqa: PLR0915
|
|||||||
request=request,
|
request=request,
|
||||||
refresh=render_maintenance.refresh,
|
refresh=render_maintenance.refresh,
|
||||||
),
|
),
|
||||||
).classes("ui-btn-primary")
|
).props("flat")
|
||||||
ui.button(
|
ui.button(
|
||||||
"Run Storage Reconciliation",
|
"Run Storage Reconciliation",
|
||||||
icon="rule",
|
icon="rule",
|
||||||
@@ -652,14 +660,7 @@ def register_page(*, settings: Settings) -> None: # noqa: PLR0915
|
|||||||
ui.button("View Log", icon="visibility", on_click=view_log).props("flat")
|
ui.button("View Log", icon="visibility", on_click=view_log).props("flat")
|
||||||
ui.button("Download Log", icon="download", on_click=download_log).props("flat")
|
ui.button("Download Log", icon="download", on_click=download_log).props("flat")
|
||||||
|
|
||||||
active_statuses = frozenset(
|
_schedule_maintenance_refresh_if_active(runs=runs, refresh=render_maintenance.refresh)
|
||||||
{
|
|
||||||
MaintenanceRunStatus.QUEUED,
|
|
||||||
MaintenanceRunStatus.PROCESSING,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
if any(run.status in active_statuses for run in runs):
|
|
||||||
ui.timer(4.0, render_maintenance.refresh, once=True)
|
|
||||||
|
|
||||||
@ui.refreshable
|
@ui.refreshable
|
||||||
async def render_runtime_settings() -> None:
|
async def render_runtime_settings() -> None:
|
||||||
@@ -842,6 +843,76 @@ async def _enqueue_maintenance_run(
|
|||||||
refresh()
|
refresh()
|
||||||
|
|
||||||
|
|
||||||
|
async def _render_gedcom_import_controls(
|
||||||
|
*,
|
||||||
|
maintenance: MaintenanceService,
|
||||||
|
request: Request,
|
||||||
|
refresh,
|
||||||
|
) -> None:
|
||||||
|
ui.label("GEDCOM Import").classes("text-sm font-semibold")
|
||||||
|
ui.label("Upload a FamilySearch GEDCOM export (.ged), then queue a GEDCOM import run.").classes(
|
||||||
|
"text-xs ui-text-muted mb-2"
|
||||||
|
)
|
||||||
|
|
||||||
|
async def on_gedcom_upload(event: events.UploadEventArguments) -> None:
|
||||||
|
payload = await event.file.read()
|
||||||
|
upload_outcome = await run_ui_action(
|
||||||
|
operation="settings.maintenance.gedcom.upload",
|
||||||
|
title="GEDCOM upload failed",
|
||||||
|
action=lambda: maintenance.store_gedcom_upload(
|
||||||
|
filename=event.file.name,
|
||||||
|
file_bytes=payload,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if not upload_outcome.ok or upload_outcome.value is None:
|
||||||
|
return
|
||||||
|
ui.notify(f"Uploaded GEDCOM file: {event.file.name}", type="positive")
|
||||||
|
refresh()
|
||||||
|
|
||||||
|
render_upload_picker(
|
||||||
|
on_upload=on_gedcom_upload,
|
||||||
|
label="Upload GEDCOM file",
|
||||||
|
extensions=GEDCOM_UPLOAD_EXTENSIONS,
|
||||||
|
)
|
||||||
|
|
||||||
|
latest_gedcom_outcome = await run_ui_action(
|
||||||
|
operation="settings.maintenance.gedcom.latest",
|
||||||
|
title="GEDCOM uploads unavailable",
|
||||||
|
action=lambda: _latest_gedcom_upload_path(maintenance=maintenance),
|
||||||
|
)
|
||||||
|
latest_gedcom_path = latest_gedcom_outcome.value if latest_gedcom_outcome.ok else None
|
||||||
|
ui.label(
|
||||||
|
f"Latest GEDCOM upload: {latest_gedcom_path}" if latest_gedcom_path else "Latest GEDCOM upload: none"
|
||||||
|
).classes("text-xs ui-text-muted mb-2")
|
||||||
|
|
||||||
|
with ui.row().classes("w-full items-center gap-2"):
|
||||||
|
ui.button(
|
||||||
|
"Run GEDCOM Import",
|
||||||
|
icon="upload_file",
|
||||||
|
on_click=lambda: _enqueue_maintenance_run(
|
||||||
|
maintenance=maintenance,
|
||||||
|
job_type=MaintenanceJobType.GEDCOM_IMPORT,
|
||||||
|
request=request,
|
||||||
|
refresh=refresh,
|
||||||
|
),
|
||||||
|
).classes("ui-btn-primary")
|
||||||
|
|
||||||
|
|
||||||
|
async def _latest_gedcom_upload_path(*, maintenance: MaintenanceService) -> str | None:
|
||||||
|
return maintenance.latest_gedcom_upload_path()
|
||||||
|
|
||||||
|
|
||||||
|
def _schedule_maintenance_refresh_if_active(*, runs: list[Any], refresh) -> None:
|
||||||
|
active_statuses = frozenset(
|
||||||
|
{
|
||||||
|
MaintenanceRunStatus.QUEUED,
|
||||||
|
MaintenanceRunStatus.PROCESSING,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if any(run.status in active_statuses for run in runs):
|
||||||
|
ui.timer(4.0, refresh, once=True)
|
||||||
|
|
||||||
|
|
||||||
def _format_timestamp(value: datetime | None) -> str:
|
def _format_timestamp(value: datetime | None) -> str:
|
||||||
if value is None:
|
if value is None:
|
||||||
return "-"
|
return "-"
|
||||||
|
|||||||
+42
@@ -0,0 +1,42 @@
|
|||||||
|
0 HEAD
|
||||||
|
1 SOUR getmyancestors
|
||||||
|
1 GEDC
|
||||||
|
2 VERS 5.5.1
|
||||||
|
1 CHAR UTF-8
|
||||||
|
0 @I1@ INDI
|
||||||
|
1 NAME John /Doe/
|
||||||
|
1 REFN KWC1-ABC
|
||||||
|
2 TYPE FSFTID
|
||||||
|
1 BIRT
|
||||||
|
2 DATE 1 JAN 1900
|
||||||
|
2 PLAC Springfield, Illinois
|
||||||
|
2 SOUR Birth Register
|
||||||
|
3 PAGE p. 12
|
||||||
|
1 DEAT
|
||||||
|
2 DATE 5 FEB 1970
|
||||||
|
2 PLAC Shelbyville, Illinois
|
||||||
|
2 SOUR Death Certificate
|
||||||
|
3 TEXT County archive
|
||||||
|
0 @I2@ INDI
|
||||||
|
1 NAME Jane /Smith/
|
||||||
|
1 _FSFTID LMN2-XYZ
|
||||||
|
1 BIRT
|
||||||
|
2 DATE 12 MAR 1905
|
||||||
|
2 PLAC Capital City, Illinois
|
||||||
|
0 @I3@ INDI
|
||||||
|
1 NAME Child /Doe/
|
||||||
|
1 REFN CHD3-123
|
||||||
|
2 TYPE FSFTID
|
||||||
|
0 @F1@ FAM
|
||||||
|
1 REFN FAM-001
|
||||||
|
2 TYPE FSFTID
|
||||||
|
1 HUSB @I1@
|
||||||
|
1 WIFE @I2@
|
||||||
|
1 CHIL @I3@
|
||||||
|
2 PEDI adopted
|
||||||
|
1 MARR
|
||||||
|
2 DATE 4 APR 1925
|
||||||
|
2 PLAC Springfield, Illinois
|
||||||
|
2 SOUR Marriage License
|
||||||
|
3 PAGE Book 9
|
||||||
|
0 TRLR
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import date
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlmodel import select
|
||||||
|
|
||||||
|
from transcription.db.models import GenealogyCitation
|
||||||
|
from transcription.db.models import GenealogyFamily
|
||||||
|
from transcription.db.models import GenealogyFamilyChild
|
||||||
|
from transcription.db.models import GenealogyPerson
|
||||||
|
from transcription.services.gedcom_import import import_gedcom_file
|
||||||
|
from transcription.services.gedcom_import import parse_gedcom
|
||||||
|
|
||||||
|
_FIXTURE_PATH = Path(__file__).resolve().parents[1] / "fixtures" / "gedcom" / "sample_familysearch.ged"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_gedcom_extracts_people_families_and_citations():
|
||||||
|
parsed = parse_gedcom(file_path=_FIXTURE_PATH)
|
||||||
|
|
||||||
|
assert len(parsed.people) == 3
|
||||||
|
john = next(person for person in parsed.people if person.fs_id == "KWC1-ABC")
|
||||||
|
assert john.full_name == "John Doe"
|
||||||
|
assert john.birth_date == date(1900, 1, 1)
|
||||||
|
assert john.birth_place == "Springfield, Illinois"
|
||||||
|
assert john.death_date == date(1970, 2, 5)
|
||||||
|
assert len(john.citations) == 2
|
||||||
|
assert "Birth Register" in john.citations[0].raw_citation_text
|
||||||
|
|
||||||
|
assert len(parsed.families) == 1
|
||||||
|
family = parsed.families[0]
|
||||||
|
assert family.fs_family_id == "FAM-001"
|
||||||
|
assert family.marriage_date == date(1925, 4, 4)
|
||||||
|
assert family.marriage_place == "Springfield, Illinois"
|
||||||
|
assert len(family.children) == 1
|
||||||
|
assert family.children[0].relationship_type == "adopted"
|
||||||
|
assert len(family.citations) == 1
|
||||||
|
assert "Marriage License" in family.citations[0].raw_citation_text
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_import_gedcom_file_is_idempotent(default_session_factory):
|
||||||
|
async with default_session_factory() as session:
|
||||||
|
first = await import_gedcom_file(session=session, file_path=_FIXTURE_PATH)
|
||||||
|
async with default_session_factory() as session:
|
||||||
|
second = await import_gedcom_file(session=session, file_path=_FIXTURE_PATH)
|
||||||
|
async with default_session_factory() as session:
|
||||||
|
people = (await session.exec(select(GenealogyPerson))).all()
|
||||||
|
families = (await session.exec(select(GenealogyFamily))).all()
|
||||||
|
children = (await session.exec(select(GenealogyFamilyChild))).all()
|
||||||
|
citations = (await session.exec(select(GenealogyCitation))).all()
|
||||||
|
|
||||||
|
assert first.new_people == 3
|
||||||
|
assert first.new_families == 1
|
||||||
|
assert first.family_children == 1
|
||||||
|
assert first.citations == 3
|
||||||
|
|
||||||
|
assert second.new_people == 0
|
||||||
|
assert second.updated_people == 0
|
||||||
|
assert second.new_families == 0
|
||||||
|
assert second.updated_families == 0
|
||||||
|
assert second.family_children == 1
|
||||||
|
assert second.citations == 3
|
||||||
|
|
||||||
|
assert len(people) == 3
|
||||||
|
assert len(families) == 1
|
||||||
|
assert len(children) == 1
|
||||||
|
assert len(citations) == 3
|
||||||
@@ -22,11 +22,13 @@ async def test_enqueue_and_list_runs(default_session_factory, default_settings):
|
|||||||
|
|
||||||
first = await service.enqueue_run(job_type=MaintenanceJobType.BACKUP, triggered_by="test")
|
first = await service.enqueue_run(job_type=MaintenanceJobType.BACKUP, triggered_by="test")
|
||||||
second = await service.enqueue_run(job_type=MaintenanceJobType.STORAGE_RECONCILIATION, triggered_by="test")
|
second = await service.enqueue_run(job_type=MaintenanceJobType.STORAGE_RECONCILIATION, triggered_by="test")
|
||||||
|
third = await service.enqueue_run(job_type=MaintenanceJobType.GEDCOM_IMPORT, triggered_by="test")
|
||||||
runs = await service.list_runs(limit=10)
|
runs = await service.list_runs(limit=10)
|
||||||
|
|
||||||
assert len(runs) == 2
|
assert len(runs) == 3
|
||||||
assert runs[0].id == second.id
|
assert runs[0].id == third.id
|
||||||
assert runs[1].id == first.id
|
assert runs[1].id == second.id
|
||||||
|
assert runs[2].id == first.id
|
||||||
assert runs[0].status == MaintenanceRunStatus.QUEUED
|
assert runs[0].status == MaintenanceRunStatus.QUEUED
|
||||||
|
|
||||||
|
|
||||||
@@ -72,3 +74,20 @@ async def test_list_runs_raises_when_table_is_missing(default_session_factory, d
|
|||||||
with pytest.raises(MaintenanceError) as exc:
|
with pytest.raises(MaintenanceError) as exc:
|
||||||
await service.list_runs(limit=10)
|
await service.list_runs(limit=10)
|
||||||
assert exc.value.category == ErrorCategory.INFRA_PERSISTENT
|
assert exc.value.category == ErrorCategory.INFRA_PERSISTENT
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_execute_run_dispatches_gedcom_import(default_session_factory, default_settings, monkeypatch):
|
||||||
|
service = MaintenanceService(session_factory=default_session_factory, settings=default_settings)
|
||||||
|
run = await service.enqueue_run(job_type=MaintenanceJobType.GEDCOM_IMPORT, triggered_by="test")
|
||||||
|
|
||||||
|
async def _fake_gedcom_import():
|
||||||
|
return MaintenanceExecution(
|
||||||
|
status=MaintenanceRunStatus.SUCCEEDED,
|
||||||
|
summary="GEDCOM imported",
|
||||||
|
output="ok",
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(service, "_execute_gedcom_import", _fake_gedcom_import)
|
||||||
|
execution = await service._execute_run(run)
|
||||||
|
assert execution.summary == "GEDCOM imported"
|
||||||
|
|||||||
@@ -29,5 +29,6 @@ class TestPageRegistration:
|
|||||||
assert "Prompts" in settings_response.text
|
assert "Prompts" in settings_response.text
|
||||||
assert "Home Page Text" in settings_response.text
|
assert "Home Page Text" in settings_response.text
|
||||||
assert "Maintenance" in settings_response.text
|
assert "Maintenance" in settings_response.text
|
||||||
|
assert "Run GEDCOM Import" in settings_response.text
|
||||||
assert "Other settings not shown here" in settings_response.text
|
assert "Other settings not shown here" in settings_response.text
|
||||||
assert "README.md" not in settings_response.text
|
assert "README.md" not in settings_response.text
|
||||||
|
|||||||
@@ -1860,6 +1860,15 @@ wheels = [
|
|||||||
{ url = "https://files.pythonhosted.org/packages/5b/96/82f6328e410515fab21d5602ba35b9377a47b5a141a0c1f9efa00ce21eb4/python_engineio-4.13.3-py3-none-any.whl", hash = "sha256:1f60ecaf1358190f0e26c48c578a60428dc02a8f1295bc3dbf53d1b31116821f", size = 59993 },
|
{ url = "https://files.pythonhosted.org/packages/5b/96/82f6328e410515fab21d5602ba35b9377a47b5a141a0c1f9efa00ce21eb4/python_engineio-4.13.3-py3-none-any.whl", hash = "sha256:1f60ecaf1358190f0e26c48c578a60428dc02a8f1295bc3dbf53d1b31116821f", size = 59993 },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "python-gedcom"
|
||||||
|
version = "1.1.0"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/2b/79/f2cb205a79b4995df74f86a0f7b3b278e4910b7a337c6fa26c5f88067117/python_gedcom-1.1.0.tar.gz", hash = "sha256:18a6d8b1832e9c0e912cbfa7958941544a88d1447e753d3aaa1f9f3c5dab1e38", size = 168754 }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/30/41/90707f85cf2585f1240a773793c0deff14b22452575c582cc0b8c8bde551/python_gedcom-1.1.0-py3-none-any.whl", hash = "sha256:98ce4f4da3949727c235e0adbe6b96360fd4840a6ccf3c8b20cab3407041f65f", size = 32730 },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "python-multipart"
|
name = "python-multipart"
|
||||||
version = "0.0.32"
|
version = "0.0.32"
|
||||||
@@ -2169,6 +2178,7 @@ dependencies = [
|
|||||||
{ name = "psycopg2-binary" },
|
{ name = "psycopg2-binary" },
|
||||||
{ name = "pydantic" },
|
{ name = "pydantic" },
|
||||||
{ name = "pydantic-settings" },
|
{ name = "pydantic-settings" },
|
||||||
|
{ name = "python-gedcom" },
|
||||||
{ name = "sqlmodel" },
|
{ name = "sqlmodel" },
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -2196,6 +2206,7 @@ requires-dist = [
|
|||||||
{ name = "psycopg2-binary", specifier = ">=2.9.12" },
|
{ name = "psycopg2-binary", specifier = ">=2.9.12" },
|
||||||
{ name = "pydantic", specifier = ">=2.13.4" },
|
{ name = "pydantic", specifier = ">=2.13.4" },
|
||||||
{ name = "pydantic-settings", specifier = ">=2.9.1" },
|
{ name = "pydantic-settings", specifier = ">=2.9.1" },
|
||||||
|
{ name = "python-gedcom", specifier = ">=1.1.0" },
|
||||||
{ name = "sqlmodel", specifier = ">=0.0.25" },
|
{ name = "sqlmodel", specifier = ">=0.0.25" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user