generated from john/python-template
V6.0 Phase 3 - Add cloudflare tunnel. Remote access to app via the tunnel now working.
Quality Gate / gate (push) Failing after 49s
Quality Gate / gate (push) Failing after 49s
This commit is contained in:
@@ -50,4 +50,5 @@ POSTGRES_USER=transcription
|
|||||||
POSTGRES_PASSWORD=replace-with-strong-password
|
POSTGRES_PASSWORD=replace-with-strong-password
|
||||||
|
|
||||||
# --- cloudflare tunnel ---
|
# --- cloudflare tunnel ---
|
||||||
|
# Required for token-based tunnel startup.
|
||||||
CLOUDFLARE_TUNNEL_TOKEN=replace-with-cloudflare-tunnel-token
|
CLOUDFLARE_TUNNEL_TOKEN=replace-with-cloudflare-tunnel-token
|
||||||
|
|||||||
@@ -27,3 +27,6 @@ data/photos/*
|
|||||||
data-migration-test/*
|
data-migration-test/*
|
||||||
.migration-bundle/*
|
.migration-bundle/*
|
||||||
|
|
||||||
|
# Cloudflare tunnel local runtime files
|
||||||
|
deploy/cloudflared/config.yml
|
||||||
|
deploy/cloudflared/credentials.json
|
||||||
|
|||||||
@@ -137,7 +137,13 @@ Services:
|
|||||||
- `app`: FastAPI + NiceGUI runtime (`RUN_EMBEDDED_WORKER=false`)
|
- `app`: FastAPI + NiceGUI runtime (`RUN_EMBEDDED_WORKER=false`)
|
||||||
- `worker`: standalone queue processor (`python -m transcription.worker_service`)
|
- `worker`: standalone queue processor (`python -m transcription.worker_service`)
|
||||||
- `postgres`: primary datastore
|
- `postgres`: primary datastore
|
||||||
- `cloudflared`: tunnel client using `CLOUDFLARE_TUNNEL_TOKEN`
|
- `cloudflared`: tunnel client using mounted ingress config + `CLOUDFLARE_TUNNEL_TOKEN`
|
||||||
|
|
||||||
|
Cloudflare setup files:
|
||||||
|
|
||||||
|
1. `copy deploy\cloudflared\config.yml.example deploy\cloudflared\config.yml`
|
||||||
|
2. set `CLOUDFLARE_TUNNEL_TOKEN` in `.env.production`
|
||||||
|
3. update ingress hostnames in `deploy\cloudflared\config.yml`
|
||||||
|
|
||||||
## How to navigate the GUI
|
## How to navigate the GUI
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
ingress:
|
||||||
|
# Primary transcription app endpoint.
|
||||||
|
- hostname: transcription.example.com
|
||||||
|
service: http://app:8000
|
||||||
|
|
||||||
|
# Optional: generic remote access endpoints for other internal services.
|
||||||
|
# Replace hostnames and targets for your LAN.
|
||||||
|
- hostname: homeassistant.example.com
|
||||||
|
service: http://192.168.1.50:8123
|
||||||
|
- hostname: pihole.example.com
|
||||||
|
service: http://192.168.1.60:80
|
||||||
|
|
||||||
|
# Required catch-all.
|
||||||
|
- service: http_status:404
|
||||||
@@ -54,17 +54,17 @@ services:
|
|||||||
timeout: 5s
|
timeout: 5s
|
||||||
retries: 10
|
retries: 10
|
||||||
start_period: 10s
|
start_period: 10s
|
||||||
ports:
|
|
||||||
- "5432:5432"
|
|
||||||
|
|
||||||
cloudflared:
|
cloudflared:
|
||||||
image: cloudflare/cloudflared:2026.8.0
|
image: cloudflare/cloudflared:2026.8.0
|
||||||
env_file:
|
env_file:
|
||||||
- .env.production
|
- .env.production
|
||||||
command: tunnel --no-autoupdate run --token ${CLOUDFLARE_TUNNEL_TOKEN}
|
command: tunnel --no-autoupdate --config /etc/cloudflared/config.yml run --token ${CLOUDFLARE_TUNNEL_TOKEN}
|
||||||
depends_on:
|
depends_on:
|
||||||
app:
|
app:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
volumes:
|
||||||
|
- ./deploy/cloudflared/config.yml:/etc/cloudflared/config.yml:ro
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Cloudflare Tunnel and Access Setup (V6.0 Phase 3)
|
||||||
|
|
||||||
|
This guide defines the repository-supported setup for exposing app and selected LAN services through Cloudflare Tunnel with Cloudflare Access protection.
|
||||||
|
|
||||||
|
## 1. Files used by this deployment
|
||||||
|
|
||||||
|
1. `deploy/cloudflared/config.yml` (local copy from `config.yml.example`)
|
||||||
|
2. `.env.production` (`CLOUDFLARE_TUNNEL_TOKEN`)
|
||||||
|
3. `docker-compose.production.yml` (`cloudflared` service reads token + mounts config)
|
||||||
|
|
||||||
|
Do not commit `config.yml` or `.env.production`.
|
||||||
|
|
||||||
|
## 2. Configure cloudflared
|
||||||
|
|
||||||
|
1. Copy `deploy/cloudflared/config.yml.example` to `deploy/cloudflared/config.yml`.
|
||||||
|
2. Update hostname -> service mappings in `ingress`.
|
||||||
|
3. Keep the final catch-all ingress `http_status:404`.
|
||||||
|
4. Set `CLOUDFLARE_TUNNEL_TOKEN` in `.env.production`.
|
||||||
|
|
||||||
|
Example app route:
|
||||||
|
|
||||||
|
- `transcription.example.com` -> `http://app:8000`
|
||||||
|
|
||||||
|
Optional generic remote-access routes:
|
||||||
|
|
||||||
|
- `homeassistant.example.com` -> `http://<home-assistant-lan-ip>:8123`
|
||||||
|
- `pihole.example.com` -> `http://<pihole-lan-ip>:80`
|
||||||
|
|
||||||
|
## 3. Cloudflare Access policy baseline
|
||||||
|
|
||||||
|
Create one Access app policy per exposed hostname:
|
||||||
|
|
||||||
|
1. Include: your allowed identities/groups only.
|
||||||
|
2. Exclude: none by default.
|
||||||
|
3. Require: identity provider login (and MFA if available).
|
||||||
|
|
||||||
|
Recommended baseline:
|
||||||
|
|
||||||
|
- App endpoint (`transcription.*`): your admin identity set.
|
||||||
|
- Other internal endpoints (`homeassistant.*`, `pihole.*`, etc.): explicit least-privilege groups.
|
||||||
|
|
||||||
|
## 4. Startup
|
||||||
|
|
||||||
|
Start production stack:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.production.yml up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
Validate tunnel container:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.production.yml logs cloudflared
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Security notes
|
||||||
|
|
||||||
|
- Keep `postgres` and other internal-only services off public hostnames unless required.
|
||||||
|
- Use distinct hostnames per service; avoid path-based multiplexing for unrelated admin surfaces.
|
||||||
|
- Rotate `CLOUDFLARE_TUNNEL_TOKEN` and Access policy memberships on a regular schedule.
|
||||||
@@ -25,6 +25,7 @@ This runbook is the operational checklist for releasing and monitoring the trans
|
|||||||
- `/healthz` responds `200`
|
- `/healthz` responds `200`
|
||||||
- if `RUN_EMBEDDED_WORKER=true`, `worker.state` is `running`
|
- if `RUN_EMBEDDED_WORKER=true`, `worker.state` is `running`
|
||||||
- if `RUN_EMBEDDED_WORKER=false`, validate `worker` container is healthy/running in Compose
|
- if `RUN_EMBEDDED_WORKER=false`, validate `worker` container is healthy/running in Compose
|
||||||
|
- validate `cloudflared` logs show active tunnel routes and no ingress errors
|
||||||
3. Execute one smoke workflow:
|
3. Execute one smoke workflow:
|
||||||
- create a document/job with at least one source
|
- create a document/job with at least one source
|
||||||
- verify terminal job outcome updates
|
- verify terminal job outcome updates
|
||||||
@@ -86,6 +87,13 @@ This runbook is the operational checklist for releasing and monitoring the trans
|
|||||||
2. Confirm failures are category-aligned (`external`/`timeout`/`internal`).
|
2. Confirm failures are category-aligned (`external`/`timeout`/`internal`).
|
||||||
3. Triage whether issue is source quality, provider, or runtime regression.
|
3. Triage whether issue is source quality, provider, or runtime regression.
|
||||||
|
|
||||||
|
### Cloudflare ingress/access failure
|
||||||
|
|
||||||
|
1. Check `cloudflared` container logs for ingress parse, DNS, or auth failures.
|
||||||
|
2. Confirm `deploy/cloudflared/config.yml` tunnel UUID and hostname mappings are correct.
|
||||||
|
3. Confirm `deploy/cloudflared/credentials.json` matches the tunnel configured in Cloudflare.
|
||||||
|
4. Confirm Cloudflare Access app policy includes the intended identity/group for that hostname.
|
||||||
|
|
||||||
## 6. Dependency upgrade policy
|
## 6. Dependency upgrade policy
|
||||||
|
|
||||||
Dependencies are declared in `pyproject.toml` and resolved through the committed
|
Dependencies are declared in `pyproject.toml` and resolved through the committed
|
||||||
|
|||||||
@@ -49,6 +49,11 @@ Implemented workflow references:
|
|||||||
2. Apply Cloudflare Access policies for identity-gated remote access.
|
2. Apply Cloudflare Access policies for identity-gated remote access.
|
||||||
3. Keep non-required administrative/internal surfaces LAN-only unless explicitly approved.
|
3. Keep non-required administrative/internal surfaces LAN-only unless explicitly approved.
|
||||||
|
|
||||||
|
Implemented workflow references:
|
||||||
|
|
||||||
|
- `deploy/cloudflared/config.yml.example`
|
||||||
|
- `docs/cloudflare_tunnel_access.md`
|
||||||
|
|
||||||
## 3.4 Backup, restore, rollback
|
## 3.4 Backup, restore, rollback
|
||||||
|
|
||||||
1. Define backup schedule, retention, and artifact naming/versioning.
|
1. Define backup schedule, retention, and artifact naming/versioning.
|
||||||
|
|||||||
@@ -26,6 +26,18 @@ from .base import ServiceBase
|
|||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _utc_now_naive() -> datetime:
|
||||||
|
"""Return current UTC as naive datetime for DB timestamp columns."""
|
||||||
|
return datetime.now(UTC).replace(tzinfo=None)
|
||||||
|
|
||||||
|
|
||||||
|
def _as_naive_utc(value: datetime) -> datetime:
|
||||||
|
"""Normalize datetimes to naive UTC for DB comparisons/binds."""
|
||||||
|
if value.tzinfo is None:
|
||||||
|
return value
|
||||||
|
return value.astimezone(UTC).replace(tzinfo=None)
|
||||||
|
|
||||||
|
|
||||||
class JobDeleteBlockedError(AppError):
|
class JobDeleteBlockedError(AppError):
|
||||||
"""Raised when a job delete operation is blocked by lifecycle policy."""
|
"""Raised when a job delete operation is blocked by lifecycle policy."""
|
||||||
|
|
||||||
@@ -201,7 +213,7 @@ class JobService(ServiceBase):
|
|||||||
await self._finalize(session=_session, caller_session=session, refresh=(job,))
|
await self._finalize(session=_session, caller_session=session, refresh=(job,))
|
||||||
return job
|
return job
|
||||||
|
|
||||||
now = datetime.now(UTC)
|
now = _utc_now_naive()
|
||||||
queued_job_id = (
|
queued_job_id = (
|
||||||
select(col(Job.id))
|
select(col(Job.id))
|
||||||
.where(col(Job.status) == JobStatus.QUEUED)
|
.where(col(Job.status) == JobStatus.QUEUED)
|
||||||
@@ -239,12 +251,15 @@ class JobService(ServiceBase):
|
|||||||
``stale_before`` are considered stale and re-queued.
|
``stale_before`` are considered stale and re-queued.
|
||||||
"""
|
"""
|
||||||
async with self._session_scope(session) as _session:
|
async with self._session_scope(session) as _session:
|
||||||
query = select(Job).where(Job.status == JobStatus.PROCESSING).where(Job.date_updated < stale_before)
|
normalized_stale_before = _as_naive_utc(stale_before)
|
||||||
|
query = (
|
||||||
|
select(Job).where(Job.status == JobStatus.PROCESSING).where(Job.date_updated < normalized_stale_before)
|
||||||
|
)
|
||||||
stale_jobs = (await _session.exec(query)).all()
|
stale_jobs = (await _session.exec(query)).all()
|
||||||
if not stale_jobs:
|
if not stale_jobs:
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
now = datetime.now(UTC)
|
now = _utc_now_naive()
|
||||||
for job in stale_jobs:
|
for job in stale_jobs:
|
||||||
job.status = JobStatus.QUEUED
|
job.status = JobStatus.QUEUED
|
||||||
job.date_updated = now
|
job.date_updated = now
|
||||||
@@ -352,7 +367,7 @@ class JobService(ServiceBase):
|
|||||||
suggestion="Use resubmit for reprocessing needs, or leave the terminal job unchanged.",
|
suggestion="Use resubmit for reprocessing needs, or leave the terminal job unchanged.",
|
||||||
)
|
)
|
||||||
|
|
||||||
now = datetime.now(UTC)
|
now = _utc_now_naive()
|
||||||
job.status = JobStatus.FAILED
|
job.status = JobStatus.FAILED
|
||||||
job.date_updated = now
|
job.date_updated = now
|
||||||
|
|
||||||
@@ -398,7 +413,7 @@ class JobService(ServiceBase):
|
|||||||
suggestion="Only failed or cancelled sources can be resubmitted.",
|
suggestion="Only failed or cancelled sources can be resubmitted.",
|
||||||
)
|
)
|
||||||
|
|
||||||
now = datetime.now(UTC)
|
now = _utc_now_naive()
|
||||||
for job_source in candidates:
|
for job_source in candidates:
|
||||||
job_source.status = JobSourceStatus.PENDING
|
job_source.status = JobSourceStatus.PENDING
|
||||||
|
|
||||||
|
|||||||
@@ -20,9 +20,24 @@ from transcription.services.jobs import JobDeleteBlockedError
|
|||||||
from transcription.services.jobs import JobNotFoundError
|
from transcription.services.jobs import JobNotFoundError
|
||||||
from transcription.services.jobs import JobResubmitBlockedError
|
from transcription.services.jobs import JobResubmitBlockedError
|
||||||
from transcription.services.jobs import JobService
|
from transcription.services.jobs import JobService
|
||||||
|
from transcription.services.jobs import _as_naive_utc
|
||||||
|
from transcription.services.jobs import _utc_now_naive
|
||||||
from transcription.services.sources import SourceService
|
from transcription.services.sources import SourceService
|
||||||
|
|
||||||
|
|
||||||
|
def test_utc_now_naive_returns_naive_datetime():
|
||||||
|
now = _utc_now_naive()
|
||||||
|
assert now.tzinfo is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_as_naive_utc_normalizes_timezone_aware_datetime():
|
||||||
|
aware = datetime.now(UTC)
|
||||||
|
normalized = _as_naive_utc(aware)
|
||||||
|
|
||||||
|
assert normalized.tzinfo is None
|
||||||
|
assert normalized == aware.replace(tzinfo=None)
|
||||||
|
|
||||||
|
|
||||||
class TestJobService:
|
class TestJobService:
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_create_and_read_job(self, job_service: JobService, document_service: DocumentService):
|
async def test_create_and_read_job(self, job_service: JobService, document_service: DocumentService):
|
||||||
|
|||||||
Reference in New Issue
Block a user