T-HUB/docs/design/2026-10-11-video-ai-architecture.md

Report saved to `research/2026-10-11-av-transcription-diarization-faces-gdpr.md`. Full findings below — everything marked "verified" was fetched from primary sources today (2026-10-11); everything else is explicitly flagged.

# Decision-ready summary

## (a) Transcription — use PeerTube's, don't build one

**Verified from PeerTube v8.3.1 source today** (current release, 2026-09-28):
- Transcription is **core since 6.2** (`video_transcription:` in `config/default.yaml`), default **off**. Engines: **`whisper-ctranslate2`** (default) or `openai-whisper`; models `tiny|base|small|medium|large-v2|large-v3` — **no `turbo` in the list**, but `model_path:` accepts a CTranslate2 model directory, so drop in `faster-whisper-large-v3-turbo` (MIT) via path. Set `remote_runners.enabled: true` and run the official `peertube-runner` on a Linux 3090 node — same runner contract as your transcode pool.
- Subtitles land as **VTT** in the caption system (storage prefix `captions/` on R2 is supported — recent changelog fixes caption CORS for object storage). API verified in `server/core/controllers/api/videos/captions.ts`: `GET /api/v1/videos/{id}/captions`, `PUT/DELETE .../captions/{lang}`, `POST .../captions/generate`.
- **No webhooks in PeerTube (verified — zero hits in repo tree)** → your NestJS api should poll captions-on-publish, or write a tiny plugin (plugin `Hooks` are imported by the caption controller; specific transcription hook names unverified — flagged).
- Model licensing is clean: **whisper-large-v3 = Apache-2.0, large-v3-turbo = MIT, faster-whisper-large-v3 = MIT** (HF API today). distil-whisper is **English-only → excluded** for ES (card check: marked UNVERIFIED-MEMORY, but nothing suggests multilingual).
- **Spanish numbers honestly missing**: current HF cards no longer publish FLEURS/CV per-language tables (checked both — absent). From memory: FLEURS-es ~5–7% for large-v3; **expect 15–35% on street audio** (wind/PA/chants). Whisper cards explicitly warn of *"repetitive texts"* (hallucinations) — a real risk on chants/music; VAD + human spot-check needed.
- VRAM (faster-whisper README, measured on a 3070 Ti): large-v3 int8 batched = **4500 MB, 5× realtime**. Turbo ≈ half that (ESTIMATE). whisper.cpp v1.9.5 (2026-10-06) verified as CPU/Windows fallback with `-ovtt` VTT output, CUDA/Vulkan, Q5_0.
- Fallback serving if you outgrow the runner: **whisper-asr-webservice v1.10.0** (faster_whisper/whisperx engines, VTT/SRT/word timestamps — README-verified; ideal for media-plane blobs that never enter PeerTube) or **Speaches** (MIT, but v0.9.0-rc.3 — pre-1.0).

## (b) Diarization — pyannote is fine for this use; the gate is not a license

- **`pyannote/speaker-diarization-community-1`: license CC-BY-4.0, gated `auto`** (HF API today); 3.1/segmentation-3.0 are **MIT, gated `auto`**; pyannote.audio code is **MIT** (CNRS). The gate = HF account + accept conditions + read token (card fetch without auth returned 401 — verified empirically). **No non-commercial clause anywhere** → OK for you, add CC attribution to a credits page.
- Quality (official DER table, community-1 vs 3.1): AMI 17.0/18.8, DIHARD 3 20.2/21.4, **MSDWild 22.8/25.4, Ego4D 46.8/51.2** — the two "wild" benchmarks are the honest proxies for street footage; **no Spanish benchmark exists** (flagged).
- **WhisperX** (BSD-2, v3.8.6) fuses whisper + word alignment + diarization; its README now requires accepting **`speaker-diarization-community-1`** + `--hf_token`. Spanish alignment model = `VOXPOPULI_ASR_BASE_10K_ES` (verified in `alignment.py`; that model's license unverified — flagged).
- **NeMo Sortformer weights are CC-BY-NC-4.0 → excluded** (non-commercial; NeMo toolkit itself is Apache-2.0). diart (MIT) is streaming-oriented, maintenance unclear.
- Speaker→name UX: `speaker_key` per video + one `video_speaker_person` row; confirmed mappings + face-track co-occurrence give cross-video suggestions (label once per speaker).

## (c) Faces — the license audit is the headline

| Component | Code | **Weights** | Verdict |
|---|---|---|---|
| InsightFace buffalo_l / ArcFace-w600k | MIT | **NON-COMMERCIAL** — README: models *"available for non-commercial research purposes only"*, applies to pip auto-download too; **2025-11-24 update: contact recognition-oss-pack@insightface.ai for licensing** (both quoted from README today) | ❌ for production index |
| SCRFD detector weights | MIT code | same non-commercial policy | ❌ if strict / ⚠️ pragmatic |
| AdaFace | MIT | weights on Google Drive, **no license stated**; WebFace4M/MS1MV2-lineage | ⚠️ gray zone |
| deepface | MIT | explicitly **inherits** wrapped-model licenses (README quote) — launders nothing | ⚠️ |
| MediaPipe (BlazeFace detect) | Apache-2.0 | per-file license not clearly stated (flag) | ✅ detection; **no recognition model exists in MediaPipe** |
| yolov8-face / yolov5-face | **GPL-3.0** | — | ❌ copyleft hazard |

**No accurate FR model in the open ecosystem has clean weights today.** Recommended posture: **detection+tracking+blur by default; identity matching opt-in, human-review-gated, collective decision recorded in writing** (non-commercial terms bar *commercialization* — whether a self-hosted activist tool qualifies is a legal question, not a technical one). Quality limits on your footage: masks (NIST IR 8331 PDF confirmed live at nvlpubs; the "5×–50× more errors" figure = UNVERIFIED-MEMORY, don't quote yet), motion blur, low light, side profiles — these push embeddings into the band where clustering **silently merges the wrong person**, which is your most dangerous failure mode (§GDPR).
Pipeline: PySceneDetect (0.7.1, BSD-3) → 1–2 keyframes/shot capped at ~2 fps → detect → track → 1 embedding/track → HDBSCAN/Chinese-Whispers offline → incremental assignment vs reviewed centroids (≥τ auto, τ-band → review queue). Cosine threshold ~0.4–0.5 same-person = UNVERIFIED-MEMORY; calibrate on your data. Store centroids in pgvector (512-d fits fine) — reuse the existing pattern.

## (d) Identity model + GDPR (the part that gates everything)

- DDL in the report: `persons` (status `auto|member_confirmed|hidden|erased`, visibility tiers), `person_face_embeddings` (reviewed centroids only), `video_face_detections` (bbox + timestamps + `assignment` state machine + **transient** embedding column wiped by retention), `transcript_segments`, `video_speaker_person`, `identity_audit` (append-only; merge/undo/erasure all reversible). Label once → `UPDATE ... WHERE match_sim >= τ` propagates; τ-band goes to `pending_review`; nothing `auto` ever renders in UI/API.
- **Verified legal text** (EUR-Lex today): Art. 4(14) definition; **Art. 9(1) prohibits processing biometric data "for the purpose of uniquely identifying"** absent a 9(2) condition — and for bystanders/police, none of (a) explicit consent, (e) "manifestly made public", (f) vital interests, (g) substantial public interest (requires specific law) applies cleanly to an identity index → **an automated FR gallery over non-consenting third parties is effectively unlawful by default; detection+blur that doesn't uniquely identify arguably isn't Art. 9 data at all**. Art. 35(3)(b) makes the **DPIA mandatory** before phase 3. Design mitigations mapped: minimization (coords-not-crops, transient embeddings, pinned models/thresholds), human-review gating, visibility tiers enforced server-side, per-person erasure workflow (index-level; originals stay), audit trail, "high-risk person" flag for police-adjacent IDs, Tailscale-only exposure.
- **Could not verify this session (flagged):** EDPB Guidelines 05/2022 (all URL attempts 404'd), AEPD guidance + Zaragoza enforcement case (**aepd.es returned 500/503 all day**), NIST mask headline figure. Re-verify before citing any of them.

## VRAM / cost (3090)
TEI+bge-m3 ~2–3 GB persistent + turbo int8 ~3 GB burst + pyannote ~2–4 GB + faces ~1–2 GB ≈ **11–13 GB concurrent worst case — fits 24 GB, but serialize jobs anyway** (queue, reuse pool-manager pressure signals; keep NVENC transcode floors). Throughput ESTIMATES: turbo ≈ 20–40× RT, large-v3 ≈ 8–12× RT; full pipeline ≈ **5–25 GPU-min per video-hour**; 500-h backfill ≈ 2–4 GPU-days for transcription alone.

## Top unknowns before you commit
1. One smoke test: does PeerTube's **runner** transcription path pick up CUDA for whisper-ctranslate2, and does `model_path` accept a turbo CT2 dir there?
2. Internal ES/EN eval set (20–30 real clips) — no published benchmark matches protest audio.
3. Collective decision + legal advice on FR weights (non-commercial) and on Art. 9 posture **before any phase-3 code**.
4. WhisperX VoxPopuli-ES alignment weights license; EDPB/AEPD doc URLs (fetch failed today).

Full details, all quotes, DDL, and per-claim source URLs are in the saved report.