From eb1b60f67d97d7e2efe9ccb592b70be13928aa1f Mon Sep 17 00:00:00 2001 From: glass beadian Date: Sun, 11 Oct 2026 07:55:01 +0000 Subject: [PATCH] Add video-AI architecture research (transcription, diarization, faces, GDPR) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Research: use PeerTube native transcription (core since 6.2, whisper-ctranslate2, remote runners on GPU nodes; large-v3-turbo MIT via model_path); pyannote community-1 diarization (CC-BY-4.0, clean); WhisperX fusion. Face recognition license audit: NO accurate open FR model has clean weights (InsightFace/SCRFD non-commercial, yolo-face GPL, AdaFace unstated) — posture is detection+tracking+blur by default; identity matching opt-in, human-review-gated, collective decision + DPIA required (Art. 9 biometric data; automated FR over non-consenting third parties effectively unlawful). DB DDL sketch: persons/person_face_embeddings/video_face_detections/ transcript_segments/video_speaker_person/identity_audit with review gating, visibility tiers, erasure workflow. --- .../2026-10-11-video-ai-architecture.md | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 docs/design/2026-10-11-video-ai-architecture.md diff --git a/docs/design/2026-10-11-video-ai-architecture.md b/docs/design/2026-10-11-video-ai-architecture.md new file mode 100644 index 0000000..6c9b94a --- /dev/null +++ b/docs/design/2026-10-11-video-ai-architecture.md @@ -0,0 +1,53 @@ +Report saved to `research/2026-10-11-av-transcription-diarization-faces-gdpr.md`. Full findings below — everything marked "verified" was fetched from primary sources today (2026-10-11); everything else is explicitly flagged. + +# Decision-ready summary + +## (a) Transcription — use PeerTube's, don't build one + +**Verified from PeerTube v8.3.1 source today** (current release, 2026-09-28): +- Transcription is **core since 6.2** (`video_transcription:` in `config/default.yaml`), default **off**. Engines: **`whisper-ctranslate2`** (default) or `openai-whisper`; models `tiny|base|small|medium|large-v2|large-v3` — **no `turbo` in the list**, but `model_path:` accepts a CTranslate2 model directory, so drop in `faster-whisper-large-v3-turbo` (MIT) via path. Set `remote_runners.enabled: true` and run the official `peertube-runner` on a Linux 3090 node — same runner contract as your transcode pool. +- Subtitles land as **VTT** in the caption system (storage prefix `captions/` on R2 is supported — recent changelog fixes caption CORS for object storage). API verified in `server/core/controllers/api/videos/captions.ts`: `GET /api/v1/videos/{id}/captions`, `PUT/DELETE .../captions/{lang}`, `POST .../captions/generate`. +- **No webhooks in PeerTube (verified — zero hits in repo tree)** → your NestJS api should poll captions-on-publish, or write a tiny plugin (plugin `Hooks` are imported by the caption controller; specific transcription hook names unverified — flagged). +- Model licensing is clean: **whisper-large-v3 = Apache-2.0, large-v3-turbo = MIT, faster-whisper-large-v3 = MIT** (HF API today). distil-whisper is **English-only → excluded** for ES (card check: marked UNVERIFIED-MEMORY, but nothing suggests multilingual). +- **Spanish numbers honestly missing**: current HF cards no longer publish FLEURS/CV per-language tables (checked both — absent). From memory: FLEURS-es ~5–7% for large-v3; **expect 15–35% on street audio** (wind/PA/chants). Whisper cards explicitly warn of *"repetitive texts"* (hallucinations) — a real risk on chants/music; VAD + human spot-check needed. +- VRAM (faster-whisper README, measured on a 3070 Ti): large-v3 int8 batched = **4500 MB, 5× realtime**. Turbo ≈ half that (ESTIMATE). whisper.cpp v1.9.5 (2026-10-06) verified as CPU/Windows fallback with `-ovtt` VTT output, CUDA/Vulkan, Q5_0. +- Fallback serving if you outgrow the runner: **whisper-asr-webservice v1.10.0** (faster_whisper/whisperx engines, VTT/SRT/word timestamps — README-verified; ideal for media-plane blobs that never enter PeerTube) or **Speaches** (MIT, but v0.9.0-rc.3 — pre-1.0). + +## (b) Diarization — pyannote is fine for this use; the gate is not a license + +- **`pyannote/speaker-diarization-community-1`: license CC-BY-4.0, gated `auto`** (HF API today); 3.1/segmentation-3.0 are **MIT, gated `auto`**; pyannote.audio code is **MIT** (CNRS). The gate = HF account + accept conditions + read token (card fetch without auth returned 401 — verified empirically). **No non-commercial clause anywhere** → OK for you, add CC attribution to a credits page. +- Quality (official DER table, community-1 vs 3.1): AMI 17.0/18.8, DIHARD 3 20.2/21.4, **MSDWild 22.8/25.4, Ego4D 46.8/51.2** — the two "wild" benchmarks are the honest proxies for street footage; **no Spanish benchmark exists** (flagged). +- **WhisperX** (BSD-2, v3.8.6) fuses whisper + word alignment + diarization; its README now requires accepting **`speaker-diarization-community-1`** + `--hf_token`. Spanish alignment model = `VOXPOPULI_ASR_BASE_10K_ES` (verified in `alignment.py`; that model's license unverified — flagged). +- **NeMo Sortformer weights are CC-BY-NC-4.0 → excluded** (non-commercial; NeMo toolkit itself is Apache-2.0). diart (MIT) is streaming-oriented, maintenance unclear. +- Speaker→name UX: `speaker_key` per video + one `video_speaker_person` row; confirmed mappings + face-track co-occurrence give cross-video suggestions (label once per speaker). + +## (c) Faces — the license audit is the headline + +| Component | Code | **Weights** | Verdict | +|---|---|---|---| +| InsightFace buffalo_l / ArcFace-w600k | MIT | **NON-COMMERCIAL** — README: models *"available for non-commercial research purposes only"*, applies to pip auto-download too; **2025-11-24 update: contact recognition-oss-pack@insightface.ai for licensing** (both quoted from README today) | ❌ for production index | +| SCRFD detector weights | MIT code | same non-commercial policy | ❌ if strict / ⚠️ pragmatic | +| AdaFace | MIT | weights on Google Drive, **no license stated**; WebFace4M/MS1MV2-lineage | ⚠️ gray zone | +| deepface | MIT | explicitly **inherits** wrapped-model licenses (README quote) — launders nothing | ⚠️ | +| MediaPipe (BlazeFace detect) | Apache-2.0 | per-file license not clearly stated (flag) | ✅ detection; **no recognition model exists in MediaPipe** | +| yolov8-face / yolov5-face | **GPL-3.0** | — | ❌ copyleft hazard | + +**No accurate FR model in the open ecosystem has clean weights today.** Recommended posture: **detection+tracking+blur by default; identity matching opt-in, human-review-gated, collective decision recorded in writing** (non-commercial terms bar *commercialization* — whether a self-hosted activist tool qualifies is a legal question, not a technical one). Quality limits on your footage: masks (NIST IR 8331 PDF confirmed live at nvlpubs; the "5×–50× more errors" figure = UNVERIFIED-MEMORY, don't quote yet), motion blur, low light, side profiles — these push embeddings into the band where clustering **silently merges the wrong person**, which is your most dangerous failure mode (§GDPR). +Pipeline: PySceneDetect (0.7.1, BSD-3) → 1–2 keyframes/shot capped at ~2 fps → detect → track → 1 embedding/track → HDBSCAN/Chinese-Whispers offline → incremental assignment vs reviewed centroids (≥τ auto, τ-band → review queue). Cosine threshold ~0.4–0.5 same-person = UNVERIFIED-MEMORY; calibrate on your data. Store centroids in pgvector (512-d fits fine) — reuse the existing pattern. + +## (d) Identity model + GDPR (the part that gates everything) + +- DDL in the report: `persons` (status `auto|member_confirmed|hidden|erased`, visibility tiers), `person_face_embeddings` (reviewed centroids only), `video_face_detections` (bbox + timestamps + `assignment` state machine + **transient** embedding column wiped by retention), `transcript_segments`, `video_speaker_person`, `identity_audit` (append-only; merge/undo/erasure all reversible). Label once → `UPDATE ... WHERE match_sim >= τ` propagates; τ-band goes to `pending_review`; nothing `auto` ever renders in UI/API. +- **Verified legal text** (EUR-Lex today): Art. 4(14) definition; **Art. 9(1) prohibits processing biometric data "for the purpose of uniquely identifying"** absent a 9(2) condition — and for bystanders/police, none of (a) explicit consent, (e) "manifestly made public", (f) vital interests, (g) substantial public interest (requires specific law) applies cleanly to an identity index → **an automated FR gallery over non-consenting third parties is effectively unlawful by default; detection+blur that doesn't uniquely identify arguably isn't Art. 9 data at all**. Art. 35(3)(b) makes the **DPIA mandatory** before phase 3. Design mitigations mapped: minimization (coords-not-crops, transient embeddings, pinned models/thresholds), human-review gating, visibility tiers enforced server-side, per-person erasure workflow (index-level; originals stay), audit trail, "high-risk person" flag for police-adjacent IDs, Tailscale-only exposure. +- **Could not verify this session (flagged):** EDPB Guidelines 05/2022 (all URL attempts 404'd), AEPD guidance + Zaragoza enforcement case (**aepd.es returned 500/503 all day**), NIST mask headline figure. Re-verify before citing any of them. + +## VRAM / cost (3090) +TEI+bge-m3 ~2–3 GB persistent + turbo int8 ~3 GB burst + pyannote ~2–4 GB + faces ~1–2 GB ≈ **11–13 GB concurrent worst case — fits 24 GB, but serialize jobs anyway** (queue, reuse pool-manager pressure signals; keep NVENC transcode floors). Throughput ESTIMATES: turbo ≈ 20–40× RT, large-v3 ≈ 8–12× RT; full pipeline ≈ **5–25 GPU-min per video-hour**; 500-h backfill ≈ 2–4 GPU-days for transcription alone. + +## Top unknowns before you commit +1. One smoke test: does PeerTube's **runner** transcription path pick up CUDA for whisper-ctranslate2, and does `model_path` accept a turbo CT2 dir there? +2. Internal ES/EN eval set (20–30 real clips) — no published benchmark matches protest audio. +3. Collective decision + legal advice on FR weights (non-commercial) and on Art. 9 posture **before any phase-3 code**. +4. WhisperX VoxPopuli-ES alignment weights license; EDPB/AEPD doc URLs (fetch failed today). + +Full details, all quotes, DDL, and per-claim source URLs are in the saved report.