hacklab/T-HUB
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Expand video-AI architecture doc to the full research report

Browse source 
281-line version with the person-identity DDL sketch, per-model license
audit table, VRAM/cost budget, GDPR Art. 9 analysis, and all primary-source
citations — replaces the 53-line executive summary.
This commit is contained in:
glass beadian 2026-10-11 08:29:30 +00:00
parent d58db63fe6
commit 777c67f7a3
1 changed files with 270 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

312
docs/design/2026-10-11-video-ai-architecture.md
View file

@ -1,53 +1,281 @@
Report saved to `research/2026-10-11-av-transcription-diarization-faces-gdpr.md`. Full findings below — everything marked "verified" was fetched from primary sources today (2026-10-11); everything else is explicitly flagged.
# A/V Intelligence Pipeline — Research Report (transcription · diarization · faces · identity · GDPR)
# Decision-ready summary
**Date:** 2026-10-11 · **Scope:** Tactical Media Broadcast Hub (PeerTube occ-tube + R2 media plane, NestJS api, pgvector, RTX 3090 fleet on Tailscale, GDPR/Spain, EN+ES, no third-party cloud APIs).
**Method:** every claim below was checked against a primary source fetched live in this session (GitHub source/release API, HuggingFace API/model cards, PyPI, EUR-Lex, nvlpubs) unless explicitly marked **UNVERIFIED-MEMORY** or **ESTIMATE**. Versions quoted are current as of today.
## (a) Transcription — use PeerTube's, don't build one
---
**Verified from PeerTube v8.3.1 source today** (current release, 2026-09-28):
- Transcription is **core since 6.2** (`video_transcription:` in `config/default.yaml`), default **off**. Engines: **`whisper-ctranslate2`** (default) or `openai-whisper`; models `tiny|base|small|medium|large-v2|large-v3` — **no `turbo` in the list**, but `model_path:` accepts a CTranslate2 model directory, so drop in `faster-whisper-large-v3-turbo` (MIT) via path. Set `remote_runners.enabled: true` and run the official `peertube-runner` on a Linux 3090 node — same runner contract as your transcode pool.
- Subtitles land as **VTT** in the caption system (storage prefix `captions/` on R2 is supported — recent changelog fixes caption CORS for object storage). API verified in `server/core/controllers/api/videos/captions.ts`: `GET /api/v1/videos/{id}/captions`, `PUT/DELETE .../captions/{lang}`, `POST .../captions/generate`.
- **No webhooks in PeerTube (verified — zero hits in repo tree)** → your NestJS api should poll captions-on-publish, or write a tiny plugin (plugin `Hooks` are imported by the caption controller; specific transcription hook names unverified — flagged).
- Model licensing is clean: **whisper-large-v3 = Apache-2.0, large-v3-turbo = MIT, faster-whisper-large-v3 = MIT** (HF API today). distil-whisper is **English-only → excluded** for ES (card check: marked UNVERIFIED-MEMORY, but nothing suggests multilingual).
- **Spanish numbers honestly missing**: current HF cards no longer publish FLEURS/CV per-language tables (checked both — absent). From memory: FLEURS-es ~5–7% for large-v3; **expect 15–35% on street audio** (wind/PA/chants). Whisper cards explicitly warn of *"repetitive texts"* (hallucinations) — a real risk on chants/music; VAD + human spot-check needed.
- VRAM (faster-whisper README, measured on a 3070 Ti): large-v3 int8 batched = **4500 MB, 5× realtime**. Turbo ≈ half that (ESTIMATE). whisper.cpp v1.9.5 (2026-10-06) verified as CPU/Windows fallback with `-ovtt` VTT output, CUDA/Vulkan, Q5_0.
- Fallback serving if you outgrow the runner: **whisper-asr-webservice v1.10.0** (faster_whisper/whisperx engines, VTT/SRT/word timestamps — README-verified; ideal for media-plane blobs that never enter PeerTube) or **Speaches** (MIT, but v0.9.0-rc.3 — pre-1.0).
## 0. Recommended architecture (TL;DR)
## (b) Diarization — pyannote is fine for this use; the gate is not a license
- **`pyannote/speaker-diarization-community-1`: license CC-BY-4.0, gated `auto`** (HF API today); 3.1/segmentation-3.0 are **MIT, gated `auto`**; pyannote.audio code is **MIT** (CNRS). The gate = HF account + accept conditions + read token (card fetch without auth returned 401 — verified empirically). **No non-commercial clause anywhere** → OK for you, add CC attribution to a credits page.
- Quality (official DER table, community-1 vs 3.1): AMI 17.0/18.8, DIHARD 3 20.2/21.4, **MSDWild 22.8/25.4, Ego4D 46.8/51.2** — the two "wild" benchmarks are the honest proxies for street footage; **no Spanish benchmark exists** (flagged).
- **WhisperX** (BSD-2, v3.8.6) fuses whisper + word alignment + diarization; its README now requires accepting **`speaker-diarization-community-1`** + `--hf_token`. Spanish alignment model = `VOXPOPULI_ASR_BASE_10K_ES` (verified in `alignment.py`; that model's license unverified — flagged).
- **NeMo Sortformer weights are CC-BY-NC-4.0 → excluded** (non-commercial; NeMo toolkit itself is Apache-2.0). diart (MIT) is streaming-oriented, maintenance unclear.
- Speaker→name UX: `speaker_key` per video + one `video_speaker_person` row; confirmed mappings + face-track co-occurrence give cross-video suggestions (label once per speaker).
## (c) Faces — the license audit is the headline
| Component | Code | **Weights** | Verdict |
| Concern | Choice | License | Why |
|---|---|---|---|
| InsightFace buffalo_l / ArcFace-w600k | MIT | **NON-COMMERCIAL** — README: models *"available for non-commercial research purposes only"*, applies to pip auto-download too; **2025-11-24 update: contact recognition-oss-pack@insightface.ai for licensing** (both quoted from README today) | ❌ for production index |
| SCRFD detector weights | MIT code | same non-commercial policy | ❌ if strict / ⚠️ pragmatic |
| AdaFace | MIT | weights on Google Drive, **no license stated**; WebFace4M/MS1MV2-lineage | ⚠️ gray zone |
| deepface | MIT | explicitly **inherits** wrapped-model licenses (README quote) — launders nothing | ⚠️ |
| MediaPipe (BlazeFace detect) | Apache-2.0 | per-file license not clearly stated (flag) | ✅ detection; **no recognition model exists in MediaPipe** |
| yolov8-face / yolov5-face | **GPL-3.0** | — | ❌ copyleft hazard |
| Transcription | **PeerTube built-in (v8.3.1), engine `whisper-ctranslate2`, model `large-v3-turbo` (int8), executed by a remote runner on the 3090** | Whisper weights MIT/Apache-2.0 (see §2) | It ships today, subtitles land in PeerTube's caption system (VTT), API retrievable, zero glue code |
| Word timestamps | faster-whisper `word_timestamps` / WhisperX alignment (phase 2) | BSD-2/MIT | VTT + speaker fusion |
| Diarization | **pyannote `speaker-diarization-community-1`** via WhisperX (or standalone) | **CC-BY-4.0** weights, MIT code, gated `auto` | Best open DER; the gate is access-control, not a license restriction |
| Face detection | **SCRFD (ONNX, via insightface pip for runtime only)** or MediaPipe BlazeFace if you want zero-contamination pipeline | weights **non-commercial** (SCRFD/buffalo) / Apache-2.0 (MediaPipe) | See §4 — this is the one genuinely dirty licensing corner; options given |
| Face recognition | **Deferred to phase 3, human-review-gated.** If run: AdaFace/ArcFace-class 512-d embeddings; store centroids, not crops | — | GDPR Art. 9 — see §6; weight licenses are non-commercial across the board for the accurate models |
| Identity model | Postgres tables in §5: `persons`, `person_face_embeddings`, `video_face_detections`, `transcript_segments` + review queue + audit | — | Label once → propagate; undo/merge; erasure |
| Orchestration | Reuse the transcode-pool pattern: PeerTube runner for transcription jobs; a small GPU worker container (like TEI) + Postgres-backed job queue for the rest | — | Few moving parts, no k8s |
**No accurate FR model in the open ecosystem has clean weights today.** Recommended posture: **detection+tracking+blur by default; identity matching opt-in, human-review-gated, collective decision recorded in writing** (non-commercial terms bar *commercialization* — whether a self-hosted activist tool qualifies is a legal question, not a technical one). Quality limits on your footage: masks (NIST IR 8331 PDF confirmed live at nvlpubs; the "5×–50× more errors" figure = UNVERIFIED-MEMORY, don't quote yet), motion blur, low light, side profiles — these push embeddings into the band where clustering **silently merges the wrong person**, which is your most dangerous failure mode (§GDPR).
Pipeline: PySceneDetect (0.7.1, BSD-3) → 1–2 keyframes/shot capped at ~2 fps → detect → track → 1 embedding/track → HDBSCAN/Chinese-Whispers offline → incremental assignment vs reviewed centroids (≥τ auto, τ-band → review queue). Cosine threshold ~0.4–0.5 same-person = UNVERIFIED-MEMORY; calibrate on your data. Store centroids in pgvector (512-d fits fine) — reuse the existing pattern.
Phase 1 = transcription+subtitles (PeerTube-native, low risk). Phase 2 = diarization + speaker naming. Phase 3 = faces/clustering with human-review gating (only after DPIA).
## (d) Identity model + GDPR (the part that gates everything)
---
- DDL in the report: `persons` (status `auto|member_confirmed|hidden|erased`, visibility tiers), `person_face_embeddings` (reviewed centroids only), `video_face_detections` (bbox + timestamps + `assignment` state machine + **transient** embedding column wiped by retention), `transcript_segments`, `video_speaker_person`, `identity_audit` (append-only; merge/undo/erasure all reversible). Label once → `UPDATE ... WHERE match_sim >= τ` propagates; τ-band goes to `pending_review`; nothing `auto` ever renders in UI/API.
- **Verified legal text** (EUR-Lex today): Art. 4(14) definition; **Art. 9(1) prohibits processing biometric data "for the purpose of uniquely identifying"** absent a 9(2) condition — and for bystanders/police, none of (a) explicit consent, (e) "manifestly made public", (f) vital interests, (g) substantial public interest (requires specific law) applies cleanly to an identity index → **an automated FR gallery over non-consenting third parties is effectively unlawful by default; detection+blur that doesn't uniquely identify arguably isn't Art. 9 data at all**. Art. 35(3)(b) makes the **DPIA mandatory** before phase 3. Design mitigations mapped: minimization (coords-not-crops, transient embeddings, pinned models/thresholds), human-review gating, visibility tiers enforced server-side, per-person erasure workflow (index-level; originals stay), audit trail, "high-risk person" flag for police-adjacent IDs, Tailscale-only exposure.
- **Could not verify this session (flagged):** EDPB Guidelines 05/2022 (all URL attempts 404'd), AEPD guidance + Zaragoza enforcement case (**aepd.es returned 500/503 all day**), NIST mask headline figure. Re-verify before citing any of them.
## 1. PeerTube transcription — verified from v8.3.1 source
## VRAM / cost (3090)
TEI+bge-m3 ~2–3 GB persistent + turbo int8 ~3 GB burst + pyannote ~2–4 GB + faces ~1–2 GB ≈ **11–13 GB concurrent worst case — fits 24 GB, but serialize jobs anyway** (queue, reuse pool-manager pressure signals; keep NVENC transcode floors). Throughput ESTIMATES: turbo ≈ 20–40× RT, large-v3 ≈ 8–12× RT; full pipeline ≈ **5–25 GPU-min per video-hour**; 500-h backfill ≈ 2–4 GPU-days for transcription alone.
Current release: **v8.3.1 (2026-09-28)** via GitHub releases API.
## Top unknowns before you commit
1. One smoke test: does PeerTube's **runner** transcription path pick up CUDA for whisper-ctranslate2, and does `model_path` accept a turbo CT2 dir there?
2. Internal ES/EN eval set (20–30 real clips) — no published benchmark matches protest audio.
3. Collective decision + legal advice on FR weights (non-commercial) and on Art. 9 posture **before any phase-3 code**.
4. WhisperX VoxPopuli-ES alignment weights license; EDPB/AEPD doc URLs (fetch failed today).
- **Built into core since 6.2** — CHANGELOG: *"Add automatic transcription of videos to generate subtitles" (PR #6303)*, *"Transcription can also be performed by PeerTube runners, as it can consume a lot of CPU"*, and *"PeerTube instance requires python >= 3.8 for transcription"*; pip3 is a listed dependency. Config (source: `config/default.yaml`, `video_transcription:`):
- `enabled: false` (default off), `engine: 'whisper-ctranslate2'` (alternative `openai-whisper`), `model: 'small'` (choices `tiny|base|small|medium|large-v2|large-v3` — **no `turbo` entry**; a CTranslate2 turbo dir can be used via `model_path`), `model_path: null`, `timeout: '6 hours'`, `remote_runners.enabled: false`.
- **Engines verified** (`packages/transcription/src/whisper/engines.ts`): `openai-whisper` (v20250625, MIT) and `whisper-ctranslate2` (>=0.4.3, MIT, CTranslate2 format). `packages/transcription` is an internal package (`@peertube/peertube-transcription`) wrapping these CLIs; transcription runs in the Node job queue locally or as a **runner job** (`server/core/lib/runners/job-handlers/transcription-job-handler.ts`; runner code: `apps/peertube-runner/.../process-transcription.ts`).
- **Where subtitles land:** caption rows in PeerTube DB + **VTT files** under `storage/captions/` (or object-storage prefix `captions/` — migrations `0875-caption-object-storage`, `0890-hls-caption`; a recent changelog line *"Fix transcription CORS issue if the caption file is stored externally"* confirms captions on R2 are a supported posture).
- **API (verified routes in `server/core/controllers/api/videos/captions.ts`):**
- `GET /api/v1/videos/{id}/captions` (public listing) → caption file URLs (`.vtt`)
- `PUT /api/v1/videos/{id}/captions/{lang}` (upload VTT, multipart)
- `DELETE /api/v1/videos/{id}/captions/{lang}`
- `POST /api/v1/videos/{id}/captions/generate` (request transcription manually; `generateTranscription` also honored on upload/import per changelog)
- **No webhooks exist in PeerTube** (repo-tree grep: zero webhook files). Completion signals available: (a) user notification `video-transcription-generated-for-owner` (not programmatic), (b) REST polling, (c) a **custom plugin** — the caption controller imports the plugin `Hooks` subsystem (specific transcription hook names NOT verified — flagged).
- Transcription widget in the web player exists (v6.3+ changelog) and re-runs happen after file replacement (v7.x changelog).
Full details, all quotes, DDL, and per-claim source URLs are in the saved report.
**Implication for us:** enable `video_transcription` with `remote_runners.enabled: true` and run the official `peertube-runner` on a Linux GPU node with the transcription capability — same runner contract as the transcode pool (companion doc). The api then watches the captions endpoint on video-publish and pulls the VTT. Whether the runner path picks up CUDA for `whisper-ctranslate2` needs one smoke test (flagged, §8).
---
## 2. Whisper family — models, licenses, Spanish, VRAM
| Model | License (HF API) | Spanish | VRAM | Notes |
|---|---|---|---|---|
| `openai/whisper-large-v3` | **Apache-2.0** | strong (paper-level; card no longer publishes the per-language table) | fp16 ~10 GB, int8 ~4.5 GB (README-measured class) | Best accuracy ceiling |
| `openai/whisper-large-v3-turbo` | **MIT** | multilingual; card states *"the exact same model, except that the number of decoding layers have reduced from 32 to 4"*, 809 M params | fp16 ~6 GB, int8 ~3 GB (ESTIMATE) | **Recommended default**: ~95% of v3 quality at a fraction of the cost |
| `distil-whisper/distil-large-v3` | MIT | **English-only** (UNVERIFIED-MEMORY: card language restriction; not re-fetched today) | ~4 GB | **Exclude** for ES |
| `Systran/faster-whisper-large-v3` (+turbo conversions) | **MIT** | = v3 | — | The CTranslate2 conversions PeerTube's engine consumes |
- **Spanish WER reality:** the current HF cards no longer carry the FLEURS/Common-Voice per-language tables (checked both cards — absent). From memory (UNVERIFIED-MEMORY): large-v3 FLEURS-es ≈ 5–7% WER, Common Voice-es ≈ 8–15%; expect **15–35%+ on street footage** (wind, PA, chants, overlapping speech, masks). Build a 20–30-clip internal ES/EN eval set (same as the search-architecture doc's eval plan) before choosing large-v3 vs turbo.
- **Whisper failure modes** (from the v3 card, verified text): *"prone to generating repetitive texts"*; hallucination risk on music/chants/silence — directly relevant to protest footage. Mitigation: VAD pre-filter (WhisperX/faster-whisper do this), temperature-fallback, human check of chant-heavy segments.
- **Runtime (faster-whisper 1.2.1, PyPI):** README-verified benchmark: large-v3 **int8, batch_size=8 → 4500 MB peak, 5× realtime on an RTX 3070 Ti 8GB**; `BatchedInferencePipeline` documented; word timestamps supported by the API (`word_timestamps=True` — standard; word output also verified downstream in whisper-asr-webservice "Word-level timestamps support").
- **whisper.cpp v1.9.5 (2026-10-06):** CPU/Windows fallback verified — `-ovtt/--output-vtt` in `examples/cli/cli.cpp`, CUDA + Vulkan backends, Q5_0 quantization. Multilingual ggml models only for ES. Use for the Windows node without WSL2 or as CPU emergency path.
- **3090 throughput (ESTIMATE, extrapolated):** turbo int8 batched ≈ 20–40× realtime (1.5–3 min per video-hour); large-v3 int8 ≈ 8–12× realtime (5–8 min per video-hour).
**Serving options (if we outgrow PeerTube's runner):**
- **Speaches** (MIT, v0.9.0-rc.3, 2025-12-27): OpenAI-compatible server over faster-whisper + TTS; modern but pre-1.0. Diarization endpoints: UNVERIFIED-MEMORY (not in README excerpt fetched).
- **whisper-asr-webservice** (v1.10.0, 2026-08-09): engines `openai_whisper | faster_whisper | whisperx`, outputs **text/JSON/VTT/SRT/TSV, word-level timestamps, diarization via WhisperX engine** (all README-verified). The most "boring and works" option; good as a standalone microservice feeding `transcript_segments` for media-plane blobs that never enter PeerTube.
---
## 3. Speaker diarization
- **pyannote.audio 4.0.7** (PyPI). **Code is MIT** (repo LICENSE, Copyright CNRS).
- **Model gating — the important nuance (verified):**
- `pyannote/speaker-diarization-community-1` (lastModified 2025-09-29, ~5.5 M downloads): **license CC-BY-4.0, gated: `auto`**.
- `pyannote/speaker-diarization-3.1` and `segmentation-3.0`: **MIT, gated: `auto`**.
- Empirically: fetching the model card **without auth returns HTTP 401** ("Access to model … is restricted. You must have access to it and be authenticated").
- `gated: auto` = you need an HF account, accept the model's user conditions once (name/email are recorded by HF), and pass a read token. It is **access control + attribution, not a use restriction** — no non-commercial clause found anywhere in the cards or the repo license. Verdict: **OK for this use** (CC-BY-4.0 needs attribution — add a models/credits page).
- The current pyannote README instructs exactly: install `pyannote.audio`, *"Accept pyannote/speaker-diarization-community-1 user conditions"*, create an HF access token, then `Pipeline.from_pretrained(..., token=...)`, runs locally, `.to("cuda")`.
- **Quality (official README benchmark table, DER %, lower better):** community-1 beats 3.1 across the board — AMI IHM 17.0 vs 18.8, DIHARD 3 20.2 vs 21.4, MSDWild 22.8 vs 25.4, Ego4D 46.8 vs 51.2. **No Spanish-specific benchmark exists in that table** — flag; expect degradation on street/overlapping speech and validate on our footage.
- **WhisperX** (BSD-2-Clause, PyPI 3.8.6): *"fast ASR (70× realtime with large-v2) with word-level timestamps and speaker diarization"*; current README: *"include your Hugging Face access token … after the `--hf_token` argument and accept the user agreement for the **speaker-diarization-community-1** model"*; `--min_speakers/--max_speakers`; Spanish word alignment via **`VOXPOPULI_ASR_BASE_10K_ES`** (verified line 36 of `whisperx/alignment.py`; license: UNVERIFIED — check before phase 2).
- **NVIDIA NeMo:** toolkit Apache-2.0, but **`nvidia/diar_sortformer_4spk-v1` is CC-BY-NC-4.0 → excluded** (non-commercial). (titanet speaker embedding is CC-BY-4.0, not gated, if ever needed.)
- **diart** (MIT, Univ. Paris-Saclay): streaming diarization over pyannote; maintenance status unclear (no fresh release evidence) — not needed for offline batch processing anyway.
- **Fusion pattern (recommended):** phase 2 = WhisperX (whisper large-v3-turbo CT2 + alignment + diarization) → `transcript_segments(video, start, end, speaker_key, text, lang, words jsonb)`. `speaker_key` = ephemeral per-video label (`SPEAKER_00…`). Speaker→person mapping lives in `video_speaker_person(video_id, speaker_key, person_id, confidence, reviewed)` so **one assignment per speaker per video, and a confirmed mapping can be suggested across videos** by pairing diarization with face-track co-occurrence (same timestamps as face detections) — cheap and surprisingly reliable.
- VRAM: pyannote community-1 on GPU ≈ 2–4 GB (ESTIMATE, no official figure published — flagged).
---
## 4. Faces — license audit (the deliverable table) + quality limits
| Model / repo | Code license | **Weights terms** | Weights provenance | Verdict for us |
|---|---|---|---|---|
| InsightFace repo (`deepinsight/insightface`) | MIT — *"no limitation for both academic and commercial usage"* (README) | **NON-COMMERCIAL** — *"The training data containing the annotation (and the models trained with these data) are available for non-commercial research purposes only"*, explicitly covering *"both manual-downloading models … and auto-downloading models with our python-library"*; **2025-11-24 update: for open-sourced FR models (e.g. buffalo_l) "contact recognition-oss-pack@insightface.ai for licensing"** | MS1M/WebFace-family research datasets | **Cannot use buffalo_l / ArcFace-w600k weights** for the collective's production index |
| `public-data/insightface` HF mirror | — | no separate license field; links back to insightface GitHub release (`buffalo_l.zip`) → same non-commercial terms | same | same |
| SCRFD detector weights | (inside insightface, MIT code) | same non-commercial policy (OneDrive-hosted) | — | detector-only contamination; replaceable |
| AdaFace (`mk-minchul/AdaFace`) | MIT | **silent** — weights on Google Drive, no license attached; trained on CASIA-WebFace/VGGFace2/WebFace4M/MS1MV2-3 (README table) | research datasets with own terms | Gray zone; not "clean" |
| deepface (`serengil/deepface`) | MIT | **does not launder weights** — README: *"License types will be inherited when you intend to utilize those models"*; wraps VGG-Face/Facenet/ArcFace/**Buffalo_L** | mixed | Convenience wrapper only; license risk unchanged |
| facenet-pytorch (`timesler`) | MIT | weights ported from davidsandberg/facenet (VGGFace2/CASIA-WebFace); no explicit weight license (README) | research datasets | Older/weaker anyway; not recommended |
| MediaPipe (`google-ai-edge/mediapipe`) | **Apache-2.0** (LICENSE verified); solution docs pages show Apache 2.0 links | model files: docs do not state a per-file license clearly (flagged) | Google self-collected | **Detection** OK-ish; **no recognition/identity embedding model exists in MediaPipe** (solutions index has detector+landmarker only) → cannot do identity with MediaPipe alone |
| yolov8-face / yolov5-face | **GPL-3.0** (both LICENSE files verified) | — | — | Copyleft hazard for a distributed self-hosted stack; avoid |
| WAFiA / PaddlePaddle FR | not audited this session | — | — | **Flagged: unexamined** |
**Honest reading:** every *accurate, realistic-footage* FR embedding model in the open ecosystem (InsightFace buffalo_l, ArcFace-webface variants, AdaFace-best) is trained on MS-Celeb-1M-lineage or WebFace research datasets whose terms are **non-commercial or undefined**. There is no verified fully-clean MIT/Apache recognition model with competitive accuracy today. Options:
1. **(Recommended) Do not run automated identity matching by default.** Run **detection + tracking + blurring/on-demand reveal**, and restrict recognition to an opt-in, DPIA-approved, human-review workflow (phase 3), under legal advice. Detection models can be MediaPipe/SCRFD (SCRFD weights non-commercial → if strict, use MediaPipe BlazeFace or YuNet (libfacedetection, MIT-code — weights terms unverified, flagged)).
2. If recognition is approved: accept the non-commercial-weights risk **in writing as a collective decision** (non-commercial terms restrict *commercialization*, and a self-hosted activist tool arguably isn't a "commercial use" — but that's a legal judgment call, not mine), and prefer AdaFace R50 (MIT code, documented weights) or InsightFace-runtime-with-explicitly-acknowledged-terms.
3. Watch for genuinely clean re-licenses (InsightFace's 2025-11-24 "contact for licensing" note suggests a relicensing channel exists).
**Quality on activist footage (limits to state publicly):** masks and occlusions massively degrade FR — NIST IR 8331 (masks) exists at `https://nvlpubs.nist.gov/nistpubs/IR/2020/NIST.IR.8331.pdf` (fetched, PDF confirmed); the often-quoted *"error rates 5×–50× higher with digitally applied masks"* figure is **UNVERIFIED-MEMORY** (PDF text extraction failed in-session — re-verify before quoting). Motion blur, low light, side profiles, helmets, and distant faces push embeddings into the low-confidence band where clustering silently merges the wrong people — which in our threat model is the *most dangerous* failure (see §6).
**Sampling strategy:** PySceneDetect (0.7.1, BSD-3) per shot → 1–2 keyframes/shot plus a hard cap (e.g. ≤2 fps) → detect → track (IoU/OSNet) → 1 embedding per track per scene. This cuts GPU cost ~10× vs naive fps and improves cluster purity.
**Clustering:** HDBSCAN (0.8.44, BSD) or Chinese Whispers on cosine similarity of 512-d embeddings; ArcFace-cosine same-person band ~0.4–0.5 (UNVERIFIED-MEMORY, calibrate on our data); then **incremental assignment**: new track → match against reviewed centroids (`person_face_embeddings`); ≥τ_accept auto-assign, τ_band → review queue, else new cluster candidate. Scale note: collection-scale kNN in pgvector works (512-d `vector` < 2000-d limit, HNSW) — reuse the existing pgvector pattern.
---
## 5. Person identity model — DDL sketch
```sql
-- Canonical identities. A person starts as an unverified cluster and is
-- only promoted to a reviewed label by a human.
CREATE TABLE persons (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
display_name text, -- null until a member labels it
status text NOT NULL DEFAULT 'auto' -- auto | member_confirmed | hidden | erased
CHECK (status IN ('auto','member_confirmed','hidden','erased')),
visibility text NOT NULL DEFAULT 'members_only'
CHECK (visibility IN ('members_only','staff_only','erased')),
centroid vector(512), -- running mean of reviewed embeddings
created_by uuid NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
-- Reference embeddings (reviewed only; see minimization note)
CREATE TABLE person_face_embeddings (
id bigserial PRIMARY KEY,
person_id uuid NOT NULL REFERENCES persons(id) ON DELETE CASCADE,
embedding vector(512) NOT NULL,
source text, -- 'review_ui' | 'backfill'
model text NOT NULL, -- e.g. 'adaface-r50' (pin it)
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON person_face_embeddings USING hnsw (embedding vector_cosine_ops);
-- Every face box ever detected. Coordinates, never crops.
CREATE TABLE video_face_detections (
id bigserial PRIMARY KEY,
video_id uuid NOT NULL, -- PeerTube uuid or media-plane blob hash
source text NOT NULL DEFAULT 'peertube' CHECK (source IN ('peertube','media_plane')),
t_start_ms int NOT NULL, t_end_ms int, -- track span
bbox numeric[4] NOT NULL, -- x,y,w,h normalized
track_id int NOT NULL, -- within-video track
person_id uuid REFERENCES persons(id), -- null = unassigned
confidence real NOT NULL, -- detector conf
match_sim real, -- cosine to assigned centroid
assignment text NOT NULL DEFAULT 'auto' -- auto | human_confirmed | rejected | pending_review
CHECK (assignment IN ('auto','human_confirmed','rejected','pending_review')),
reviewed_by uuid, reviewed_at timestamptz,
embedding vector(512), -- TRANSIENT: nulled by retention job after review/SLA
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON video_face_detections (video_id, t_start_ms);
CREATE INDEX ON video_face_detections (person_id);
CREATE INDEX ON video_face_detections (assignment) WHERE assignment = 'pending_review';
-- Phase 1/2: transcripts (feeds chat_media_archive + pgvector via the search pipeline)
CREATE TABLE transcript_segments (
id bigserial PRIMARY KEY,
video_id uuid NOT NULL,
source text NOT NULL DEFAULT 'peertube',
lang text NOT NULL, -- 'en' | 'es' (also ca/eu occur in Spain!)
start_ms int NOT NULL, end_ms int NOT NULL,
speaker_key text, -- 'SPEAKER_00' from diarization
person_id uuid REFERENCES persons(id), -- resolved speaker (phase 2)
text text NOT NULL,
words jsonb, -- word-level timestamps
confidence real,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (video_id, start_ms, lang)
);
-- Speaker-name bridge (label once per speaker per video; suggestions across videos)
CREATE TABLE video_speaker_person (
video_id uuid NOT NULL, speaker_key text NOT NULL,
person_id uuid REFERENCES persons(id),
confidence real, assignment text NOT NULL DEFAULT 'auto',
PRIMARY KEY (video_id, speaker_key)
);
-- Audit trail: every label action is reversible
CREATE TABLE identity_audit (
id bigserial PRIMARY KEY,
actor uuid NOT NULL, action text NOT NULL, -- assign|merge|unmerge|erase|relabel
person_id uuid, other_person_id uuid, payload jsonb,
at timestamptz NOT NULL DEFAULT now()
);
```
**Propagation rules**
- Label a face once (human confirms track → `persons.status='member_confirmed'`, centroid updated): backfill = `UPDATE video_face_detections SET person_id=$p WHERE match_sim >= τ_accept AND person_id IS NULL` (τ_accept ~0.6 calibrated); everything in the τ band goes to `pending_review`.
- **Merge personas:** union → recompute centroid → single `identity_audit` row; undo by replaying audit (append-only, never destructive).
- **Erasure (Art. 17):** `persons.status='erased'` → delete person embeddings + null all `video_face_detections.person_id` + remove speaker mappings + purge embeddings column for that person's detections; detections revert to anonymous boxes. Name/label never appears again.
- **Minimization:** `embedding` on detections is a *transient* column, wiped by a retention job (e.g. 72h after processing) — persist only reviewed centroids. No face crops on disk at rest (processing temp dir, tmpfs, TTL).
---
## 6. GDPR / threat model (Spain)
**Verified legal text (EUR-Lex CELEX:32016R0679, fetched today):**
- **Art. 4(14):** biometric data = *"personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, which allow or confirm the unique identification of that natural person, such as facial images…"*
- **Art. 9(1) prohibition:** processing of *"biometric data for the purpose of uniquely identifying a natural person"* is prohibited unless a 9(2) condition applies.
- 9(2) candidates for third parties in footage: **(a) explicit consent** (bystanders/police never gave it), **(e) manifestly made public by the data subject** (appearing in a protest video is not "manifestly making public" one's biometric template — analysis, not legal advice), **(f) vital interests** (narrow emergencies), **(g) substantial public interest — requires a Union/Member-State law providing the basis** (no obvious one for us). **Analysis: none applies cleanly to automated identity indexing of non-consenting third parties.** Note also that face *detection/blur* that does not uniquely identify arguably isn't Art. 9 data at all; the moment we build a gallery to identify people, it is.
- **Art. 35(3)(b):** DPIA is mandatory for *"processing on a large scale of special categories of personal data"* → **DPIA before phase 3**.
- Art. 5 (minimization/purpose limitation), Art. 17 (erasure), Art. 25 (protection by design), Art. 32 (security), Art. 85 (freedom of expression exemption — relevant for *publishing* footage, and LOPDGDD (LO 3/2018) develops it; article-level mapping **flagged: not verified against BOE this session**).
**EDPB / AEPD sources — fetch status this session (flagged):**
- EDPB *Guidelines 05/2022 on processing personal data through facial recognition technology in law enforcement* — known document; **all attempted URLs 404'd in this session → URL unverified** (search "EDPB Guidelines 05/2022 facial recognition" before citing).
- **AEPD (aepd.es) returned 500/503 all session** — the AEPD+CAT *guía de tratamiento de datos biométricos* and enforcement cases could not be retrieved. **Flagged as unverified.** From memory (unverified): AEPD sanctioned Ayuntamiento de Zaragoza over facial-recognition cameras (2019, small fine) — do not cite without checking.
- NIST IR 8331 masks study: PDF confirmed live (URL above); headline figure unverified (see §4).
**The specific danger: mislabeling.** A wrong person_id in activist footage (a) wrongly associates a bystander with a collective, (b) creates a target list for state/paramilitary actors if the index leaks, (c) is effectively irreversible reputationally. Therefore:
**Design mitigations (concrete):**
1. **Default = detect, don't identify.** Phase 3 ships with face *detection* + blurring/redaction tooling and a **kill switch: identity matching is an opt-in feature flag per instance**, default off, behind DPIA sign-off.
2. **Human-review gating:** nothing from `auto` assignment is ever exposed in any UI or API; only `human_confirmed` labels render, and only to roles with `identity:read`.
3. **Visibility tiers:** `persons.visibility` (members_only / staff_only) enforced at the API/query layer, not the client; face-label search endpoints excluded from `chat_media_archive` public surfaces.
4. **Minimization:** coordinates not crops (§5); transient embeddings wiped; centroids only for confirmed persons; model + threshold pinned and logged (reproducibility).
5. **Erasure workflow:** one click → §5 erasure; plus per-person "scrub from index" independent of video deletion; document that PeerTube/R2 originals stay (erasure applies to the *index*, per minimization the index is what we must justify).
6. **Audit trail** (§5) — who labeled what, when; undo/merge reversible.
7. **Blast-radius controls:** index DB on Tailscale-only interface; no face data in backups of public shares; consider per-person "high-risk" flag (police-adjacent identifications) that hard-restricts visibility to staff and logs every read.
8. **Co-official languages:** Spain footage will contain Catalan/Basque/Galician — Whisper supports them (`ca`, `eu` verified in the turbo model card language list) but with worse WER than ES; transcribe+flag language per segment rather than assuming EN/ES.
---
## 7. Pipeline integration + VRAM/cost budget
**Data flow**
```
Phase 1: PeerTube upload → transcode (existing pool) → video published
→ video_transcription (remote runner on 3090, whisper-ctranslate2 + large-v3-turbo CT2 int8)
→ captions (VTT) on R2 (prefix captions/)
→ api poller (no webhooks in PeerTube — verified): GET /api/v1/videos/{id}/captions
→ fetch VTT → parse → transcript_segments → TEI/bge-m3 → pgvector (existing search doc)
Phase 1b (media-plane blobs): upload-finalized api event → job queue → whisper-asr-webservice
(faster_whisper engine, VTT out) → same transcript_segments path
Phase 2: api event → diarization job (WhisperX: turbo CT2 + align + community-1)
→ speaker_key per segment → review UI "who is this speaker?" → video_speaker_person
Phase 3: api event → face job: PySceneDetect → keyframes → detect (BlazeFace/SCRFD) → track
→ embed (only if flag on) → assign vs reviewed centroids → pending_review queue
→ human review → persons / centroid update → backfill propagation (§5)
```
Orchestration reuses the transcode-pool pattern: PeerTube runner protocol for transcription; everything else is a small GPU worker container + Postgres-backed queue (like the api's existing transcode queue), jobs idempotent, resumable per video.
**VRAM budget (24 GB 3090, Linux node — per the transcode-pool doc, production NVIDIA work stays off WSL2; the Windows box is feasible for ASR-only via WSL2 *if* Windows-side VRAM tenants are cleared — measured 23.8/24 GB occupied there on 2026-10-09)**
| Component | fp16 | int8 | Duty |
|---|---|---|---|
| TEI + bge-m3 (already deployed) | ~2–3 GB | — | persistent |
| whisper turbo (CT2) | ~6 GB | ~3 GB | burst |
| whisper large-v3 (CT2) | ~10 GB | ~4.5 GB (README-measured on 8 GB card) | burst, quality tier |
| WhisperX + pyannote community-1 | +2–4 GB (EST.) | — | burst, phase 2 |
| SCRFD/BlazeFace + embed ONNX | ~1–2 GB (EST.) | — | burst, phase 3 |
Concurrent worst case (large-v3 + diarization + faces) ≈ 16 GB → fits; but **serialize per job class** anyway (queue). Turbo + diarization + faces ≈ 11–13 GB. Always keep the NVENC transcode pool's VRAM floor in mind on shared nodes — schedule A/V-intelligence jobs on the same nodes only when transcode queue depth is low (reuse the pool manager's pressure signal).
**Cost per video-hour (3090, ESTIMATES):** transcription turbo int8 1.5–3 min (large-v3 5–12 min) · diarization 1–3 min · keyframe faces 3–10 min (sampler-capped) → **~5–25 GPU-min per video-hour** full pipeline; backfilling 500 h ≈ 2–4 GPU-days for transcription alone.
---
## 8. Honest gaps & unverified items (all flagged in-text; consolidated)
1. **Spanish ASR on street audio** — no published benchmark matches our domain (noise/PA/chants/code-switching); FLEURS numbers absent from current HF cards. Internal eval set is mandatory.
2. **Hallucination on chants/music** — documented Whisper weakness; needs VAD + segment-level confidence and human spot-checks before any published subtitle.
3. **Diarization has no Spanish benchmark** in pyannote's table; overlapping-protest-speech DER will be far worse than AMI numbers.
4. **Face weights licensing is dirty everywhere** (§4) — the accurate models are non-commercial/undefined; no clean equal exists today. Decision needed at collective level + legal advice; SCRFD/YuNet/MediaPipe weight-file license status partially unverified.
5. **NIST mask figure**, **EDPB 05/2022 URL**, **AEPD guidance/enforcement** — could not be fully retrieved this session; re-verify before citing (aepd.es was 5xx all day).
6. **PeerTube runner + CUDA pickup** for `whisper-ctranslate2`, and whether `model_path` accepts a turbo CT2 dir in the runner path — one smoke test needed.
7. **PeerTube plugin transcription hook names** — plugin system exists (verified import), specific hook unverified.
8. **Speaches diarization endpoints**, **diart maintenance status**, **chinese-whispers package**, **pyannote community-1 VRAM**, **3090 RTF figures** — ESTIMATE/UNVERIFIED-MEMORY as marked.
9. **WhisperX Spanish alignment model license** (VoxPopuli weights) — check before phase 2.
## Key primary sources (fetched 2026-10-11)
- PeerTube tree/CHANGELOG/source: https://github.com/Chocobozzz/PeerTube (v8.3.1; `config/default.yaml`, `packages/transcription/*`, `server/core/controllers/api/videos/captions.ts`, runner `process-transcription.ts`)
- HF API/cards: openai/whisper-large-v3 (Apache-2.0), openai/whisper-large-v3-turbo (MIT, 32→4 layers, 809M), distil-whisper/distil-large-v3 (MIT), Systran/faster-whisper-large-v3 (MIT), pyannote/speaker-diarization-community-1 (CC-BY-4.0, gated auto), pyannote/speaker-diarization-3.1 + segmentation-3.0 (MIT, gated auto), nvidia/diar_sortformer_4spk-v1 (**CC-BY-NC-4.0**)
- faster-whisper README (VRAM/RTF benchmarks), whisper.cpp v1.9.5 (VTT/CUDA/Vulkan/Q5_0), whisperX README + alignment.py (community-1, `VOXPOPULI_ASR_BASE_10K_ES`), Speaches v0.9.0-rc.3 (MIT), whisper-asr-webservice v1.10.0
- insightface README (MIT code / non-commercial models + 2025-11-24 relicensing note), AdaFace (MIT code, Drive weights), deepface (MIT + inherited-weights disclaimer), MediaPipe (Apache-2.0), yolov8/yolov5-face (GPL-3.0), facenet-pytorch (MIT), PySceneDetect 0.7.1 (BSD-3), hdbscan 0.8.44 (BSD)
- EUR-Lex GDPR (Art. 4(14), 9, 35(3)(b) quoted), NIST IR 8331 PDF (exists; figure unverified), EDPB/AEPD (unreachable this session — flagged)