hacklab/T-HUB
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
T-HUB/docs/design/2026-10-11-video-ai-architecture.md
glass beadian eb1b60f67d Add video-AI architecture research (transcription, diarization, faces, GDPR) 
Research: use PeerTube
native transcription (core since 6.2, whisper-ctranslate2, remote runners on
GPU nodes; large-v3-turbo MIT via model_path); pyannote community-1
diarization (CC-BY-4.0, clean); WhisperX fusion.

Face recognition license audit: NO accurate open FR model has clean weights
(InsightFace/SCRFD non-commercial, yolo-face GPL, AdaFace unstated) —
posture is detection+tracking+blur by default; identity matching opt-in,
human-review-gated, collective decision + DPIA required (Art. 9 biometric
data; automated FR over non-consenting third parties effectively unlawful).

DB DDL sketch: persons/person_face_embeddings/video_face_detections/
transcript_segments/video_speaker_person/identity_audit with review gating,
visibility tiers, erasure workflow.
2026-10-11 07:55:01 +00:00

8.7 KiB
Raw Blame History

Report saved to research/2026-10-11-av-transcription-diarization-faces-gdpr.md. Full findings below — everything marked "verified" was fetched from primary sources today (2026-10-11); everything else is explicitly flagged.

Decision-ready summary

(a) Transcription — use PeerTube's, don't build one

Verified from PeerTube v8.3.1 source today (current release, 2026-09-28):

  • Transcription is core since 6.2 (video_transcription: in config/default.yaml), default off. Engines: whisper-ctranslate2 (default) or openai-whisper; models tiny|base|small|medium|large-v2|large-v3 — no turbo in the list, but model_path: accepts a CTranslate2 model directory, so drop in faster-whisper-large-v3-turbo (MIT) via path. Set remote_runners.enabled: true and run the official peertube-runner on a Linux 3090 node — same runner contract as your transcode pool.
  • Subtitles land as VTT in the caption system (storage prefix captions/ on R2 is supported — recent changelog fixes caption CORS for object storage). API verified in server/core/controllers/api/videos/captions.ts: GET /api/v1/videos/{id}/captions, PUT/DELETE .../captions/{lang}, POST .../captions/generate.
  • No webhooks in PeerTube (verified — zero hits in repo tree) → your NestJS api should poll captions-on-publish, or write a tiny plugin (plugin Hooks are imported by the caption controller; specific transcription hook names unverified — flagged).
  • Model licensing is clean: whisper-large-v3 = Apache-2.0, large-v3-turbo = MIT, faster-whisper-large-v3 = MIT (HF API today). distil-whisper is English-only → excluded for ES (card check: marked UNVERIFIED-MEMORY, but nothing suggests multilingual).
  • Spanish numbers honestly missing: current HF cards no longer publish FLEURS/CV per-language tables (checked both — absent). From memory: FLEURS-es ~5–7% for large-v3; expect 15–35% on street audio (wind/PA/chants). Whisper cards explicitly warn of "repetitive texts" (hallucinations) — a real risk on chants/music; VAD + human spot-check needed.
  • VRAM (faster-whisper README, measured on a 3070 Ti): large-v3 int8 batched = 4500 MB, 5× realtime. Turbo ≈ half that (ESTIMATE). whisper.cpp v1.9.5 (2026-10-06) verified as CPU/Windows fallback with -ovtt VTT output, CUDA/Vulkan, Q5_0.
  • Fallback serving if you outgrow the runner: whisper-asr-webservice v1.10.0 (faster_whisper/whisperx engines, VTT/SRT/word timestamps — README-verified; ideal for media-plane blobs that never enter PeerTube) or Speaches (MIT, but v0.9.0-rc.3 — pre-1.0).

(b) Diarization — pyannote is fine for this use; the gate is not a license

  • pyannote/speaker-diarization-community-1: license CC-BY-4.0, gated auto (HF API today); 3.1/segmentation-3.0 are MIT, gated auto; pyannote.audio code is MIT (CNRS). The gate = HF account + accept conditions + read token (card fetch without auth returned 401 — verified empirically). No non-commercial clause anywhere → OK for you, add CC attribution to a credits page.
  • Quality (official DER table, community-1 vs 3.1): AMI 17.0/18.8, DIHARD 3 20.2/21.4, MSDWild 22.8/25.4, Ego4D 46.8/51.2 — the two "wild" benchmarks are the honest proxies for street footage; no Spanish benchmark exists (flagged).
  • WhisperX (BSD-2, v3.8.6) fuses whisper + word alignment + diarization; its README now requires accepting speaker-diarization-community-1 + --hf_token. Spanish alignment model = VOXPOPULI_ASR_BASE_10K_ES (verified in alignment.py; that model's license unverified — flagged).
  • NeMo Sortformer weights are CC-BY-NC-4.0 → excluded (non-commercial; NeMo toolkit itself is Apache-2.0). diart (MIT) is streaming-oriented, maintenance unclear.
  • Speaker→name UX: speaker_key per video + one video_speaker_person row; confirmed mappings + face-track co-occurrence give cross-video suggestions (label once per speaker).

(c) Faces — the license audit is the headline

Component Code Weights Verdict
InsightFace buffalo_l / ArcFace-w600k MIT NON-COMMERCIAL — README: models "available for non-commercial research purposes only", applies to pip auto-download too; 2025-11-24 update: contact recognition-oss-pack@insightface.ai for licensing (both quoted from README today) ❌ for production index
SCRFD detector weights MIT code same non-commercial policy ❌ if strict / ⚠️ pragmatic
AdaFace MIT weights on Google Drive, no license stated; WebFace4M/MS1MV2-lineage ⚠️ gray zone
deepface MIT explicitly inherits wrapped-model licenses (README quote) — launders nothing ⚠️
MediaPipe (BlazeFace detect) Apache-2.0 per-file license not clearly stated (flag) ✅ detection; no recognition model exists in MediaPipe
yolov8-face / yolov5-face GPL-3.0 — ❌ copyleft hazard

No accurate FR model in the open ecosystem has clean weights today. Recommended posture: detection+tracking+blur by default; identity matching opt-in, human-review-gated, collective decision recorded in writing (non-commercial terms bar commercialization — whether a self-hosted activist tool qualifies is a legal question, not a technical one). Quality limits on your footage: masks (NIST IR 8331 PDF confirmed live at nvlpubs; the "5×–50× more errors" figure = UNVERIFIED-MEMORY, don't quote yet), motion blur, low light, side profiles — these push embeddings into the band where clustering silently merges the wrong person, which is your most dangerous failure mode (§GDPR). Pipeline: PySceneDetect (0.7.1, BSD-3) → 1–2 keyframes/shot capped at ~2 fps → detect → track → 1 embedding/track → HDBSCAN/Chinese-Whispers offline → incremental assignment vs reviewed centroids (≥τ auto, τ-band → review queue). Cosine threshold ~0.4–0.5 same-person = UNVERIFIED-MEMORY; calibrate on your data. Store centroids in pgvector (512-d fits fine) — reuse the existing pattern.

(d) Identity model + GDPR (the part that gates everything)

  • DDL in the report: persons (status auto|member_confirmed|hidden|erased, visibility tiers), person_face_embeddings (reviewed centroids only), video_face_detections (bbox + timestamps + assignment state machine + transient embedding column wiped by retention), transcript_segments, video_speaker_person, identity_audit (append-only; merge/undo/erasure all reversible). Label once → UPDATE ... WHERE match_sim >= τ propagates; τ-band goes to pending_review; nothing auto ever renders in UI/API.
  • Verified legal text (EUR-Lex today): Art. 4(14) definition; Art. 9(1) prohibits processing biometric data "for the purpose of uniquely identifying" absent a 9(2) condition — and for bystanders/police, none of (a) explicit consent, (e) "manifestly made public", (f) vital interests, (g) substantial public interest (requires specific law) applies cleanly to an identity index → an automated FR gallery over non-consenting third parties is effectively unlawful by default; detection+blur that doesn't uniquely identify arguably isn't Art. 9 data at all. Art. 35(3)(b) makes the DPIA mandatory before phase 3. Design mitigations mapped: minimization (coords-not-crops, transient embeddings, pinned models/thresholds), human-review gating, visibility tiers enforced server-side, per-person erasure workflow (index-level; originals stay), audit trail, "high-risk person" flag for police-adjacent IDs, Tailscale-only exposure.
  • Could not verify this session (flagged): EDPB Guidelines 05/2022 (all URL attempts 404'd), AEPD guidance + Zaragoza enforcement case (aepd.es returned 500/503 all day), NIST mask headline figure. Re-verify before citing any of them.

VRAM / cost (3090)

TEI+bge-m3 ~2–3 GB persistent + turbo int8 ~3 GB burst + pyannote ~2–4 GB + faces ~1–2 GB ≈ 11–13 GB concurrent worst case — fits 24 GB, but serialize jobs anyway (queue, reuse pool-manager pressure signals; keep NVENC transcode floors). Throughput ESTIMATES: turbo ≈ 20–40× RT, large-v3 ≈ 8–12× RT; full pipeline ≈ 5–25 GPU-min per video-hour; 500-h backfill ≈ 2–4 GPU-days for transcription alone.

Top unknowns before you commit

  1. One smoke test: does PeerTube's runner transcription path pick up CUDA for whisper-ctranslate2, and does model_path accept a turbo CT2 dir there?
  2. Internal ES/EN eval set (20–30 real clips) — no published benchmark matches protest audio.
  3. Collective decision + legal advice on FR weights (non-commercial) and on Art. 9 posture before any phase-3 code.
  4. WhisperX VoxPopuli-ES alignment weights license; EDPB/AEPD doc URLs (fetch failed today).

Full details, all quotes, DDL, and per-claim source URLs are in the saved report.