glass beadian
777c67f7a3
281-line version with the person-identity DDL sketch, per-model license audit table, VRAM/cost budget, GDPR Art. 9 analysis, and all primary-source citations — replaces the 53-line executive summary.
281 lines
31 KiB
Markdown
281 lines
31 KiB
Markdown
# A/V Intelligence Pipeline — Research Report (transcription · diarization · faces · identity · GDPR)
|
||
|
||
**Date:** 2026-10-11 · **Scope:** Tactical Media Broadcast Hub (PeerTube occ-tube + R2 media plane, NestJS api, pgvector, RTX 3090 fleet on Tailscale, GDPR/Spain, EN+ES, no third-party cloud APIs).
|
||
**Method:** every claim below was checked against a primary source fetched live in this session (GitHub source/release API, HuggingFace API/model cards, PyPI, EUR-Lex, nvlpubs) unless explicitly marked **UNVERIFIED-MEMORY** or **ESTIMATE**. Versions quoted are current as of today.
|
||
|
||
---
|
||
|
||
## 0. Recommended architecture (TL;DR)
|
||
|
||
| Concern | Choice | License | Why |
|
||
|---|---|---|---|
|
||
| Transcription | **PeerTube built-in (v8.3.1), engine `whisper-ctranslate2`, model `large-v3-turbo` (int8), executed by a remote runner on the 3090** | Whisper weights MIT/Apache-2.0 (see §2) | It ships today, subtitles land in PeerTube's caption system (VTT), API retrievable, zero glue code |
|
||
| Word timestamps | faster-whisper `word_timestamps` / WhisperX alignment (phase 2) | BSD-2/MIT | VTT + speaker fusion |
|
||
| Diarization | **pyannote `speaker-diarization-community-1`** via WhisperX (or standalone) | **CC-BY-4.0** weights, MIT code, gated `auto` | Best open DER; the gate is access-control, not a license restriction |
|
||
| Face detection | **SCRFD (ONNX, via insightface pip for runtime only)** or MediaPipe BlazeFace if you want zero-contamination pipeline | weights **non-commercial** (SCRFD/buffalo) / Apache-2.0 (MediaPipe) | See §4 — this is the one genuinely dirty licensing corner; options given |
|
||
| Face recognition | **Deferred to phase 3, human-review-gated.** If run: AdaFace/ArcFace-class 512-d embeddings; store centroids, not crops | — | GDPR Art. 9 — see §6; weight licenses are non-commercial across the board for the accurate models |
|
||
| Identity model | Postgres tables in §5: `persons`, `person_face_embeddings`, `video_face_detections`, `transcript_segments` + review queue + audit | — | Label once → propagate; undo/merge; erasure |
|
||
| Orchestration | Reuse the transcode-pool pattern: PeerTube runner for transcription jobs; a small GPU worker container (like TEI) + Postgres-backed job queue for the rest | — | Few moving parts, no k8s |
|
||
|
||
Phase 1 = transcription+subtitles (PeerTube-native, low risk). Phase 2 = diarization + speaker naming. Phase 3 = faces/clustering with human-review gating (only after DPIA).
|
||
|
||
---
|
||
|
||
## 1. PeerTube transcription — verified from v8.3.1 source
|
||
|
||
Current release: **v8.3.1 (2026-09-28)** via GitHub releases API.
|
||
|
||
- **Built into core since 6.2** — CHANGELOG: *"Add automatic transcription of videos to generate subtitles" (PR #6303)*, *"Transcription can also be performed by PeerTube runners, as it can consume a lot of CPU"*, and *"PeerTube instance requires python >= 3.8 for transcription"*; pip3 is a listed dependency. Config (source: `config/default.yaml`, `video_transcription:`):
|
||
- `enabled: false` (default off), `engine: 'whisper-ctranslate2'` (alternative `openai-whisper`), `model: 'small'` (choices `tiny|base|small|medium|large-v2|large-v3` — **no `turbo` entry**; a CTranslate2 turbo dir can be used via `model_path`), `model_path: null`, `timeout: '6 hours'`, `remote_runners.enabled: false`.
|
||
- **Engines verified** (`packages/transcription/src/whisper/engines.ts`): `openai-whisper` (v20250625, MIT) and `whisper-ctranslate2` (>=0.4.3, MIT, CTranslate2 format). `packages/transcription` is an internal package (`@peertube/peertube-transcription`) wrapping these CLIs; transcription runs in the Node job queue locally or as a **runner job** (`server/core/lib/runners/job-handlers/transcription-job-handler.ts`; runner code: `apps/peertube-runner/.../process-transcription.ts`).
|
||
- **Where subtitles land:** caption rows in PeerTube DB + **VTT files** under `storage/captions/` (or object-storage prefix `captions/` — migrations `0875-caption-object-storage`, `0890-hls-caption`; a recent changelog line *"Fix transcription CORS issue if the caption file is stored externally"* confirms captions on R2 are a supported posture).
|
||
- **API (verified routes in `server/core/controllers/api/videos/captions.ts`):**
|
||
- `GET /api/v1/videos/{id}/captions` (public listing) → caption file URLs (`.vtt`)
|
||
- `PUT /api/v1/videos/{id}/captions/{lang}` (upload VTT, multipart)
|
||
- `DELETE /api/v1/videos/{id}/captions/{lang}`
|
||
- `POST /api/v1/videos/{id}/captions/generate` (request transcription manually; `generateTranscription` also honored on upload/import per changelog)
|
||
- **No webhooks exist in PeerTube** (repo-tree grep: zero webhook files). Completion signals available: (a) user notification `video-transcription-generated-for-owner` (not programmatic), (b) REST polling, (c) a **custom plugin** — the caption controller imports the plugin `Hooks` subsystem (specific transcription hook names NOT verified — flagged).
|
||
- Transcription widget in the web player exists (v6.3+ changelog) and re-runs happen after file replacement (v7.x changelog).
|
||
|
||
**Implication for us:** enable `video_transcription` with `remote_runners.enabled: true` and run the official `peertube-runner` on a Linux GPU node with the transcription capability — same runner contract as the transcode pool (companion doc). The api then watches the captions endpoint on video-publish and pulls the VTT. Whether the runner path picks up CUDA for `whisper-ctranslate2` needs one smoke test (flagged, §8).
|
||
|
||
---
|
||
|
||
## 2. Whisper family — models, licenses, Spanish, VRAM
|
||
|
||
| Model | License (HF API) | Spanish | VRAM | Notes |
|
||
|---|---|---|---|---|
|
||
| `openai/whisper-large-v3` | **Apache-2.0** | strong (paper-level; card no longer publishes the per-language table) | fp16 ~10 GB, int8 ~4.5 GB (README-measured class) | Best accuracy ceiling |
|
||
| `openai/whisper-large-v3-turbo` | **MIT** | multilingual; card states *"the exact same model, except that the number of decoding layers have reduced from 32 to 4"*, 809 M params | fp16 ~6 GB, int8 ~3 GB (ESTIMATE) | **Recommended default**: ~95% of v3 quality at a fraction of the cost |
|
||
| `distil-whisper/distil-large-v3` | MIT | **English-only** (UNVERIFIED-MEMORY: card language restriction; not re-fetched today) | ~4 GB | **Exclude** for ES |
|
||
| `Systran/faster-whisper-large-v3` (+turbo conversions) | **MIT** | = v3 | — | The CTranslate2 conversions PeerTube's engine consumes |
|
||
|
||
- **Spanish WER reality:** the current HF cards no longer carry the FLEURS/Common-Voice per-language tables (checked both cards — absent). From memory (UNVERIFIED-MEMORY): large-v3 FLEURS-es ≈ 5–7% WER, Common Voice-es ≈ 8–15%; expect **15–35%+ on street footage** (wind, PA, chants, overlapping speech, masks). Build a 20–30-clip internal ES/EN eval set (same as the search-architecture doc's eval plan) before choosing large-v3 vs turbo.
|
||
- **Whisper failure modes** (from the v3 card, verified text): *"prone to generating repetitive texts"*; hallucination risk on music/chants/silence — directly relevant to protest footage. Mitigation: VAD pre-filter (WhisperX/faster-whisper do this), temperature-fallback, human check of chant-heavy segments.
|
||
- **Runtime (faster-whisper 1.2.1, PyPI):** README-verified benchmark: large-v3 **int8, batch_size=8 → 4500 MB peak, 5× realtime on an RTX 3070 Ti 8GB**; `BatchedInferencePipeline` documented; word timestamps supported by the API (`word_timestamps=True` — standard; word output also verified downstream in whisper-asr-webservice "Word-level timestamps support").
|
||
- **whisper.cpp v1.9.5 (2026-10-06):** CPU/Windows fallback verified — `-ovtt/--output-vtt` in `examples/cli/cli.cpp`, CUDA + Vulkan backends, Q5_0 quantization. Multilingual ggml models only for ES. Use for the Windows node without WSL2 or as CPU emergency path.
|
||
- **3090 throughput (ESTIMATE, extrapolated):** turbo int8 batched ≈ 20–40× realtime (1.5–3 min per video-hour); large-v3 int8 ≈ 8–12× realtime (5–8 min per video-hour).
|
||
|
||
**Serving options (if we outgrow PeerTube's runner):**
|
||
- **Speaches** (MIT, v0.9.0-rc.3, 2025-12-27): OpenAI-compatible server over faster-whisper + TTS; modern but pre-1.0. Diarization endpoints: UNVERIFIED-MEMORY (not in README excerpt fetched).
|
||
- **whisper-asr-webservice** (v1.10.0, 2026-08-09): engines `openai_whisper | faster_whisper | whisperx`, outputs **text/JSON/VTT/SRT/TSV, word-level timestamps, diarization via WhisperX engine** (all README-verified). The most "boring and works" option; good as a standalone microservice feeding `transcript_segments` for media-plane blobs that never enter PeerTube.
|
||
|
||
---
|
||
|
||
## 3. Speaker diarization
|
||
|
||
- **pyannote.audio 4.0.7** (PyPI). **Code is MIT** (repo LICENSE, Copyright CNRS).
|
||
- **Model gating — the important nuance (verified):**
|
||
- `pyannote/speaker-diarization-community-1` (lastModified 2025-09-29, ~5.5 M downloads): **license CC-BY-4.0, gated: `auto`**.
|
||
- `pyannote/speaker-diarization-3.1` and `segmentation-3.0`: **MIT, gated: `auto`**.
|
||
- Empirically: fetching the model card **without auth returns HTTP 401** ("Access to model … is restricted. You must have access to it and be authenticated").
|
||
- `gated: auto` = you need an HF account, accept the model's user conditions once (name/email are recorded by HF), and pass a read token. It is **access control + attribution, not a use restriction** — no non-commercial clause found anywhere in the cards or the repo license. Verdict: **OK for this use** (CC-BY-4.0 needs attribution — add a models/credits page).
|
||
- The current pyannote README instructs exactly: install `pyannote.audio`, *"Accept pyannote/speaker-diarization-community-1 user conditions"*, create an HF access token, then `Pipeline.from_pretrained(..., token=...)`, runs locally, `.to("cuda")`.
|
||
- **Quality (official README benchmark table, DER %, lower better):** community-1 beats 3.1 across the board — AMI IHM 17.0 vs 18.8, DIHARD 3 20.2 vs 21.4, MSDWild 22.8 vs 25.4, Ego4D 46.8 vs 51.2. **No Spanish-specific benchmark exists in that table** — flag; expect degradation on street/overlapping speech and validate on our footage.
|
||
- **WhisperX** (BSD-2-Clause, PyPI 3.8.6): *"fast ASR (70× realtime with large-v2) with word-level timestamps and speaker diarization"*; current README: *"include your Hugging Face access token … after the `--hf_token` argument and accept the user agreement for the **speaker-diarization-community-1** model"*; `--min_speakers/--max_speakers`; Spanish word alignment via **`VOXPOPULI_ASR_BASE_10K_ES`** (verified line 36 of `whisperx/alignment.py`; license: UNVERIFIED — check before phase 2).
|
||
- **NVIDIA NeMo:** toolkit Apache-2.0, but **`nvidia/diar_sortformer_4spk-v1` is CC-BY-NC-4.0 → excluded** (non-commercial). (titanet speaker embedding is CC-BY-4.0, not gated, if ever needed.)
|
||
- **diart** (MIT, Univ. Paris-Saclay): streaming diarization over pyannote; maintenance status unclear (no fresh release evidence) — not needed for offline batch processing anyway.
|
||
- **Fusion pattern (recommended):** phase 2 = WhisperX (whisper large-v3-turbo CT2 + alignment + diarization) → `transcript_segments(video, start, end, speaker_key, text, lang, words jsonb)`. `speaker_key` = ephemeral per-video label (`SPEAKER_00…`). Speaker→person mapping lives in `video_speaker_person(video_id, speaker_key, person_id, confidence, reviewed)` so **one assignment per speaker per video, and a confirmed mapping can be suggested across videos** by pairing diarization with face-track co-occurrence (same timestamps as face detections) — cheap and surprisingly reliable.
|
||
- VRAM: pyannote community-1 on GPU ≈ 2–4 GB (ESTIMATE, no official figure published — flagged).
|
||
|
||
---
|
||
|
||
## 4. Faces — license audit (the deliverable table) + quality limits
|
||
|
||
| Model / repo | Code license | **Weights terms** | Weights provenance | Verdict for us |
|
||
|---|---|---|---|---|
|
||
| InsightFace repo (`deepinsight/insightface`) | MIT — *"no limitation for both academic and commercial usage"* (README) | **NON-COMMERCIAL** — *"The training data containing the annotation (and the models trained with these data) are available for non-commercial research purposes only"*, explicitly covering *"both manual-downloading models … and auto-downloading models with our python-library"*; **2025-11-24 update: for open-sourced FR models (e.g. buffalo_l) "contact recognition-oss-pack@insightface.ai for licensing"** | MS1M/WebFace-family research datasets | **Cannot use buffalo_l / ArcFace-w600k weights** for the collective's production index |
|
||
| `public-data/insightface` HF mirror | — | no separate license field; links back to insightface GitHub release (`buffalo_l.zip`) → same non-commercial terms | same | same |
|
||
| SCRFD detector weights | (inside insightface, MIT code) | same non-commercial policy (OneDrive-hosted) | — | detector-only contamination; replaceable |
|
||
| AdaFace (`mk-minchul/AdaFace`) | MIT | **silent** — weights on Google Drive, no license attached; trained on CASIA-WebFace/VGGFace2/WebFace4M/MS1MV2-3 (README table) | research datasets with own terms | Gray zone; not "clean" |
|
||
| deepface (`serengil/deepface`) | MIT | **does not launder weights** — README: *"License types will be inherited when you intend to utilize those models"*; wraps VGG-Face/Facenet/ArcFace/**Buffalo_L** | mixed | Convenience wrapper only; license risk unchanged |
|
||
| facenet-pytorch (`timesler`) | MIT | weights ported from davidsandberg/facenet (VGGFace2/CASIA-WebFace); no explicit weight license (README) | research datasets | Older/weaker anyway; not recommended |
|
||
| MediaPipe (`google-ai-edge/mediapipe`) | **Apache-2.0** (LICENSE verified); solution docs pages show Apache 2.0 links | model files: docs do not state a per-file license clearly (flagged) | Google self-collected | **Detection** OK-ish; **no recognition/identity embedding model exists in MediaPipe** (solutions index has detector+landmarker only) → cannot do identity with MediaPipe alone |
|
||
| yolov8-face / yolov5-face | **GPL-3.0** (both LICENSE files verified) | — | — | Copyleft hazard for a distributed self-hosted stack; avoid |
|
||
| WAFiA / PaddlePaddle FR | not audited this session | — | — | **Flagged: unexamined** |
|
||
|
||
**Honest reading:** every *accurate, realistic-footage* FR embedding model in the open ecosystem (InsightFace buffalo_l, ArcFace-webface variants, AdaFace-best) is trained on MS-Celeb-1M-lineage or WebFace research datasets whose terms are **non-commercial or undefined**. There is no verified fully-clean MIT/Apache recognition model with competitive accuracy today. Options:
|
||
1. **(Recommended) Do not run automated identity matching by default.** Run **detection + tracking + blurring/on-demand reveal**, and restrict recognition to an opt-in, DPIA-approved, human-review workflow (phase 3), under legal advice. Detection models can be MediaPipe/SCRFD (SCRFD weights non-commercial → if strict, use MediaPipe BlazeFace or YuNet (libfacedetection, MIT-code — weights terms unverified, flagged)).
|
||
2. If recognition is approved: accept the non-commercial-weights risk **in writing as a collective decision** (non-commercial terms restrict *commercialization*, and a self-hosted activist tool arguably isn't a "commercial use" — but that's a legal judgment call, not mine), and prefer AdaFace R50 (MIT code, documented weights) or InsightFace-runtime-with-explicitly-acknowledged-terms.
|
||
3. Watch for genuinely clean re-licenses (InsightFace's 2025-11-24 "contact for licensing" note suggests a relicensing channel exists).
|
||
|
||
**Quality on activist footage (limits to state publicly):** masks and occlusions massively degrade FR — NIST IR 8331 (masks) exists at `https://nvlpubs.nist.gov/nistpubs/IR/2020/NIST.IR.8331.pdf` (fetched, PDF confirmed); the often-quoted *"error rates 5×–50× higher with digitally applied masks"* figure is **UNVERIFIED-MEMORY** (PDF text extraction failed in-session — re-verify before quoting). Motion blur, low light, side profiles, helmets, and distant faces push embeddings into the low-confidence band where clustering silently merges the wrong people — which in our threat model is the *most dangerous* failure (see §6).
|
||
|
||
**Sampling strategy:** PySceneDetect (0.7.1, BSD-3) per shot → 1–2 keyframes/shot plus a hard cap (e.g. ≤2 fps) → detect → track (IoU/OSNet) → 1 embedding per track per scene. This cuts GPU cost ~10× vs naive fps and improves cluster purity.
|
||
|
||
**Clustering:** HDBSCAN (0.8.44, BSD) or Chinese Whispers on cosine similarity of 512-d embeddings; ArcFace-cosine same-person band ~0.4–0.5 (UNVERIFIED-MEMORY, calibrate on our data); then **incremental assignment**: new track → match against reviewed centroids (`person_face_embeddings`); ≥τ_accept auto-assign, τ_band → review queue, else new cluster candidate. Scale note: collection-scale kNN in pgvector works (512-d `vector` < 2000-d limit, HNSW) — reuse the existing pgvector pattern.
|
||
|
||
---
|
||
|
||
## 5. Person identity model — DDL sketch
|
||
|
||
```sql
|
||
-- Canonical identities. A person starts as an unverified cluster and is
|
||
-- only promoted to a reviewed label by a human.
|
||
CREATE TABLE persons (
|
||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
display_name text, -- null until a member labels it
|
||
status text NOT NULL DEFAULT 'auto' -- auto | member_confirmed | hidden | erased
|
||
CHECK (status IN ('auto','member_confirmed','hidden','erased')),
|
||
visibility text NOT NULL DEFAULT 'members_only'
|
||
CHECK (visibility IN ('members_only','staff_only','erased')),
|
||
centroid vector(512), -- running mean of reviewed embeddings
|
||
created_by uuid NOT NULL,
|
||
created_at timestamptz NOT NULL DEFAULT now(),
|
||
updated_at timestamptz NOT NULL DEFAULT now()
|
||
);
|
||
|
||
-- Reference embeddings (reviewed only; see minimization note)
|
||
CREATE TABLE person_face_embeddings (
|
||
id bigserial PRIMARY KEY,
|
||
person_id uuid NOT NULL REFERENCES persons(id) ON DELETE CASCADE,
|
||
embedding vector(512) NOT NULL,
|
||
source text, -- 'review_ui' | 'backfill'
|
||
model text NOT NULL, -- e.g. 'adaface-r50' (pin it)
|
||
created_at timestamptz NOT NULL DEFAULT now()
|
||
);
|
||
CREATE INDEX ON person_face_embeddings USING hnsw (embedding vector_cosine_ops);
|
||
|
||
-- Every face box ever detected. Coordinates, never crops.
|
||
CREATE TABLE video_face_detections (
|
||
id bigserial PRIMARY KEY,
|
||
video_id uuid NOT NULL, -- PeerTube uuid or media-plane blob hash
|
||
source text NOT NULL DEFAULT 'peertube' CHECK (source IN ('peertube','media_plane')),
|
||
t_start_ms int NOT NULL, t_end_ms int, -- track span
|
||
bbox numeric[4] NOT NULL, -- x,y,w,h normalized
|
||
track_id int NOT NULL, -- within-video track
|
||
person_id uuid REFERENCES persons(id), -- null = unassigned
|
||
confidence real NOT NULL, -- detector conf
|
||
match_sim real, -- cosine to assigned centroid
|
||
assignment text NOT NULL DEFAULT 'auto' -- auto | human_confirmed | rejected | pending_review
|
||
CHECK (assignment IN ('auto','human_confirmed','rejected','pending_review')),
|
||
reviewed_by uuid, reviewed_at timestamptz,
|
||
embedding vector(512), -- TRANSIENT: nulled by retention job after review/SLA
|
||
created_at timestamptz NOT NULL DEFAULT now()
|
||
);
|
||
CREATE INDEX ON video_face_detections (video_id, t_start_ms);
|
||
CREATE INDEX ON video_face_detections (person_id);
|
||
CREATE INDEX ON video_face_detections (assignment) WHERE assignment = 'pending_review';
|
||
|
||
-- Phase 1/2: transcripts (feeds chat_media_archive + pgvector via the search pipeline)
|
||
CREATE TABLE transcript_segments (
|
||
id bigserial PRIMARY KEY,
|
||
video_id uuid NOT NULL,
|
||
source text NOT NULL DEFAULT 'peertube',
|
||
lang text NOT NULL, -- 'en' | 'es' (also ca/eu occur in Spain!)
|
||
start_ms int NOT NULL, end_ms int NOT NULL,
|
||
speaker_key text, -- 'SPEAKER_00' from diarization
|
||
person_id uuid REFERENCES persons(id), -- resolved speaker (phase 2)
|
||
text text NOT NULL,
|
||
words jsonb, -- word-level timestamps
|
||
confidence real,
|
||
created_at timestamptz NOT NULL DEFAULT now(),
|
||
UNIQUE (video_id, start_ms, lang)
|
||
);
|
||
|
||
-- Speaker-name bridge (label once per speaker per video; suggestions across videos)
|
||
CREATE TABLE video_speaker_person (
|
||
video_id uuid NOT NULL, speaker_key text NOT NULL,
|
||
person_id uuid REFERENCES persons(id),
|
||
confidence real, assignment text NOT NULL DEFAULT 'auto',
|
||
PRIMARY KEY (video_id, speaker_key)
|
||
);
|
||
|
||
-- Audit trail: every label action is reversible
|
||
CREATE TABLE identity_audit (
|
||
id bigserial PRIMARY KEY,
|
||
actor uuid NOT NULL, action text NOT NULL, -- assign|merge|unmerge|erase|relabel
|
||
person_id uuid, other_person_id uuid, payload jsonb,
|
||
at timestamptz NOT NULL DEFAULT now()
|
||
);
|
||
```
|
||
|
||
**Propagation rules**
|
||
- Label a face once (human confirms track → `persons.status='member_confirmed'`, centroid updated): backfill = `UPDATE video_face_detections SET person_id=$p WHERE match_sim >= τ_accept AND person_id IS NULL` (τ_accept ~0.6 calibrated); everything in the τ band goes to `pending_review`.
|
||
- **Merge personas:** union → recompute centroid → single `identity_audit` row; undo by replaying audit (append-only, never destructive).
|
||
- **Erasure (Art. 17):** `persons.status='erased'` → delete person embeddings + null all `video_face_detections.person_id` + remove speaker mappings + purge embeddings column for that person's detections; detections revert to anonymous boxes. Name/label never appears again.
|
||
- **Minimization:** `embedding` on detections is a *transient* column, wiped by a retention job (e.g. 72h after processing) — persist only reviewed centroids. No face crops on disk at rest (processing temp dir, tmpfs, TTL).
|
||
|
||
---
|
||
|
||
## 6. GDPR / threat model (Spain)
|
||
|
||
**Verified legal text (EUR-Lex CELEX:32016R0679, fetched today):**
|
||
- **Art. 4(14):** biometric data = *"personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, which allow or confirm the unique identification of that natural person, such as facial images…"*
|
||
- **Art. 9(1) prohibition:** processing of *"biometric data for the purpose of uniquely identifying a natural person"* is prohibited unless a 9(2) condition applies.
|
||
- 9(2) candidates for third parties in footage: **(a) explicit consent** (bystanders/police never gave it), **(e) manifestly made public by the data subject** (appearing in a protest video is not "manifestly making public" one's biometric template — analysis, not legal advice), **(f) vital interests** (narrow emergencies), **(g) substantial public interest — requires a Union/Member-State law providing the basis** (no obvious one for us). **Analysis: none applies cleanly to automated identity indexing of non-consenting third parties.** Note also that face *detection/blur* that does not uniquely identify arguably isn't Art. 9 data at all; the moment we build a gallery to identify people, it is.
|
||
- **Art. 35(3)(b):** DPIA is mandatory for *"processing on a large scale of special categories of personal data"* → **DPIA before phase 3**.
|
||
- Art. 5 (minimization/purpose limitation), Art. 17 (erasure), Art. 25 (protection by design), Art. 32 (security), Art. 85 (freedom of expression exemption — relevant for *publishing* footage, and LOPDGDD (LO 3/2018) develops it; article-level mapping **flagged: not verified against BOE this session**).
|
||
|
||
**EDPB / AEPD sources — fetch status this session (flagged):**
|
||
- EDPB *Guidelines 05/2022 on processing personal data through facial recognition technology in law enforcement* — known document; **all attempted URLs 404'd in this session → URL unverified** (search "EDPB Guidelines 05/2022 facial recognition" before citing).
|
||
- **AEPD (aepd.es) returned 500/503 all session** — the AEPD+CAT *guía de tratamiento de datos biométricos* and enforcement cases could not be retrieved. **Flagged as unverified.** From memory (unverified): AEPD sanctioned Ayuntamiento de Zaragoza over facial-recognition cameras (2019, small fine) — do not cite without checking.
|
||
- NIST IR 8331 masks study: PDF confirmed live (URL above); headline figure unverified (see §4).
|
||
|
||
**The specific danger: mislabeling.** A wrong person_id in activist footage (a) wrongly associates a bystander with a collective, (b) creates a target list for state/paramilitary actors if the index leaks, (c) is effectively irreversible reputationally. Therefore:
|
||
|
||
**Design mitigations (concrete):**
|
||
1. **Default = detect, don't identify.** Phase 3 ships with face *detection* + blurring/redaction tooling and a **kill switch: identity matching is an opt-in feature flag per instance**, default off, behind DPIA sign-off.
|
||
2. **Human-review gating:** nothing from `auto` assignment is ever exposed in any UI or API; only `human_confirmed` labels render, and only to roles with `identity:read`.
|
||
3. **Visibility tiers:** `persons.visibility` (members_only / staff_only) enforced at the API/query layer, not the client; face-label search endpoints excluded from `chat_media_archive` public surfaces.
|
||
4. **Minimization:** coordinates not crops (§5); transient embeddings wiped; centroids only for confirmed persons; model + threshold pinned and logged (reproducibility).
|
||
5. **Erasure workflow:** one click → §5 erasure; plus per-person "scrub from index" independent of video deletion; document that PeerTube/R2 originals stay (erasure applies to the *index*, per minimization the index is what we must justify).
|
||
6. **Audit trail** (§5) — who labeled what, when; undo/merge reversible.
|
||
7. **Blast-radius controls:** index DB on Tailscale-only interface; no face data in backups of public shares; consider per-person "high-risk" flag (police-adjacent identifications) that hard-restricts visibility to staff and logs every read.
|
||
8. **Co-official languages:** Spain footage will contain Catalan/Basque/Galician — Whisper supports them (`ca`, `eu` verified in the turbo model card language list) but with worse WER than ES; transcribe+flag language per segment rather than assuming EN/ES.
|
||
|
||
---
|
||
|
||
## 7. Pipeline integration + VRAM/cost budget
|
||
|
||
**Data flow**
|
||
```
|
||
Phase 1: PeerTube upload → transcode (existing pool) → video published
|
||
→ video_transcription (remote runner on 3090, whisper-ctranslate2 + large-v3-turbo CT2 int8)
|
||
→ captions (VTT) on R2 (prefix captions/)
|
||
→ api poller (no webhooks in PeerTube — verified): GET /api/v1/videos/{id}/captions
|
||
→ fetch VTT → parse → transcript_segments → TEI/bge-m3 → pgvector (existing search doc)
|
||
Phase 1b (media-plane blobs): upload-finalized api event → job queue → whisper-asr-webservice
|
||
(faster_whisper engine, VTT out) → same transcript_segments path
|
||
Phase 2: api event → diarization job (WhisperX: turbo CT2 + align + community-1)
|
||
→ speaker_key per segment → review UI "who is this speaker?" → video_speaker_person
|
||
Phase 3: api event → face job: PySceneDetect → keyframes → detect (BlazeFace/SCRFD) → track
|
||
→ embed (only if flag on) → assign vs reviewed centroids → pending_review queue
|
||
→ human review → persons / centroid update → backfill propagation (§5)
|
||
```
|
||
Orchestration reuses the transcode-pool pattern: PeerTube runner protocol for transcription; everything else is a small GPU worker container + Postgres-backed queue (like the api's existing transcode queue), jobs idempotent, resumable per video.
|
||
|
||
**VRAM budget (24 GB 3090, Linux node — per the transcode-pool doc, production NVIDIA work stays off WSL2; the Windows box is feasible for ASR-only via WSL2 *if* Windows-side VRAM tenants are cleared — measured 23.8/24 GB occupied there on 2026-10-09)**
|
||
|
||
| Component | fp16 | int8 | Duty |
|
||
|---|---|---|---|
|
||
| TEI + bge-m3 (already deployed) | ~2–3 GB | — | persistent |
|
||
| whisper turbo (CT2) | ~6 GB | ~3 GB | burst |
|
||
| whisper large-v3 (CT2) | ~10 GB | ~4.5 GB (README-measured on 8 GB card) | burst, quality tier |
|
||
| WhisperX + pyannote community-1 | +2–4 GB (EST.) | — | burst, phase 2 |
|
||
| SCRFD/BlazeFace + embed ONNX | ~1–2 GB (EST.) | — | burst, phase 3 |
|
||
|
||
Concurrent worst case (large-v3 + diarization + faces) ≈ 16 GB → fits; but **serialize per job class** anyway (queue). Turbo + diarization + faces ≈ 11–13 GB. Always keep the NVENC transcode pool's VRAM floor in mind on shared nodes — schedule A/V-intelligence jobs on the same nodes only when transcode queue depth is low (reuse the pool manager's pressure signal).
|
||
|
||
**Cost per video-hour (3090, ESTIMATES):** transcription turbo int8 1.5–3 min (large-v3 5–12 min) · diarization 1–3 min · keyframe faces 3–10 min (sampler-capped) → **~5–25 GPU-min per video-hour** full pipeline; backfilling 500 h ≈ 2–4 GPU-days for transcription alone.
|
||
|
||
---
|
||
|
||
## 8. Honest gaps & unverified items (all flagged in-text; consolidated)
|
||
|
||
1. **Spanish ASR on street audio** — no published benchmark matches our domain (noise/PA/chants/code-switching); FLEURS numbers absent from current HF cards. Internal eval set is mandatory.
|
||
2. **Hallucination on chants/music** — documented Whisper weakness; needs VAD + segment-level confidence and human spot-checks before any published subtitle.
|
||
3. **Diarization has no Spanish benchmark** in pyannote's table; overlapping-protest-speech DER will be far worse than AMI numbers.
|
||
4. **Face weights licensing is dirty everywhere** (§4) — the accurate models are non-commercial/undefined; no clean equal exists today. Decision needed at collective level + legal advice; SCRFD/YuNet/MediaPipe weight-file license status partially unverified.
|
||
5. **NIST mask figure**, **EDPB 05/2022 URL**, **AEPD guidance/enforcement** — could not be fully retrieved this session; re-verify before citing (aepd.es was 5xx all day).
|
||
6. **PeerTube runner + CUDA pickup** for `whisper-ctranslate2`, and whether `model_path` accepts a turbo CT2 dir in the runner path — one smoke test needed.
|
||
7. **PeerTube plugin transcription hook names** — plugin system exists (verified import), specific hook unverified.
|
||
8. **Speaches diarization endpoints**, **diart maintenance status**, **chinese-whispers package**, **pyannote community-1 VRAM**, **3090 RTF figures** — ESTIMATE/UNVERIFIED-MEMORY as marked.
|
||
9. **WhisperX Spanish alignment model license** (VoxPopuli weights) — check before phase 2.
|
||
|
||
## Key primary sources (fetched 2026-10-11)
|
||
- PeerTube tree/CHANGELOG/source: https://github.com/Chocobozzz/PeerTube (v8.3.1; `config/default.yaml`, `packages/transcription/*`, `server/core/controllers/api/videos/captions.ts`, runner `process-transcription.ts`)
|
||
- HF API/cards: openai/whisper-large-v3 (Apache-2.0), openai/whisper-large-v3-turbo (MIT, 32→4 layers, 809M), distil-whisper/distil-large-v3 (MIT), Systran/faster-whisper-large-v3 (MIT), pyannote/speaker-diarization-community-1 (CC-BY-4.0, gated auto), pyannote/speaker-diarization-3.1 + segmentation-3.0 (MIT, gated auto), nvidia/diar_sortformer_4spk-v1 (**CC-BY-NC-4.0**)
|
||
- faster-whisper README (VRAM/RTF benchmarks), whisper.cpp v1.9.5 (VTT/CUDA/Vulkan/Q5_0), whisperX README + alignment.py (community-1, `VOXPOPULI_ASR_BASE_10K_ES`), Speaches v0.9.0-rc.3 (MIT), whisper-asr-webservice v1.10.0
|
||
- insightface README (MIT code / non-commercial models + 2025-11-24 relicensing note), AdaFace (MIT code, Drive weights), deepface (MIT + inherited-weights disclaimer), MediaPipe (Apache-2.0), yolov8/yolov5-face (GPL-3.0), facenet-pytorch (MIT), PySceneDetect 0.7.1 (BSD-3), hdbscan 0.8.44 (BSD)
|
||
- EUR-Lex GDPR (Art. 4(14), 9, 35(3)(b) quoted), NIST IR 8331 PDF (exists; figure unverified), EDPB/AEPD (unreachable this session — flagged)
|