hacklab/T-HUB
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
T-HUB/docs/design/2026-10-11-video-ai-architecture.md
glass beadian 777c67f7a3 Expand video-AI architecture doc to the full research report 
281-line version with the person-identity DDL sketch, per-model license
audit table, VRAM/cost budget, GDPR Art. 9 analysis, and all primary-source
citations — replaces the 53-line executive summary.
2026-10-11 08:29:30 +00:00

31 KiB
Raw Permalink Blame History

A/V Intelligence Pipeline — Research Report (transcription · diarization · faces · identity · GDPR)

Date: 2026-10-11 · Scope: Tactical Media Broadcast Hub (PeerTube occ-tube + R2 media plane, NestJS api, pgvector, RTX 3090 fleet on Tailscale, GDPR/Spain, EN+ES, no third-party cloud APIs). Method: every claim below was checked against a primary source fetched live in this session (GitHub source/release API, HuggingFace API/model cards, PyPI, EUR-Lex, nvlpubs) unless explicitly marked UNVERIFIED-MEMORY or ESTIMATE. Versions quoted are current as of today.

0. Recommended architecture (TL;DR)

Concern Choice License Why
Transcription PeerTube built-in (v8.3.1), engine whisper-ctranslate2, model large-v3-turbo (int8), executed by a remote runner on the 3090 Whisper weights MIT/Apache-2.0 (see §2) It ships today, subtitles land in PeerTube's caption system (VTT), API retrievable, zero glue code
Word timestamps faster-whisper word_timestamps / WhisperX alignment (phase 2) BSD-2/MIT VTT + speaker fusion
Diarization pyannote speaker-diarization-community-1 via WhisperX (or standalone) CC-BY-4.0 weights, MIT code, gated auto Best open DER; the gate is access-control, not a license restriction
Face detection SCRFD (ONNX, via insightface pip for runtime only) or MediaPipe BlazeFace if you want zero-contamination pipeline weights non-commercial (SCRFD/buffalo) / Apache-2.0 (MediaPipe) See §4 — this is the one genuinely dirty licensing corner; options given
Face recognition Deferred to phase 3, human-review-gated. If run: AdaFace/ArcFace-class 512-d embeddings; store centroids, not crops — GDPR Art. 9 — see §6; weight licenses are non-commercial across the board for the accurate models
Identity model Postgres tables in §5: persons, person_face_embeddings, video_face_detections, transcript_segments + review queue + audit — Label once → propagate; undo/merge; erasure
Orchestration Reuse the transcode-pool pattern: PeerTube runner for transcription jobs; a small GPU worker container (like TEI) + Postgres-backed job queue for the rest — Few moving parts, no k8s

Phase 1 = transcription+subtitles (PeerTube-native, low risk). Phase 2 = diarization + speaker naming. Phase 3 = faces/clustering with human-review gating (only after DPIA).

1. PeerTube transcription — verified from v8.3.1 source

Current release: v8.3.1 (2026-09-28) via GitHub releases API.

  • Built into core since 6.2 — CHANGELOG: "Add automatic transcription of videos to generate subtitles" (PR #6303), "Transcription can also be performed by PeerTube runners, as it can consume a lot of CPU", and "PeerTube instance requires python >= 3.8 for transcription"; pip3 is a listed dependency. Config (source: config/default.yaml, video_transcription:):
    • enabled: false (default off), engine: 'whisper-ctranslate2' (alternative openai-whisper), model: 'small' (choices tiny|base|small|medium|large-v2|large-v3 — no turbo entry; a CTranslate2 turbo dir can be used via model_path), model_path: null, timeout: '6 hours', remote_runners.enabled: false.
  • Engines verified (packages/transcription/src/whisper/engines.ts): openai-whisper (v20250625, MIT) and whisper-ctranslate2 (>=0.4.3, MIT, CTranslate2 format). packages/transcription is an internal package (@peertube/peertube-transcription) wrapping these CLIs; transcription runs in the Node job queue locally or as a runner job (server/core/lib/runners/job-handlers/transcription-job-handler.ts; runner code: apps/peertube-runner/.../process-transcription.ts).
  • Where subtitles land: caption rows in PeerTube DB + VTT files under storage/captions/ (or object-storage prefix captions/ — migrations 0875-caption-object-storage, 0890-hls-caption; a recent changelog line "Fix transcription CORS issue if the caption file is stored externally" confirms captions on R2 are a supported posture).
  • API (verified routes in server/core/controllers/api/videos/captions.ts):
    • GET /api/v1/videos/{id}/captions (public listing) → caption file URLs (.vtt)
    • PUT /api/v1/videos/{id}/captions/{lang} (upload VTT, multipart)
    • DELETE /api/v1/videos/{id}/captions/{lang}
    • POST /api/v1/videos/{id}/captions/generate (request transcription manually; generateTranscription also honored on upload/import per changelog)
  • No webhooks exist in PeerTube (repo-tree grep: zero webhook files). Completion signals available: (a) user notification video-transcription-generated-for-owner (not programmatic), (b) REST polling, (c) a custom plugin — the caption controller imports the plugin Hooks subsystem (specific transcription hook names NOT verified — flagged).
  • Transcription widget in the web player exists (v6.3+ changelog) and re-runs happen after file replacement (v7.x changelog).

Implication for us: enable video_transcription with remote_runners.enabled: true and run the official peertube-runner on a Linux GPU node with the transcription capability — same runner contract as the transcode pool (companion doc). The api then watches the captions endpoint on video-publish and pulls the VTT. Whether the runner path picks up CUDA for whisper-ctranslate2 needs one smoke test (flagged, §8).

2. Whisper family — models, licenses, Spanish, VRAM

Model License (HF API) Spanish VRAM Notes
openai/whisper-large-v3 Apache-2.0 strong (paper-level; card no longer publishes the per-language table) fp16 ~10 GB, int8 ~4.5 GB (README-measured class) Best accuracy ceiling
openai/whisper-large-v3-turbo MIT multilingual; card states "the exact same model, except that the number of decoding layers have reduced from 32 to 4", 809 M params fp16 ~6 GB, int8 ~3 GB (ESTIMATE) Recommended default: ~95% of v3 quality at a fraction of the cost
distil-whisper/distil-large-v3 MIT English-only (UNVERIFIED-MEMORY: card language restriction; not re-fetched today) ~4 GB Exclude for ES
Systran/faster-whisper-large-v3 (+turbo conversions) MIT = v3 — The CTranslate2 conversions PeerTube's engine consumes
  • Spanish WER reality: the current HF cards no longer carry the FLEURS/Common-Voice per-language tables (checked both cards — absent). From memory (UNVERIFIED-MEMORY): large-v3 FLEURS-es ≈ 5–7% WER, Common Voice-es ≈ 8–15%; expect 15–35%+ on street footage (wind, PA, chants, overlapping speech, masks). Build a 20–30-clip internal ES/EN eval set (same as the search-architecture doc's eval plan) before choosing large-v3 vs turbo.
  • Whisper failure modes (from the v3 card, verified text): "prone to generating repetitive texts"; hallucination risk on music/chants/silence — directly relevant to protest footage. Mitigation: VAD pre-filter (WhisperX/faster-whisper do this), temperature-fallback, human check of chant-heavy segments.
  • Runtime (faster-whisper 1.2.1, PyPI): README-verified benchmark: large-v3 int8, batch_size=8 → 4500 MB peak, 5× realtime on an RTX 3070 Ti 8GB; BatchedInferencePipeline documented; word timestamps supported by the API (word_timestamps=True — standard; word output also verified downstream in whisper-asr-webservice "Word-level timestamps support").
  • whisper.cpp v1.9.5 (2026-10-06): CPU/Windows fallback verified — -ovtt/--output-vtt in examples/cli/cli.cpp, CUDA + Vulkan backends, Q5_0 quantization. Multilingual ggml models only for ES. Use for the Windows node without WSL2 or as CPU emergency path.
  • 3090 throughput (ESTIMATE, extrapolated): turbo int8 batched ≈ 20–40× realtime (1.5–3 min per video-hour); large-v3 int8 ≈ 8–12× realtime (5–8 min per video-hour).

Serving options (if we outgrow PeerTube's runner):

  • Speaches (MIT, v0.9.0-rc.3, 2025-12-27): OpenAI-compatible server over faster-whisper + TTS; modern but pre-1.0. Diarization endpoints: UNVERIFIED-MEMORY (not in README excerpt fetched).
  • whisper-asr-webservice (v1.10.0, 2026-08-09): engines openai_whisper | faster_whisper | whisperx, outputs text/JSON/VTT/SRT/TSV, word-level timestamps, diarization via WhisperX engine (all README-verified). The most "boring and works" option; good as a standalone microservice feeding transcript_segments for media-plane blobs that never enter PeerTube.

3. Speaker diarization

  • pyannote.audio 4.0.7 (PyPI). Code is MIT (repo LICENSE, Copyright CNRS).
  • Model gating — the important nuance (verified):
    • pyannote/speaker-diarization-community-1 (lastModified 2025-09-29, ~5.5 M downloads): license CC-BY-4.0, gated: auto.
    • pyannote/speaker-diarization-3.1 and segmentation-3.0: MIT, gated: auto.
    • Empirically: fetching the model card without auth returns HTTP 401 ("Access to model … is restricted. You must have access to it and be authenticated").
    • gated: auto = you need an HF account, accept the model's user conditions once (name/email are recorded by HF), and pass a read token. It is access control + attribution, not a use restriction — no non-commercial clause found anywhere in the cards or the repo license. Verdict: OK for this use (CC-BY-4.0 needs attribution — add a models/credits page).
    • The current pyannote README instructs exactly: install pyannote.audio, "Accept pyannote/speaker-diarization-community-1 user conditions", create an HF access token, then Pipeline.from_pretrained(..., token=...), runs locally, .to("cuda").
  • Quality (official README benchmark table, DER %, lower better): community-1 beats 3.1 across the board — AMI IHM 17.0 vs 18.8, DIHARD 3 20.2 vs 21.4, MSDWild 22.8 vs 25.4, Ego4D 46.8 vs 51.2. No Spanish-specific benchmark exists in that table — flag; expect degradation on street/overlapping speech and validate on our footage.
  • WhisperX (BSD-2-Clause, PyPI 3.8.6): "fast ASR (70× realtime with large-v2) with word-level timestamps and speaker diarization"; current README: "include your Hugging Face access token … after the --hf_token argument and accept the user agreement for the speaker-diarization-community-1 model"; --min_speakers/--max_speakers; Spanish word alignment via VOXPOPULI_ASR_BASE_10K_ES (verified line 36 of whisperx/alignment.py; license: UNVERIFIED — check before phase 2).
  • NVIDIA NeMo: toolkit Apache-2.0, but nvidia/diar_sortformer_4spk-v1 is CC-BY-NC-4.0 → excluded (non-commercial). (titanet speaker embedding is CC-BY-4.0, not gated, if ever needed.)
  • diart (MIT, Univ. Paris-Saclay): streaming diarization over pyannote; maintenance status unclear (no fresh release evidence) — not needed for offline batch processing anyway.
  • Fusion pattern (recommended): phase 2 = WhisperX (whisper large-v3-turbo CT2 + alignment + diarization) → transcript_segments(video, start, end, speaker_key, text, lang, words jsonb). speaker_key = ephemeral per-video label (SPEAKER_00…). Speaker→person mapping lives in video_speaker_person(video_id, speaker_key, person_id, confidence, reviewed) so one assignment per speaker per video, and a confirmed mapping can be suggested across videos by pairing diarization with face-track co-occurrence (same timestamps as face detections) — cheap and surprisingly reliable.
  • VRAM: pyannote community-1 on GPU ≈ 2–4 GB (ESTIMATE, no official figure published — flagged).

4. Faces — license audit (the deliverable table) + quality limits

Model / repo Code license Weights terms Weights provenance Verdict for us
InsightFace repo (deepinsight/insightface) MIT — "no limitation for both academic and commercial usage" (README) NON-COMMERCIAL — "The training data containing the annotation (and the models trained with these data) are available for non-commercial research purposes only", explicitly covering "both manual-downloading models … and auto-downloading models with our python-library"; 2025-11-24 update: for open-sourced FR models (e.g. buffalo_l) "contact recognition-oss-pack@insightface.ai for licensing" MS1M/WebFace-family research datasets Cannot use buffalo_l / ArcFace-w600k weights for the collective's production index
public-data/insightface HF mirror — no separate license field; links back to insightface GitHub release (buffalo_l.zip) → same non-commercial terms same same
SCRFD detector weights (inside insightface, MIT code) same non-commercial policy (OneDrive-hosted) — detector-only contamination; replaceable
AdaFace (mk-minchul/AdaFace) MIT silent — weights on Google Drive, no license attached; trained on CASIA-WebFace/VGGFace2/WebFace4M/MS1MV2-3 (README table) research datasets with own terms Gray zone; not "clean"
deepface (serengil/deepface) MIT does not launder weights — README: "License types will be inherited when you intend to utilize those models"; wraps VGG-Face/Facenet/ArcFace/Buffalo_L mixed Convenience wrapper only; license risk unchanged
facenet-pytorch (timesler) MIT weights ported from davidsandberg/facenet (VGGFace2/CASIA-WebFace); no explicit weight license (README) research datasets Older/weaker anyway; not recommended
MediaPipe (google-ai-edge/mediapipe) Apache-2.0 (LICENSE verified); solution docs pages show Apache 2.0 links model files: docs do not state a per-file license clearly (flagged) Google self-collected Detection OK-ish; no recognition/identity embedding model exists in MediaPipe (solutions index has detector+landmarker only) → cannot do identity with MediaPipe alone
yolov8-face / yolov5-face GPL-3.0 (both LICENSE files verified) — — Copyleft hazard for a distributed self-hosted stack; avoid
WAFiA / PaddlePaddle FR not audited this session — — Flagged: unexamined

Honest reading: every accurate, realistic-footage FR embedding model in the open ecosystem (InsightFace buffalo_l, ArcFace-webface variants, AdaFace-best) is trained on MS-Celeb-1M-lineage or WebFace research datasets whose terms are non-commercial or undefined. There is no verified fully-clean MIT/Apache recognition model with competitive accuracy today. Options:

  1. (Recommended) Do not run automated identity matching by default. Run detection + tracking + blurring/on-demand reveal, and restrict recognition to an opt-in, DPIA-approved, human-review workflow (phase 3), under legal advice. Detection models can be MediaPipe/SCRFD (SCRFD weights non-commercial → if strict, use MediaPipe BlazeFace or YuNet (libfacedetection, MIT-code — weights terms unverified, flagged)).
  2. If recognition is approved: accept the non-commercial-weights risk in writing as a collective decision (non-commercial terms restrict commercialization, and a self-hosted activist tool arguably isn't a "commercial use" — but that's a legal judgment call, not mine), and prefer AdaFace R50 (MIT code, documented weights) or InsightFace-runtime-with-explicitly-acknowledged-terms.
  3. Watch for genuinely clean re-licenses (InsightFace's 2025-11-24 "contact for licensing" note suggests a relicensing channel exists).

Quality on activist footage (limits to state publicly): masks and occlusions massively degrade FR — NIST IR 8331 (masks) exists at https://nvlpubs.nist.gov/nistpubs/IR/2020/NIST.IR.8331.pdf (fetched, PDF confirmed); the often-quoted "error rates 5×–50× higher with digitally applied masks" figure is UNVERIFIED-MEMORY (PDF text extraction failed in-session — re-verify before quoting). Motion blur, low light, side profiles, helmets, and distant faces push embeddings into the low-confidence band where clustering silently merges the wrong people — which in our threat model is the most dangerous failure (see §6).

Sampling strategy: PySceneDetect (0.7.1, BSD-3) per shot → 1–2 keyframes/shot plus a hard cap (e.g. ≤2 fps) → detect → track (IoU/OSNet) → 1 embedding per track per scene. This cuts GPU cost ~10× vs naive fps and improves cluster purity.

Clustering: HDBSCAN (0.8.44, BSD) or Chinese Whispers on cosine similarity of 512-d embeddings; ArcFace-cosine same-person band ~0.4–0.5 (UNVERIFIED-MEMORY, calibrate on our data); then incremental assignment: new track → match against reviewed centroids (person_face_embeddings); ≥τ_accept auto-assign, τ_band → review queue, else new cluster candidate. Scale note: collection-scale kNN in pgvector works (512-d vector < 2000-d limit, HNSW) — reuse the existing pgvector pattern.

5. Person identity model — DDL sketch

-- Canonical identities. A person starts as an unverified cluster and is
-- only promoted to a reviewed label by a human.
CREATE TABLE persons (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  display_name  text,                          -- null until a member labels it
  status        text NOT NULL DEFAULT 'auto'   -- auto | member_confirmed | hidden | erased
                CHECK (status IN ('auto','member_confirmed','hidden','erased')),
  visibility    text NOT NULL DEFAULT 'members_only'
                CHECK (visibility IN ('members_only','staff_only','erased')),
  centroid      vector(512),                   -- running mean of reviewed embeddings
  created_by    uuid NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  updated_at    timestamptz NOT NULL DEFAULT now()
);

-- Reference embeddings (reviewed only; see minimization note)
CREATE TABLE person_face_embeddings (
  id          bigserial PRIMARY KEY,
  person_id   uuid NOT NULL REFERENCES persons(id) ON DELETE CASCADE,
  embedding   vector(512) NOT NULL,
  source      text,                            -- 'review_ui' | 'backfill'
  model       text NOT NULL,                   -- e.g. 'adaface-r50' (pin it)
  created_at  timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON person_face_embeddings USING hnsw (embedding vector_cosine_ops);

-- Every face box ever detected. Coordinates, never crops.
CREATE TABLE video_face_detections (
  id            bigserial PRIMARY KEY,
  video_id      uuid NOT NULL,                -- PeerTube uuid or media-plane blob hash
  source        text NOT NULL DEFAULT 'peertube' CHECK (source IN ('peertube','media_plane')),
  t_start_ms    int NOT NULL, t_end_ms int,   -- track span
  bbox          numeric[4] NOT NULL,          -- x,y,w,h normalized
  track_id      int NOT NULL,                 -- within-video track
  person_id     uuid REFERENCES persons(id),  -- null = unassigned
  confidence    real NOT NULL,                -- detector conf
  match_sim     real,                         -- cosine to assigned centroid
  assignment    text NOT NULL DEFAULT 'auto'  -- auto | human_confirmed | rejected | pending_review
                CHECK (assignment IN ('auto','human_confirmed','rejected','pending_review')),
  reviewed_by   uuid, reviewed_at timestamptz,
  embedding     vector(512),                  -- TRANSIENT: nulled by retention job after review/SLA
  created_at    timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON video_face_detections (video_id, t_start_ms);
CREATE INDEX ON video_face_detections (person_id);
CREATE INDEX ON video_face_detections (assignment) WHERE assignment = 'pending_review';

-- Phase 1/2: transcripts (feeds chat_media_archive + pgvector via the search pipeline)
CREATE TABLE transcript_segments (
  id          bigserial PRIMARY KEY,
  video_id    uuid NOT NULL,
  source      text NOT NULL DEFAULT 'peertube',
  lang        text NOT NULL,                  -- 'en' | 'es' (also ca/eu occur in Spain!)
  start_ms    int NOT NULL, end_ms int NOT NULL,
  speaker_key text,                           -- 'SPEAKER_00' from diarization
  person_id   uuid REFERENCES persons(id),    -- resolved speaker (phase 2)
  text        text NOT NULL,
  words       jsonb,                          -- word-level timestamps
  confidence  real,
  created_at  timestamptz NOT NULL DEFAULT now(),
  UNIQUE (video_id, start_ms, lang)
);

-- Speaker-name bridge (label once per speaker per video; suggestions across videos)
CREATE TABLE video_speaker_person (
  video_id uuid NOT NULL, speaker_key text NOT NULL,
  person_id uuid REFERENCES persons(id),
  confidence real, assignment text NOT NULL DEFAULT 'auto',
  PRIMARY KEY (video_id, speaker_key)
);

-- Audit trail: every label action is reversible
CREATE TABLE identity_audit (
  id bigserial PRIMARY KEY,
  actor uuid NOT NULL, action text NOT NULL,   -- assign|merge|unmerge|erase|relabel
  person_id uuid, other_person_id uuid, payload jsonb,
  at timestamptz NOT NULL DEFAULT now()
);

Propagation rules

  • Label a face once (human confirms track → persons.status='member_confirmed', centroid updated): backfill = UPDATE video_face_detections SET person_id=$p WHERE match_sim >= τ_accept AND person_id IS NULL (τ_accept ~0.6 calibrated); everything in the τ band goes to pending_review.
  • Merge personas: union → recompute centroid → single identity_audit row; undo by replaying audit (append-only, never destructive).
  • Erasure (Art. 17): persons.status='erased' → delete person embeddings + null all video_face_detections.person_id + remove speaker mappings + purge embeddings column for that person's detections; detections revert to anonymous boxes. Name/label never appears again.
  • Minimization: embedding on detections is a transient column, wiped by a retention job (e.g. 72h after processing) — persist only reviewed centroids. No face crops on disk at rest (processing temp dir, tmpfs, TTL).

6. GDPR / threat model (Spain)

Verified legal text (EUR-Lex CELEX:32016R0679, fetched today):

  • Art. 4(14): biometric data = "personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, which allow or confirm the unique identification of that natural person, such as facial images…"
  • Art. 9(1) prohibition: processing of "biometric data for the purpose of uniquely identifying a natural person" is prohibited unless a 9(2) condition applies.
  • 9(2) candidates for third parties in footage: (a) explicit consent (bystanders/police never gave it), (e) manifestly made public by the data subject (appearing in a protest video is not "manifestly making public" one's biometric template — analysis, not legal advice), (f) vital interests (narrow emergencies), (g) substantial public interest — requires a Union/Member-State law providing the basis (no obvious one for us). Analysis: none applies cleanly to automated identity indexing of non-consenting third parties. Note also that face detection/blur that does not uniquely identify arguably isn't Art. 9 data at all; the moment we build a gallery to identify people, it is.
  • Art. 35(3)(b): DPIA is mandatory for "processing on a large scale of special categories of personal data" → DPIA before phase 3.
  • Art. 5 (minimization/purpose limitation), Art. 17 (erasure), Art. 25 (protection by design), Art. 32 (security), Art. 85 (freedom of expression exemption — relevant for publishing footage, and LOPDGDD (LO 3/2018) develops it; article-level mapping flagged: not verified against BOE this session).

EDPB / AEPD sources — fetch status this session (flagged):

  • EDPB Guidelines 05/2022 on processing personal data through facial recognition technology in law enforcement — known document; all attempted URLs 404'd in this session → URL unverified (search "EDPB Guidelines 05/2022 facial recognition" before citing).
  • AEPD (aepd.es) returned 500/503 all session — the AEPD+CAT guía de tratamiento de datos biométricos and enforcement cases could not be retrieved. Flagged as unverified. From memory (unverified): AEPD sanctioned Ayuntamiento de Zaragoza over facial-recognition cameras (2019, small fine) — do not cite without checking.
  • NIST IR 8331 masks study: PDF confirmed live (URL above); headline figure unverified (see §4).

The specific danger: mislabeling. A wrong person_id in activist footage (a) wrongly associates a bystander with a collective, (b) creates a target list for state/paramilitary actors if the index leaks, (c) is effectively irreversible reputationally. Therefore:

Design mitigations (concrete):

  1. Default = detect, don't identify. Phase 3 ships with face detection + blurring/redaction tooling and a kill switch: identity matching is an opt-in feature flag per instance, default off, behind DPIA sign-off.
  2. Human-review gating: nothing from auto assignment is ever exposed in any UI or API; only human_confirmed labels render, and only to roles with identity:read.
  3. Visibility tiers: persons.visibility (members_only / staff_only) enforced at the API/query layer, not the client; face-label search endpoints excluded from chat_media_archive public surfaces.
  4. Minimization: coordinates not crops (§5); transient embeddings wiped; centroids only for confirmed persons; model + threshold pinned and logged (reproducibility).
  5. Erasure workflow: one click → §5 erasure; plus per-person "scrub from index" independent of video deletion; document that PeerTube/R2 originals stay (erasure applies to the index, per minimization the index is what we must justify).
  6. Audit trail (§5) — who labeled what, when; undo/merge reversible.
  7. Blast-radius controls: index DB on Tailscale-only interface; no face data in backups of public shares; consider per-person "high-risk" flag (police-adjacent identifications) that hard-restricts visibility to staff and logs every read.
  8. Co-official languages: Spain footage will contain Catalan/Basque/Galician — Whisper supports them (ca, eu verified in the turbo model card language list) but with worse WER than ES; transcribe+flag language per segment rather than assuming EN/ES.

7. Pipeline integration + VRAM/cost budget

Data flow

Phase 1: PeerTube upload → transcode (existing pool) → video published
         → video_transcription (remote runner on 3090, whisper-ctranslate2 + large-v3-turbo CT2 int8)
         → captions (VTT) on R2 (prefix captions/)
         → api poller (no webhooks in PeerTube — verified): GET /api/v1/videos/{id}/captions
         → fetch VTT → parse → transcript_segments → TEI/bge-m3 → pgvector (existing search doc)
Phase 1b (media-plane blobs): upload-finalized api event → job queue → whisper-asr-webservice
         (faster_whisper engine, VTT out) → same transcript_segments path
Phase 2: api event → diarization job (WhisperX: turbo CT2 + align + community-1)
         → speaker_key per segment → review UI "who is this speaker?" → video_speaker_person
Phase 3: api event → face job: PySceneDetect → keyframes → detect (BlazeFace/SCRFD) → track
         → embed (only if flag on) → assign vs reviewed centroids → pending_review queue
         → human review → persons / centroid update → backfill propagation (§5)

Orchestration reuses the transcode-pool pattern: PeerTube runner protocol for transcription; everything else is a small GPU worker container + Postgres-backed queue (like the api's existing transcode queue), jobs idempotent, resumable per video.

VRAM budget (24 GB 3090, Linux node — per the transcode-pool doc, production NVIDIA work stays off WSL2; the Windows box is feasible for ASR-only via WSL2 if Windows-side VRAM tenants are cleared — measured 23.8/24 GB occupied there on 2026-10-09)

Component fp16 int8 Duty
TEI + bge-m3 (already deployed) ~2–3 GB — persistent
whisper turbo (CT2) ~6 GB ~3 GB burst
whisper large-v3 (CT2) ~10 GB ~4.5 GB (README-measured on 8 GB card) burst, quality tier
WhisperX + pyannote community-1 +2–4 GB (EST.) — burst, phase 2
SCRFD/BlazeFace + embed ONNX ~1–2 GB (EST.) — burst, phase 3

Concurrent worst case (large-v3 + diarization + faces) ≈ 16 GB → fits; but serialize per job class anyway (queue). Turbo + diarization + faces ≈ 11–13 GB. Always keep the NVENC transcode pool's VRAM floor in mind on shared nodes — schedule A/V-intelligence jobs on the same nodes only when transcode queue depth is low (reuse the pool manager's pressure signal).

Cost per video-hour (3090, ESTIMATES): transcription turbo int8 1.5–3 min (large-v3 5–12 min) · diarization 1–3 min · keyframe faces 3–10 min (sampler-capped) → ~5–25 GPU-min per video-hour full pipeline; backfilling 500 h ≈ 2–4 GPU-days for transcription alone.

8. Honest gaps & unverified items (all flagged in-text; consolidated)

  1. Spanish ASR on street audio — no published benchmark matches our domain (noise/PA/chants/code-switching); FLEURS numbers absent from current HF cards. Internal eval set is mandatory.
  2. Hallucination on chants/music — documented Whisper weakness; needs VAD + segment-level confidence and human spot-checks before any published subtitle.
  3. Diarization has no Spanish benchmark in pyannote's table; overlapping-protest-speech DER will be far worse than AMI numbers.
  4. Face weights licensing is dirty everywhere (§4) — the accurate models are non-commercial/undefined; no clean equal exists today. Decision needed at collective level + legal advice; SCRFD/YuNet/MediaPipe weight-file license status partially unverified.
  5. NIST mask figure, EDPB 05/2022 URL, AEPD guidance/enforcement — could not be fully retrieved this session; re-verify before citing (aepd.es was 5xx all day).
  6. PeerTube runner + CUDA pickup for whisper-ctranslate2, and whether model_path accepts a turbo CT2 dir in the runner path — one smoke test needed.
  7. PeerTube plugin transcription hook names — plugin system exists (verified import), specific hook unverified.
  8. Speaches diarization endpoints, diart maintenance status, chinese-whispers package, pyannote community-1 VRAM, 3090 RTF figures — ESTIMATE/UNVERIFIED-MEMORY as marked.
  9. WhisperX Spanish alignment model license (VoxPopuli weights) — check before phase 2.

Key primary sources (fetched 2026-10-11)

  • PeerTube tree/CHANGELOG/source: https://github.com/Chocobozzz/PeerTube (v8.3.1; config/default.yaml, packages/transcription/*, server/core/controllers/api/videos/captions.ts, runner process-transcription.ts)
  • HF API/cards: openai/whisper-large-v3 (Apache-2.0), openai/whisper-large-v3-turbo (MIT, 32→4 layers, 809M), distil-whisper/distil-large-v3 (MIT), Systran/faster-whisper-large-v3 (MIT), pyannote/speaker-diarization-community-1 (CC-BY-4.0, gated auto), pyannote/speaker-diarization-3.1 + segmentation-3.0 (MIT, gated auto), nvidia/diar_sortformer_4spk-v1 (CC-BY-NC-4.0)
  • faster-whisper README (VRAM/RTF benchmarks), whisper.cpp v1.9.5 (VTT/CUDA/Vulkan/Q5_0), whisperX README + alignment.py (community-1, VOXPOPULI_ASR_BASE_10K_ES), Speaches v0.9.0-rc.3 (MIT), whisper-asr-webservice v1.10.0
  • insightface README (MIT code / non-commercial models + 2025-11-24 relicensing note), AdaFace (MIT code, Drive weights), deepface (MIT + inherited-weights disclaimer), MediaPipe (Apache-2.0), yolov8/yolov5-face (GPL-3.0), facenet-pytorch (MIT), PySceneDetect 0.7.1 (BSD-3), hdbscan 0.8.44 (BSD)
  • EUR-Lex GDPR (Art. 4(14), 9, 35(3)(b) quoted), NIST IR 8331 PDF (exists; figure unverified), EDPB/AEPD (unreachable this session — flagged)