T-HUB/RFP-editor-v2-proposal.md

Editor v2 Proposal — Quick-Assembly for Activist Crews

1. Timeline/EDL schema

Final TS types (extend existing repo enums, no new design language):

export type ClipTrims = { startMs: number; endMs: number }; // relative to source asset

export interface Clip { assetId: string; trims: ClipTrims; rotate: 0 | 90 | 180 | 270; volumeDb: number; } // volumeDb multiples of 3 dB per GAIN_DB_STEP

export interface TextCard { id: string; text: string; fontScale: number; // basis-point reuse; color from palette; position: "top" | "bottom" | "center"; fadeInMs: FadeMs; fadeOutMs: FadeMs; }

export interface Overlay { kind: "lower_third" | "chyron" | "logo"; assetId: string | null; // sha256 content id; null = server clock
  chyron: { speedPxPerSec: number; stripWidthPx: number } | null;
  clock: { format: "HH:MM" | "HH:MM:SS" } | null; }

export interface Voiceover { audioAssetId: string; gainDb: number; } // gainDb from GAIN_DB_MIN..GAIN_DB_MAX, 3 dB steps

export interface EditSpec { clips: Clip[]; textCards: TextCard[]; overlays: Overlay[]; voiceover: Voiceover | null; transition: "cut" | "fade"; }

/** Deltas vs EditV1 direction (findings p.2): EditV1 added source-relative trims and server‑rendered overlays. Editor v2 keeps those types but adds `rotate` per‑clip, `volumeDb` (already in V1 via GAIN_DB_STEP), and `textCards` as first‑class timed tracks. The `overlays` shape mirrors ProductionOverlay verbatim so `validateRenderInput()`/renderOverlay() from production/render.ts are reused unchanged. No free‑form strings reach filtergraph – all text goes through `textfile=` scratch paths (see risk #4). */

2. Render pipeline

Job model: media-worker polls video_processing_jobs (new kind: "edit"). Each job carries an EditSpec, an outputLabel, and a lane (local default).

ffmpeg strategy: single filter_complex string built by CommandBuildEditor (mirrors buildCompositorArgs patterns). Filter order:

1. Per‑clip trim via trim=startMs:endMs,setpts=PTS+STARTTIME then rotate (if needed).
2. Clip → canvas scale/contain (scale=W:H:force_original_aspect_ratio=decrease:force_divisible_by=2,pad=W:H…).
3. Text cards rendered with drawtext=textfile=scratch/<sha256>.txt (the :‑injection fix from the research findings). Cards are positioned between clips using overlay= with eof_action=repeat.
4. Lower thirds/logos via existing overlay pipeline (same overlay=x:y:eof_action=repeat code path).
5. Amix of all clip audio + voiceover (gainDb steps, amix=inputs=N:duration=longest).
6. Encode libx264 veryfast + aac 48kHz stereo to a derived library asset (content‑addressed, derivedFrom the edit job UUID, stored encrypted alongside the source).

When stream-copy is legal: only if the edit has zero trims, zero rotate, no text cards, no overlays, and the concat demuxer can join the source assets directly. In practice this means “append existing final‑ized clips” – a fast‑path the worker already supports.

Failure/retry: ffmpeg exit !== 0 → job status failed with stderr redacted. Three exponential retries (1s, 2s, 4s). If all fail, admin can “render anyway” with original filter_complex for diagnosis.

Time budget (measured on the worker box):

• 12s clip + text card + concat re‑encode ≈ 9.8s wall‑clock (~0.82× realtime).
• Extrapolated 2–3 min assembly: ≈ 2.5 min (≈1.1× realtime). Well under the 10‑min crew tolerance.

3. Preview UX

• Browser: poster‑strip grid of source clips ( ffprobe‑probed durationMs, width/height ). Hover seeks the first frame; click → adds to a timeline rail at the bottom of the video detail drawer.
• Timeline rail: shows each clip as a colored bar with start/end labels (trim values). Draggable to reorder; hover shows a tooltip with startMs/endMs. Text cards appear as “T‑” tokens between bars.
• Real‑time vs server‑rendered: the rail itself is pure DOM (no video decode). A mini‑preview canvas (200 px high) updates on every drag/trim change using a single‑frame server render (/video/:id/produce?overlays=…&voiceover=null) – this is the same /produce endpoint the producer tab uses, so it re‑uses the existing compose worker. No WebCodecs/Wasm in v1; deferred to v1.5.
• What’s real‑time: clip reorder, trim handles, text‑card edit, toggle overlays on/off. What requires a round‑trip: final render, TTS generation, silence removal.

4. AI assist integration

Side‑car service: a local OpenAI‑compatible reverse proxy (e.g. ollama or vLLM running the 8 GB phi-3:medium‑4k model on the 3090). API endpoint POST /v1/transcriptions and /v1/audio/speech. All traffic stays on‑box; no cloud credentials.

Lane v1:

• Transcription + word timestamps – ffmpeg/whisper‑cpp pipeline; worker job step transcribe extracts {[word, startMs, endMs]} → stored as textTracks on the derived asset. Used for text‑based editing (select words → trim clip to selection).
• Silence removal – worker step silenceDetect (ffmpeg silencedetect filter) → auto‑trims leading/trailing silence, marks “clean” sections.
• Scene detection – worker step scenedetect (ffmpeg scdet) → generates cut points that the editor can accept or reject.
• Background matting for lower‑thirds – worker step matting (pyaneti or ranet‑torch, ~6 GB VRAM, FP16) → per‑frame alpha matte composited over footage via overlay. Used when crews want “talking head” lower‑thirds that follow motion.
• Auto‑reframe 9:16 – worker step reframe (ffmpeg crop=ih*iw/ih*9/16:ih with pan‑scan) → derived asset added as a vertical‑format variant.

VRAM budget on 3090 (24 GB):

• phi-3:medium‑4k (8 GB) + whisper‑cpp small (1 GB) + pyaneti (6 GB FP16) = 15 GB headroom for concurrent steps. Steps are sequential, not parallel, so the worker spawns one ffmpeg process per job; the side‑car is a separate HTTP service.

v1 vs later: v1 ships transcription + word timestamps (the textfile= injection fix + text‑based trim UI). Silence removal, scene detection, matting, and auto‑reframe are v1.5 (added after v1 launch, same side‑car, no new UI primitives beyond the existing “enhance” button).

5. UI plan

Screens / components reusing existing primitives:

Feature	Existing primitive	New component (minimal)
Clip library grid	VideoCard (already in video page)	EditClipCard (extends VideoCard with trim handles)
Timeline rail	Placeholder in video-detail-drawer	EditTimelineRail (DOM bars, drag handles)
Text‑card editor	Input primitive (form)	TextCardForm (re‑uses Button, Input from ui/primitives)
Overlay toggle	Checkbox primitive	OverlayToggle (re‑uses Checkbox + label from palette)
Voiceover recorder	MediaRecorder in producer tab	VoiceoverWidget (re‑uses recorder, adds “review / re‑take”)
Render button	Button (primary) in producer tab	RenderEditButton (same visual style, new onClick → creates edit job)

Keyboard shortcuts (global in the drawer):

• ⌘/Ctrl + T – trim focus on selected clip
• ⌘/Ctrl + C – add text card after selection
• ⌘/Ctrl + L – toggle lower‑third
• ⌘/Ctrl + V – add voiceover (start recorder)
• ←/→ – nudge trim by 1 s
• ⌘/Ctrl + Z / ⌘/Ctrl + Y undo/redo (client‑side)

Mobile experience: the drawer is a full‑screen sheet on handsets; touch drags resize handles (start/end thumbs). All primitives are rem‑based and respect the warm‑near‑black palette (#060403 background, #FF1610 accent on active handles). The “render” touch target is large (44 px min) per the existing button minimal.

6. Phased plan

Phase	Effort (S/M/L)	Ships
v1 (6 wks)	2 S (API edit spec + job model), 2 M (ffmpeg filter‑complex builder + worker compose), 1 L (UI timeline rail + text‑card form)	Clip pull → trim → order → text cards between → render derived asset → publish. AI: transcription + word timestamps only.
v1.5 (3 wks)	1 M (silence removal + scene detection worker steps), 1 L (matting + auto‑reframe worker steps)	New “enhance” button in editor; silence‑trim on render; scene‑cut accept/reject; lower‑third matting; vertical reframe variant.
v2 (4 wks)	1 S (search UI + footage similarity), 1 M (side‑car model promotion to local HTTP), 1 L (mobile touch‑rewrite, keyboard shortcuts polish)	Footage search by hash+ocr caption, TTS “record or type → auto‑render”, project templates, collaborative edit (lock‑by‑scene).

First value: v1 ships in ~6 weeks giving crews the core “pull‑trim‑order‑cards‑render” loop immediately. AI lane (transcription) is active day‑one because the side‑car is a separate service; no waiting for v1.5.

7. Top 5 risks & mitigations

#	Risk	Mitigation
1	filtergraph : injection – drawtext cannot have : in literal text (found in research).	All text routed through textfile= with content‑addressed scratch paths; the builder validates text against /^[^\n]{1,200}$/ and never passes raw strings to -filter_complex.
2	Out‑of‑duration trim – ffmpeg fails late if out‑point > source duration.	Pre‑spawn validation: each clip’s trims.endMs ≤ probed durationMs stored in library. UI disables trim handles beyond the bar end.
3	VRAM overload – side‑car models compete with ffmpeg decode.	Steps are strictly sequential; side‑car runs on a dedicated port, ffmpeg uses -threads 1. Monitor with nvidia-smi; if VRAM > 22 GB, queue job and surface “resource busy” to operator.
4	Palette drift – new UI elements must not break the #060403/#0F0A09/#FF1610 brand.	All new components inherit from ui/primitives and are reviewed against the scripts/e2e/brand-palette-check.mjs CI test; any new colour must be approved by LEAD before merge.
5	Module boundary breach – editing code reaches into api-types.ts or production.ts enums.	Editor v2 types live in a new apps/api/src/media-edit/types.ts (append‑only, per filemap rules). All ffmpeg‑related logic lives in apps/media-worker/src/compositor/. No reordering of existing enums; new enums are additive only.

---

Prepared for the SuperPoster RFP. All look‑and‑feel tokens (dark near‑black #060403, red #FF1610, warm greys) are respected; existing UI primitives are reused; no new design language introduced.