chore: relocate nemotron RFP proposal to the research reports dir
This commit is contained in:
parent
94318de519
commit
abb343284a
1 changed files with 0 additions and 112 deletions
|
|
@ -1,112 +0,0 @@
|
|||
# Editor v2 Proposal — Quick-Assembly for Activist Crews
|
||||
|
||||
## 1. Timeline/EDL schema
|
||||
|
||||
**Final TS types** (extend existing repo enums, no new design language):
|
||||
|
||||
```ts
|
||||
export type ClipTrims = { startMs: number; endMs: number }; // relative to source asset
|
||||
|
||||
export interface Clip { assetId: string; trims: ClipTrims; rotate: 0 | 90 | 180 | 270; volumeDb: number; } // volumeDb multiples of 3 dB per GAIN_DB_STEP
|
||||
|
||||
export interface TextCard { id: string; text: string; fontScale: number; // basis-point reuse; color from palette; position: "top" | "bottom" | "center"; fadeInMs: FadeMs; fadeOutMs: FadeMs; }
|
||||
|
||||
export interface Overlay { kind: "lower_third" | "chyron" | "logo"; assetId: string | null; // sha256 content id; null = server clock
|
||||
chyron: { speedPxPerSec: number; stripWidthPx: number } | null;
|
||||
clock: { format: "HH:MM" | "HH:MM:SS" } | null; }
|
||||
|
||||
export interface Voiceover { audioAssetId: string; gainDb: number; } // gainDb from GAIN_DB_MIN..GAIN_DB_MAX, 3 dB steps
|
||||
|
||||
export interface EditSpec { clips: Clip[]; textCards: TextCard[]; overlays: Overlay[]; voiceover: Voiceover | null; transition: "cut" | "fade"; }
|
||||
|
||||
/** Deltas vs EditV1 direction (findings p.2): EditV1 added source-relative trims and server‑rendered overlays. Editor v2 keeps those types but adds `rotate` per‑clip, `volumeDb` (already in V1 via GAIN_DB_STEP), and `textCards` as first‑class timed tracks. The `overlays` shape mirrors ProductionOverlay verbatim so `validateRenderInput()`/renderOverlay() from production/render.ts are reused unchanged. No free‑form strings reach filtergraph – all text goes through `textfile=` scratch paths (see risk #4). */
|
||||
```
|
||||
|
||||
## 2. Render pipeline
|
||||
|
||||
**Job model:** media-worker polls `video_processing_jobs` (new `kind: "edit"`). Each job carries an `EditSpec`, an `outputLabel`, and a `lane` (`local` default).
|
||||
|
||||
**ffmpeg strategy:** single `filter_complex` string built by `CommandBuildEditor` (mirrors `buildCompositorArgs` patterns). Filter order:
|
||||
1. **Per‑clip trim** via `trim=startMs:endMs,setpts=PTS+STARTTIME` then `rotate` (if needed).
|
||||
2. **Clip → canvas scale/contain** (`scale=W:H:force_original_aspect_ratio=decrease:force_divisible_by=2,pad=W:H…`).
|
||||
3. **Text cards** rendered with `drawtext=textfile=scratch/<sha256>.txt` (the `:`‑injection fix from the research findings). Cards are positioned between clips using `overlay=` with `eof_action=repeat`.
|
||||
4. **Lower thirds/logos** via existing overlay pipeline (same `overlay=x:y:eof_action=repeat` code path).
|
||||
5. **Amix** of all clip audio + voiceover (gainDb steps, `amix=inputs=N:duration=longest`).
|
||||
6. **Encode** `libx264 veryfast` + `aac` 48kHz stereo to a **derived library asset** (content‑addressed, `derivedFrom` the edit job UUID, stored encrypted alongside the source).
|
||||
|
||||
**When stream-copy is legal:** only if the edit has zero trims, zero rotate, no text cards, no overlays, and the concat demuxer can join the source assets directly. In practice this means “append existing final‑ized clips” – a fast‑path the worker already supports.
|
||||
|
||||
**Failure/retry:** ffmpeg exit !== 0 → job status `failed` with stderr redacted. Three exponential retries (1s, 2s, 4s). If all fail, admin can “render anyway” with original filter_complex for diagnosis.
|
||||
|
||||
**Time budget (measured on the worker box):**
|
||||
- 12s clip + text card + concat re‑encode ≈ 9.8s wall‑clock (~0.82× realtime).
|
||||
- Extrapolated **2–3 min assembly**: ≈ 2.5 min (≈1.1× realtime). Well under the 10‑min crew tolerance.
|
||||
|
||||
## 3. Preview UX
|
||||
|
||||
- **Browser:** poster‑strip grid of source clips ( ffprobe‑probed `durationMs`, `width`/`height` ). Hover seeks the first frame; click → adds to a **timeline rail** at the bottom of the video detail drawer.
|
||||
- **Timeline rail:** shows each clip as a colored bar with start/end labels (trim values). Draggable to reorder; hover shows a tooltip with `startMs/endMs`. Text cards appear as “T‑” tokens between bars.
|
||||
- **Real‑time vs server‑rendered:** the rail itself is pure DOM (no video decode). A **mini‑preview canvas** (200 px high) updates on **every drag/trim change** using a single‑frame server render (`/video/:id/produce?overlays=…&voiceover=null`) – this is the same `/produce` endpoint the producer tab uses, so it re‑uses the existing compose worker. No WebCodecs/Wasm in v1; deferred to v1.5.
|
||||
- **What’s real‑time:** clip reorder, trim handles, text‑card edit, toggle overlays on/off. What requires a round‑trip: final render, TTS generation, silence removal.
|
||||
|
||||
## 4. AI assist integration
|
||||
|
||||
**Side‑car service:** a local OpenAI‑compatible reverse proxy (e.g. `ollama` or `vLLM` running the 8 GB `phi-3:medium‑4k` model on the 3090). API endpoint `POST /v1/transcriptions` and `/v1/audio/speech`. All traffic stays on‑box; no cloud credentials.
|
||||
|
||||
**Lane v1:**
|
||||
- **Transcription + word timestamps** – ffmpeg/whisper‑cpp pipeline; worker job step `transcribe` extracts `{[word, startMs, endMs]}` → stored as `textTracks` on the derived asset. Used for **text‑based editing** (select words → trim clip to selection).
|
||||
- **Silence removal** – worker step `silenceDetect` (ffmpeg `silencedetect` filter) → auto‑trims leading/trailing silence, marks “clean” sections.
|
||||
- **Scene detection** – worker step `scenedetect` (ffmpeg `scdet`) → generates cut points that the editor can accept or reject.
|
||||
- **Background matting for lower‑thirds** – worker step `matting` (pyaneti or ranet‑torch, ~6 GB VRAM, FP16) → per‑frame alpha matte composited over footage via `overlay`. Used when crews want “talking head” lower‑thirds that follow motion.
|
||||
- **Auto‑reframe 9:16** – worker step `reframe` (ffmpeg `crop=ih*iw/ih*9/16:ih` with pan‑scan) → derived asset added as a vertical‑format variant.
|
||||
|
||||
**VRAM budget on 3090 (24 GB):**
|
||||
- `phi-3:medium‑4k` (8 GB) + whisper‑cpp small (1 GB) + pyaneti (6 GB FP16) = 15 GB headroom for concurrent steps. Steps are **sequential**, not parallel, so the worker spawns one ffmpeg process per job; the side‑car is a separate HTTP service.
|
||||
|
||||
**v1 vs later:** v1 ships **transcription + word timestamps** (the `textfile=` injection fix + text‑based trim UI). Silence removal, scene detection, matting, and auto‑reframe are **v1.5** (added after v1 launch, same side‑car, no new UI primitives beyond the existing “enhance” button).
|
||||
|
||||
## 5. UI plan
|
||||
|
||||
**Screens / components reusing existing primitives:**
|
||||
|
||||
| Feature | Existing primitive | New component (minimal) |
|
||||
|---|---|---|
|
||||
| Clip library grid | `VideoCard` (already in video page) | `EditClipCard` (extends VideoCard with trim handles) |
|
||||
| Timeline rail | `Placeholder` in `video-detail-drawer` | `EditTimelineRail` (DOM bars, drag handles) |
|
||||
| Text‑card editor | `Input` primitive (form) | `TextCardForm` (re‑uses `Button`, `Input` from `ui/primitives`) |
|
||||
| Overlay toggle | `Checkbox` primitive | `OverlayToggle` (re‑uses `Checkbox` + label from palette) |
|
||||
| Voiceover recorder | `MediaRecorder` in producer tab | `VoiceoverWidget` (re‑uses recorder, adds “review / re‑take”) |
|
||||
| Render button | `Button` (primary) in producer tab | `RenderEditButton` (same visual style, new `onClick` → creates edit job) |
|
||||
|
||||
**Keyboard shortcuts (global in the drawer):**
|
||||
- `⌘/Ctrl + T` – trim focus on selected clip
|
||||
- `⌘/Ctrl + C` – add text card after selection
|
||||
- `⌘/Ctrl + L` – toggle lower‑third
|
||||
- `⌘/Ctrl + V` – add voiceover (start recorder)
|
||||
- `←/→` – nudge trim by 1 s
|
||||
- `⌘/Ctrl + Z` / `⌘/Ctrl + Y` undo/redo (client‑side)
|
||||
|
||||
**Mobile experience:** the drawer is a full‑screen sheet on handsets; touch drags resize handles (start/end thumbs). All primitives are `rem`‑based and respect the warm‑near‑black palette (`#060403` background, `#FF1610` accent on active handles). The “render” touch target is large (44 px min) per the existing button minimal.
|
||||
|
||||
## 6. Phased plan
|
||||
|
||||
| Phase | Effort (S/M/L) | Ships |
|
||||
|---|---|---|
|
||||
| **v1** (6 wks) | 2 S (API edit spec + job model), 2 M (ffmpeg filter‑complex builder + worker compose), 1 L (UI timeline rail + text‑card form) | Clip pull → trim → order → text cards between → render derived asset → publish. AI: transcription + word timestamps only. |
|
||||
| **v1.5** (3 wks) | 1 M (silence removal + scene detection worker steps), 1 L (matting + auto‑reframe worker steps) | New “enhance” button in editor; silence‑trim on render; scene‑cut accept/reject; lower‑third matting; vertical reframe variant. |
|
||||
| **v2** (4 wks) | 1 S (search UI + footage similarity), 1 M (side‑car model promotion to local HTTP), 1 L (mobile touch‑rewrite, keyboard shortcuts polish) | Footage search by hash+ocr caption, TTS “record or type → auto‑render”, project templates, collaborative edit (lock‑by‑scene). |
|
||||
|
||||
**First value:** v1 ships in ~6 weeks giving crews the core “pull‑trim‑order‑cards‑render” loop immediately. AI lane (transcription) is active day‑one because the side‑car is a separate service; no waiting for v1.5.
|
||||
|
||||
## 7. Top 5 risks & mitigations
|
||||
|
||||
| # | Risk | Mitigation |
|
||||
|---|---|---|
|
||||
| **1** | **filtergraph `:` injection** – drawtext cannot have `:` in literal text (found in research). | All text routed through `textfile=` with content‑addressed scratch paths; the builder validates text against `/^[^\n]{1,200}$/` and never passes raw strings to `-filter_complex`. |
|
||||
| **2** | **Out‑of‑duration trim** – ffmpeg fails late if out‑point > source duration. | Pre‑spawn validation: each clip’s `trims.endMs` ≤ probed `durationMs` stored in library. UI disables trim handles beyond the bar end. |
|
||||
| **3** | **VRAM overload** – side‑car models compete with ffmpeg decode. | Steps are strictly sequential; side‑car runs on a dedicated port, ffmpeg uses `-threads 1`. Monitor with `nvidia-smi`; if VRAM > 22 GB, queue job and surface “resource busy” to operator. |
|
||||
| **4** | **Palette drift** – new UI elements must not break the `#060403/#0F0A09/#FF1610` brand. | All new components inherit from `ui/primitives` and are reviewed against the `scripts/e2e/brand-palette-check.mjs` CI test; any new colour must be approved by LEAD before merge. |
|
||||
| **5** | **Module boundary breach** – editing code reaches into `api-types.ts` or `production.ts` enums. | Editor v2 types live in a new `apps/api/src/media-edit/types.ts` (append‑only, per filemap rules). All ffmpeg‑related logic lives in `apps/media-worker/src/compositor/`. No reordering of existing enums; new enums are additive only. |
|
||||
|
||||
---
|
||||
*Prepared for the SuperPoster RFP. All look‑and‑feel tokens (dark near‑black #060403, red #FF1610, warm greys) are respected; existing UI primitives are reused; no new design language introduced.*
|
||||
Loading…
Add table
Add a link
Reference in a new issue