T-HUB/docs/design/streaming-architecture.md

Streaming Architecture — Control Room, Overlays, Fallback, Chat, Telegram Ingest

Status: decision document (synthesis of four WP4 research reports). No code was changed while writing it. Date: 2026-10-07 Inputs:

• [flash] runs/wp4/control-room-research-flash.md
• [glm] runs/wp4/control-room-research-glm.md
• [as] (integration research, kept outside the repository)
• [tg] runs/wp4/telegram-amplify-research.md (WP15 middle)
• [scope] runs/wp4/control-room-scope.md; [dm] docs/design/design-merged.md; [wb] docs/work-breakdown.md WP9–WP13, WP15; [prop] docs/proposal.md §4.4/§4.5

Provenance legend. [flash], [glm], [as], [tg] mark the report a decision comes from. [both] means flash and glm agree. [SYN] marks a decision this document makes where neither report settled the question, or where the two disagree. Each [SYN] explains why it wins.

---

0. Decisions at a glance

#	Question	Decision	Source
D1	Where is program composited?	On the server, in the media worker's FFmpeg compositor. The browser is a control surface and a preview consumer only.	[both]
D2	One FFmpeg process per channel, or split processes?	Split: ingest×N → compositor (replaceable) → frame gate (clock master) → persistent encoder → push relays. A graph change starts a new compositor (A/B swap). The encoder and the RTMP session are never restarted.	[SYN] over [flash]'s one-process invariant
D3	Multiview	Every tile plays the worker's preview tap, never the remote source. 2x1/2x2 tiles are live LL-HLS. 3x3/4x4 tiles are thumbnails, with live LL-HLS only on the PGM tile and the focused tile. Hard cap of 4 live decodes on desktop and 1 on mobile.	[both]; caps [SYN]
D4	Audio	Monitor mix runs in the browser (one shared AudioContext, gain per tile, same-origin preview audio). Program mix runs in the compositor (amix with weights from desired state, gain in 3 dB steps).	[both]; steps [glm]
D5	Overlay rendering	Pre-rendered assets. The API renders an SVG template to a PNG with alpha. The worker composites it with overlay. A chyron is one wide PNG strip scrolled by an overlay x-expression. FFmpeg never sees user text.	[flash] over [glm]'s drawtext+textfile
D6	Playlist fallback	On the worker. A playlist is an ordinary ingest kind feeding a SLATE/PLAYLIST scene into the same encoder. PeerTube runs as permanentLive. PeerTube-native playlists are only the programming source of truth.	[both]; "Tier-2 PeerTube-only mode" from [glm] deferred
D7	Auto Director	Rules only: a priority ladder over monitor states, plus dwell time per source class, ending in the fallback scene. Automation writes the same buses through the same transaction as a human TAKE.	[both]; dwell-by-class [glm]
D8	PGM/PST transitions	v1 ships CUT. FADE follows behind a gate, as a transition graph swapped twice (old+new with xfade, then new only). The transition is part of desired state, not an imperative command.	[SYN] from [flash] Q6 + [glm] Q6
D9	Chat transport to the app	WebSocket: subscribe/ack, auth sent in the first frame, lastSeq resume from a server ring buffer. REST cursor polling (?after=seq) is the fallback. No FCM.	[as]; polling fallback aligned with [dm] §7
D10	Telegram "join and amplify"	Bot-as-channel-admin webhook → archived_feed_items → review draft → existing amplify(). MTProto stays a flagged Phase 2 and is not in this plan.	[tg]

---

1. THE composition decision

1.1 Options

• (A) Server-side FFmpeg compositor. Runs inside apps/media-worker under the supervisor and receives commands over the existing Ed25519 signed-snapshot bus.
• (B) Browser-composited program (the StreamYard model). The browser draws onto a canvas, then captureStream()/WebCodecs → WebRTC/WHIP upstream → the worker transcodes to RTMP.

1.2 Evidence

Criterion	(A) server compositor	(B) browser compositor	Source
Operator laptop sleeps, tab crashes, Wi-Fi drops	Program keeps running. The supervisor, output lease and reconcile loop already live on the worker.	Program ends. For a self-hosted product there is no cloud fallback, so the operator's machine becomes a single point of failure for a 24/7 channel.	[flash] Q3.1, [glm] Q3
24/7 premise (scope item 5: a playlist plays when nothing is live)	Fallback is just another scene, and the encoder never idles.	Unattended hours have no browser to run in, so a second, server-side path is needed anyway.	[glm] Q3 "disqualifying on its own"
Mobile (later)	The phone is a pure control-plane client sending desired-state commands. No media runs on the device.	HTMLMediaElement.captureStream is false on Safari. WebCodecs AudioEncoder arrives only in Safari 26. Firefox Android has no VideoEncoder.	[flash] Q3 (BCD), [glm] Q3 (BCD)
Signed command bus	Fits 1:1. A TAKE writes a bus row, bumps desiredVersion, writes an outbox row and publishes a signed full snapshot. The worker converges idempotently. The fence, seen-set, reconcile and assignment epochs all carry over unchanged.	Needs a second "what is program" authority in the browser, a WHIP ingest trust boundary, and a WebRTC signalling path outside egress mediation, secret handling and the reconcile gate.	[flash] Q3.2, [glm] Q3 "Fit with the signed-command worker"
Latency at the viewer	Adds about one encode generation.	Can be under a second from glass to RTMP.	Both say PeerTube's live→HLS chain dominates (latency modes 1/2/3), so the gap mostly disappears at the viewer. [flash] Q3.4, Q5
CPU placement	Server decodes and encodes. This is already priced in encoder units by WP9 admission.	Moves CPU to the operator machine.	[glm] Q3
Security	Secrets, sandbox and egress pinning stay server-side, where the proposal already solved them.	The browser must decode every source at full quality, which breaks the "operator never pulls the remote source" rule.	[glm] Q3, [prop] §4.4 previews

1.3 Decision — (A), server-side compositor [both]

The worker composes. The browser only watches previews and sends desired-state commands [glm]. A WebRTC/WHIP low-latency path, if ever needed, enters as one more source kind feeding the server compositor. Composition stays server-side either way [flash].

Dissenting view (recorded). Option (B) is the right answer for interactive call-in shows: sub-second guest interaction, zero server encode cost, and WYSIWYG editing on the operator's screen. StreamYard/Restream ship it (their internals are not verified in either report). We reject it because our product is a continuous Virtual Channel, not a studio session. If call-in guests become a requirement, the shape is vMix Call's [glm]: browsers contribute sources over WHIP, and the mix still happens on our worker.

1.4 Where the two reports disagree inside (A): process topology [SYN]

• [flash] proposes the invariant "one FFmpeg process per channel", with every TAKE, layout or overlay change rebuilding the graph.
• [glm] and [prop] §4.4 use separate ingest / compositor / persistent encoder / pusher processes: "scene switch = graph rewiring… encoder and pushers are never restarted".

Decision: split topology, plus a frame gate. FFmpeg cannot rewire a running filtergraph. Runtime commands only change parameters of filters that already exist. So in flash's single process, every scene change restarts the encoder and the RTMP publish. That breaks WP9's "output that does not stop" and the WP11/WP12 continuity acceptance. flash's own Risk 2 ("a bad rebuild drops frames or kills program") is the symptom. The split topology moves the rebuild away from the encoder:

ingest_i (normalize → raw video + PCM over pipes, per source; also emits preview taps)
        │  (N inputs, plus a playlist ingest and overlay asset inputs)
        ▼
compositor[A]  ── active graph for desiredVersion v
compositor[B]  ── started for v+1, warmed until its first frame is ready
        │ rawvideo (yuv420p, fixed frame size) + s16le 48 kHz (fixed bytes per frame)
        ▼
FRAME GATE  (clock master: paced at the profile fps; on compositor stall it repeats the
             last frame and writes silence; switches A→B at a frame boundary once B
             has produced K frames; then SIGTERMs A)
        ▼
encoder (persistent; reads gate pipes; never restarted by scene changes)
        ▼
push relay ×M (WP9/WP13: lease-bound key over fd, pinned destination)

• Why the gate is the clock master. Raw video and PCM have fixed byte sizes per frame, so the gate needs no demuxer and no timestamp parsing. Frames are counted, and the encoder sees a perfectly regular input with no gaps. Sync between audio and video holds by construction.
• Cost. About 93 MB/s of raw 1080p30 through one more pipe hop, plus a brief doubled compositor CPU during each swap. The doubled CPU is admission-accounted as a transient burst inside the worker's 20 % failover reserve.
• Spike gate (first task of S2). A gate written in Node with preallocated ring buffers must sustain 1080p60 at under 5 % of a core with no GC pauses above one frame interval. If it fails, the gate becomes a small static Rust binary behind the same pipe contract. The WP9 GStreamer decision gate ([wb] WP9) is folded into this spike: if A/B swap with the gate is not frame-continuous across 1 000 swaps, escalate to GStreamer input-selector.

flash's argument for a single process was simplicity and keeping "WP11's continuity risk class" in one place. The gate keeps that risk in one place, the gate, while making continuity a property of the design instead of a hope about rebuild speed.

---

2. Multiview + audio spec

2.1 What a tile plays [both]

Tiles never touch remote HLS. Each active ingest emits one shared preview tap ([prop] §4.4):

• a JPEG thumbnail, every 2 s by default (configurable 1–10 s);
• an on-demand 360p / ≤500 kbps LL-HLS rendition;
• [SYN] an on-demand audio-only rendition (AAC 64 kbps LL-HLS). Thumbnails carry no audio, so without it a 4x4 operator could not hear a thumbnail tile.

These are served from the authenticated preview endpoint with ETag, single-flight fill, and an LRU of 6 segments or fewer per source. The endpoint is same-origin with correct CORS. This is load-bearing: a cross-origin MediaElementAudioSourceNode outputs silence per the Web Audio spec [flash] Q2 and [glm] Q2.

The program tap is the same mechanism applied to the gate output (one extra tap per channel). The PGM tile always shows what the encoder actually receives.

2.2 Grid tiers and the decode budget

No browser documents a hard limit on concurrent decoders; both reports flag this as not verified. The binding constraints are pixel throughput, per-tab MSE memory and mobile hardware-decoder sessions [glm] Q1. Budget: live decodes ≤ 4 on desktop, ≤ 1 on mobile [SYN]. This is flash's "never more than 4" plus glm's mobile "thumbnails + one live tile".

Layout	Desktop tiles	Live (LL-HLS 360p) tiles	Mobile
2x1	2	both live	PGM live, other thumbnail
2x2	4	all 4 live	PGM live, others thumbnail
3x3	9	PGM + focused live; the other 7 are thumbnails	1 live (PGM or focused)
4x4	16	PGM + focused live; the other 14 are thumbnails	1 live (PGM or focused)

Additional rules:

• "Upgrade more tiles" is an explicit user action, capped at 4 total, with a capability warning [glm].
• Thumbnail cadence: 2 s by default and 1 s for the PST-selected tile. Off-screen tiles stop polling (IntersectionObserver + visibilitychange) [prop].
• hls.js config per tile [glm] Q1:
  • lowLatencyMode: true;
  • maxBufferLength: 4, maxMaxBufferLength: 8, backBufferLength: 4 (the default Infinity is wrong for a multiview);
  • liveSyncDurationCount: 2.
• Player selection [flash] Q1: if Hls.isSupported(), use MSE. Otherwise, if ManagedMediaSource exists and canPlayType(m3u8), use native HLS (Safari/iOS 17.1+). Otherwise the client is thumbnail-only.
• Autoplay: every tile boots muted + playsinline. One "Arm audio" gesture unlocks the shared AudioContext [flash] Q1.
• Tile chrome: monitor badge (LIVE/UNSTABLE/OFFLINE/… from the live-monitor WS fan-out), tally border (red = PGM, green = PST), and a tile-state label ("thumbnail · 2 s", "live preview").
  • Tally reflects only state stamped with observed_version. It is never optimistic [glm] Q6, Risk 4.
• PST rendering [SYN] (adopting [glm] Q3's browser preview compositor):
  • The PST monitor is rendered client-side onto a canvas, from the preview taps of the slots in the PST scene plus the scene's overlay SVGs. These are the same SVGs the server rasterizes (§4), so the look matches.
  • The worker does not composite a second (preview) program. That would double compositor CPU for every channel.
  • Caveat shown in the UI: "PST is a look-ahead; PGM is the truth".

2.3 Audio: two mixes, two owners [both]

Monitor mix (what the operator hears; browser):

• Graph: one AudioContext per page. Each audible tile's media element → MediaElementAudioSourceNode → its own GainNode → master GainNode → destination [glm] Q2.
• Modes:
  • Follow program (default): plays the program tap's audio, which is exactly what viewers hear, delayed by roughly the tap latency.
  • Solo source X: plays that source's audio rendition.
  • Custom monitor mix: up to 4 sources with independent faders. These are continuous UI faders, because the mix is local.
• Bounds: never more than 4 audio renditions plus the program tap. Thumbnails gain audio by attaching the audio-only rendition (§2.1).
• Privacy: per operator and never sent to the server, so no command traffic.
• Mobile: the monitor collapses to "the one tile you are watching" [glm].

Program mix (what viewers hear; compositor):

• Per-slot audio intent: scene_slots.audio_mode ∈ off|on|primary, with at most one primary per scene (partial unique index, [prop] §4.4). The default is primary-only, meaning one feed's audio. "Mix N feeds" means several on slots.
• Graph: volume per input → amix inputs=N duration=longest dropout_transition=2 normalize=1 [glm] Q2. A dead source fades out instead of cutting.
• Gain is desired state, in 3 dB steps from −24 to 0 dB. No continuous program fader, because continuous faders would turn the signed bus into a UI-rate control channel [glm] Q2.
• Applying a change: a program gain change bumps desiredVersion and triggers an A/B compositor swap (§1.4). It is inaudible because the gate splices at a frame boundary.
• Deferred optimization: amix/volume runtime commands via azmq on an ipc:// socket in job scratch [flash] Q2. Not verified: whether our pinned FFmpeg has libzmq, and whether it is compatible with the seccomp profile.

---

3. Chat unification contract (Opticast ↔ Tactical Media Broadcast Hub)

3.1 Roles [as] §2.1

Tactical Media Broadcast Hub is the chat aggregation authority. Connectors ingest server-side into a normalized stream. A new chat-hub module persists, dedupes, sequences and fans out. Opticast, the web Control Room and the Tactical Media Broadcast Hub mobile web shell are all thin authenticated realtime clients. The phone never holds Twitch/YouTube/PeerTube credentials.

The privacy pledge in Opticast's README ("no network access beyond the configured server") remains true. The integration is opt-in and off by default, and the only endpoint is a user-configured, self-hosted Tactical Media Broadcast Hub.

3.2 Ingestion (platform → LiveChatMessage) [as] §2.3

Platform	Read	Reply	Moderate	Mechanism / governance
Twitch	✅	✅	delete, timeout, ban	EventSub WS channel.chat.message plus delete/clear events. Helix POST /chat/messages for replies; Helix moderation and ban endpoints. No replay after a WS drop. The hub ring buffer covers clients; the platform gap is accepted and marked with a system gap message. Token bucket of 100 per 30 s (elevated).
YouTube	✅	✅	delete, ban	liveChatMessages.list long-poll honouring pollingIntervalMillis, one shared poller per broadcast. Quota ledger: list = 1 unit, insert/delete/ban = 50, 10 000 per day by default. Quota exhaustion is a first-class paused:quota state.
PeerTube + livechat plugin, C2S on	✅	✅	MUC moderation	Server-side XMPP client joins as the streamer with the livechat long-term token, which is "not officially supported" per the plugin docs.
PeerTube, embed only	read-only	❌	❌	Read-only embed pane, web only.
PeerTube, no plugin	comments	comment reply	✅	Existing getComments on the connector, labelled "comments".
Mastodon	mentions	status reply	delete own	Streaming /api/v1/streaming (user), source: "social-reply".
Mock livestream	✅	✅	✅	Contract and E2E harness.

Capability probing. Each connection carries a probed capability row, with pt-chat: xmpp-c2s | readonly-embed | comments-only | none for PeerTube.

Storage. live_chat_messages is partitioned monthly per the [wb] retention row. live_chat_seen_events uses ON CONFLICT DO NOTHING, mirroring monitor_seen_events. Dedup key: (workspace_id, provider, provider_message_id). Ordering: a per-live monotonic seq assigned at hub ingress. Provider timestamps are display-only, because clocks skew [as] §2.4.

Local display filters. Hide !commands, hide bots, mute user, word filter. These run once, hub-side, so web and mobile inherit them [as] §2.8.

3.3 Transport decision [SYN] (from [as] §2.5 + [dm] §7)

Option	Verdict
WebSocket (subscribe/ack + resume)	Primary. Chat volume is far beyond review-chat volume. OkHttp WS on Android can authenticate in the first frame. The existing authorizeSubscription port is the gate.
SSE	Rejected. EventSource cannot send our Bearer header ([dm] §7 rationale), and it offers no upstream channel for acks.
REST cursor polling	Fallback for constrained clients and for resync after RESYNC_REQUIRED. It uses the same after=seq semantics as [dm] review_events, so one cursor discipline applies across the codebase.
FCM push	Rejected for v1, because it breaks Opticast's privacy pledge. A later opt-in UnifiedPush or self-hosted web-push is possible.

WS lifetime on the phone. The socket stays open only while the chat sheet is open, or while StreamingService, the existing camera/mic foreground service, is live. There is no background socket when idle. Backoff from 1 s to 8 s mirrors ReconnectController [as] §2.5, Risk 4.

Fan-out. The hub publishes to Redis chat:{workspaceId}:{liveId}. The gateway subscribes only to authorized and viewed topics, using WP11 fan-out rules: bounded per-socket queues, RESYNC_REQUIRED on overflow. Prerequisite: the API-side RespControlRedis has PUBLISH but not SUBSCRIBE, so a subscribe-capable client must land first [as] §2.2.

3.4 Device pairing [SYN]

Opticast authenticates with a device token, not a user session cookie.

1. The web UI calls POST /devices/pairings and shows a QR code containing {serverUrl, pairingCode} (TTL 5 min, single use).
2. The app redeems the code with POST /devices/pairings/redeem and receives a device token. The token is bound to (workspaceId, userId), scoped to livechat.view|send|moderate + stream.view, epoch-gated, and revocable from the web UI.
3. The app stores the token in EncryptedSecretStore [as] §4 Phase 2.

3.5 Endpoint shapes (both apps implement)

Routes follow repo convention: no /workspaces/{wid} prefix, because the workspace comes from the session or device token; compare @Controller("publishing/drafts"). [as] Phase 0 shapes are adapted accordingly.

REST (Tactical Media Broadcast Hub API; RbacGuard + RLS; Idempotency-Key on POSTs)
GET  /live-chat/lives                                   → { items: LiveSummary[] }               # stream.view
GET  /live-chat/lives/{liveId}/messages?after=<seq>&limit=<1..500, default 100>&platform=<p>
     → { items: ChatMessage[], nextCursor?: number }                                            # livechat.view
POST /live-chat/lives/{liveId}/messages
     body { text (1..500 chars), targets: "all" | connectionId[], replyTo?: messageId }
     → 202 { message: ChatMessage, delivery: Delivery[] }                                       # livechat.send
POST /live-chat/lives/{liveId}/messages/{messageId}/moderation
     body { action: "delete"|"timeout"|"ban", durationSeconds?: 1..1209600 }
     → 202 { delivery: Delivery[] }   (audited via audit_append)                                # livechat.moderate
GET  /live-chat/lives/{liveId}/capabilities
     → [{ connectionId, provider, read, reply, moderate, state: "ok"|"paused:quota"|"paused:rate"|"degraded", reason? }]
POST /devices/pairings                 → { pairingCode, qrPayload, expiresAt }                  # session user
POST /devices/pairings/redeem          body { pairingCode, deviceName } → { deviceToken, workspaceId }
DELETE /devices/{deviceId}             (revoke; bumps the device's epoch)

Delivery = { connectionId, provider, status: "queued"|"sent"|"failed"|"unsupported", error?: string }

WS  /realtime   (one socket, multiplexed topics; also used by the Control Room)
C→S { op:"auth", token }                                     # first frame; never in the URL
S→C { op:"ready" } | { op:"close", reason }
C→S { op:"subscribe", id, topic:"livechat", liveId, lastSeq? }
C→S { op:"subscribe", id, topic:"channel", channelId }       # Control Room: buses/monitor/observed
S→C { op:"ack", id, fromSeq } | { op:"nack", id, reason }    # reason ∈ authorizeSubscription reasons
S→C { op:"msg", topic, seq, type, payload }
    type ∈ chat.message | chat.moderated | chat.delivery | chat.gap | stream.state | capability
S→C { op:"resync", topic }                                   # client re-fetches via REST ?after=
C→S { op:"unsubscribe", id } | { op:"ping" }  S→C { op:"pong" }

ChatMessage = { seq, id, liveStreamId, connectionId, provider, providerMessageId,
                author: { handle, displayName?, avatarUrl?, badges?: string[], color },
                text, sentAt, receivedAt, isModerated, replyTo?, source: "live-chat"|"social-reply"|"system" }

Contract rules:

• Resume. lastSeq replays from a ring buffer of the last 500 messages per live, held in Redis. If lastSeq is older than the ring, the server sends resync and the client pages via REST.
• Epoch. An authz-epoch bump (role change, device revoke) closes every socket of that principal within 5 s, matching the WP11 revocation SLO.
• Reply semantics. The composer target defaults to "All capable". An incapable target is refused client-side with its reason. Delivery ticks per target reconcile from chat.delivery events, following Restream's pattern [as] §2.7.
• Identity. No cross-platform identity fusion. The author colour is a hash of provider:handle [as] §2.6.
• Licensing. Opticast is GPLv3 and the Tactical Media Broadcast Hub licence is not verified. The relationship stays API-only, with no shared code [as] Risk 5.

3.6 Opticast UX (for the client slice) [as] §3

• Chat is a bottom sheet on LiveScreen with snap points at 40/70/100 %. It is not a third route.
• One unified list with platform badges, filter chips All · Twitch · YouTube · PeerTube · Mentions, and swipe moderation (left = delete, right = ban; long-press for timeout).
• A "↓ N new" pill appears when scrolled up.
• A ChatViewModel sits beside StreamViewModel. Streaming code is untouched.
• Opticast as a camera source for the Control Room needs a push ingest kind (rtmp-push/srt-listen) that Tactical Media Broadcast Hub lacks today (source.kind is "hls" only). It is out of scope for this plan and listed as a follow-on in §6.4.

---

4. Overlay system

4.1 The disagreement

• [glm] drawtext + textfile + reload=1. Text edits land without a graph restart, and scrolling uses x = main_w − mod(t·speed, main_w + text_w). [prop] §4.4 also mentions "overlay text → written to a textfile".
• [flash] pre-rendered assets. SVG template → PNG with alpha (via resvg/sharp) → overlay. The chyron is one wide strip scrolled by an overlay x-expression. FFmpeg never sees text.

4.2 Decision — pre-rendered assets [flash], with glm's no-restart property recovered by the A/B swap [SYN]

Why pre-rendered assets win:

1. Same pixels in PST and PGM. The browser PST look-ahead (§2.2) renders the same SVG the server rasterizes. With drawtext, the browser would have to imitate FreeType layout and would never match.
2. Brand-quality typography. The geometric design language (#FF1610, solid disk mark, rounded/angled boxes) is trivial in SVG and painful in drawtext box parameters.
3. Smaller attack surface inside the sandbox. drawtext needs freetype, harfbuzz, fontconfig and fribidi in the pinned FFmpeg, plus font files mounted into every compositor. [glm] Risk 3 itself names "font bombs". Rasterizing once in an API-side sandboxed job keeps fonts and SVG parsing out of the 24/7 process.
4. Cost and determinism. A static PNG overlay is cheaper than per-frame glyph rendering. [glm] flags drawtext scroll CPU at 1080p60 as a risk, and its own fallback is "render the band into a strip", which is flash's design. PNGs are also golden-frame deterministic.
5. glm's advantage is no longer needed. glm's argument for drawtext (an edit without a graph restart) matters only if restarts are visible. With the frame gate (§1.4), a graph swap is frame-continuous, so an overlay edit is just another desiredVersion.

drawtext survives in one narrow role: a server-generated clock/countdown using expansion=normal with localtime. The format string is a typed enum, never user text.

4.3 Pipeline

Operator edits a lower third (web/mobile)
   │  POST /production/overlays/{overlayId}  { templateId, fields: {title ≤ 80, subtitle ≤ 120}, style enum }
   ▼
API validation: length, Unicode category allowlist, no control characters → overlay_revisions row
   ▼
render job (sandboxed WP4 probe class): SVG template (server-owned, typed field slots, bundled fonts)
   → resvg (no scripts, no external fetch) → PNG with alpha at output resolution
   → StorageService (sha256 key, dedup) → asset id
   ▼
1 tx: scene overlay slot → new assetId + desired_version++ + outbox row   (same path as a TAKE)
   ▼
signed snapshot → worker → compositor B built with the new overlay input → gate swap at a frame boundary

Overlay kinds:

• Lower third: PNG composited with overlay=x:y. There is no timed enable= expression. Toggling is a slot flag in desired state, so the next graph simply includes or omits it.
• Chyron: render the whole ticker strip (all items with separators) as one wide PNG. The height of the band is fixed. Width is capped at 16 384 px [SYN]; longer text is split into a carousel of strips. Scroll with overlay=x='W-mod(t*S\,w+W)':y=Y, where speed S is a typed integer of 40–400 px/s.
  • A chyron text edit renders a new strip and swaps graphs. The scroll position restarts, and that is accepted.
• Logo from an SVG input: a tenant upload goes through the WP4 sandboxed probe and is rasterized once per asset version at 1×/2× sizes. Position and size are typed values (corner enum + margin, or explicit numbers).

Security gates:

• Injection. Overlay text never reaches argv or a graph, because the graph references assetId only. The injection tests from [wb] WP12 apply to the render job too: SVG text nodes are XML-escaped, and the template engine is typed field-slot substitution, not string concatenation.
• Worker storage access. The worker fetches assets through presigned GETs scoped to its assignment ([prop] §4.5 object-storage tenant ACLs).

Not verified:

• resvg and sharp APIs. flash fetched the resvg README; glm did not verify either. Pick resvg-js.
• Whether the image2 demuxer re-reads a file on -loop 1. flash and glm do not discuss it. It is no longer needed, because edits go through the swap.

---

5. Playlist fallback + Auto Director

5.1 Fallback venue [both]

Decision: worker-side fallback. The PeerTube live is permanentLive: true, so the channel URL stays live forever. "Nothing is live" means the director puts the FALLBACK scene on program.

• No live/VOD flapping, no RTMP reconnect, and no change of PeerTube latency mode. Overlays and program audio stay consistent [flash] Q5, [glm] Q5.
• There is no PeerTube API to play a playlist into a live [flash] Q5 (verified absence). Switching a channel between live and VOD relies on unverified viewer-UI behaviour [glm] Q5.
• permanentLive and saveReplay are mutually exclusive (verified 400 error). Fallback channels therefore run without replay, and the channel config UI says so [flash] Risk 5.

Deferred. [glm]'s Tier-2 mode ("live windows + PeerTube playlist page", where the worker stops pushing when dark) is a cost-saving option for channels that do not need continuity. It is not in this plan. Revisit it after we have measured the 24/7 encoder cost.

Mechanics [SYN] (fits the WP9 sandbox: FFmpeg reads only pipes):

• A playlist is a source kind (kind: "playlist"). A worker-side asset feeder walks the ordered asset list:
  • assets are presigned GETs through the ingest egress relay;
  • they are piped one after another into a normal ingest process (-re, normalize stage);
  • the compositor sees a continuous input like any other source.
  • Asset boundaries are hidden by the ingest normalize stage plus the gate (PTS regenerated). Concat-protocol tricks are not used; they are banned by the WP9 protocol rules.
• Assets are pre-normalized at upload to the channel profile by the existing WP4 video pipeline. Loops never transcode. Pricing therefore follows the [glm] "normalize-on-ingest, not in the loop" note.
• Programming source of truth. The FALLBACK scene's playlist is either a local asset list or mirrored from a PeerTube channel playlist through the connector (GET /video-playlists/{id}/videos), downloaded to S3 and normalized [flash] Q5.
• Slate floor. A static slate (logo + "back soon" + optional clock) is generated by the overlay renderer. It is always available, independent of asset availability. When the playlist ingest is unhealthy (asset missing, decode error), the scene degrades to the slate automatically.

5.2 Auto Director rules [both]

The evaluation context and plumbing already exist and are not redesigned here:

• Monitor states: the reducer emits OFFLINE|PROBING|LIVE|UNSTABLE|ERROR, with STARTING/ENDING/UNKNOWN added in the DB. Hysteresis: liveConfirmations=2, offlineConfirmations=3, noMediaMs=10 s, minHoldMs=2 s, cooldownMs=10 s (apps/api/src/live-monitor/reducer.ts).
• Evaluation plumbing: WP10's 250 ms debounce, one evaluation in flight per channel, and continue-as-new.

Rule ladder, evaluated top-down. The first match writes program.

1. Program lock / operator override (four WP10 release modes). Automation never undoes a human TAKE while an override or lock holds [both].
2. Media-health failover: if the program source's media-health tier reports failure, the debounce is bypassed. Fail over to the next eligible source, otherwise to FALLBACK. This works during an API outage (WP10).
3. Scheduled windows: WP13 schedules can force a scene inside a window.
4. Priority list: the highest-priority source in LIVE wins. A source already on program that turns UNSTABLE keeps program until noMediaMs plus confirmations expire.
5. Terminal: the FALLBACK scene (playlist, else slate).

Anti-flap at the director level [glm] Q7:

• Dwell time by source class. Defaults: a live source holds at least 15 s; FALLBACK holds at least 60 s before a newly LIVE source may preempt it, unless that source's priority is "interrupt". This complements, and does not duplicate, monitor hysteresis.
• Cooldown of 10 s per source after it leaves program before it may return automatically.
• Out of scope: "real direction" such as audio-activity speaker switching or content-aware cuts. Future work adds it as new conditions in the rule condition jsonb, not as a new engine [both].

Monitor-state integration in the Control Room. The UI renders:

• AUTO/MANUAL/LOCKED status;
• for each rule, a "would fire now" indicator computed from the same monitor snapshot, so operators see why the director acted [flash] Q7;
• a dwell countdown on the program tile;
• RETURN TO AUTO, which re-arms the ladder.

Every automated action uses the same bus transaction as a human TAKE: one desired_version bump and one outbox row. Automation has no special-cased path [glm] Q7.

5.3 PGM/PST control set (web now, same commands on mobile later) [both]

The v1 control set:

• PST select (tile or scene);
• CUT (TAKE, transition: {type:"cut"});
• overlay on/off for each overlay slot (armed on PST, live after TAKE);
• program audio primary pick and gain steps;
• monitor audio (local only);
• LOCK;
• RETURN TO AUTO.

Layout picker, position swap and PIP all operate on the PST scene and reach program only through TAKE [flash] Q6.

FADE ({type:"fade", ms: 300|500|1000|2000}) is added in S4 behind the continuity gate (D8). If a fade graph fails to start within 1 s, the worker degrades to CUT and reports TRANSITION_DEGRADED [flash] Risk 2.

Every command carries an Idempotency-Key and is checked against director.*. The target is under 300 ms from command to observed (WP11).

Mobile "go live from preview" is the same PST select followed by TAKE. It needs no new server surface.

---

6. Phased build plan

Six dispatchable slices. Migration numbers are pre-assigned so parallel slices do not collide. 011 stays reserved for the review workflow ([dm] §4), and 012 exists.

Default gate for every slice: pnpm typecheck && pnpm test green from the repo root (turbo), plus the slice-specific gates below. Each slice writes runs/<wp>/<slice>-report.md.

         ┌── S2 worker compositor ──┐
S1 ──────┤                          ├── S4 fallback + director + FADE
         └── S3 control room ───────┘
S5 chat hub + contract   (independent; starts day 1)  ── S5b Opticast client (anarchostreamer repo, after S5 contract freeze)
S6 telegram WP15 middle  (independent; starts day 1)

Parallelism: S1, S5 and S6 can start together. S2 and S3 run in parallel once S1 lands. S3 develops against a fake worker that echoes realized versions. S4 needs both S2 and S3.

S1 — Desired-state v2 + production schema (small; blocks S2/S3)

• Scope:
  • Extend DesiredChannel/DesiredSpec from a single source:{kind:"hls"} to { scenes, buses, sources[], overlays[], transition, audio }. Old v1 snapshots must still validate.
  • Keep the strict key allowlist and secret rejection.
  • Add migration 013_production.sql: production_scenes, scene_slots (layout enum, geometry CHECKs, audio_mode with the one-primary partial unique index, gain_db in −24…0 in steps of 3), production_buses (preview|program, unique per channel), overlay_slots/overlay_revisions (by asset id), and source.kind gaining playlist.
  • All tables use channel-scoped composite FKs for the same-channel invariant, plus RLS.
• Files: apps/api/src/media-control/spec.ts, wire.ts, apps/media-worker/src/types.ts, apps/api/migrations/013_production.sql, plus tests.
• Gates:
  • Old snapshots round-trip.
  • The [wb] WP12 same-channel invariant tests fail at the DB constraint.
  • Injection payloads in labels (;[,', movie=) are rejected by the type layer.
  • The RLS denial suite covers the new tables.

S2 — Worker compositor + frame gate + taps (XL, highest risk; after S1, ∥ S3)

• Scope:
  • The frame-gate spike comes first (§1.4 gate: Node vs Rust; 1 000-swap continuity).
  • MediaCommandBuilder graph builders emit xstack (2x1/2x2/3x3/4x4 plus custom), PIP overlay, overlay asset inputs, chyron x-expression, and volume+amix, all from typed params only.
  • Ingest×N normalize stage; compositor A/B lifecycle in the supervisor; persistent encoder fed by the gate; playlist feeder ingest.
  • Preview taps: JPEG, 360p LL-HLS, audio-only, and the program tap.
  • The supervisor/bus/fence stay untouched apart from the meaning of a RUNNING snapshot [glm] Phase 1.
• Files: apps/media-worker/src/compositor/* (new), src/gate/* (new), src/relay/ffmpeg.ts (generalized), src/supervisor/index.ts (A/B lifecycle), src/preview/* (new).
• Gates:
  • Golden-frame hashes for each layout.
  • Zero gap at the encoder input across 1 000 scripted scene, overlay and gain swaps (frame counter plus PCM continuity check).
  • A 4-input 1080p30 CPU budget test on the reference host.
  • kill -9 of the compositor mid-swap leaves the encoder and RTMP session up.
  • The ≥8 h soak is wired into the nightly run (not blocking merge).
  • Existing WP9 sandbox, bus and secret-leak suites stay green.

S3 — Control plane + Control Room UI + overlay pipeline (XL; after S1, ∥ S2)

• Scope:
  • Bus writes: TAKE/CUT/PST-select/lock/override in media-control.service.ts. One tx = bus write + desired_version++ + outbox, with Idempotency-Key and director.*.
  • Preview endpoint: ETag, single-flight fill, same-origin, signed short TTL.
  • Realtime gateway:
    • wire /realtime for real, including the subscribe-capable Redis client, which is shared with S5;
    • S3 owns the gateway and S5 builds on it;
    • if S5 starts first, S5 lands the gateway and S3 rebases.
  • Web: multiview tiers per §2.2, Web Audio monitor per §2.3, PST canvas look-ahead, tally from observed state only, overlay editor.
  • Overlay pipeline: SVG templates → resvg-js render job → StorageService → asset id → desired-state bump.
• Files: apps/api/src/media-control/*, apps/api/src/realtime/*, apps/api/src/production/* (new: scenes, overlays, preview), apps/web/src/app/control-room/*, apps/web/src/components/control-room/*.
• Gates:
  • Idempotency matrix: duplicate, concurrent duplicate, crash mid-tx.
  • Command → observed under 300 ms against the fake worker.
  • Playwright E2E: PST → CUT → tally flips only on observed state.
  • Grid tier caps enforced: a 5th live tile is refused.
  • Overlay injection tests through the render job.
  • Revocation evicts the operator within 5 s.

S4 — Fallback scene + Auto Director v1 + FADE (L; after S2 + S3)

• Scope:
  • FALLBACK scene: playlist source mirrored from a PeerTube channel playlist via packages/connector-peertube, plus the slate floor.
  • permanentLive wiring and the replay conflict surfaced in channel config.
  • Rule ladder per §5.2 with dwell-by-class and cooldown, consuming live-monitor outbox events (WP10 intake rules).
  • "Would fire now" indicators and RETURN TO AUTO.
  • FADE via the two-swap transition graph, degrading to CUT.
• Files: apps/api/src/director/* (new), apps/scheduler/* (Temporal evaluator workflow), packages/connector-peertube/src/* (playlist mirror), apps/media-worker/src/compositor/transition.ts, and the Control Room AUTO panel.
• Gates:
  • [wb] WP10 matrix: 0/1/2/3+ live, high-priority interrupt, TAKE not undone, lock stability, flap damping, failover without output kill, API-outage scenario.
  • Event-storm test (50 sources at 10 Hz → ≤4 evaluations/s).
  • Playlist asset removed mid-loop → slate within 2 s with no output gap.
  • FADE continuity holds across 200 fades.

S5 — Chat hub + unified contract + web chat panel (L; independent)

• Scope:
  • Phase 0 contract (§3.5) frozen in packages/core as types and JSON schema.
  • Migration 014_live_chat.sql: live_chat_messages partitioned monthly, live_chat_seen_events, devices, device_pairings.
  • apps/api/src/live-chat/ (new):
    • Twitch EventSub WS and YouTube shared-poller ingestion;
    • seq assignment and ring buffer;
    • send governors (token buckets, YT quota ledger);
    • reply fan-out and moderation fan-out with per-target Delivery;
    • display filters.
  • Twitch connector gains Helix send/moderate.
  • Device pairing.
  • Web chat panel in the Control Room.
  • PeerTube XMPP adapter = S5 phase 2. Ship capability-probe + read-only-embed + comments first.
• Files: apps/api/src/live-chat/*, apps/api/src/devices/*, apps/api/src/realtime/* (shared with S3, see S3), packages/connector-twitch/src/index.ts, packages/connector-mock-livestream/*, apps/web/src/components/live-chat/*, apps/api/migrations/014_live_chat.sql.
• Gates:
  • Mock-livestream E2E: ingest → hub → Redis → WS → client, including lastSeq resume and the resync path.
  • Dedupe: the same provider id twice gives one row.
  • Authz-epoch bump closes sockets within 5 s.
  • The YT quota ledger refuses a send below budget and surfaces paused:quota.
  • Delivery unsupported per target never fails the batch.
  • Retention sweep test per the [wb] cross-cutting row.
• S5b — Opticast client (separate repo anarchostreamer; dispatch after the S5 contract freeze):
  • OkHttp (its only new network dependency), a pairing settings screen, ChatViewModel + bottom sheet, WS lifetime tied to StreamingService, the composer with target pill, and swipe moderation.
  • Gate: testDebugUnitTest assembleDebug green, plus a contract test against a recorded S5 WS transcript.

S6 — WP15 Telegram middle (M; independent)

• Scope [tg] §5 Phase 1:
  • Migration 015_archive.sql: feed_subscriptions ({type:"telegram_channel", connectionId, channelId}) and archived_feed_items (RLS; dedup on (workspace_id, remote_ref=chat_id:message_id) and content hash).
  • Channel-post normalizer: media_group_id album debounce of 2 s, edited_channel_post → revision, forward_origin and author_signature kept as attribution.
  • my_chat_member → subscription health ("bot removed").
  • Media harvest via getFile into StorageService. Files over 20 MB are recorded as "Bot API cannot fetch", not failed.
  • "Create draft from archived item" → the existing review states in publishing.service.ts.
  • Approval calls the existing amplify(). Protected content shows "amplify unavailable".
  • Rights note in the UI, and attribution on by default.
  • No MTProto.
• Dependency note. The draft binding targets today's editing → in_review → approved states. When the review workflow (011, [dm] WP5.1–5.2) lands, the archived-item draft enters the review cycle unchanged, because it is just a draft. Amplifying others' content must never auto-publish [tg] §2.3.
• Files: apps/api/src/archive/* (new), packages/connector-telegram/src/index.ts (normalizer hook only), apps/api/src/publishing/* (draft-from-item), apps/web/src/app/archive/*, apps/api/migrations/015_archive.sql.
• Gates:
  • Webhook fixture → one archived row for a 3-photo album.
  • The same post via two subscriptions → one row.
  • The same image → one blob.
  • Tenant B cannot read tenant A's rows despite a shared blob.
  • A protected-content forward surfaces ConnectorError.
  • E2E: post → archive → draft → approve → forwardMessage on the mock.

6.4 Not in this plan (follow-ons)

• Opticast as a camera source: an rtmp-push/srt-listen ingest kind with per-channel ingest keys.
• WHIP guest contribution.
• Tier-2 PeerTube-playlist mode.
• MTProto grabber ([tg] Phase 2: flagged, ToS disclosure).
• Chat relay bot.
• Runtime azmq gain control.
• Mobile Control Room screens. These reuse the S3 endpoints and need no new server surface.

---

7. Top risks and mitigations

#	Risk	Likelihood / impact	Mitigation
1	The frame gate / A/B swap is not frame-continuous, or costs too much CPU. Every TAKE, overlay and gain change depends on it.	Medium / critical	It is the first task of S2 and gates the whole slice: 1 000-swap continuity plus a CPU budget; a Rust gate if Node fails; GStreamer input-selector as the escalation path (the WP9 decision gate). The transient doubled-compositor CPU is reserved inside the 20 % failover headroom.
2	Multiview expectations versus browser physics. Users expect 16 live tiles. Decode limits are undocumented ([both]: not verified), and iPhone MSE exists only from 17.1.	High / medium	Thumbnail-default tiers with a hard cap of 4 (1 on mobile) and explicit tile-state copy. Run a device-matrix test before any marketing claim. Tiles never decode remote sources.
3	Worker CPU at scale. A 1080p30 N-input grid plus overlays and amix, plus an always-on fallback encoder 24/7 for every channel.	High / high	Input caps per profile in admission. Playlist assets pre-normalized, never transcoded in the loop. FALLBACK priced in capacity units, not treated as free. Tier-2 "dark" mode as the documented future cost lever. CI CPU-budget test in S2.
4	Chat platform fragility: Twitch has no EventSub replay and harsh rate enforcement; the YouTube quota is 10k/day; the PeerTube livechat plugin varies by instance and its bot token is unsupported.	High / medium	Ring-buffer resume plus chat.gap markers. Token buckets and the quota ledger with paused:* states. Capability probe with tiered PeerTube degradation (C2S → read-only embed → comments). Request a YouTube quota increase for production.
5	PeerTube coupling: permanentLive excludes replay, Small latency disables P2P, and live↔VOD switching is unmeasured.	Medium / high	Worker-side FALLBACK from day one. Replay and latency mode are explicit channel config with in-UI consequences. Measure live-end→reconnect on a test instance before promising Tier-2.
6	Overlay and asset attack surface: SVG parsing, fonts, render-job resource exhaustion.	Medium / high	Server-owned templates with typed field slots. resvg with no scripts and no external fetch, run in the WP4 sandbox with time and memory limits. Rendered PNGs are content-addressed and cached. FFmpeg only ever sees asset ids. Injection tests in S1 and S3.
7	Telegram amplification ethics and ToS: re-posting others' content, protected channels, the 20 MB bot cap.	Certain / medium	A human review gate is mandatory. Attribution is on by default. Protected content and oversize media are surfaced honestly. MTProto stays out of this plan.
8	Opticast privacy pledge erosion: pressure to add FCM, analytics or a cloud relay.	Medium / medium	The contract offers no push and only the self-hosted endpoint. The integration is opt-in and off by default. Any push must be UnifiedPush/self-hosted and opt-in. The relationship stays API-only, so the unresolved GPLv3 question remains moot.

---

Appendix — where the two control-room reports disagree

Topic	[flash]	[glm]	Pick	Why
Process topology	One FFmpeg per channel	Split ingest/compositor/encoder/pusher ([prop])	Split + frame gate	FFmpeg cannot rewire graphs, so a single process restarts the RTMP session on every change.
Overlay text	Pre-rendered PNG (SVG→resvg)	drawtext + textfile reload	PNG	PST/PGM pixel parity, brand typography, fonts kept out of the 24/7 sandbox, cheaper CPU. glm's no-restart edge is covered by the A/B swap.
Live tile cap at 3x3/4x4	Focused + PST live (≤4 total)	1–2 live (focus + PGM)	PGM + focused live, cap 4 on explicit upgrade	The PGM tile must be real, so its truth comes from the program tap. PST is a client-side look-ahead, not a decoded stream.
Fallback modes	Worker-side only	Worker-side Tier 1 + PeerTube-playlist Tier 2	Tier 1 now, Tier 2 deferred	Tier 2 depends on unverified PeerTube UI behaviour; it stays as a later cost lever.
Program gain changes	amix runtime commands	Desired-state bump in coarse 3 dB steps	3 dB steps via swap; runtime commands later	azmq availability is unverified, and swaps are already continuous.
PST rendering	Not specified (implied worker)	Browser preview compositor (canvas)	Browser canvas look-ahead	Avoids a second server composite for each channel.
Transitions	CUT plus FADE presets; fall back to CUT	CUT plus 300 ms fade default	CUT in v1; FADE in S4, degrading to CUT	FADE needs a two-swap transition graph and must be proven continuous first.