T-HUB/docs/engineering-guide.md

Tactical Media Broadcast Hub — Engineering Guide

Covers: everything built as of 499c9fe (WP0–WP3-in-flight), plus Level-2 systems as marked (planned). Companion docs: admin-guide.md · user-guide.md.

---

1. System overview

Tactical Media Broadcast Hub is a self-hosted, multi-tenant social-media management platform (Level 1: post & schedule) growing into a video/livestream master-control system (Level 2, planned). Three planes:

CONTROL PLANE                     MEDIA PLANE (planned, WP9+)
  apps/web  (Next.js UI)            media-worker: ingest → compositor →
  apps/api  (NestJS)                  persistent encoder → outputs
  apps/scheduler (Temporal)        FFmpeg-first; never through the API
  PostgreSQL · Redis · Temporal
STORAGE PLANE
  S3-compatible object storage (MinIO dev / any S3 prod)
  ALL media client-side encrypted; content-addressed

Hard rules enforced in code: media bytes never transit API processes; provider branching outside connector packages fails lint; passwords never transit the system (OAuth tokens / bot tokens only); every tenant table is RLS-isolated with FORCE ROW LEVEL SECURITY.

2. Repository layout

apps/
  web/            Next.js 14 App Router frontend (all screens, mock-registry layer)
  api/            NestJS control plane (health, auth, rbac, vault, audit, publishing, storage)
  scheduler/      Temporal worker (publish workflow + activities)
  media-worker/   health-only stub (WP9 fills it)
packages/
  core/           normalized domain types (permissions, connections, publishing, VC family)
  connector-sdk/  PlatformConnector interface, manifest, registry, HTTP client, contract kits
  connector-example/ connector-mock-social/ connector-mock-video/ connector-mock-livestream/
  connector-mastodon/ connector-telegram/          (WP3, in flight)
  testing/        live-source simulator (dod-flap scenario) + mock RTMP sink
infra/            docker-compose (postgres:5433, redis, temporal, minio, otel, prometheus, grafana)
docs/             this documentation set

Monorepo: pnpm workspaces + Turborepo. pnpm lint|typecheck|build|test at root runs all 13+ packages.

3. Identity, tenancy & security (WP2)

3.1 Database roles — two-pool discipline

• Owner pool (SUPER_DATABASE_URL, the postgres superuser in dev): runs migrations, provisions. Never serves requests.
• App pool (DATABASE_URL, role sp_app): neither superuser nor table owner → FORCE ROW LEVEL SECURITY binds for real. Created by migration 003_app_role.sql with table/sequence grants + default privileges.

3.2 Row-level security

Every tenant table (memberships, connections, credentials, audit_log, drafts, publications, scheduled_publications) has:

USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)

NULLIF matters: pooled sessions carry an empty-string GUC placeholder after SET LOCAL commits; without it the cast throws instead of failing closed. Tenant context is set per-transaction via withTenant(workspaceId, fn) (apps/api/src/db/client.ts) using set_config(..., true) — transaction-local, cannot leak.

3.3 SECURITY DEFINER discipline (audit finding #3)

Cross-tenant functions take an explicit p_workspace_id and call fn_assert_tenant(), which asserts the parameter against the session's tenant context. No unguarded EXECUTE grants for sp_app.

3.4 Continuous authorization (audit finding #2)

Any membership insert/update/delete fires bump_authz_epoch() — COALESCE(NEW.workspace_id, OLD.workspace_id) (OLD is NULL on INSERT; the original bug silently no-oped). Sessions embed the epoch at issue; RbacGuard compares against the live workspaces.authz_epoch and fails closed (401, epoch stale) on mismatch. Revocation propagates to live sessions without waiting for re-login.

3.5 Sessions, logout & the Hub as OIDC issuer

HMAC-SHA256 signed tokens (apps/api/src/auth/session.ts), 12h TTL, constant-time verify. Since migration 034, every session also carries a sid backed by an auth_sessions row: login/reissue write the row, POST /auth/logout revokes it, and reissue refuses both revoked rows and pre-034 tokens (no sid) so revocation applies from the next epoch bump. Request validation (RbacGuard) stays stateless — rows are consulted only by reissue and the OIDC refresh path.

The Hub is also its own OpenID Connect provider for oCIS (apps/api/src/oidc/, issuer https://api.occupymedia.cc/oidc): discovery, JWKS (ES256, keys sealed via the vault pattern), /oidc/authorize → SPA consent → single-use code (60 s, PKCE S256 enforced) → tokens. Only the first-party client web (the oCIS web app) is registered; anything else fails closed. Phase 2 added session-bound rotating refresh tokens (oidc_refresh_tokens, SHA-256 at rest, family_id chains): a chain lives and dies with the consenting Hub session, reuse of a consumed token revokes the whole family (OAuth BCP), and every accept re-checks that the session row is live and the membership active — so Hub logout propagates to oCIS at the next refresh instead of at the next silent authorize. The unauthenticated token endpoint resolves tokens through the definer-rights oidc_refresh_lookup() (the audit_append pattern). Matrix SSO (apps/api/src/matrix-sso/) is a separate 2-minute m.login.token broker and intentionally out of propagation scope: Element holds its own Matrix-native session after mint.

3.6 Credential vault

Envelope encryption (vault/vault.ts): per-workspace DEK (AES-256-GCM) wrapped by master KEK (SUPERPOSTER_KEK, 32B base64; production-required). Rotation re-wraps; old DEK dropped → pre-rotation ciphertext unreadable (tested). Connectors receive opaque CredentialRef handles via CredentialAccessor — never ciphertext, never raw tokens.

3.7 Audit log

Hash-chained per branch (audit_append for tenant rows, audit_append_system for workspaceless events). Branch selection is by tenant-context presence, never by role (findings #1/#6 — sp_scheduler with tenant context appends to the tenant chain). audit_verify_chain(workspace) recomputes hashes — tampering is detectable (tested). Secret-shaped keys are scrubbed to [REDACTED] before append (audit/audit.service.ts).

3.8 Storage encryption (user mandate)

storage/storage.ts — client-side envelope: body = SPENC1 | iv(12) | KEK-wrapped CEK(48) | AES-256-GCM ciphertext+tag. S3_KEK required in production. SSE-at-rest header set additionally. Keys are the plaintext sha256 (sha256/<hash>.<ext>), so put-dedup survives; getObject re-verifies the hash → storage tampering throws. Cross-workspace media dedup is storage-level only; row visibility stays tenant-scoped.

4. Connector architecture (WP1)

4.1 The contract

PlatformConnector (connector-sdk/src/types.ts): manifest(), authorize(), getIdentity(), optional publish/uploadMedia/uploadVideo/getFeed/getComments/reply/getMessages/amplify/unamplify/subscribeWebhooks/handleWebhook/.... Every list method returns Page<T, cursor>.

4.2 Manifest & capabilities

ConnectorManifest declares id, displayName, capabilities (30+ booleans incl. amplifyRepost/amplifyQuote/amplifyShare), connectionFields (e.g. Mastodon instanceUrl, Telegram botToken secret), oauth flavor. The manifest validator rejects password fields outright. UI renders strictly from capabilities — an undeclared capability renders nothing, never a grayed-out lie.

4.3 Registry & the no-branching rule

ConnectorRegistry.register/get — unknown provider throws UnknownProviderError. The ESLint rule no-restricted-syntax (root eslint.config.mjs) fails any provider string literal outside packages/connector-*/, packages/testing/, and apps/web/src/lib/ (the mock-registry boundary standing in for the real registry until the API wires up — same discipline, documented in the config).

4.4 HTTP discipline

RateLimitedHttpClient: per-host token bucket (default 1 rps; Mastodon 0.5), exponential backoff with jitter on 429/5xx, Retry-After honored.

4.5 Amplify truth table (verified per platform, 2026)

Platform	Repost	Quote	Share	Notes
Mastodon	✅ reblog	✅ quoted_status_id (≥4.5)	—	native scheduling via scheduled_at
Telegram	—	—	✅ forward/copy	has_protected_content blocks
Facebook	⚠️ no native reshare API	—	link-share approximation	manifest marks degraded
Instagram	❌	❌	own-media stories only	API exposes no amplify
Bluesky / Threads / LinkedIn	✅ / ✅ / ✅(reshare w/ commentary)	✅ / ✅ / —	—	Level 1b wave
Reddit	—	—	crosspost (stable-but-undocumented) + vote	no vote automation ever
X	✅ (metered)	❌ self-serve (Apr 2026)	—	deferred; pay-per-post model when added

4.6 Contract test kits

connectorContractTests / videoConnectorContractTests / livestreamConnectorContractTests — capability-aware: declared capability without implementing method = FAIL; methods behind undeclared capabilities must be absent. Kits caught two real violations during development.

5. Publishing pipeline (WP3, sol-delivered & verified)

• State machine (publishing.service.ts): editing → in_review → approved → scheduled → published (+ rejected, cancelled). Every transition permission-checked (approval.request / approval.grant / publication.schedule) and audited.
• Durability (006_publishing.sql): scheduled_publications carries a UNIQUE idempotency_key; the DB row commits first; if Temporal is unreachable the row stays pending and the worker's startup sweep picks it up.
• Temporal (scheduler/src/workflows/publish.workflow.ts): scheduledPublish(id) activity (5-min timeout) claims pending-only, resolves the connector via registry, publishes, records the publication, idempotent on re-run.
• Receipts: GET /publishing/drafts/:draftId/publications returns per-attempt outcomes (ready/in_progress/completed/failed/unknown/abandoned, remoteId/remoteUrl); the calendar page renders the latest outcome per scheduled draft as a badge with an open-remote link.

5.1 Per-connection publish governance (migration 033)

connections.fields.publishGovernance decides what happens when an approved draft targets that connection:

Setting	Behavior on approval
unset	today's manual flow: a publication.publish holder calls POST :draftId/publish
auto	the approval itself schedules the publish (console approve or passing proposal poll)
admin1	a second chatgov poll (subject_kind='draft_publish') opens; one admin yes releases
admin_majority	same, quorum [{admins, x: floor(n/2)+1}] snapshotted at open
admin_consensus	same, [{admins, x: n}] join all, veto allowed

Route: PATCH /connections/:id/publish-governance (requires chatgov.manage). The publish route answers 202 {state:"awaiting_publish_approval", pollId, closesAt, rule} while held; a passing poll schedules the publish as the original requester (re-checked at claim). The scheduler's E3 pre-dispatch check refuses any dispatch to a gated destination without a passed publish poll for the current approved revision — including schedules created before the gate was switched on. One open publish poll per (draft, destination), index-enforced, never colliding with review polls.

5.2 Governed boost (amplify)

POST /publishing/drafts/:draftId/amplify {connectionId, kind: "repost"|"quote"} executes the connector's amplify against providerMetadata.sourceRemoteRef recorded at draft creation — never a caller-supplied id. Target gates: draft state approved + the connection's publish governance. A gated boost holds with 202 awaiting_publish_approval on the same publish-phase poll and is released by re-invoking amplify after the pass (hold-and-retry: no durable amplify queue exists, and the poll-close transaction must not do network I/O). Unset/auto connections behave exactly as before (200 + execute).

6. Frontend (apps/web)

Tailwind v4 @theme tokens (dark-first OKLCH, @theme inline brand hook for tenant theming), hand-rolled primitives, cmdk palette, TanStack Virtual streams (500+ items/column, scroll-into-view), Tiptap compose with per-platform char counters/previews. Streams flagship: capability-gated amplify bars, keyboard j/k navigate, r repost, q quote. The Control Room (app/(main)/control-room/) runs on the live API — signed preview taps, realtime tally over the websocket client, fixture fallback when offline. QA: apps/web/scripts/visual-qa.mjs.

7. Testing strategy

Suite	What it proves	Runs
connector contract kits	connectors don't lie about capabilities	CI, always
pure units (vault, session, oidc, scrub, storage-crypto, key-scheme)	crypto/format invariants incl. tamper detection	CI, always
tenancy.integration.test.ts	RLS isolation, tenant-branch audit, epoch bump, chain tamper, immutability triggers	needs DATABASE_URL (sp_app) + SUPER_DATABASE_URL (owner), else skips
storage.integration.test.ts	S3 roundtrip, dedup, presign	needs S3_ENDPOINT, else skips
visual-qa (playwright)	screens render, interactions fire, console clean	local/CI with chromium

Live-DB invocation (from repo root, postgres on :5433):

DATABASE_URL='postgresql://sp_app:sp_app_dev@localhost:5433/tmbh' \
SUPER_DATABASE_URL='postgresql://tmbh:tmbh_dev@localhost:5433/tmbh' \
pnpm --filter @tactical-media-broadcast-hub/api test

8. Observability & ops surface

/healthz (liveness), /readyz (TCP probes: postgres/redis/temporal with per-component status). OTel OTLP tracing skeleton (disabled-safe without collector). Structured pino logs with credential-key redaction. Grafana (3001) provisioned against Prometheus (9090). Full metrics rollout is WP14.

9. Level 2 systems — status

Shipped and wired end-to-end:

• Media control / director (apps/api/src/media-control/, apps/api/src/production/): the PUT desired / take / cut / pst-select / lock / override surface. Each command is a single PostgreSQL transaction — channel row FOR UPDATE, idempotency receipt lookup, director-state fold (locked/override), bus moves, capacity admission (§4.4 quota; over-quota rolls back everything, a stop drains the pending queue), desired-version floor across DB+Redis, and an audit_append receipt (director.<op>, the transactional outbox). The worker observes desired snapshots over signed channels and reports the realized version; tally in the UI is worker-observed state, never optimistic.
• Production spec (production.ts, mirrored in apps/media-worker): scenes (single/pip/2x1/2x2/3x3/4x4/custom, per-slot fit, per-slot audio off/on/primary + gain, ≤1 primary slot), sources (HLS URL or video-asset playlist), overlays (lower_third/chyron/logo/clock, layers 0–7, rendered revisions as content-addressed PNGs, chyron speed/strip params, server clock), transitions (cut or fade 300–2000 ms), program master gain. Strict parsers reject injectable strings (;, [, ,, ', movie=) so nothing rides along into FFmpeg graphs.
• Control Room UI (apps/web/src/app/(main)/control-room/): multiview grid (2x1–4x4/custom, up to 4 live HLS tiles, tap-to-focus, visibility-gated 1 s/2 s preview polling), switcher bar (PST/TAKE-at-frame-boundary/CUT-immediate, lock, override modes until_released/five_minutes/until_source_offline/until_next_event), PST preview, audio monitor, overlay editor (director.change_layout), fixture mode when offline. Permission gates: media.view, director.view, director.take, director.lock_program, director.override_auto, director.change_layout, media.manage.
• Signed previews (production/preview.ts + controller): capability URLs (expires+HMAC) minted by the authenticated endpoint; the public read route is same-origin only, serves plaintext renders, never redirects encrypted S3 objects, ETag/304-aware.
• Worker fence (apps/api/src/director/): internal state/fence/apply endpoints the worker uses to apply desired snapshots under a fencing token; web ingest, PeerTube and the transcode pool per their own docs.

Still planned: media-engine graph rewiring beyond the current FFmpeg-normalize path, LLM video AI (design only), content search rollout.

10. Producer & media ingest (October 2026)

10.1 Media attachments for publishing

• POST /media (multipart file); content type is magic-byte sniffed (jpeg/png/webp/mp4/webm) — the extension is never trusted.
• Caps: image 10 MiB, video 64 MiB, and at most 4 attachments per post (MAX_MEDIA_PER_POST).
• Read back via GET /media/:id (auth-scoped). Bytes live in R2 object storage — SPENC1 envelope-encrypted, content-addressed sha256 keys (§3.8 scheme).

10.2 Alt texts

• Drafts carry altTexts positionally aligned with mediaAssetIds (0–4 entries; null = no alt text for that slot).
• At publish, the scheduler maps them into providerMetadata.altText (asset-id keyed); Mastodon media uploads receive them as descriptions.

10.3 Video producer

• POST /video/:id/produce accepts { overlays?, voiceover?, outputLabel?, lane? } — closed enums with a strict key allowlist; unknown keys are rejected. Overlay kinds: lower_third | logo | clock; overlay text runs through the filtergraph-metacharacter blocklist before it reaches an FFmpeg graph.
• Jobs execute in the produce-worker (its own container, MEDIA_WORKER_LANE=local): claim/lease via polling with SKIP LOCKED; stale leases are re-queued.
• The output is a derived library asset (derived_from), publishable like any other asset.

10.4 Audio-less sources

• The command-builder synthesizes an anullsrc silence bed — configured at the output preset's rate/layout — instead of emitting [0:a] maps that ffmpeg rejects.

10.5 Chunked video ingest

• POST /video/upload/init → PUT /video/upload/:uploadId/chunk/:index → POST /video/upload/:uploadId/complete.
• 10 MiB chunks (CHUNK_SIZE_BYTES) sidestep Cloudflare's ~100 MB proxy body cap. The server reassembles, seals, stores to R2, then runs the normal probe pipeline.
• Max chunked upload: 100 GiB (MAX_CHUNKED_UPLOAD_BYTES).

10.7 Direct-to-R2 video ingest (worker-probed)

• GET /video/direct/config → POST /video/direct/init (presigned PUT) → POST /video/direct/complete (HEAD-gated register + queued probe job). The browser seals SPENC2 and PUTs straight to object storage — the api box never touches video bytes.
• The produce worker claims the probe job (042 widened produce_job_claim), fetches the sealed object off R2, opens by magic (SPENC1 kek / SPENC2 direct key via DIRECT_EC_PRIVATE_KEY), ffprobes, writes metadata, completes. Compose/mix jobs read direct objects the same way.
• Requirements: DIRECT_EC_PRIVATE_KEY on api + media-worker + produce-worker (compose appenv).

10.8 Files direct-to-R2 + media viewer

• GET /files/direct/config, POST /files/direct/init|complete, POST /files/:id/direct-download (presigned GET + CEK ticket; the browser decrypts — the server never streams direct/ payloads).
• Viewer suite: image lightbox (zoom/pan/rotate + filmstrip), brand video player (scrubber/buffered/volume/speed/PiP/fullscreen + keyboard), image EDITOR (crop with aspect presets, rotate 90s, flip H/V, brightness/contrast/saturation — saves as a NEW library file), video TRIM (in/out → the cut renders into the Video library via the assemble lane; the original is never modified), PDF (native iframe), text (+copy), HTML (sandboxed allow-scripts run + source toggle).
• Thumbnails decrypt client-side; video stills are captured in-browser. Session cache dedupes decryption per item.
• Browser memory caps the direct path at 512 MiB (sealing holds plaintext + frame); larger files fall back to the server path (Cloudflare-capped ~100 MB) until multipart lands.

10.9 Multi-file upload queues

• Both dropzones accept multiple files: sequential queue, per-file status (queued / % / done / failed), one failure never aborts the queue.
• Video per-file transport: direct (when enabled, ≤512 MiB) → chunked (>10 MiB) → whole-body (≤10 MiB). Files per-file transport: direct (when enabled, ≤512 MiB) → server multipart.

11. Key file map

• Whole-body video upload cap: 2 GiB (MAX_VIDEO_UPLOAD_BYTES).
• Files group upload cap: 2 GiB (was 20 MiB).

11. Key file map

Concern	Path
Tenant context / pools	apps/api/src/db/client.ts
Migrations 001–034	apps/api/migrations/
Vault / storage crypto	apps/api/src/vault/vault.ts, apps/api/src/storage/storage.ts
Sessions / OIDC provider (issuer + refresh)	apps/api/src/auth/, apps/api/src/oidc/
RBAC guard	apps/api/src/rbac/rbac.guard.ts
Audit chain	migrations/001_tenants.sql (SQL) + apps/api/src/audit/
Publishing + publish governance + amplify	apps/api/src/publishing/, apps/scheduler/src/activities/dispatch-gate.ts
Media control / director / production spec	apps/api/src/media-control/, apps/api/src/production/, apps/media-worker/src/production.ts
Control Room UI	apps/web/src/app/(main)/control-room/, apps/web/src/components/control-room/
Matrix SSO broker	apps/api/src/matrix-sso/
Connector SDK	packages/connector-sdk/src/
Frontend mock-registry	apps/web/src/lib/
Lint: no-provider-branching	eslint.config.mjs
E2E: governance sim (S1–S9), brand palette guard	scripts/e2e/governance-sim.mjs, scripts/e2e/brand-palette-check.mjs