glass beadian
aa5e262f3a
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
151 lines
11 KiB
Markdown
151 lines
11 KiB
Markdown
# Tactical Media Broadcast Hub — Engineering Guide
|
||
|
||
**Covers:** everything built as of `499c9fe` (WP0–WP3-in-flight), plus Level-2 systems as marked *(planned)*. Companion docs: [admin-guide.md](./admin-guide.md) · [user-guide.md](./user-guide.md).
|
||
|
||
---
|
||
|
||
## 1. System overview
|
||
|
||
Tactical Media Broadcast Hub is a self-hosted, multi-tenant social-media management platform (Level 1: post & schedule) growing into a video/livestream master-control system (Level 2, planned). Three planes:
|
||
|
||
```
|
||
CONTROL PLANE MEDIA PLANE (planned, WP9+)
|
||
apps/web (Next.js UI) media-worker: ingest → compositor →
|
||
apps/api (NestJS) persistent encoder → outputs
|
||
apps/scheduler (Temporal) FFmpeg-first; never through the API
|
||
PostgreSQL · Redis · Temporal
|
||
STORAGE PLANE
|
||
S3-compatible object storage (MinIO dev / any S3 prod)
|
||
ALL media client-side encrypted; content-addressed
|
||
```
|
||
|
||
Hard rules enforced in code: media bytes never transit API processes; provider branching outside connector packages fails lint; passwords never transit the system (OAuth tokens / bot tokens only); every tenant table is RLS-isolated with `FORCE ROW LEVEL SECURITY`.
|
||
|
||
## 2. Repository layout
|
||
|
||
```
|
||
apps/
|
||
web/ Next.js 14 App Router frontend (all screens, mock-registry layer)
|
||
api/ NestJS control plane (health, auth, rbac, vault, audit, publishing, storage)
|
||
scheduler/ Temporal worker (publish workflow + activities)
|
||
media-worker/ health-only stub (WP9 fills it)
|
||
packages/
|
||
core/ normalized domain types (permissions, connections, publishing, VC family)
|
||
connector-sdk/ PlatformConnector interface, manifest, registry, HTTP client, contract kits
|
||
connector-example/ connector-mock-social/ connector-mock-video/ connector-mock-livestream/
|
||
connector-mastodon/ connector-telegram/ (WP3, in flight)
|
||
testing/ live-source simulator (dod-flap scenario) + mock RTMP sink
|
||
infra/ docker-compose (postgres:5433, redis, temporal, minio, otel, prometheus, grafana)
|
||
docs/ this documentation set
|
||
```
|
||
|
||
Monorepo: pnpm workspaces + Turborepo. `pnpm lint|typecheck|build|test` at root runs all 13+ packages.
|
||
|
||
## 3. Identity, tenancy & security (WP2)
|
||
|
||
### 3.1 Database roles — two-pool discipline
|
||
- **Owner pool** (`SUPER_DATABASE_URL`, the postgres superuser in dev): runs migrations, provisions. Never serves requests.
|
||
- **App pool** (`DATABASE_URL`, role `sp_app`): neither superuser nor table owner → `FORCE ROW LEVEL SECURITY` binds for real. Created by migration `003_app_role.sql` with table/sequence grants + default privileges.
|
||
|
||
### 3.2 Row-level security
|
||
Every tenant table (memberships, connections, credentials, audit_log, drafts, publications, scheduled_publications) has:
|
||
```sql
|
||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||
```
|
||
`NULLIF` matters: pooled sessions carry an empty-string GUC placeholder after `SET LOCAL` commits; without it the cast throws instead of failing closed. Tenant context is set per-transaction via `withTenant(workspaceId, fn)` (`apps/api/src/db/client.ts`) using `set_config(..., true)` — transaction-local, cannot leak.
|
||
|
||
### 3.3 SECURITY DEFINER discipline (audit finding #3)
|
||
Cross-tenant functions take an explicit `p_workspace_id` and call `fn_assert_tenant()`, which asserts the parameter against the session's tenant context. No unguarded `EXECUTE` grants for `sp_app`.
|
||
|
||
### 3.4 Continuous authorization (audit finding #2)
|
||
Any membership insert/update/delete fires `bump_authz_epoch()` — `COALESCE(NEW.workspace_id, OLD.workspace_id)` (OLD is NULL on INSERT; the original bug silently no-oped). Sessions embed the epoch at issue; `RbacGuard` compares against the live `workspaces.authz_epoch` and fails closed (`401, epoch stale`) on mismatch. Revocation propagates to live sessions without waiting for re-login.
|
||
|
||
### 3.5 Sessions & OIDC
|
||
HMAC-SHA256 signed tokens (`apps/api/src/auth/session.ts`), 12h TTL, constant-time verify. OIDC: generic code flow + PKCE S256, discovery-document driven (`auth/oidc.ts`) — Authentik/Keycloak/any compliant issuer via `OIDC_*` env.
|
||
|
||
### 3.6 Credential vault
|
||
Envelope encryption (`vault/vault.ts`): per-workspace DEK (AES-256-GCM) wrapped by master KEK (`SUPERPOSTER_KEK`, 32B base64; production-required). Rotation re-wraps; old DEK dropped → pre-rotation ciphertext unreadable (tested). Connectors receive opaque `CredentialRef` handles via `CredentialAccessor` — never ciphertext, never raw tokens.
|
||
|
||
### 3.7 Audit log
|
||
Hash-chained per branch (`audit_append` for tenant rows, `audit_append_system` for workspaceless events). Branch selection is by **tenant-context presence, never by role** (findings #1/#6 — `sp_scheduler` with tenant context appends to the tenant chain). `audit_verify_chain(workspace)` recomputes hashes — tampering is detectable (tested). Secret-shaped keys are scrubbed to `[REDACTED]` before append (`audit/audit.service.ts`).
|
||
|
||
### 3.8 Storage encryption (user mandate)
|
||
`storage/storage.ts` — client-side envelope: body = `SPENC1 | iv(12) | KEK-wrapped CEK(48) | AES-256-GCM ciphertext+tag`. `S3_KEK` required in production. SSE-at-rest header set additionally. Keys are the **plaintext** sha256 (`sha256/<hash>.<ext>`), so put-dedup survives; `getObject` re-verifies the hash → storage tampering throws. Cross-workspace media dedup is storage-level only; row visibility stays tenant-scoped.
|
||
|
||
## 4. Connector architecture (WP1)
|
||
|
||
### 4.1 The contract
|
||
`PlatformConnector` (`connector-sdk/src/types.ts`): `manifest()`, `authorize()`, `getIdentity()`, optional `publish/uploadMedia/uploadVideo/getFeed/getComments/reply/getMessages/amplify/unamplify/subscribeWebhooks/handleWebhook/...`. Every list method returns `Page<T, cursor>`.
|
||
|
||
### 4.2 Manifest & capabilities
|
||
`ConnectorManifest` declares `id`, `displayName`, `capabilities` (30+ booleans incl. `amplifyRepost/amplifyQuote/amplifyShare`), `connectionFields` (e.g. Mastodon `instanceUrl`, Telegram `botToken` secret), `oauth` flavor. **The manifest validator rejects password fields outright.** UI renders strictly from capabilities — an undeclared capability renders nothing, never a grayed-out lie.
|
||
|
||
### 4.3 Registry & the no-branching rule
|
||
`ConnectorRegistry.register/get` — unknown provider throws `UnknownProviderError`. The ESLint rule `no-restricted-syntax` (root `eslint.config.mjs`) fails any provider string literal outside `packages/connector-*/`, `packages/testing/`, and `apps/web/src/lib/` (the mock-registry boundary standing in for the real registry until the API wires up — same discipline, documented in the config).
|
||
|
||
### 4.4 HTTP discipline
|
||
`RateLimitedHttpClient`: per-host token bucket (default 1 rps; Mastodon 0.5), exponential backoff with jitter on 429/5xx, `Retry-After` honored.
|
||
|
||
### 4.5 Amplify truth table (verified per platform, 2026)
|
||
| Platform | Repost | Quote | Share | Notes |
|
||
|---|---|---|---|---|
|
||
| Mastodon | ✅ reblog | ✅ `quoted_status_id` (≥4.5) | — | native scheduling via `scheduled_at` |
|
||
| Telegram | — | — | ✅ forward/copy | `has_protected_content` blocks |
|
||
| Facebook | ⚠️ no native reshare API | — | link-share approximation | manifest marks degraded |
|
||
| Instagram | ❌ | ❌ | own-media stories only | API exposes no amplify |
|
||
| Bluesky / Threads / LinkedIn | ✅ / ✅ / ✅(reshare w/ commentary) | ✅ / ✅ / — | — | Level 1b wave |
|
||
| Reddit | — | — | crosspost (stable-but-undocumented) + vote | no vote automation ever |
|
||
| X | ✅ (metered) | ❌ self-serve (Apr 2026) | — | deferred; pay-per-post model when added |
|
||
|
||
### 4.6 Contract test kits
|
||
`connectorContractTests / videoConnectorContractTests / livestreamConnectorContractTests` — capability-aware: declared capability without implementing method = FAIL; methods behind undeclared capabilities must be absent. Kits caught two real violations during development.
|
||
|
||
## 5. Publishing pipeline (WP3, sol-delivered & verified)
|
||
|
||
- **State machine** (`publishing.service.ts`): `editing → in_review → approved → scheduled → published` (+ `rejected`, `cancelled`). Every transition permission-checked (`approval.request` / `approval.grant` / `publication.schedule`) and audited.
|
||
- **Durability** (`006_publishing.sql`): `scheduled_publications` carries a `UNIQUE idempotency_key`; the DB row commits first; if Temporal is unreachable the row stays `pending` and the worker's startup sweep picks it up.
|
||
- **Temporal** (`scheduler/src/workflows/publish.workflow.ts`): `scheduledPublish(id)` activity (5-min timeout) claims pending-only, resolves the connector via registry, publishes, records the publication, idempotent on re-run.
|
||
|
||
## 6. Frontend (apps/web)
|
||
|
||
Tailwind v4 `@theme` tokens (dark-first OKLCH, `@theme inline` brand hook for tenant theming), hand-rolled primitives, cmdk palette, TanStack Virtual streams (500+ items/column, scroll-into-view), Tiptap compose with per-platform char counters/previews. **Streams flagship:** capability-gated amplify bars, keyboard `j/k` navigate, `r` repost, `q` quote. All data from `src/lib/` mock-registry (fixtures are seeded-deterministic). QA: `apps/web/scripts/visual-qa.mjs` — 14 states, 0 console errors; byte-diff confirms every interaction renders.
|
||
|
||
## 7. Testing strategy
|
||
|
||
| Suite | What it proves | Runs |
|
||
|---|---|---|
|
||
| connector contract kits | connectors don't lie about capabilities | CI, always |
|
||
| pure units (vault, session, oidc, scrub, storage-crypto, key-scheme) | crypto/format invariants incl. tamper detection | CI, always |
|
||
| `tenancy.integration.test.ts` | RLS isolation, tenant-branch audit, epoch bump, chain tamper, immutability triggers | needs `DATABASE_URL` (sp_app) + `SUPER_DATABASE_URL` (owner), else skips |
|
||
| `storage.integration.test.ts` | S3 roundtrip, dedup, presign | needs `S3_ENDPOINT`, else skips |
|
||
| visual-qa (playwright) | screens render, interactions fire, console clean | local/CI with chromium |
|
||
|
||
Live-DB invocation (from repo root, postgres on :5433):
|
||
```
|
||
DATABASE_URL='postgresql://sp_app:sp_app_dev@localhost:5433/tmbh' \
|
||
SUPER_DATABASE_URL='postgresql://tmbh:tmbh_dev@localhost:5433/tmbh' \
|
||
pnpm --filter @tactical-media-broadcast-hub/api test
|
||
```
|
||
|
||
## 8. Observability & ops surface
|
||
|
||
`/healthz` (liveness), `/readyz` (TCP probes: postgres/redis/temporal with per-component status). OTel OTLP tracing skeleton (disabled-safe without collector). Structured pino logs with credential-key redaction. Grafana (3001) provisioned against Prometheus (9090). Full metrics rollout is WP14.
|
||
|
||
## 9. Level 2 systems *(planned — designs in proposal.md §4.4)*
|
||
|
||
Live Source Monitor (adaptive polling, anti-flapping, per-host budgets); media engine (persistent encoder, graph-rewire switching, GStreamer decision gate); Auto Director (policies/rules/override/program-lock); control room (Preview/Program, TAKE/CUT); archive/grabber/crosspost (S3-mandated, encrypted, content-deduped).
|
||
|
||
## 10. Key file map
|
||
|
||
| Concern | Path |
|
||
|---|---|
|
||
| Tenant context / pools | `apps/api/src/db/client.ts` |
|
||
| Migrations 001–006 | `apps/api/migrations/` |
|
||
| Vault / storage crypto | `apps/api/src/vault/vault.ts`, `apps/api/src/storage/storage.ts` |
|
||
| Sessions / OIDC | `apps/api/src/auth/` |
|
||
| RBAC guard | `apps/api/src/rbac/rbac.guard.ts` |
|
||
| Audit chain | `migrations/001_tenants.sql` (SQL) + `apps/api/src/audit/` |
|
||
| Publishing | `apps/api/src/publishing/`, `apps/scheduler/src/` |
|
||
| Connector SDK | `packages/connector-sdk/src/` |
|
||
| Frontend mock-registry | `apps/web/src/lib/` |
|
||
| Lint: no-provider-branching | `eslint.config.mjs` |
|