glass beadian
ced5e5da63
The engineering guide still called the control room 'planned' while it ships: multiview grid (2x1-4x4/custom, 4 live HLS tiles, tap-to-focus, adaptive preview polling), broadcast switcher (PST/TAKE/CUT semantics, lock, override modes), per-slot + master audio, overlay editor (lower_third/chyron/logo/clock, layers, rendered revisions), signed previews and the transactional director behind it. User manual gets the operator walkthrough (§7.1); engineering guide documents the media- control architecture and corrects the Level-2 status section; admin guide gets permissions, PUBLIC_ORIGIN, capacity and tally notes.
183 lines
12 KiB
Markdown
183 lines
12 KiB
Markdown
# Tactical Media Broadcast Hub — Administrator Guide
|
||
|
||
For operators running Tactical Media Broadcast Hub for one or more workspaces. Engineering detail lives in [engineering-guide.md](./engineering-guide.md); end-user help in [user-guide.md](./user-guide.md).
|
||
|
||
---
|
||
|
||
## 1. Quick start (development)
|
||
|
||
```bash
|
||
pnpm install
|
||
docker compose -f infra/docker-compose.yml up -d postgres redis temporal
|
||
# dev extras: --profile dev (adds media-worker stub, mock live source, mock RTMP sink)
|
||
|
||
pnpm --filter @tactical-media-broadcast-hub/api dev # API on :3000
|
||
pnpm --filter @tactical-media-broadcast-hub/web dev -- -p 3001 # UI on :3001
|
||
pnpm --filter @tactical-media-broadcast-hub/scheduler dev # Temporal worker
|
||
|
||
curl localhost:3000/healthz # {"status":"alive"}
|
||
curl localhost:3000/readyz # per-component: postgres/redis/temporal
|
||
```
|
||
|
||
Default dev ports: API **3000** · Web **3001** · media-worker **3100** · Postgres **5433** (host 5432 was already taken) · Redis 6379 · Temporal 7233 · MinIO 9000 (console 9001) · Prometheus 9090 · Grafana **3001→3000**.
|
||
|
||
> Docker image pulls can fail when the registry route is degraded (observed: minio/minio denied). Storage/S3 integration tests skip gracefully without an endpoint; everything else runs.
|
||
|
||
## 2. Environment variables
|
||
|
||
### API (`apps/api`)
|
||
|
||
| Variable | Required | Purpose |
|
||
|---|---|---|
|
||
| `DATABASE_URL` | prod | **App pool** — connect as `sp_app` (RLS-bound). Never the owner. |
|
||
| `SUPER_DATABASE_URL` | prod | **Owner pool** — migrations + provisioning only. |
|
||
| `SESSION_SECRET` | prod | HMAC key for session tokens. |
|
||
| `SUPERPOSTER_KEK` | prod | Vault master KEK, 32 bytes base64. Wraps per-workspace DEKs. |
|
||
| `S3_KEK` | prod | Media-encryption KEK, 32 bytes base64. Seals every S3 object. |
|
||
| `S3_ENDPOINT` / `S3_REGION` | prod | S3-compatible endpoint (MinIO, AWS, Spaces…). |
|
||
| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | prod | Storage credentials. |
|
||
| `S3_BUCKET` | no | Default `tactical-media-broadcast-hub-media`. |
|
||
| `S3_FORCE_PATH_STYLE` | no | `true` (default) for MinIO; `false` for AWS virtual-host style. |
|
||
| `OIDC_ISSUER` / `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` / `OIDC_REDIRECT_URI` | prod | Any compliant OIDC IdP (Authentik, Keycloak). |
|
||
| `TEMPORAL_ADDRESS` / `TEMPORAL_NAMESPACE` | no | Default `localhost:7233` / `default`. |
|
||
| `POSTGRES_HOST/PORT`, `REDIS_HOST/PORT` | no | Readiness probes. Default localhost, pg port **5433**. |
|
||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | no | Tracing to the collector; absent = disabled. |
|
||
| `LOG_LEVEL` | no | pino level, default `info`. |
|
||
|
||
In development, missing KEKs derive stable local fallbacks; **in production the API refuses to start without both KEKs and `SESSION_SECRET`** — encryption cannot be silently skipped.
|
||
|
||
## 3. Key management (critical)
|
||
|
||
Generate each key once, store in your secret manager, and back them up **separately from the database**:
|
||
|
||
```bash
|
||
openssl rand -base64 32 # → SUPERPOSTER_KEK (credential vault)
|
||
openssl rand -base64 32 # → S3_KEK (media encryption)
|
||
openssl rand -base64 32 # → SESSION_SECRET
|
||
```
|
||
|
||
- **Lose `SUPERPOSTER_KEK`** → all stored provider tokens are unrecoverable; users must reconnect accounts.
|
||
- **Lose `S3_KEK`** → the entire media archive is unrecoverable even though the S3 bytes remain.
|
||
- **Rotation:** `SUPERPOSTER_KEK` rotation re-wraps DEKs via the vault's `rotate()`. `S3_KEK` rotation re-wraps per-object CEKs without re-encrypting bodies (roadmap item; see engineering guide §3.8).
|
||
- Both KEKs are distinct from database credentials; changing DB passwords never affects them.
|
||
|
||
## 4. Database
|
||
|
||
### 4.1 Roles
|
||
- `tmbh` (docker `POSTGRES_USER`) — superuser, **owner pool only** (migrations via `SUPER_DATABASE_URL`).
|
||
- `sp_app` — created by migration `003_app_role.sql`; `NOSUPERUSER`, grants on all tables/sequences. **All runtime connections.** `FORCE ROW LEVEL SECURITY` means even the table owner cannot silently bypass tenancy.
|
||
|
||
### 4.2 Migrations
|
||
Ordered SQL files in `apps/api/migrations/` — currently **001–034** (001 tenants + RLS + audit chain · 002 RBAC seed · 003 sp_app role · 004 NULLIF policy fix · 005 epoch trigger fix · 006 publishing · 019 publishing integration · 024 chatgov · 029 group files · 031 personal workspaces · 033 publish governance · 034 auth sessions + OIDC refresh tokens). Idempotent runner (`apps/api/src/db/migrate.ts`) tracks `schema_migrations`:
|
||
|
||
```bash
|
||
cd apps/api && SUPER_DATABASE_URL='postgresql://tmbh:…@localhost:5433/tmbh' \
|
||
pnpm exec tsx -e 'import {runMigrations} from "./src/db/migrate.js"; runMigrations("./migrations").then(m => console.log("applied:", m))'
|
||
```
|
||
Integration tests apply migrations automatically when `SUPER_DATABASE_URL` is set.
|
||
|
||
### 4.3 Backups
|
||
- Nightly `pg_dump` of the `tmbh` DB.
|
||
- S3 bucket lifecycle/replication per your provider (objects are already encrypted client-side; provider snapshots never see plaintext).
|
||
- **KEKs** (§3) — without them backups are unrecoverable.
|
||
|
||
## 5. Rate & capacity settings (per connection)
|
||
|
||
Post/rate caps are **settings, never hardcoded**. Three layers, highest wins:
|
||
|
||
1. **Per-connection override** (workspace settings) — e.g. a news account with elevated API tier declares `dailyPostCap: 100`.
|
||
2. **Deployment env** — `CONNECTOR_RATE_<PROVIDER>_RPS`, `CONNECTOR_RATE_<PROVIDER>_DAILY_POSTS`, `CONNECTOR_RATE_<PROVIDER>_COMMENTS_PER_HOUR`, `CONNECTOR_RATE_<PROVIDER>_TIER_NOTE` (provider upper-cased, dashes→underscores: `CONNECTOR_RATE_INSTAGRAM_DAILY_POSTS`).
|
||
3. **Documented provider defaults** (standard-access caps: Instagram 25 posts/24h, Threads 250/24h — Meta enforces these server-side).
|
||
|
||
Truth model: raising a value **does not bypass the provider** — it declares what your tier actually permits (news-publisher/advanced-access programs raise real limits) so the scheduler paces within true capacity. The UI warns (`exceedsDocumentedCap`) when an override exceeds the documented cap: the provider will reject the excess unless your tier genuinely allows it.
|
||
|
||
## 6. Workspaces, roles & permissions
|
||
|
||
37 permissions across 10 seeded roles (Owner, Admin, Publisher, Editor, Approver, Analyst, Moderator, VideoPublisher, StreamOperator, BroadcastOperator). Role changes bump the workspace `authz_epoch`; affected live sessions fail closed on their next request. Audit every membership change (hash-chained; `audit_verify_chain()` detects tampering).
|
||
|
||
## 6.1 Publish governance (who signs off a post)
|
||
|
||
Each social connection carries a publish-governance setting. Manage per connection (requires `chatgov.manage`):
|
||
|
||
```bash
|
||
curl -X PATCH https://api.occupymedia.cc/connections/$CONN/publish-governance \
|
||
-H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
|
||
-d '{"publishGovernance": "admin1"}' # auto | admin1 | admin_majority | admin_consensus | null
|
||
```
|
||
|
||
| Mode | Who signs off the publish |
|
||
|---|---|
|
||
| `null` (unset) | manual: anyone with `publication.publish` |
|
||
| `auto` | nobody — dispatch fires when the draft's approval lands |
|
||
| `admin1` | one admin, via a publish poll |
|
||
| `admin_majority` | majority of admins (snapshot at poll open) |
|
||
| `admin_consensus` | every admin; one veto blocks at once |
|
||
|
||
While held, the publish/boost route answers `202 awaiting_publish_approval` with the poll id; the poll closes after a day and rejects unanswered. Reconnecting an account keeps the setting. Changing the setting does not cancel an already-open poll. The Mastodon **boost** route runs behind the same gate.
|
||
|
||
## 6.2 Control Room & media workers
|
||
|
||
The Live Control Room drives real encoders — gate it carefully. Permissions: `media.view`/`director.view` (watch), `director.take` (TAKE/CUT/PST), `director.lock_program`, `director.override_auto`, `director.change_layout` (overlays), `media.manage` (desired spec).
|
||
|
||
Operational notes:
|
||
|
||
- A channel must be **assigned to a media worker** before any switch succeeds ("channel is not assigned to a media worker"). Configure ingest sources and scenes first; select the preview bus.
|
||
- Capacity admission is part of every command transaction: an over-quota start is queued (`PENDING_CAPACITY`) and a stop offers its freed capacity to that queue.
|
||
- Preview URLs are signed capabilities on the API origin — set `PUBLIC_ORIGIN` correctly or same-origin checks reject the browser's preview requests.
|
||
- The worker reports the **realized version**; the UI's tally says "Tally unknown" until the worker confirms. If panels show thumbnails but never go live, check native HLS support and that preview signatures aren't being stripped by a proxy.
|
||
- Every switch is idempotent (`Idempotency-Key`) and lands in the audit chain as `director.<operation>` — incident review replays commands from the audit log.
|
||
|
||
## 6.3 Files SSO (Hub as the OIDC issuer for oCIS)
|
||
`files.occupymedia.cc` signs in through the Hub: the issuer is `https://api.occupymedia.cc/oidc`, client id `web`, PKCE only, auto-consent for that first-party client. Phase 2 (migration 034) makes sessions revocable and adds rotating refresh tokens bound to the Hub session — Hub logout reaches Drive within one access-token TTL (≤1 h). Operational notes:
|
||
|
||
- `OIDC_KEY_DIR` must be a persistent volume (the API generates and seals its ES256 keyset there on first boot; losing it logs every Drive session out).
|
||
- `FILES_OCID_USER` and `OIDC_RESERVED_USERNAMES` are never claimable as files accounts (`admin` is reserved by default).
|
||
- Sign-in requires an **active membership** and a username usable as an account name; otherwise the consent endpoint fails closed.
|
||
|
||
## 6.4 Verification harnesses
|
||
|
||
```bash
|
||
RADAR_EMAIL=… RADAR_PASSWORD=… node scripts/e2e/governance-sim.mjs # S1–S9 governance scenarios, live
|
||
node scripts/e2e/brand-palette-check.mjs # fails on blue-family pixels in embedded surfaces
|
||
```
|
||
|
||
The governance sim is re-runnable (all fixtures prefixed `e2e-gov-`) and writes a report to `/tmp/opencode/workers/GOV-SIM-REPORT.md` by default.
|
||
|
||
## 7. Monitoring & health
|
||
|
||
- `GET /healthz` — liveness (no dependencies).
|
||
- `GET /readyz` — readiness: TCP probes for postgres/redis/temporal, `503` + component detail when degraded.
|
||
- Grafana `http://localhost:3001` (anonymous admin in dev) with Prometheus datasource pre-provisioned; scrape targets for API and media-worker are in `infra/prometheus/prometheus.yml`.
|
||
- Logs: structured JSON; credential-shaped keys are redacted at the logger. Never log raw tokens — the scrubbing is defense-in-depth, not permission.
|
||
|
||
## 8. Security operations checklist
|
||
|
||
- [ ] Both KEKs generated, stored in secret manager, backed up off-box.
|
||
- [ ] `DATABASE_URL` uses `sp_app`, never the owner role.
|
||
- [ ] OIDC issuer uses HTTPS; redirect URI exact.
|
||
- [ ] Postgres not exposed beyond the host (compose binds 5433 locally).
|
||
- [ ] Audit-chain verification scheduled (a cron calling `audit_verify_chain` per workspace and alerting on `false`).
|
||
- [ ] Connector credentials rotate on schedule (vault `rotate()`).
|
||
- [ ] S3 bucket: no public access; presigned URLs are the only sharing path.
|
||
|
||
## 9. Troubleshooting
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---|---|---|
|
||
| API won't start, "S3_KEK is required" | production without KEKs | set both KEKs (§3) |
|
||
| `tenant context mismatch` errors | SECURITY DEFINER call without matching `withTenant` context | code bug — file an issue; do not grant sp_app more privileges |
|
||
| Sessions mass-invalidated after role change | `authz_epoch` bump (by design) | users re-login once |
|
||
| Scheduler idle, posts stuck `pending` | Temporal down when scheduled | worker sweeps pending rows on startup; verify `TEMPORAL_ADDRESS` |
|
||
| Postgres port conflict | host 5432 occupied | compose maps 5433; set `POSTGRES_PORT` accordingly |
|
||
| MinIO image pull denied | registry degradation | retry later or point `S3_ENDPOINT` at any S3 |
|
||
| Streams columns empty in UI | mock-registry data (API not wired yet) | expected pre-WP3-integration |
|
||
|
||
## 10. Upgrading
|
||
|
||
1. Back up DB + confirm KEK backups (§4.3).
|
||
2. `git pull && pnpm install`.
|
||
3. Run migrations (§4.2) — additive by design; expand/contract policy documented per migration. **Order matters**: 033 must exist before the scheduler that queries its columns rolls, or the E3 pre-dispatch check fails open.
|
||
4. Restart API, then scheduler.
|
||
5. Watch `/readyz` and the audit chain verification.
|
||
|
||
**Session note (migration 034+):** tokens issued before 034 carry no `sid`; they keep working until expiry but `reissue` refuses them — after upgrading, users re-login at their next epoch bump or within 12 h at the latest. Logout only revokes server-side for tokens issued after the upgrade.
|