hacklab/T-HUB
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
T-HUB/docs/admin-guide.md

15 KiB
Raw Permalink Blame History

Tactical Media Broadcast Hub — Administrator Guide

For operators running Tactical Media Broadcast Hub for one or more workspaces. Engineering detail lives in engineering-guide.md; end-user help in user-guide.md.

1. Quick start (development)

pnpm install
docker compose -f infra/docker-compose.yml up -d postgres redis temporal
# dev extras: --profile dev  (adds media-worker stub, mock live source, mock RTMP sink)

pnpm --filter @tactical-media-broadcast-hub/api dev          # API on :3000
pnpm --filter @tactical-media-broadcast-hub/web dev -- -p 3001   # UI on :3001
pnpm --filter @tactical-media-broadcast-hub/scheduler dev   # Temporal worker

curl localhost:3000/healthz                # {"status":"alive"}
curl localhost:3000/readyz                 # per-component: postgres/redis/temporal

Default dev ports: API 3000 · Web 3001 · media-worker 3100 · Postgres 5433 (host 5432 was already taken) · Redis 6379 · Temporal 7233 · MinIO 9000 (console 9001) · Prometheus 9090 · Grafana 3001→3000.

Docker image pulls can fail when the registry route is degraded (observed: minio/minio denied). Storage/S3 integration tests skip gracefully without an endpoint; everything else runs.

2. Environment variables

API (apps/api)

Variable Required Purpose
DATABASE_URL prod App pool — connect as sp_app (RLS-bound). Never the owner.
SUPER_DATABASE_URL prod Owner pool — migrations + provisioning only.
SESSION_SECRET prod HMAC key for session tokens.
SUPERPOSTER_KEK prod Vault master KEK, 32 bytes base64. Wraps per-workspace DEKs.
S3_KEK prod Media-encryption KEK, 32 bytes base64. Seals every S3 object.
S3_ENDPOINT / S3_REGION prod S3-compatible endpoint (MinIO, AWS, Spaces…).
S3_ACCESS_KEY / S3_SECRET_KEY prod Storage credentials.
S3_BUCKET no Default tactical-media-broadcast-hub-media.
S3_FORCE_PATH_STYLE no true (default) for MinIO; false for AWS virtual-host style.
OIDC_ISSUER / OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_REDIRECT_URI prod Any compliant OIDC IdP (Authentik, Keycloak).
TEMPORAL_ADDRESS / TEMPORAL_NAMESPACE no Default localhost:7233 / default.
POSTGRES_HOST/PORT, REDIS_HOST/PORT no Readiness probes. Default localhost, pg port 5433.
OTEL_EXPORTER_OTLP_ENDPOINT no Tracing to the collector; absent = disabled.
LOG_LEVEL no pino level, default info.

In development, missing KEKs derive stable local fallbacks; in production the API refuses to start without both KEKs and SESSION_SECRET — encryption cannot be silently skipped.

3. Key management (critical)

Generate each key once, store in your secret manager, and back them up separately from the database:

openssl rand -base64 32   # → SUPERPOSTER_KEK  (credential vault)
openssl rand -base64 32   # → S3_KEK           (media encryption)
openssl rand -base64 32   # → SESSION_SECRET
  • Lose SUPERPOSTER_KEK → all stored provider tokens are unrecoverable; users must reconnect accounts.
  • Lose S3_KEK → the entire media archive is unrecoverable even though the S3 bytes remain.
  • Rotation: SUPERPOSTER_KEK rotation re-wraps DEKs via the vault's rotate(). S3_KEK rotation re-wraps per-object CEKs without re-encrypting bodies (roadmap item; see engineering guide §3.8).
  • Both KEKs are distinct from database credentials; changing DB passwords never affects them.

4. Database

4.1 Roles

  • tmbh (docker POSTGRES_USER) — superuser, owner pool only (migrations via SUPER_DATABASE_URL).
  • sp_app — created by migration 003_app_role.sql; NOSUPERUSER, grants on all tables/sequences. All runtime connections. FORCE ROW LEVEL SECURITY means even the table owner cannot silently bypass tenancy.

4.2 Migrations

Ordered SQL files in apps/api/migrations/ — currently 001–034 (001 tenants + RLS + audit chain · 002 RBAC seed · 003 sp_app role · 004 NULLIF policy fix · 005 epoch trigger fix · 006 publishing · 019 publishing integration · 024 chatgov · 029 group files · 031 personal workspaces · 033 publish governance · 034 auth sessions + OIDC refresh tokens). Idempotent runner (apps/api/src/db/migrate.ts) tracks schema_migrations:

cd apps/api && SUPER_DATABASE_URL='postgresql://tmbh:…@localhost:5433/tmbh' \
  pnpm exec tsx -e 'import {runMigrations} from "./src/db/migrate.js"; runMigrations("./migrations").then(m => console.log("applied:", m))'

Integration tests apply migrations automatically when SUPER_DATABASE_URL is set.

4.3 Backups

  • Nightly pg_dump of the tmbh DB.
  • S3 bucket lifecycle/replication per your provider (objects are already encrypted client-side; provider snapshots never see plaintext).
  • KEKs (§3) — without them backups are unrecoverable.

5. Rate & capacity settings (per connection)

Post/rate caps are settings, never hardcoded. Three layers, highest wins:

  1. Per-connection override (workspace settings) — e.g. a news account with elevated API tier declares dailyPostCap: 100.
  2. Deployment env — CONNECTOR_RATE_<PROVIDER>_RPS, CONNECTOR_RATE_<PROVIDER>_DAILY_POSTS, CONNECTOR_RATE_<PROVIDER>_COMMENTS_PER_HOUR, CONNECTOR_RATE_<PROVIDER>_TIER_NOTE (provider upper-cased, dashes→underscores: CONNECTOR_RATE_INSTAGRAM_DAILY_POSTS).
  3. Documented provider defaults (standard-access caps: Instagram 25 posts/24h, Threads 250/24h — Meta enforces these server-side).

Truth model: raising a value does not bypass the provider — it declares what your tier actually permits (news-publisher/advanced-access programs raise real limits) so the scheduler paces within true capacity. The UI warns (exceedsDocumentedCap) when an override exceeds the documented cap: the provider will reject the excess unless your tier genuinely allows it.

6. Workspaces, roles & permissions

37 permissions across 10 seeded roles (Owner, Admin, Publisher, Editor, Approver, Analyst, Moderator, VideoPublisher, StreamOperator, BroadcastOperator). Role changes bump the workspace authz_epoch; affected live sessions fail closed on their next request. Audit every membership change (hash-chained; audit_verify_chain() detects tampering).

6.1 Publish governance (who signs off a post)

Each social connection carries a publish-governance setting. Manage per connection (requires chatgov.manage):

curl -X PATCH https://api.occupymedia.cc/connections/$CONN/publish-governance \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"publishGovernance": "admin1"}'     # auto | admin1 | admin_majority | admin_consensus | null
Mode Who signs off the publish
null (unset) manual: anyone with publication.publish
auto nobody — dispatch fires when the draft's approval lands
admin1 one admin, via a publish poll
admin_majority majority of admins (snapshot at poll open)
admin_consensus every admin; one veto blocks at once

While held, the publish/boost route answers 202 awaiting_publish_approval with the poll id; the poll closes after a day and rejects unanswered. Reconnecting an account keeps the setting. Changing the setting does not cancel an already-open poll. The Mastodon boost route runs behind the same gate.

6.2 Control Room & media workers

The Live Control Room drives real encoders — gate it carefully. Permissions: media.view/director.view (watch), director.take (TAKE/CUT/PST), director.lock_program, director.override_auto, director.change_layout (overlays), media.manage (desired spec).

Operational notes:

  • A channel must be assigned to a media worker before any switch succeeds ("channel is not assigned to a media worker"). Configure ingest sources and scenes first; select the preview bus.
  • Capacity admission is part of every command transaction: an over-quota start is queued (PENDING_CAPACITY) and a stop offers its freed capacity to that queue.
  • Preview URLs are signed capabilities on the API origin — set PUBLIC_ORIGIN correctly or same-origin checks reject the browser's preview requests.
  • The worker reports the realized version; the UI's tally says "Tally unknown" until the worker confirms. If panels show thumbnails but never go live, check native HLS support and that preview signatures aren't being stripped by a proxy.
  • Every switch is idempotent (Idempotency-Key) and lands in the audit chain as director.<operation> — incident review replays commands from the audit log.

6.3 Files SSO (Hub as the OIDC issuer for oCIS)

Retired (October 2026): oCIS is out of the payload path and the in-app Drive view is gone — see §10.1. This section remains only as reference for the legacy files.occupymedia.cc host. files.occupymedia.cc signs in through the Hub: the issuer is https://api.occupymedia.cc/oidc, client id web, PKCE only, auto-consent for that first-party client. Phase 2 (migration 034) makes sessions revocable and adds rotating refresh tokens bound to the Hub session — Hub logout reaches Drive within one access-token TTL (≤1 h). Operational notes:

  • OIDC_KEY_DIR must be a persistent volume (the API generates and seals its ES256 keyset there on first boot; losing it logs every Drive session out).
  • FILES_OCID_USER and OIDC_RESERVED_USERNAMES are never claimable as files accounts (admin is reserved by default).
  • Sign-in requires an active membership and a username usable as an account name; otherwise the consent endpoint fails closed.

6.4 Verification harnesses

RADAR_EMAIL=… RADAR_PASSWORD=… node scripts/e2e/governance-sim.mjs   # S1–S9 governance scenarios, live
node scripts/e2e/brand-palette-check.mjs                             # fails on blue-family pixels in embedded surfaces

The governance sim is re-runnable (all fixtures prefixed e2e-gov-) and writes a report to /tmp/opencode/workers/GOV-SIM-REPORT.md by default.

7. Monitoring & health

  • GET /healthz — liveness (no dependencies).
  • GET /readyz — readiness: TCP probes for postgres/redis/temporal, 503 + component detail when degraded.
  • Grafana http://localhost:3001 (anonymous admin in dev) with Prometheus datasource pre-provisioned; scrape targets for API and media-worker are in infra/prometheus/prometheus.yml.
  • Logs: structured JSON; credential-shaped keys are redacted at the logger. Never log raw tokens — the scrubbing is defense-in-depth, not permission.

8. Security operations checklist

  • Both KEKs generated, stored in secret manager, backed up off-box.
  • DATABASE_URL uses sp_app, never the owner role.
  • OIDC issuer uses HTTPS; redirect URI exact.
  • Postgres not exposed beyond the host (compose binds 5433 locally).
  • Audit-chain verification scheduled (a cron calling audit_verify_chain per workspace and alerting on false).
  • Connector credentials rotate on schedule (vault rotate()).
  • S3 bucket: no public access; presigned URLs are the only sharing path.

9. Troubleshooting

Symptom Cause Fix
API won't start, "S3_KEK is required" production without KEKs set both KEKs (§3)
tenant context mismatch errors SECURITY DEFINER call without matching withTenant context code bug — file an issue; do not grant sp_app more privileges
Sessions mass-invalidated after role change authz_epoch bump (by design) users re-login once
Scheduler idle, posts stuck pending Temporal down when scheduled worker sweeps pending rows on startup; verify TEMPORAL_ADDRESS
Postgres port conflict host 5432 occupied compose maps 5433; set POSTGRES_PORT accordingly
MinIO image pull denied registry degradation retry later or point S3_ENDPOINT at any S3
Streams columns empty in UI mock-registry data (API not wired yet) expected pre-WP3-integration

10. Operations changes (October 2026)

10.1 oCIS retired from the payload path

  • All media — Files tab, video, submissions, media posts, produced files — now stores in the R2 bucket (S3_* env, S3_KEK envelope encryption). oCIS holds no payloads.
  • files.occupymedia.cc / the Drive tab is gone from the web UI.
  • 192.168.13.210 is no longer in the serving path and may be decommissioned.

10.2 Payload pointer (migration 041)

  • chat_media_archive.storage_key is the authoritative payload pointer.
  • ocis_resource_id remains only as a legacy column — do not build against it.

10.3 Publishing media

  • The scheduler resolves media at dispatch (PublishingMediaResolver: RLS workspace lookup + R2 download + SPENC1 open).
  • MASTODON_PUBLISH_DEDUPE_ORIGINS=https://social.occupymedia.cc must stay set. Media posts are two provider mutations (media upload + status create) and the dispatch gate refuses the second without a verified dedupe origin.

10.4 Produce worker

  • Runs as a separate container (produce-worker) and needs ffmpeg + a font — DejaVuSans-Bold at /usr/share/fonts/truetype/dejavu/.
  • MEDIA_DEBUG_STDERR=1 routes ffmpeg stderr to /tmp/ffmpeg-debug.log for debugging.

10.5 Known ops note

  • Restarting the api container repeatedly can trip the homeserver login rate limit (M_LIMIT_EXCEEDED) and crash bootstrap — wait ~2–3 min between api restarts.
  • DIRECT_EC_PRIVATE_KEY (P-256 PKCS8 base64) is required by the api, media-worker and produce-worker (compose appenv) — it unwraps client-sealed SPENC2 frames. Rotation needs a re-seal migration (not yet designed).
  • The AI transcription sidecar (faster-whisper large-v3-turbo, OpenAI-compatible) runs on the moxexleyhp1-1 WSL GPU at http://100.70.240.7:9000 (systemd unit tmbh-transcribe); cuBLAS/cuDNN come from pip wheels (nvidia-cublas-cu12, nvidia-cudnn-cu12) + /usr/lib/wsl/lib.
  • LLM serving for agent workflows: LiteLLM at http://100.96.13.16:4000 (opencode3), key sk-hub1-… (hub1 virtual key), serving qwen3.8-27b.
  • Restarting the api container repeatedly can trip the homeserver login rate limit (M_LIMIT_EXCEEDED) and crash bootstrap — wait ~2–3 min between api restarts.

11. Upgrading

  1. Back up DB + confirm KEK backups (§4.3).
  2. git pull && pnpm install.
  3. Run migrations (§4.2) — additive by design; expand/contract policy documented per migration. Order matters: 033 must exist before the scheduler that queries its columns rolls, or the E3 pre-dispatch check fails open.
  4. Restart API, then scheduler.
  5. Watch /readyz and the audit chain verification.

Session note (migration 034+): tokens issued before 034 carry no sid; they keep working until expiry but reissue refuses them — after upgrading, users re-login at their next epoch bump or within 12 h at the latest. Logout only revokes server-side for tokens issued after the upgrade.