Tactical Media Broadcast Hub — initial release
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
This commit is contained in:
commit
aa5e262f3a
880 changed files with 111042 additions and 0 deletions
131
packages/egress/README.md
Normal file
131
packages/egress/README.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# @tactical-media-broadcast-hub/egress
|
||||
|
||||
Shared HTTP(S) egress policy and PostgreSQL host budgets for WP7 callers. This slice
|
||||
only adds the package; API and media-worker endpoint validators are migrated later.
|
||||
|
||||
```ts
|
||||
import { Pool } from "pg";
|
||||
import { createEgressClient, EgressError } from "@tactical-media-broadcast-hub/egress";
|
||||
|
||||
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
|
||||
const { egressFetch, policy } = createEgressClient({ pool });
|
||||
const response = await egressFetch("https://instance.example/api/v1/videos", {}, {
|
||||
hostClass: "monitor",
|
||||
pinnedOrigin: true,
|
||||
});
|
||||
```
|
||||
|
||||
The caller owns/closes the pool. `egressFetch(url, init, {pool, hostClass, ...config})`
|
||||
is also exported for one-off calls. `createEgressClient` reuses configuration and
|
||||
returns `{egressFetch, policy, budgets}`. No global pool, application migration or
|
||||
in-process budget is created. Requests use Node HTTP(S), bypass proxy environment
|
||||
variables, and open a fresh connection for each hop. There is no hidden retry.
|
||||
|
||||
## Policy and requests
|
||||
|
||||
`validateUrl(url, options)` and `policy.validateUrl(url)` check syntax, literals and
|
||||
ports without DNS or database access. `await policy.validate(url)` returns
|
||||
`{url, pinnedAddress, family, hostKey}` after resolving once, rejecting every
|
||||
non-public answer (including mixed public/private answers), and choosing one
|
||||
address. The request lookup returns only this IP. The original hostname remains
|
||||
in Host and TLS certificate verification/SNI. A failed address is not replaced
|
||||
by an unvalidated address.
|
||||
|
||||
HTTP port 80 and HTTPS port 443 are allowed by default. Operator configuration can
|
||||
set per-scheme allowlists, e.g. `ports: {"https:": [443, 8443]}`. Credentials in
|
||||
URLs, localhost names, whitespace/control characters and backslashes are rejected.
|
||||
Query strings are supported; fragments are stripped. Both local blocklists are
|
||||
covered, including metadata/link-local, CGNAT, reserved/documentation/benchmark,
|
||||
multicast, IPv4-mapped IPv6 and IPv6 transition ranges. Legacy IPv4-compatible,
|
||||
site-local and local-use NAT64 ranges are also denied.
|
||||
|
||||
301/302/303/307/308 redirects get fresh validation, DNS pins and budget consumption
|
||||
per hop. Cycles and more than five redirects fail. Fetch `redirect: "manual"`
|
||||
returns the redirect without following; `"error"` rejects it. Cross-origin
|
||||
redirects strip Authorization, Proxy-Authorization and cookies; redirects that
|
||||
would replay a body across origins are rejected. Use `pinnedOrigin: true` for
|
||||
secret-bearing requests and discovery/JWKS fetches: every off-origin redirect is
|
||||
rejected before DNS or connection, including a different scheme/port. A URL value
|
||||
for `pinnedOrigin` also checks the initial request against that origin.
|
||||
|
||||
Responses are buffered into a standard `Response` with final `url` and
|
||||
`redirected`. Defaults: 15-second total deadline (DNS, DB, redirects and body),
|
||||
8 MiB request/response limits, five redirects. Configure `timeoutMs`,
|
||||
`maxRequestBytes`, `maxResponseBytes`, `maxRedirects`; caller abort signals are
|
||||
honored. The transport requests identity encoding and rejects upstream compressed
|
||||
bodies rather than returning incorrectly decoded bytes. It does not maintain a
|
||||
cookie jar. No application-specific authorization is inferred from `hostClass`.
|
||||
|
||||
`EgressError.code` distinguishes `DENIED_PRIVATE_ADDRESS`, `SCHEME_DENIED`,
|
||||
`PORT_DENIED`, `DNS_FAILED`, `BUDGET_EXHAUSTED`, `REDIRECT_LOOP`, `ORIGIN_DENIED`,
|
||||
`TIMEOUT`, `ABORTED`, size limits and invalid input/configuration. Budget errors
|
||||
include `retryAfterMs`; messages omit URLs and upstream exception strings.
|
||||
|
||||
## Migration and shared buckets
|
||||
|
||||
The application migration owner runs `migrations/001_host_budgets.sql` inside a
|
||||
transaction, recording **`@tactical-media-broadcast-hub/egress/001_host_budgets.sql`** in the
|
||||
existing `schema_migrations(name, applied_at)` ledger in the same transaction.
|
||||
Use a namespaced ledger key to avoid colliding with the API's `001_tenants.sql`.
|
||||
The SQL is exported via `@tactical-media-broadcast-hub/egress/migrations/001_host_budgets.sql`.
|
||||
It neither creates nor writes a separate migration ledger. Grant the runtime
|
||||
pool role SELECT/INSERT/UPDATE on `host_budgets` as part of application deployment.
|
||||
This is infrastructure state, so no workspace column or RLS policy is installed.
|
||||
|
||||
Keys are IDNA/punycode lowercase `host:effective-port` (IPv6 bracketed); trailing
|
||||
DNS dots are stripped. All callers, replicas and host classes draw from the same
|
||||
row for a key. Default capacity is 3, continuous refill is 10 tokens per fixed
|
||||
10-second interval. `hostClasses` configures initial capacity/refill for new hosts,
|
||||
e.g. `{monitor: {capacity: 3, refillPerInterval: 10}}`. The first initializer wins;
|
||||
later callers cannot enlarge or reset an existing bucket. Administrative changes
|
||||
to existing rows must be applied explicitly in the database.
|
||||
|
||||
Initialization, row locking, refill and conditional UPDATE/debit occur in one
|
||||
transaction. Time is sampled from PostgreSQL after acquiring the lock, so worker
|
||||
clock skew and lock waits cannot overfill the bucket. The optional `now` callback
|
||||
is a test clock; omit it in production.
|
||||
|
||||
429, Retry-After (seconds or HTTP date), and X-RateLimit-Remaining=0 park tokens at
|
||||
zero using a future `updated_at`. X-RateLimit-Reset accepts Unix seconds,
|
||||
milliseconds, or small delta-second values. Missing/invalid 429 hints default to
|
||||
ten seconds. Parking deadlines never shorten due to concurrent responses; refill
|
||||
starts at the deadline and the next token becomes available after its refill
|
||||
time. Remaining/Limit hints only tighten tokens/capacity/rate; absent a longer
|
||||
reset window, Limit is conservatively treated as tokens per ten seconds. Hints
|
||||
never increase local limits. Parking is applied at response headers, even if the
|
||||
body later fails its size limit. Callers decide when to retry budget errors.
|
||||
|
||||
## HLS
|
||||
|
||||
`await validateMasterPlaylist(text, playlistUrl, policy)` requires a master
|
||||
playlist and returns validated/pinned child endpoints, including variant lines,
|
||||
renditions, iframe variants, keys/maps and other URI-bearing attributes.
|
||||
`await rewriteSegmentUris(text, playlistUrl, policy)` accepts master/media
|
||||
playlists, validates their base and every child, and returns absolute URLs.
|
||||
Relative, root-relative, protocol-relative and absolute references are supported.
|
||||
Quoted commas, other attribute values, comments and LF/CRLF style are preserved.
|
||||
Content-steering SERVER-URI is validated too. Malformed/unquoted/duplicate URI
|
||||
attributes and unexpanded HLS variables are rejected.
|
||||
|
||||
Rewritten URLs must still be fetched through `egressFetch`, which validates and
|
||||
pins at connection time. These helpers do not fetch playlists or hand remote
|
||||
URLs to media processes. Relay-local IDs and RTMP/SRT mediation belong to later
|
||||
integration work.
|
||||
|
||||
## Verification
|
||||
|
||||
From this package, with Node 22+ and pnpm available:
|
||||
|
||||
```sh
|
||||
pnpm install --filter @tactical-media-broadcast-hub/egress --lockfile=false --ignore-scripts
|
||||
pnpm exec tsc --noEmit
|
||||
pnpm exec vitest run
|
||||
```
|
||||
|
||||
Database tests use a unique temporary schema and drop it afterward. Set
|
||||
`EGRESS_TEST_DATABASE_URL` to a PostgreSQL 15+ database whose test role can create
|
||||
schemas; the default is the repository Compose development database at
|
||||
`postgres://superposter:superposter_dev@127.0.0.1:5433/superposter`.
|
||||
No production tables or application migrations are touched. Tests cover the
|
||||
address matrix, DNS pinning/rebinding, redirect policy, PostgreSQL concurrency,
|
||||
fake-clock refill, Retry-After/rate-limit parking, request caps and HLS rewriting.
|
||||
Loading…
Add table
Add a link
Reference in a new issue