T-HUB/packages/egress/README.md

@tactical-media-broadcast-hub/egress

Shared HTTP(S) egress policy and PostgreSQL host budgets for WP7 callers. This slice only adds the package; API and media-worker endpoint validators are migrated later.

import { Pool } from "pg";
import { createEgressClient, EgressError } from "@tactical-media-broadcast-hub/egress";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const { egressFetch, policy } = createEgressClient({ pool });
const response = await egressFetch("https://instance.example/api/v1/videos", {}, {
  hostClass: "monitor",
  pinnedOrigin: true,
});

The caller owns/closes the pool. egressFetch(url, init, {pool, hostClass, ...config}) is also exported for one-off calls. createEgressClient reuses configuration and returns {egressFetch, policy, budgets}. No global pool, application migration or in-process budget is created. Requests use Node HTTP(S), bypass proxy environment variables, and open a fresh connection for each hop. There is no hidden retry.

Policy and requests

validateUrl(url, options) and policy.validateUrl(url) check syntax, literals and ports without DNS or database access. await policy.validate(url) returns {url, pinnedAddress, family, hostKey} after resolving once, rejecting every non-public answer (including mixed public/private answers), and choosing one address. The request lookup returns only this IP. The original hostname remains in Host and TLS certificate verification/SNI. A failed address is not replaced by an unvalidated address.

HTTP port 80 and HTTPS port 443 are allowed by default. Operator configuration can set per-scheme allowlists, e.g. ports: {"https:": [443, 8443]}. Credentials in URLs, localhost names, whitespace/control characters and backslashes are rejected. Query strings are supported; fragments are stripped. Both local blocklists are covered, including metadata/link-local, CGNAT, reserved/documentation/benchmark, multicast, IPv4-mapped IPv6 and IPv6 transition ranges. Legacy IPv4-compatible, site-local and local-use NAT64 ranges are also denied.

301/302/303/307/308 redirects get fresh validation, DNS pins and budget consumption per hop. Cycles and more than five redirects fail. Fetch redirect: "manual" returns the redirect without following; "error" rejects it. Cross-origin redirects strip Authorization, Proxy-Authorization and cookies; redirects that would replay a body across origins are rejected. Use pinnedOrigin: true for secret-bearing requests and discovery/JWKS fetches: every off-origin redirect is rejected before DNS or connection, including a different scheme/port. A URL value for pinnedOrigin also checks the initial request against that origin.

Responses are buffered into a standard Response with final url and redirected. Defaults: 15-second total deadline (DNS, DB, redirects and body), 8 MiB request/response limits, five redirects. Configure timeoutMs, maxRequestBytes, maxResponseBytes, maxRedirects; caller abort signals are honored. The transport requests identity encoding and rejects upstream compressed bodies rather than returning incorrectly decoded bytes. It does not maintain a cookie jar. No application-specific authorization is inferred from hostClass.

EgressError.code distinguishes DENIED_PRIVATE_ADDRESS, SCHEME_DENIED, PORT_DENIED, DNS_FAILED, BUDGET_EXHAUSTED, REDIRECT_LOOP, ORIGIN_DENIED, TIMEOUT, ABORTED, size limits and invalid input/configuration. Budget errors include retryAfterMs; messages omit URLs and upstream exception strings.

Migration and shared buckets

The application migration owner runs migrations/001_host_budgets.sql inside a transaction, recording @tactical-media-broadcast-hub/egress/001_host_budgets.sql in the existing schema_migrations(name, applied_at) ledger in the same transaction. Use a namespaced ledger key to avoid colliding with the API's 001_tenants.sql. The SQL is exported via @tactical-media-broadcast-hub/egress/migrations/001_host_budgets.sql. It neither creates nor writes a separate migration ledger. Grant the runtime pool role SELECT/INSERT/UPDATE on host_budgets as part of application deployment. This is infrastructure state, so no workspace column or RLS policy is installed.

Keys are IDNA/punycode lowercase host:effective-port (IPv6 bracketed); trailing DNS dots are stripped. All callers, replicas and host classes draw from the same row for a key. Default capacity is 3, continuous refill is 10 tokens per fixed 10-second interval. hostClasses configures initial capacity/refill for new hosts, e.g. {monitor: {capacity: 3, refillPerInterval: 10}}. The first initializer wins; later callers cannot enlarge or reset an existing bucket. Administrative changes to existing rows must be applied explicitly in the database.

Initialization, row locking, refill and conditional UPDATE/debit occur in one transaction. Time is sampled from PostgreSQL after acquiring the lock, so worker clock skew and lock waits cannot overfill the bucket. The optional now callback is a test clock; omit it in production.

429, Retry-After (seconds or HTTP date), and X-RateLimit-Remaining=0 park tokens at zero using a future updated_at. X-RateLimit-Reset accepts Unix seconds, milliseconds, or small delta-second values. Missing/invalid 429 hints default to ten seconds. Parking deadlines never shorten due to concurrent responses; refill starts at the deadline and the next token becomes available after its refill time. Remaining/Limit hints only tighten tokens/capacity/rate; absent a longer reset window, Limit is conservatively treated as tokens per ten seconds. Hints never increase local limits. Parking is applied at response headers, even if the body later fails its size limit. Callers decide when to retry budget errors.

HLS

await validateMasterPlaylist(text, playlistUrl, policy) requires a master playlist and returns validated/pinned child endpoints, including variant lines, renditions, iframe variants, keys/maps and other URI-bearing attributes. await rewriteSegmentUris(text, playlistUrl, policy) accepts master/media playlists, validates their base and every child, and returns absolute URLs. Relative, root-relative, protocol-relative and absolute references are supported. Quoted commas, other attribute values, comments and LF/CRLF style are preserved. Content-steering SERVER-URI is validated too. Malformed/unquoted/duplicate URI attributes and unexpanded HLS variables are rejected.

Rewritten URLs must still be fetched through egressFetch, which validates and pins at connection time. These helpers do not fetch playlists or hand remote URLs to media processes. Relay-local IDs and RTMP/SRT mediation belong to later integration work.

Verification

From this package, with Node 22+ and pnpm available:

pnpm install --filter @tactical-media-broadcast-hub/egress --lockfile=false --ignore-scripts
pnpm exec tsc --noEmit
pnpm exec vitest run

Database tests use a unique temporary schema and drop it afterward. Set EGRESS_TEST_DATABASE_URL to a PostgreSQL 15+ database whose test role can create schemas; the default is the repository Compose development database at postgres://superposter:superposter_dev@127.0.0.1:5433/superposter. No production tables or application migrations are touched. Tests cover the address matrix, DNS pinning/rebinding, redirect policy, PostgreSQL concurrency, fake-clock refill, Retry-After/rate-limit parking, request caps and HLS rewriting.