Tactical Media Broadcast Hub — initial release
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
This commit is contained in:
commit
aa5e262f3a
880 changed files with 111042 additions and 0 deletions
BIN
docs/2026-10-09-tweetdeck-on-steroids-call-for-help-ES.pdf
Normal file
BIN
docs/2026-10-09-tweetdeck-on-steroids-call-for-help-ES.pdf
Normal file
Binary file not shown.
94
docs/2026-10-09-tweetdeck-on-steroids-call-for-help.md
Normal file
94
docs/2026-10-09-tweetdeck-on-steroids-call-for-help.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# TweetDeck on Steroids — the full feature set
|
||||
|
||||
**Call for help — 2026-10-09.** We're looking for **testers** across the whole flow and **intake hackers** for the hard problem at the end of this document. Everything below in the "running today" sections is built and running — this is an invitation to use it, break it, and help finish the rest.
|
||||
|
||||
**A self-hosted, multi-platform broadcast newsroom for collectives: a column wall that watches every platform at once, chats that organize themselves, votes that turn into posts, and a live channel behind it all.**
|
||||
|
||||
TweetDeck died when X locked its doors. We built what should have replaced it — and went much further. Watch everything, decide together, publish everywhere, mix live, and keep an archive no platform can take away. Self-hosted. No algorithm in between. No platform holding your keys.
|
||||
|
||||
---
|
||||
|
||||
## The column wall — every platform in one wall
|
||||
|
||||
Live columns watch all your connected accounts at once: Mastodon, Telegram, Bluesky, PeerTube, Twitch, and more. The whole collective sees the same picture at the same moment.
|
||||
|
||||
Every post is rendered safe (hostile HTML is parsed into a typed format — scripts, tracking pixels and trick links are stripped at ingest and tested against an attack corpus), media is proxied through your server, and duplicates are deduplicated into the archive automatically.
|
||||
|
||||
Columns are becoming **editable**: a column can be a platform feed, a **keyword search across platforms** (one query like "acampada26" pulling from every network that allows it), a curated **list of accounts across platforms**, or a saved search. Columns persist per group, can be reordered, paused, and shared.
|
||||
|
||||
## Collaborative chats
|
||||
|
||||
Chats from every platform bridge into one shared overlay and one web chat — viewers, contributors and moderators in the same room, with flood governors and moderation tools keeping it usable during action.
|
||||
|
||||
The chat is a coordination surface, not just a mirror: it sits next to the live program, next to the file store, next to the drafts. What people say in the chat drives what happens in the system.
|
||||
|
||||
## Voting and decision making — in the chat, automatically
|
||||
|
||||
Run a poll inside any bridged chat: **"post this?"** — and set the rules for what counts as a pass. Configurable per poll:
|
||||
|
||||
- approved by **x of y admins**
|
||||
- approved by **x of y participants**
|
||||
- mixed quorum (admins + participants), veto rights, timeboxes, anonymous or named voting
|
||||
|
||||
Votes come in through Telegram buttons and the web overlay — live tally bars, quorum meters, countdowns. Ineligible votes are stored but never counted; vetoes fail the poll instantly; revotes are supported; everything lands in a tamper-evident audit trail.
|
||||
|
||||
**When the quorum passes, the posting bot fires automatically** — the approved content goes out to the channel, topic, or account the poll was about. No admin phone in the loop. Decisions happen at the speed of the chat, with the full trail preserved.
|
||||
|
||||
## Configurable posting bots — every account, its own rules
|
||||
|
||||
Every connected account carries its own settable rules for what it takes to post:
|
||||
|
||||
- **immediate** — no gate, post as today
|
||||
- **admin-only** — an admin's approval is the whole quorum
|
||||
- **quorum** — configurable thresholds of admins and/or participants
|
||||
- per **topic** and per **content type** overrides, dry-run shadow mode, timeboxes, reset-on-edit
|
||||
|
||||
A rules editor shows the resolved effective policy before you save it — and a blocked queue shows every post waiting on its quorum with a live gap meter. Nothing is ambiguous about why something did or didn't go out.
|
||||
|
||||
## Subchats and topics — structure that follows the work
|
||||
|
||||
Forum topics become **subchats**: working groups inside a channel, created and managed by the bot. Accounts are **assigned to topics** — the press crew on the press topic, the stream crew on the stream topic — and each topic carries its own posting rules and its own view of the store. Topics can be bound to broadcast channels, so a topic isn't just conversation: it's a control surface.
|
||||
|
||||
## Chat-linked files — the store organizes itself
|
||||
|
||||
Files posted into the chat land in the **group's file store automatically** — content-addressed, encrypted, deduplicated (the same clip shared five times is stored once).
|
||||
|
||||
Admins organize: assign items to folders, copy them across subfolder trees, refile by type or date — from **chat commands** (`/ls`, `/find`, `/move`) or from the **mobile file screen** that lives next to the chat, with deep links from bot messages straight into the filtered folder view.
|
||||
|
||||
Bot listings, search across the whole library, and an audit of every move: the file storage stays organized because the people in the chat keep it organized, from their phones.
|
||||
|
||||
## Cross-amplification and retweets — one approval, every platform
|
||||
|
||||
One approved draft fans out to every connected account, native to each platform. Where a platform has native quoting (Mastodon quote posts, Bluesky, Telegram forwards), it's used. Where it doesn't, the system generates a proper attributed quote — author, instance, link back — never a silent copy-paste.
|
||||
|
||||
Amplifying a sister collective's post keeps their names on it by default. Scheduling, per-account timing, and rate awareness come along for free.
|
||||
|
||||
## Live video mixing and channel production
|
||||
|
||||
The **Control Room** runs in the browser: a multiview grid of every feed, TAKE/CUT between preview and program, scene layouts (single, picture-in-picture, two-up, splits, grid), lower-thirds, scrolling chyrons, clocks and logos as overlays, per-source audio mixing with a tally of what's hot.
|
||||
|
||||
**Field streamers** push cameras from a phone straight to your server. The director mixes in the browser; the program goes to your **PeerTube channel**; and when no one is live, the channel keeps broadcasting a scheduled playlist so the station never goes dark.
|
||||
|
||||
**Auto-Director** switches sources by priority rules — a streamer goes down, the next source fades in, the fallback slate catches anything else.
|
||||
|
||||
**Distributed transcode workers:** laptops and GPUs join as render workers over Tailscale — an RTX 3090 laptop is already transcoding real PeerTube jobs with hardware encoding, verified end to end. More workers join the pool as load rises.
|
||||
|
||||
## The backbone
|
||||
|
||||
- **Encrypted vault** for every platform credential — the database never sees a plaintext key.
|
||||
- **Hash-chained audit log** — every editorial action tamper-evident, forever.
|
||||
- **DST-proof scheduling** — the occurrence engine survives clock changes; two scheduled posts that collapse onto the same instant fire exactly once.
|
||||
- **Media sandbox** — every FFmpeg job runs in a per-job jail: unprivileged, no network, resource-capped.
|
||||
- **An archive that can't be purged** — content-addressed, encrypted, deduplicated. Deleted threads stay citable; yesterday's footage is tomorrow's playlist.
|
||||
- **Teams without password-sharing** — roles per account, invite links, people can remove their own access when they move on.
|
||||
- **Self-hosted end to end** — no corporate cloud, no accounts needed to run the system itself.
|
||||
|
||||
---
|
||||
|
||||
## What we're asking for
|
||||
|
||||
**Testers.** Take the whole flow for a spin with your real feeds and your real accounts: post through the approval flow, run the column wall, mix a stream in the Control Room, vote in a bridged chat, break the scheduler across timezones, try to break the archive. Every finding makes the next collective's deployment safer.
|
||||
|
||||
**Intake hackers.** The hard problem is pulling content from the locked platforms — hashtags, accounts, search columns on X, Instagram, TikTok — without burning accounts or exposing the collective. We have a verified architecture for the sanctioned sources (RSS, fediverse, Bluesky firehose, Telegram, PeerTube, YouTube RSS) and per-URL fetchers for single public posts. What we want is your design input and build help on: bridge services, community-submitted link intake, archive harvesters, and honest assessments of what is and isn't worth the risk. If you've built scrapers, run intake pipelines, or know the platform-integrity arms race from the inside — this is the problem to bring your scars to.
|
||||
|
||||
**Stack if you want to hack:** Next.js + NestJS + Postgres (RLS) + Temporal + Redis + S3 + FFmpeg/PeerTube. Self-hosted end to end, multi-tenant, everything audited.
|
||||
BIN
docs/2026-10-09-tweetdeck-on-steroids-call-for-help.pdf
Normal file
BIN
docs/2026-10-09-tweetdeck-on-steroids-call-for-help.pdf
Normal file
Binary file not shown.
26
docs/2026-10-09-tweetdeck-on-steroids-pitch.md
Normal file
26
docs/2026-10-09-tweetdeck-on-steroids-pitch.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# TweetDeck on Steroids
|
||||
|
||||
**A self-hosted, multi-platform broadcast newsroom for collectives — a column wall that watches every platform at once, wired into a full live broadcast pipeline.**
|
||||
|
||||
TweetDeck died when X locked its doors. We built what should have replaced it: columns that watch Mastodon, Telegram, Bluesky, PeerTube, Twitch and more in one wall — wired into publishing, approvals, live mixing, and an archive that can't be purged. Self-hosted. No algorithm in between. No platform holding your keys.
|
||||
|
||||
## How people work together in it
|
||||
|
||||
- **The wall is the room.** Live columns watch all your connected accounts at once — every post rendered safe, media proxied, duplicates deduplicated into the archive. The whole collective sees the same picture.
|
||||
- **Draft once, approve together, publish everywhere.** Anyone can draft into any account. Every draft goes through configurable group approval before it moves. On sign-off it fans out natively per platform — proper quotes, attribution intact, tamper-evident audit of who decided what.
|
||||
- **Roles, not passwords.** Editors draft, reviewers approve, publishers publish — per account, per channel. Invite links bring people in; leaving removes access. No shared logins, no bottlenecks.
|
||||
- **Schedule like a station.** Timezone-correct scheduled posts with a DST-proof occurrence engine. Cancel, review, re-schedule — the calendar survives clock changes.
|
||||
- **Live control room in the browser.** Multiview grid, TAKE/CUT, preview/program, lower-thirds and chyrons, audio mixing — program out to your PeerTube channel, 24/7 playlist when you're dark. Works from a laptop.
|
||||
- **Auto-Director.** Priority rules switch sources automatically so the stream never dies mid-action.
|
||||
- **Chat from every platform** bridged into one overlay — viewers heard, moderators acting, the streamer focused on the shot.
|
||||
- **Your own transcode workers.** Laptops and GPUs join as render workers over Tailscale — an RTX 3090 laptop is already transcoding real PeerTube jobs. More workers, more channels, all self-hosted.
|
||||
- **An archive that can't be purged.** Content-addressed, encrypted, deduplicated. Platforms purge; the archive remembers. Deleted threads stay citable.
|
||||
- **Analytics.** Month-by-month snapshots: what published where, what failed, what got archived.
|
||||
|
||||
## What we need: testers and scrapers
|
||||
|
||||
We're looking for **testers** across the flow — post through approvals, run a column wall against your real feeds, mix a stream, break the scheduler across timezones. And we're looking for help on the **hard problem: intake at scale.** Watching RSS, the fediverse and Telegram is solved. What we want next: reliable ways to pull from the locked platforms — hashtags, accounts, search columns — without burning accounts. If you've built scrapers, run intake pipelines, or know the platform-integrity arms race from the inside, help us design the intake layer: per-URL fetchers, community-submitted links, archive harvesters, bridge services.
|
||||
|
||||
## Stack
|
||||
|
||||
Next.js + NestJS + Postgres (RLS) + Temporal + Redis + S3 + FFmpeg/PeerTube. Self-hosted end to end, multi-tenant, everything audited.
|
||||
133
docs/admin-guide.md
Normal file
133
docs/admin-guide.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
# Tactical Media Broadcast Hub — Administrator Guide
|
||||
|
||||
For operators running Tactical Media Broadcast Hub for one or more workspaces. Engineering detail lives in [engineering-guide.md](./engineering-guide.md); end-user help in [user-guide.md](./user-guide.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Quick start (development)
|
||||
|
||||
```bash
|
||||
pnpm install
|
||||
docker compose -f infra/docker-compose.yml up -d postgres redis temporal
|
||||
# dev extras: --profile dev (adds media-worker stub, mock live source, mock RTMP sink)
|
||||
|
||||
pnpm --filter @tactical-media-broadcast-hub/api dev # API on :3000
|
||||
pnpm --filter @tactical-media-broadcast-hub/web dev -- -p 3001 # UI on :3001
|
||||
pnpm --filter @tactical-media-broadcast-hub/scheduler dev # Temporal worker
|
||||
|
||||
curl localhost:3000/healthz # {"status":"alive"}
|
||||
curl localhost:3000/readyz # per-component: postgres/redis/temporal
|
||||
```
|
||||
|
||||
Default dev ports: API **3000** · Web **3001** · media-worker **3100** · Postgres **5433** (host 5432 was already taken) · Redis 6379 · Temporal 7233 · MinIO 9000 (console 9001) · Prometheus 9090 · Grafana **3001→3000**.
|
||||
|
||||
> Docker image pulls can fail when the registry route is degraded (observed: minio/minio denied). Storage/S3 integration tests skip gracefully without an endpoint; everything else runs.
|
||||
|
||||
## 2. Environment variables
|
||||
|
||||
### API (`apps/api`)
|
||||
|
||||
| Variable | Required | Purpose |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | prod | **App pool** — connect as `sp_app` (RLS-bound). Never the owner. |
|
||||
| `SUPER_DATABASE_URL` | prod | **Owner pool** — migrations + provisioning only. |
|
||||
| `SESSION_SECRET` | prod | HMAC key for session tokens. |
|
||||
| `SUPERPOSTER_KEK` | prod | Vault master KEK, 32 bytes base64. Wraps per-workspace DEKs. |
|
||||
| `S3_KEK` | prod | Media-encryption KEK, 32 bytes base64. Seals every S3 object. |
|
||||
| `S3_ENDPOINT` / `S3_REGION` | prod | S3-compatible endpoint (MinIO, AWS, Spaces…). |
|
||||
| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | prod | Storage credentials. |
|
||||
| `S3_BUCKET` | no | Default `tactical-media-broadcast-hub-media`. |
|
||||
| `S3_FORCE_PATH_STYLE` | no | `true` (default) for MinIO; `false` for AWS virtual-host style. |
|
||||
| `OIDC_ISSUER` / `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` / `OIDC_REDIRECT_URI` | prod | Any compliant OIDC IdP (Authentik, Keycloak). |
|
||||
| `TEMPORAL_ADDRESS` / `TEMPORAL_NAMESPACE` | no | Default `localhost:7233` / `default`. |
|
||||
| `POSTGRES_HOST/PORT`, `REDIS_HOST/PORT` | no | Readiness probes. Default localhost, pg port **5433**. |
|
||||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | no | Tracing to the collector; absent = disabled. |
|
||||
| `LOG_LEVEL` | no | pino level, default `info`. |
|
||||
|
||||
In development, missing KEKs derive stable local fallbacks; **in production the API refuses to start without both KEKs and `SESSION_SECRET`** — encryption cannot be silently skipped.
|
||||
|
||||
## 3. Key management (critical)
|
||||
|
||||
Generate each key once, store in your secret manager, and back them up **separately from the database**:
|
||||
|
||||
```bash
|
||||
openssl rand -base64 32 # → SUPERPOSTER_KEK (credential vault)
|
||||
openssl rand -base64 32 # → S3_KEK (media encryption)
|
||||
openssl rand -base64 32 # → SESSION_SECRET
|
||||
```
|
||||
|
||||
- **Lose `SUPERPOSTER_KEK`** → all stored provider tokens are unrecoverable; users must reconnect accounts.
|
||||
- **Lose `S3_KEK`** → the entire media archive is unrecoverable even though the S3 bytes remain.
|
||||
- **Rotation:** `SUPERPOSTER_KEK` rotation re-wraps DEKs via the vault's `rotate()`. `S3_KEK` rotation re-wraps per-object CEKs without re-encrypting bodies (roadmap item; see engineering guide §3.8).
|
||||
- Both KEKs are distinct from database credentials; changing DB passwords never affects them.
|
||||
|
||||
## 4. Database
|
||||
|
||||
### 4.1 Roles
|
||||
- `tmbh` (docker `POSTGRES_USER`) — superuser, **owner pool only** (migrations via `SUPER_DATABASE_URL`).
|
||||
- `sp_app` — created by migration `003_app_role.sql`; `NOSUPERUSER`, grants on all tables/sequences. **All runtime connections.** `FORCE ROW LEVEL SECURITY` means even the table owner cannot silently bypass tenancy.
|
||||
|
||||
### 4.2 Migrations
|
||||
Ordered SQL files in `apps/api/migrations/` (001 tenants + RLS + audit chain · 002 RBAC seed · 003 sp_app role · 004 NULLIF policy fix · 005 epoch trigger fix · 006 publishing). Idempotent runner (`apps/api/src/db/migrate.ts`) tracks `schema_migrations`:
|
||||
|
||||
```bash
|
||||
cd apps/api && SUPER_DATABASE_URL='postgresql://tmbh:…@localhost:5433/tmbh' \
|
||||
pnpm exec tsx -e 'import {runMigrations} from "./src/db/migrate.js"; runMigrations("./migrations").then(m => console.log("applied:", m))'
|
||||
```
|
||||
Integration tests apply migrations automatically when `SUPER_DATABASE_URL` is set.
|
||||
|
||||
### 4.3 Backups
|
||||
- Nightly `pg_dump` of the `tmbh` DB.
|
||||
- S3 bucket lifecycle/replication per your provider (objects are already encrypted client-side; provider snapshots never see plaintext).
|
||||
- **KEKs** (§3) — without them backups are unrecoverable.
|
||||
|
||||
## 5. Rate & capacity settings (per connection)
|
||||
|
||||
Post/rate caps are **settings, never hardcoded**. Three layers, highest wins:
|
||||
|
||||
1. **Per-connection override** (workspace settings) — e.g. a news account with elevated API tier declares `dailyPostCap: 100`.
|
||||
2. **Deployment env** — `CONNECTOR_RATE_<PROVIDER>_RPS`, `CONNECTOR_RATE_<PROVIDER>_DAILY_POSTS`, `CONNECTOR_RATE_<PROVIDER>_COMMENTS_PER_HOUR`, `CONNECTOR_RATE_<PROVIDER>_TIER_NOTE` (provider upper-cased, dashes→underscores: `CONNECTOR_RATE_INSTAGRAM_DAILY_POSTS`).
|
||||
3. **Documented provider defaults** (standard-access caps: Instagram 25 posts/24h, Threads 250/24h — Meta enforces these server-side).
|
||||
|
||||
Truth model: raising a value **does not bypass the provider** — it declares what your tier actually permits (news-publisher/advanced-access programs raise real limits) so the scheduler paces within true capacity. The UI warns (`exceedsDocumentedCap`) when an override exceeds the documented cap: the provider will reject the excess unless your tier genuinely allows it.
|
||||
|
||||
## 6. Workspaces, roles & permissions
|
||||
|
||||
37 permissions across 10 seeded roles (Owner, Admin, Publisher, Editor, Approver, Analyst, Moderator, VideoPublisher, StreamOperator, BroadcastOperator). Role changes bump the workspace `authz_epoch`; affected live sessions fail closed on their next request. Audit every membership change (hash-chained; `audit_verify_chain()` detects tampering).
|
||||
|
||||
## 7. Monitoring & health
|
||||
|
||||
- `GET /healthz` — liveness (no dependencies).
|
||||
- `GET /readyz` — readiness: TCP probes for postgres/redis/temporal, `503` + component detail when degraded.
|
||||
- Grafana `http://localhost:3001` (anonymous admin in dev) with Prometheus datasource pre-provisioned; scrape targets for API and media-worker are in `infra/prometheus/prometheus.yml`.
|
||||
- Logs: structured JSON; credential-shaped keys are redacted at the logger. Never log raw tokens — the scrubbing is defense-in-depth, not permission.
|
||||
|
||||
## 8. Security operations checklist
|
||||
|
||||
- [ ] Both KEKs generated, stored in secret manager, backed up off-box.
|
||||
- [ ] `DATABASE_URL` uses `sp_app`, never the owner role.
|
||||
- [ ] OIDC issuer uses HTTPS; redirect URI exact.
|
||||
- [ ] Postgres not exposed beyond the host (compose binds 5433 locally).
|
||||
- [ ] Audit-chain verification scheduled (a cron calling `audit_verify_chain` per workspace and alerting on `false`).
|
||||
- [ ] Connector credentials rotate on schedule (vault `rotate()`).
|
||||
- [ ] S3 bucket: no public access; presigned URLs are the only sharing path.
|
||||
|
||||
## 9. Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| API won't start, "S3_KEK is required" | production without KEKs | set both KEKs (§3) |
|
||||
| `tenant context mismatch` errors | SECURITY DEFINER call without matching `withTenant` context | code bug — file an issue; do not grant sp_app more privileges |
|
||||
| Sessions mass-invalidated after role change | `authz_epoch` bump (by design) | users re-login once |
|
||||
| Scheduler idle, posts stuck `pending` | Temporal down when scheduled | worker sweeps pending rows on startup; verify `TEMPORAL_ADDRESS` |
|
||||
| Postgres port conflict | host 5432 occupied | compose maps 5433; set `POSTGRES_PORT` accordingly |
|
||||
| MinIO image pull denied | registry degradation | retry later or point `S3_ENDPOINT` at any S3 |
|
||||
| Streams columns empty in UI | mock-registry data (API not wired yet) | expected pre-WP3-integration |
|
||||
|
||||
## 10. Upgrading
|
||||
|
||||
1. Back up DB + confirm KEK backups (§4.3).
|
||||
2. `git pull && pnpm install`.
|
||||
3. Run migrations (§4.2) — additive by design; expand/contract policy documented per migration.
|
||||
4. Restart API, then scheduler.
|
||||
5. Watch `/readyz` and the audit chain verification.
|
||||
97
docs/asset-creation-guide.md
Normal file
97
docs/asset-creation-guide.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
# Tactical Media Broadcast Hub — Design Language & Asset Creation Guide
|
||||
|
||||
**Brand asset:** `apps/web/public/brand/logo.svg` — the anarchist circle-A with outward-pointing arrows
|
||||
**Palette:** Black (#000-#0a0a0b) + Warm Red-Orange (#FF1610 → oklch(0.62 0.22 30)) + semantic colors (green=live, amber=warn)
|
||||
**Direction:** Bold, defiant, activist-poster aesthetic. Black t-shirts with red screen-printing. High contrast, no apologies.
|
||||
|
||||
---
|
||||
|
||||
## The Design DNA
|
||||
|
||||
Your logo has five visual elements that define the entire brand:
|
||||
|
||||
| Element | What it communicates | How to reuse it |
|
||||
|---|---|---|
|
||||
| **Circle** | Community, enclosure, the "O" in anarchy | Use for avatars, badges, loading spinners |
|
||||
| **The A** | The core symbol — Tactical Media Broadcast Hub, anarchy, action | Use alone for favicons, watermarks, empty states |
|
||||
| **Arrows** (5x) | Outward direction — decentralization, spreading the message | Use as accent motifs on buttons, cards, headers |
|
||||
| **Black background** | The void we fill | Always the base |
|
||||
| **Red #FF1610** | Energy, urgency, defiance | The only accent color |
|
||||
|
||||
## Rules for creating new assets
|
||||
|
||||
### Rule 1: Two colors only (plus semantic)
|
||||
Every asset uses black + #FF1610 (or the warm accent oklch equivalent). Green (live), amber (warn), and deeper red (danger) are reserved for semantic status — never decoration.
|
||||
|
||||
### Rule 2: Bold strokes, no gradients
|
||||
Minimum stroke weight: 4px at 100px viewport. Think screen-print, not watercolor. No gradients, no shadows, no glow effects.
|
||||
|
||||
### Rule 3: Geometric construction
|
||||
The logo is pure geometry: circle + triangle + rectangles. New assets should use the same primitives. No organic shapes, no curves that aren't circles/arcs.
|
||||
|
||||
### Rule 4: Arrows point outward
|
||||
The five arrows on the logo all point away from center. This is the brand's core gesture: spreading outward, pushing content to the edges (the platforms). Reuse this directionality.
|
||||
|
||||
### Rule 5: High contrast or nothing
|
||||
Every element must be readable at 16px favicon size OR be a hero asset. No subtle textures, no mid-tones for decoration.
|
||||
|
||||
---
|
||||
|
||||
## Assets to create (in priority order)
|
||||
|
||||
### 1. Favicon (32×32 SVG)
|
||||
Simplify the logo: just the circle outline + the A path, red on transparent. Drop the arrows and the internal marks.
|
||||
```svg
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 1200 1200">
|
||||
<g fill="none" stroke="#FF1610" stroke-linecap="round" stroke-linejoin="round">
|
||||
<circle cx="600" cy="610" r="330" stroke-width="120"/>
|
||||
<path d="M 365 915 L 600 250 L 835 915" stroke-width="140"/>
|
||||
<path d="M 250 610 L 950 610" stroke-width="120"/>
|
||||
</g>
|
||||
</svg>
|
||||
```
|
||||
|
||||
### 2. Empty State Illustration (200×200 SVG)
|
||||
The circle-A outline at low opacity (30%) with a single red arrow pointing outward. Message sits beside it. This communicates "nothing here yet, but you're ready to broadcast."
|
||||
|
||||
### 3. Loading Spinner (48×48 SVG)
|
||||
The circle from the logo (no A), rotating. Red stroke, black background. CSS `animation: rotate 1s linear infinite`.
|
||||
|
||||
### 4. Hero/Background Image (1920×1080)
|
||||
If you want the aerial shot: take a real aerial photo, overlay a black gradient (bottom-heavy), then place the circle-A mark at 15% opacity in the background. The UI sits on top. Alternatively, create a pattern of the circle-A repeated at varying scales/opacities on black.
|
||||
|
||||
### 5. Platform Connection Icons
|
||||
Each platform (Mastodon, Telegram, etc.) gets a small circular icon in the brand style:
|
||||
- Black circle with red stroke
|
||||
- The platform's simplified symbol inside (drawn in red strokes, same weight)
|
||||
- 32×32 base size
|
||||
|
||||
### 6. Banner for Onboarding (1200×400)
|
||||
The full logo on the left, "Tactical Media Broadcast Hub" in bold red on the right, on black. Can include the tagline: "Broadcast everywhere. Answer to no one."
|
||||
|
||||
---
|
||||
|
||||
## How to create assets in Canva (matching this language)
|
||||
|
||||
1. **Open your Canva design kit** (the links you shared)
|
||||
2. **Set the canvas background to pure black** (#000000)
|
||||
3. **Use only these colors:**
|
||||
- Primary: #FF1610 (bright red — the brand color from your SVG)
|
||||
- Warm variant: #E8501E (more orange, for hover states)
|
||||
- Background: #000000 or #0A0A0B
|
||||
4. **Use geometric shapes only:** circles, triangles, rectangles, straight lines
|
||||
5. **Stroke weight:** thick (equivalent to 8-12px at 1000px canvas)
|
||||
6. **No:** gradients, shadows, blur, photos (except for the aerial background), curved lines that aren't circles, pastel colors, blue/purple/green
|
||||
7. **Export as:** SVG for icons/illustrations, PNG (1920×1080) for backgrounds
|
||||
|
||||
## File format guide
|
||||
|
||||
| Asset type | Format | Where to put it |
|
||||
|---|---|---|
|
||||
| Icons, logos, illustrations | SVG | `apps/web/public/brand/` |
|
||||
| Background images | PNG or WebP | `apps/web/public/brand/` |
|
||||
| Favicon | SVG (32×32) | `apps/web/public/brand/favicon.svg` |
|
||||
| Hero/banner | PNG (1200×400 or 1920×1080) | `apps/web/public/brand/` |
|
||||
| Animations | CSS (not GIF) | in the component code directly |
|
||||
|
||||
Once you create assets and drop them in `apps/web/public/brand/`, tell me what each one is and I'll wire them into the UI.
|
||||
22
docs/decisions/2026-10-09-intake-posture.md
Normal file
22
docs/decisions/2026-10-09-intake-posture.md
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Decision: Content intake posture — 2026-10-09
|
||||
|
||||
**Decision:** metadata-only + community submissions + Tier-1 sanctioned sources. **No gray-market account farming.**
|
||||
|
||||
## Context
|
||||
The collective wants an intake firehose: continuous content ingestion from social platforms for media work — hashtags, accounts, search — feeding Streams columns, Compose, and the archive.
|
||||
|
||||
## Research base (live-verified 2026-10-09; the research archive lives outside this repository)
|
||||
- `2026-10-09-intake-firehose-architecture.md` — per-source feasibility, verified volumes (Jetstream ~52 posts/s measured, Mastodon 300 req/5min anon verified, YouTube channel RSS keyless verified), architecture, storage math (~37 GB/day modeled), phased rollout.
|
||||
- `2026-10-09-scraping-feasibility.md` — farm-account + browser-automation scraping: every platform rated Brittle or worse; enforcement record (X C&D'd Nitter Aug 2026; Meta won injunctions specifically on fake-account scraping).
|
||||
- `2026-10-09-gray-market-services.md` — the vendor layer exists and is priced ($1–50 aged accounts, SMS pools, captcha farms, proxies, anti-detect browsers, pre-scraped datasets) but delivers +15–20% coverage for 10–30× the cost of the alternative posture, with 60–90% farm-detection probability within a year, material contagion risk to legitimate accounts, and documented identity-crime supply chains.
|
||||
|
||||
## Decision
|
||||
1. **Tier 1 (build now, zero ban risk, no accounts):** RSS/Atom (incl. YouTube channel RSS), Mastodon public tag/account timelines, Bluesky Jetstream with server-side keyword gate, PeerTube REST, per-URL public fetchers (gallery-dl/instaloader-class).
|
||||
2. **Tier 2 (small bounded pool, later):** 1–3 Telegram MTProto session accounts for public channel reading (quiet, warmed, never org-linked); optionally 3–5 read-only X accounts, dedicated egress each, 90-day sunset, kill switches.
|
||||
3. **Rejected:** mass account farming, hashtag firehoses via farms, gray-market datasets, managed-scraping retainers — 10–30× cost, existential tail risk, documented identity-crime supply chain (see gray-market ledger §9.4).
|
||||
4. **Per-URL ingestion** covers locked platforms: community-submitted public post URLs are fetched individually (gallery-dl/instaloader-class) — categorically different from authenticated crawling.
|
||||
|
||||
## Consequences
|
||||
- Intake adapters draw from `packages/egress` per-host budgets.
|
||||
- Locked-platform columns show community-submitted + metadata-only content; full firehose coverage concentrates on sanctioned sources.
|
||||
- Decision revisitable if platform APIs open or a funded data vendor passes legal review.
|
||||
385
docs/design/2026-10-09-chat-governance-design.md
Normal file
385
docs/design/2026-10-09-chat-governance-design.md
Normal file
|
|
@ -0,0 +1,385 @@
|
|||
# Chat Governance Design — Polls · Approval-Gated Posting · Forum Topics · Multi-Account Policies · Telegram Bot Control
|
||||
*Design doc: 2026-10-09 · Codebase: `apps/api/src/live-chat/**`, `packages/connector-telegram/src`, `apps/api/src/publishing`, `apps/api/src/media-control`.*
|
||||
|
||||
---
|
||||
|
||||
## 0. Decision summary (the ten calls this design makes)
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| D1 | **Polls are OURS, not Telegram's.** Ballots live in `poll_votes`; the Telegram card is an inline-keyboard *mirror* edited in place. Native `sendPoll` is allowed **only** for audience sentiment (≤600 s, anonymous, aggregate-only) and can never gate a post. Rationale: 600 s ceiling, anonymous-by-default, no voter roster API, no reopen (polls report §1). |
|
||||
| D2 | **Chat-governance bot = grammY 1.46 gateway module** inside the API (`apps/api/src/chat-gov/`), webhook in prod (secret per binding), `@grammyjs/runner` long-poll in dev. Publishing *sends* continue to flow through the existing connector-telegram + Temporal pipeline — the gateway never bulk-posts. |
|
||||
| D3 | **Forum topics are a binding registry, not a mirror of Telegram state.** Bot API has no list-topics → we persist `message_thread_id` at creation; enabling forum mode stays a human/MTProto runbook step (documented, no userbot in v1). |
|
||||
| D4 | **Posting rules are data, evaluated at three gates (E1/E2/E3)** with precedence `topic > contentType > account > workspace`, whole-block replacement, compiled snapshot + `policy_version` pinned at save and at every decision. **E3 pre-dispatch re-check is mandatory.** |
|
||||
| D5 | **Poll-pass NEVER side-channels a platform post.** Pass → `approve`-equivalent transition on a draft (system actor `system:chat-poll-gate`) → existing schedule/publish machinery → E3. Chat-originated posts are auto-materialized as drafts first. |
|
||||
| D6 | **Chat commands map 1:1 onto existing `DirectorOperation`** (`take|cut|pst-select|lock|override|desired`); authorization reuses workspace RBAC permissions (`director.take`, `director.lock_program`, …) resolved from immutable Telegram numeric `from.id` → `chat_identities` → membership role. Chat is UX; the control plane stays the enforcement point. |
|
||||
| D7 | **Destructive commands are confirmation-gated**: inline CONFIRM/ABORT, 30 s TTL, requester-only callback, dry-run diff echo, nonce; ephemeral `replace_callback_query_message` (Bot API 10.3) when available. |
|
||||
| D8 | **Every account/destination evaluates its own policy.** One draft → N destination decision records; partial fan-out (A sent, B blocked) is a first-class outcome. |
|
||||
| D9 | **Votes from ineligible users are stored, never counted** (GitLab pattern) and surfaced with `INELIGIBLE_ACTOR`; eligibility is re-read at close time. |
|
||||
| D10 | **Anti-abuse reuses what exists**: `SendGovernors`/`TokenBucket` in front of every outbound mirror/ack, grammY throttler (30/s global, 1/s per chat, 20/min group) with `bypassThrottler` for acks, `update_id` dedupe, per-user per-command-class token bucket, and the existing `live_chat_requests` idempotency pattern for close→publish. |
|
||||
|
||||
---
|
||||
|
||||
## 1. Feature set
|
||||
|
||||
### 1a. Polls in bridged chats — configurable pass criteria
|
||||
|
||||
**Subject of a poll.** A poll gates a *thing*: a `draft` (existing DraftPost), a *chat post* (text/media composed via `/post` → materialized as a draft, D5), or nothing (free-standing sentiment/procedural vote). Polls are created from the web UI, the overlay, or `/post`+`/poll` in a bound chat.
|
||||
|
||||
**Pass criteria — compiled rule object** (stored on the poll at open; never re-resolved mid-flight):
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"version": 1,
|
||||
"mode": "quorum", // immediate | quorum | admin_only
|
||||
"arms": [
|
||||
{ "kind": "admins", "x": 2 }, // x of workspace Admin/Owner set
|
||||
{ "kind": "participants","x": 3, "of": "observed" } // of: observed | eligible | none(count-only)
|
||||
],
|
||||
"join": "all", // all (AND, default) | any (OR)
|
||||
"allowVeto": true, // one veto fails regardless of quorum
|
||||
"minDwellSeconds": 45, // early close only when quorum ∧ dwell (anti-railroading)
|
||||
"timebox": { "duration": "PT2H", "onExpire": "reject" }, // approve | reject | escalateOwner
|
||||
"anonymity": "named", // named | anonymous (never mixed silently)
|
||||
"allowSelfApproval": false, // draft author's yes is stored, not counted
|
||||
"resetOnChange": true, // draft edit ⇒ approvals reset, poll re-arms
|
||||
"dryRun": false // evaluate + log, never block/fire
|
||||
}
|
||||
```
|
||||
|
||||
- **admins arm**: eligibility = workspace membership role ∈ {Owner, Admin} (or `chat_identities.role_override` ≥ that) *evaluated at close time* (TOCTOU guard).
|
||||
- **participants arm**: `of:observed` = distinct chat identities seen in the channel since `opens_at` (roster derived from `live_chat_messages` — we already own that table); `of:eligible` = explicit roster bound to the topic/channel; `of:none` = pure count.
|
||||
- **mixed** = conjunctive arms (`join:"all"`), the GitHub count+code-owner shape; each arm keeps its own `x`.
|
||||
- **State machine**: `open → passed | failed | vetoed | quorum_unreachable | cancelled`. `quorum_unreachable` fires eagerly when remaining eligible voters < needed (grey-the-card UI). Closed is terminal; a re-vote is a **new poll with `parent_poll_id`**.
|
||||
- **Close events**: (1) quorum ∧ minDwell → early close; (2) `closes_at` timebox expiry → `onExpire` action; (3) veto → immediate. Evaluation is event-driven (every ballot re-evaluates); the timebox is the only timer (Temporal `pollTimebox` workflow + lazy re-eval on read).
|
||||
- **Idempotent close**: state flip and downstream trigger live in one transaction guarded by `poll_outbox` exactly-once claim; double-close cannot double-publish.
|
||||
- **Native Telegram polls**: `/sentiment <question> ; <opt1> ; <opt2>…` sends `sendPoll` (anonymous, `open_period` ≤600 s), aggregate mirrored into the overlay card, **never linked to a draft**. Capability-gated: `poll.sentiment`.
|
||||
|
||||
### 1b. Poll-pass → automatic posting bot
|
||||
|
||||
Two wiring modes per destination policy:
|
||||
|
||||
1. **Gate mode (default)**: poll arms = the E2 gate on a draft. Pass → approve transition (system actor, §4) → E3 per destination → connector publish (`sendMessage`/`sendPhoto`/`sendMediaGroup`, `message_thread_id` when topic-bound) → `message_id` stored for edit/withdraw. Fail/veto → `reject` with poll reference.
|
||||
2. **Echo mode (small chats)**: poll subject is a chat post with no scheduled fan-out; pass posts the *same content* back into the originating chat/topic via the announce binding. Internally still materializes a draft + one destination (auditability, D5) — the "direct post" is a fan-out of size 1.
|
||||
|
||||
Poll-pass can also **trigger compose**: `/post` with attachments creates the draft in `editing`; on pass with `triggerCompose: true` the UI opens the composer pre-loaded with the approved draft for crosspost (WP15 path: fetch media → asset → draft quoting original → normal policies).
|
||||
|
||||
### 1c. Forum topics as subchats
|
||||
|
||||
- **`topic_bindings`** registry: `(workspace, connection, chat_id, message_thread_id)` → role `announce | control | audit`, plus assigned accounts and linked virtual channel. A topic binding behaves as a **subchat**: it has its own live-chat channel key (its own seq/fanout topic), its own posting policy (highest precedence), its own member roster.
|
||||
- **Bot operations**: create (`createForumTopic`, needs `can_manage_topics`), rename (`editForumTopic`), close/reopen (`closeForumTopic`/`reopenForumTopic`), plus `unpinAllForumTopicMessages` on archive. **`deleteForumTopic` is never exposed from chat** (deletes all messages; console-only with confirmation). Enabling forum mode on a supergroup: runbook step for a human admin (Bot API cannot; see research §2). Topic listing/backfill would need MTProto — **deferred**; our registry is authoritative for what we created.
|
||||
- **Accounts assigned to topics**: `topic_bindings.account_ids[]` — a destination account may be bound to zero+ topics; posts to that account ride `message_thread_id` of the binding. Unbound announce channels post to the channel root.
|
||||
- Auto-topic-per-series: on first approval pass for a show/series, broadcaster auto-creates a color-coded topic (icon colors from the fixed Telegram palette) and binds it; archive closes it. An **audit topic** is auto-created per workspace and kept append-only (every governance decision mirrored there).
|
||||
|
||||
### 1d. Multiple accounts per platform, each with posting rules
|
||||
|
||||
- Any number of connections per platform (each Telegram connection = one bot token; Mastodon = one handle; …). Policies anchor to **(platform, accountId)** — the account is the policy anchor, platform is an attribute.
|
||||
- **Modes**: `immediate` (no gate, still audited) · `quorum` (arms as §1a; `admins`/`participants`/mixed) · `admin_only` (single-admin gate: `quorum:[{kind:admins,x:1}]`, no participant arm, poll optional).
|
||||
- Cross-cutting knobs per policy: `timebox.onExpire`, `allowSelfApproval:false`, `resetOnChange:true`, `dryRun`, `fallback:block`.
|
||||
- **Precedence**: `topic > contentType > account > workspace default`, most-specific-wins, **whole-block replacement** (no field merges). Compiled at save-time into a resolved snapshot + `policy_version` hash persisted on the binding; evaluation reads only snapshots. Every rule supports `dryRun` (shadow) for onboarding a new policy.
|
||||
- Fan-out law (D8): one draft → N destinations → N independent decision records; "fully sent" = all destinations fired.
|
||||
|
||||
### 1e. Telegram bot broadcast control
|
||||
|
||||
Command grammar (control binding / locked `#control` topic only):
|
||||
|
||||
| Command | Maps to | Permission | Confirm? |
|
||||
|---|---|---|---|
|
||||
| `/tally` | read director state | `director.view` | no |
|
||||
| `/pst <sceneId>` | `{type:"pst-select"}` | `director.take` | no |
|
||||
| `/take [sceneId]` | `{type:"take"}` | `director.take` | no |
|
||||
| `/cut [sceneId]` | `{type:"cut"}` | `director.take` | **yes** |
|
||||
| `/lock on` | `{type:"lock",enabled:true}` | `director.lock_program` | no (protective) |
|
||||
| `/lock off` / `/unlock` | `{type:"lock",enabled:false}` | `director.lock_program` | **yes** |
|
||||
| `/override <mode|off>` | `{type:"override"}` | `director.override_auto` | **yes** |
|
||||
| `/start` `/stop` | desired RUNNING/STOPPED | `director.take` | stop: **yes** |
|
||||
| `/topics new\|close\|list` | topic registry ops | `chatgov.manage` | no |
|
||||
| `/post <text> [+media reply]` | draft + poll | `drafts.create` | no |
|
||||
| `/poll` (reply) | open vote on a draft/chat post | `chatgov.vote` | no |
|
||||
| `/sentiment …` | native sendPoll | `chatgov.sentiment` | no |
|
||||
| `/register` | claim identity → console approval | — | n/a (approval in console) |
|
||||
| `/help` | — | — | no |
|
||||
|
||||
Channel binding: each control binding carries `linked_channel_id` (the virtual channel commands act on); `/tally` replies with program/preview/lock/override snapshot. Locked channel ⇒ `/take`/`/cut` refused with the interlock reason (state machine enforced server-side in `media-control`, chat only displays).
|
||||
|
||||
**Confirmation UX (destructive)**: (1) dry-run diff echo ("program: SRC-A → SRC-B · ch s27-main"); (2) inline `CONFIRM / ABORT` keyboard, `callback_data = cf:<nonce>` (≤64 B); (3) callback accepted **only** from the requesting `from.id`; (4) 30 s TTL → panel self-erases via `replace_callback_query_message` (10.3) or edits to "expired"; (5) `answerCallbackQuery` ack; (6) execution goes through the same idempotent `director()` call the console uses, with the chat nonce as `Idempotency-Key`; (7) result + actor mirrored to the audit topic.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 2. Data model
|
||||
|
||||
All tables: `workspace_id uuid NOT NULL`, RLS `ENABLE`+`FORCE` with the standard tenant policy (migration lint enforces), all access via `withTenant`, each added to the guards-bypassed RLS denial suite (WP2 convention).
|
||||
|
||||
```sql
|
||||
-- 001_chatgov.sql (sketch)
|
||||
CREATE TABLE polls (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
channel_id uuid, -- live-chat channel (nullable: console-only poll)
|
||||
topic_binding_id uuid, -- vote scoped to a subchat when set
|
||||
subject_kind text NOT NULL CHECK (subject_kind IN ('draft','chat_post','none')),
|
||||
draft_id uuid, -- FK drafts(id) when subject_kind='draft'
|
||||
question text NOT NULL, -- 1..300
|
||||
parent_poll_id uuid, -- revote lineage
|
||||
rule jsonb NOT NULL, -- compiled §1a object (snapshot at open)
|
||||
policy_version text NOT NULL, -- hash of rule at open
|
||||
state text NOT NULL DEFAULT 'open'
|
||||
CHECK (state IN ('open','passed','failed','vetoed','quorum_unreachable','cancelled')),
|
||||
opens_at timestamptz NOT NULL DEFAULT now(),
|
||||
closes_at timestamptz NOT NULL, -- opens_at + timebox.duration
|
||||
closed_at timestamptz,
|
||||
close_reason text, -- quorum_early | timebox | veto | cancelled
|
||||
outbox_state text NOT NULL DEFAULT 'none' -- none|claimed|done (exactly-once close)
|
||||
CHECK (outbox_state IN ('none','claimed','done')),
|
||||
created_by uuid NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE UNIQUE INDEX polls_open_one_per_draft ON polls (workspace_id, draft_id)
|
||||
WHERE state = 'open' AND draft_id IS NOT NULL; -- one open gate per draft
|
||||
CREATE INDEX polls_ws_state ON polls (workspace_id, state, closes_at);
|
||||
|
||||
CREATE TABLE poll_votes (
|
||||
poll_id uuid NOT NULL REFERENCES polls(id),
|
||||
workspace_id uuid NOT NULL,
|
||||
voter_kind text NOT NULL CHECK (voter_kind IN ('user','chat_identity','system')),
|
||||
voter_user_id uuid, -- resolved workspace user, when known
|
||||
voter_remote_id text NOT NULL, -- provider:from.id (numeric for TG)
|
||||
voter_handle text NOT NULL,
|
||||
ballot text NOT NULL CHECK (ballot IN ('yes','no','veto','abstain')),
|
||||
eligible boolean NOT NULL DEFAULT true, -- eligibility AT CAST (informational)
|
||||
counted boolean NOT NULL DEFAULT true, -- ineligible votes stored, not counted
|
||||
ineligible_reason text, -- INELIGIBLE_ACTOR | SELF_APPROVAL_DENIED
|
||||
cast_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (poll_id, voter_remote_id) -- idempotent upsert; revote = replace ballot
|
||||
);
|
||||
CREATE INDEX poll_votes_poll ON poll_votes (poll_id, counted);
|
||||
|
||||
CREATE TABLE posting_rules (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
scope text NOT NULL CHECK (scope IN ('workspace','account','content_type','topic')),
|
||||
account_id uuid, -- connection id; NULL for workspace
|
||||
topic_binding_id uuid, -- when scope='topic'
|
||||
content_type text, -- 'alert' | 'video' | ... when scope='content_type'
|
||||
policy jsonb NOT NULL, -- §1a-shaped rule (minus subject)
|
||||
policy_version text NOT NULL, -- compiled snapshot hash at save
|
||||
dry_run boolean NOT NULL DEFAULT false,
|
||||
created_by uuid NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
superseded_by uuid,
|
||||
UNIQUE (workspace_id, scope, account_id, topic_binding_id, content_type)
|
||||
);
|
||||
-- Effective policy resolution is done at SAVE time into:
|
||||
CREATE TABLE posting_policy_snapshots (
|
||||
workspace_id uuid NOT NULL,
|
||||
destination_kind text NOT NULL, -- 'account' | 'topic'
|
||||
account_id uuid NOT NULL,
|
||||
topic_binding_id uuid,
|
||||
effective jsonb NOT NULL,
|
||||
policy_version text NOT NULL,
|
||||
resolved_from jsonb NOT NULL, -- audit: which scopes contributed
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (workspace_id, account_id, topic_binding_id)
|
||||
);
|
||||
|
||||
CREATE TABLE topic_bindings (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
connection_id uuid NOT NULL, -- telegram connection (bot)
|
||||
chat_id text NOT NULL, -- numeric -100… preferred (usernames drift)
|
||||
message_thread_id text, -- NULL = channel root / whole group
|
||||
binding_type text NOT NULL CHECK (binding_type IN ('announce','control','audit')),
|
||||
name text NOT NULL, -- 1..128 (Telegram topic name limit)
|
||||
icon_color integer, -- one of the 6 fixed TG colors
|
||||
state text NOT NULL DEFAULT 'active' CHECK (state IN ('active','closed','archived')),
|
||||
account_ids uuid[] NOT NULL DEFAULT '{}', -- accounts that post here
|
||||
linked_channel_id uuid, -- control: VC this binding commands
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (workspace_id, connection_id, chat_id, message_thread_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX topic_bindings_one_control
|
||||
ON topic_bindings (workspace_id, linked_channel_id)
|
||||
WHERE binding_type = 'control' AND linked_channel_id IS NOT NULL;
|
||||
|
||||
CREATE TABLE chat_identities (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
provider text NOT NULL,
|
||||
remote_user_id text NOT NULL, -- immutable numeric TG id — NEVER username
|
||||
user_id uuid, -- linked workspace user (after /register approval)
|
||||
display_name text,
|
||||
status text NOT NULL DEFAULT 'pending' -- pending|approved|rejected|banned
|
||||
CHECK (status IN ('pending','approved','rejected','banned')),
|
||||
approved_by uuid,
|
||||
first_seen_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (workspace_id, provider, remote_user_id)
|
||||
);
|
||||
|
||||
CREATE TABLE chat_command_audit (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
update_id bigint NOT NULL, -- Telegram update id (dedupe)
|
||||
binding_id uuid, -- topic binding command arrived on
|
||||
provider text NOT NULL,
|
||||
remote_user_id text NOT NULL,
|
||||
user_id uuid, -- resolved identity, if any
|
||||
command text NOT NULL, -- '/cut'
|
||||
args jsonb NOT NULL DEFAULT '{}',
|
||||
permission text, -- RBAC permission required
|
||||
decision text NOT NULL CHECK (decision IN ('accepted','rejected','unauthorized','unknown_command','rate_limited')),
|
||||
reason text, -- machine code: INELIGIBLE_ACTOR | LOCKED_CHANNEL | CONFIRM_EXPIRED | ...
|
||||
nonce text, -- confirmation nonce when confirm-gated
|
||||
idempotency_key text, -- chat nonce used as director key
|
||||
result_ref jsonb, -- receipt fingerprint / poll id / draft id
|
||||
received_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (workspace_id, provider, update_id)
|
||||
);
|
||||
CREATE INDEX chat_command_audit_ws_time ON chat_command_audit (workspace_id, received_at DESC);
|
||||
|
||||
-- Governance decision records (one per destination per evaluation, §4)
|
||||
CREATE TABLE gate_decisions (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
draft_id uuid NOT NULL,
|
||||
destination jsonb NOT NULL, -- {platform, accountId, topicBindingId?}
|
||||
gate text NOT NULL CHECK (gate IN ('E1','E2','E3')),
|
||||
policy_version text NOT NULL,
|
||||
outcome text NOT NULL CHECK (outcome IN ('fired','blocked','pending')),
|
||||
inputs jsonb NOT NULL, -- approvals, poll tally, quorum gaps
|
||||
reason_codes text[] NOT NULL DEFAULT '{}',
|
||||
actor text NOT NULL, -- 'system:gate-e3' | 'user:<id>' | 'chat_poll:<pollId>'
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX gate_decisions_draft ON gate_decisions (workspace_id, draft_id, gate);
|
||||
|
||||
-- Poll mirror bookkeeping (card = edited, never re-sent)
|
||||
CREATE TABLE poll_mirrors (
|
||||
poll_id uuid NOT NULL,
|
||||
workspace_id uuid NOT NULL,
|
||||
binding_id uuid NOT NULL,
|
||||
message_remote_id text NOT NULL, -- for editMessageText
|
||||
last_edit_at timestamptz NOT NULL DEFAULT now(),
|
||||
last_render_hash text NOT NULL, -- edit-on-change guard
|
||||
PRIMARY KEY (poll_id, binding_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Migration sketch** — single forward migration `001_chatgov.sql`: create the tables above + RLS enable/force + tenant policies + indexes; register retention classes (`poll_votes`, `chat_command_audit`, `gate_decisions` append-hot, monthly-partition candidates per WP2 convention; `polls`/`posting_rules` stay unpartitioned FK parents). Rollback = drop schema objects (no data migration in v1). Decision-record events additionally append to the existing `audit_append` chain (`poll.*`, `policy.*`, `chatcmd.*` actions) so governance lands in the tamper-evident spine.
|
||||
|
||||
---
|
||||
|
||||
## 3. Bot architecture
|
||||
|
||||
**Library: grammY 1.46** (+ `@grammyjs/transformer-throttler`, `@grammyjs/conversations`, `@grammyjs/menu`, `@grammyjs/runner`). TS-first, tracks Bot API 10.3, throttler ships compliant budgets (30/s global · 1/s per chat · 20/min group) and honors 429 `retry_after`; `bypassThrottler` fast-lanes command acks. Telegraf is stale (2024); node-telegram-bot-api is an immature RC — both rejected per research.
|
||||
|
||||
**Topology — module, not a new service.** `apps/api/src/chat-gov/`:
|
||||
|
||||
```
|
||||
chat-gov/
|
||||
telegram-gateway.ts grammY Bot per approved token; webhook in prod
|
||||
(POST /webhooks/chatgov/:bindingId, secret_token per
|
||||
connection), runner+long-poll in dev; 1 poller per token
|
||||
command-router.ts grammar → RBAC check → confirm gate → control-plane call
|
||||
poll-mirror.ts poll card render/edit (edit-on-change, 2–5 s throttle)
|
||||
poll-service.ts open/vote/close/evaluate; emits governance WS frames
|
||||
policy-service.ts rule CRUD + compile to posting_policy_snapshots
|
||||
gate-evaluator.ts E1/E2/E3 evaluation + decision records
|
||||
chat-gov.module.ts
|
||||
```
|
||||
|
||||
- **Bridge to live-chat WS**: inbound TG messages in bound chats → `NormalizedChatEvent` → existing `LiveChatHubService.ingest` (same dedupe, seq, ring buffer); a topic binding supplies its own `channel_id` so each subchat sequences and fans out independently on the existing `chat:<ws>:<channel>` Redis topic. Outbound, `poll-mirror` and acks go through `SendGovernors`-wrapped transports — a new `ChatTransport` impl for telegram whose `send`/`edit` consume the same token buckets. Governance frames ride the same fanout as `kind:"chat"`: new `kind:"poll"` (snapshot + delta), `kind:"poll_closed"`, `kind:"tally"` — same subscribe/ack/resume contract, server-side debounce (≥250 ms coalesce) so vote storms don't flood the overlay.
|
||||
- **Capability plumbing**: extend `LiveChatCapability` in `packages/core/src/chat-contract.ts` with `poll: "none" | "vote" | "sentiment"` and `topics: boolean`; `capability-probe.ts` verifies a callback_query round-trip before advertising `vote` (inline keyboards are always sendable; the *callback* is what must work).
|
||||
- **Command router pipeline** (order fixed): dedupe on `update_id` → resolve binding (chat_id + thread → topic_bindings) → resolve identity (`chat_identities`, numeric id) → RBAC map (identity.user → membership role → permission via existing permission table; role sources are ours, TG admin flags are display hints only) → per-user per-command-class token bucket → parse/validate args → confirm gate for destructive → call control plane (`media-control.director()`, `publishing`, or chat-gov service) with chat nonce as `Idempotency-Key` → `chat_command_audit` row → reply + audit-topic mirror.
|
||||
- **Permission mapping**: `/tally`→`director.view`; `/pst|/take|/cut|/start|/stop`→`director.take`; `/lock`→`director.lock_program`; `/override`→`director.override_auto`; `/topics`→`chatgov.manage`; `/post`→`drafts.create`; `/poll`→`chatgov.vote`; `/register`→self-service + console approval (`chatgov.manage` approves). Unregistered identity ⇒ everything except `/help`, `/register`, `/tally` is `unauthorized`. Unlinked control binding ⇒ commands rejected with `NO_CHANNEL_BOUND`.
|
||||
- **Rate limiting, layered**: grammY throttler (outbound API shape) → `SendGovernors` token buckets (our policy: mirror edits ≤1 per 3 s per binding, priority lanes control > notify > bulk) → per-user per-command-class bucket (e.g. `/cut` ≤3/min) → 429 global backoff with jitter. `poll-mirror` edits only when `render_hash` changes.
|
||||
|
||||
---
|
||||
|
||||
## 4. Quorum evaluation ↔ publishing pipeline
|
||||
|
||||
**Draft states today**: `editing → in_review → approved → rejected → scheduled → published`, with `approved_hash`/`approved_revision` pinned at approve time, and `audit_append` per transition. Chat governance plugs into exactly these seams — no parallel pipeline.
|
||||
|
||||
**Gates (Argo sync-wave order):**
|
||||
|
||||
| Gate | Where | Signal | Fires |
|
||||
|---|---|---|---|
|
||||
| E1 | `review → approved` transition | RBAC approvals on the draft | Editor submits / console approve |
|
||||
| E2 | **Chat poll gate** | `polls.state` outcome | ballot cast (event-driven) or timebox (timer) |
|
||||
| E3 | Immediately before each platform dispatch | compiled snapshot + approvals + poll state | connector publish call |
|
||||
|
||||
**Flow for a quorum-gated destination:**
|
||||
|
||||
1. Draft created (console or `/post`) with N destinations → each destination's effective snapshot resolved → `gate_decisions` rows created `pending`.
|
||||
2. Poll opened (auto on submit when an arm requires chat signal, or manually): rule compiled from the destination's most-gating arm; `polls.rule` snapshot + `policy_version` pinned.
|
||||
3. **Every ballot** upserts `poll_votes` (idempotent on `(poll_id, voter_remote_id)`) and re-evaluates in the same transaction: eligibility re-read, arms computed, `quorum_unreachable` checked eagerly. Card edits flow through mirror throttle.
|
||||
4. **Close** (quorum ∧ minDwell | timebox `onExpire` | veto): claim `outbox_state` (exactly-once), set final state, then in the *same transaction*: pass ⇒ `publishingService` transition `in_review → approved` with **system actor** `system:chat-poll-gate:<pollId>`; veto/fail ⇒ `→ rejected` with reason. The poll id doubles as the idempotency key — replayed closes are no-ops. `approved_hash` pinning stays intact: a post-close draft edit re-validates at E3 and is blocked (`RESET_ON_CHANGE` / hash mismatch).
|
||||
5. **E3 pre-dispatch** (Temporal `scheduledPublish` worker, per destination): re-read snapshot + approvals + poll state + `approved_hash`; evaluate; write the `gate_decisions` outcome (`fired`/`blocked` + reason codes); only then connector publish. Partial fan-out is normal: A fires, B blocked ⇒ B's destination shows in the blocked queue with its reason codes and gap-to-quorum.
|
||||
6. **Audit trail**: `poll.opened/voted/closed` + `policy.evaluated` + `release.fired/blocked` all append via `audit_append` (tamper-evident chain, actor = system identity with poll ref); every decision record embeds `policy_version`. `admin_only` mode = same path with a 1-arm poll or an E1-only console approval. `immediate` mode = E1 minimal + E3 always.
|
||||
|
||||
**Reason codes** (machine-readable, attached to the blocked post object, not just logs): `INSUFFICIENT_QUORUM` (got x, need y, per arm) · `INELIGIBLE_ACTOR` · `SELF_APPROVAL_DENIED` · `TIMEBOX_EXPIRED_AUTO_REJECT` / `..._AUTO_APPROVE` · `RESET_ON_CHANGE` · `PENDING_POLL` · `VETOED` · `OUTSIDE_POLICY` · `POLICY_DRY_RUN` · `OVERRIDDEN_BY_OWNER` · `LOCKED_CHANNEL` · `NO_CHANNEL_BOUND` · `CONFIRM_EXPIRED`.
|
||||
|
||||
---
|
||||
|
||||
## 5. UI spec
|
||||
|
||||
**Poll card (one React component, three mount points: web chat pane, OBS/browser overlay, mobile).**
|
||||
- Header: question + author avatar + status chip (`collecting votes → quorum met → passed → publishing` phase ribbon).
|
||||
- Body: horizontal **tally bars** per option, width = share, count + % right-aligned; named mode renders **avatar stacks** (max 5 + `+n`) under each bar; anonymous renders a lock glyph and "votes hidden" — never mixed.
|
||||
- **Quorum strip**: visually distinct segmented meter per arm — `2/3 admins ●●○ · 1/3 participants ●○○` (dots, not bars: a gate, not an option). Greyed card + "quorum unreachable — 1 eligible voter remaining" when eager-check trips.
|
||||
- **Countdown chip** on the card edge; `closing soon` (<10% left) gets emphasis. Terminal state: bars freeze, winner highlighted, outcome ribbon `PASS ✓ / FAIL ✗ / VETO ⛔`, auto-collapse to one line in history after 60 s (overlay immediately).
|
||||
- **Interaction**: web/mobile tap a bar (or 👍/👎/veto buttons); Telegram inline keyboard `[👍 2] [👎 1] [⛔ veto]` — the *card*, not chat messages, is the only surface that changes. Revote overwrites; keyboard row re-renders counts on edit.
|
||||
- **Overlay mount**: 320–400 px corner card, same component, fed by the same WS frames; server-debounced deltas; resolved state collapses to a single line.
|
||||
|
||||
**Vote flow (mobile)**: tap → optimistic bar update (client) → WS confirm (authoritative, replaces optimistic) → haptic + checkmark on your bar. Offline/late frames: card always renders from the last authoritative snapshot (`kind:"poll"` snapshot after `kind:"poll_delta"` catch-up on resume).
|
||||
|
||||
**Posting-rules editor** (console, per account and per topic):
|
||||
- Left: destination tree (platform → accounts → topics). Right: rule card with mode selector (`immediate / quorum / admin_only`), arm builder rows (`kind` + `x` + `of`), `join` toggle, veto switch, min-dwell, timebox duration + `onExpire` select, anonymity, self-approval, reset-on-change, **dry-run toggle with a badge**.
|
||||
- Save action shows the **resolved preview** ("effective for @newsbot: 2-of-admins AND 3 participants, veto on, 2 h → auto-reject") before commit; the compiled `policy_version` hash is displayed. Every save writes `policy.revised` to the audit chain; edits require `chatgov.manage`.
|
||||
- **Blocked queue view**: drafts blocked at E2/E3 as first-class cards with gap-to-quorum meter (live), reason-code chips, and actions (nudge voters / open re-vote with `parent_poll_id` / owner override-with-reason).
|
||||
|
||||
**Topic management screen**:
|
||||
- Grid of topic bindings per connection: name, icon color chip, type badge (announce/control/audit), state (active/closed/archived), assigned accounts, linked VC, live policy summary chip.
|
||||
- Actions: create (calls bot `createForumTopic`, persists thread id), rename, close/reopen, bind/unbind accounts, set control link, edit policy (jumps to rules editor scoped to topic), archive (close + unpin). `delete` is deliberately absent (console-only runbook with typed confirmation; destructive to history).
|
||||
- Header banner when forum mode is off on the target supergroup: runbook steps (toggle Topics, re-promote bot with `can_manage_topics`) — we cannot detect or fix this from the Bot API; create attempts surface the Telegram error verbatim.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security
|
||||
|
||||
- **Token vault**: bot tokens and any MTProto session are opaque **vault credential handles** (existing envelope-encryption vault, same boundary as stream keys — `credential-handles.ts` pattern). Tokens never appear in DTOs, logs, WS frames, argv, or chat. Gateway reads handles at boot and on rotation; every credential read is audited. Rotation runbook: BotFather `/revoke` → re-vault → gateway reload → old token dead.
|
||||
- **Identity**: authorization keyed on immutable numeric `from.id` → `chat_identities` (status `approved` only) → membership role → RBAC permission. **Usernames are never trusted.** Unregistered senders get `/register`, which creates a `pending` identity an admin approves in console — no self-elevation.
|
||||
- **Least privilege on-platform**: bot promoted with only `can_post_messages`, `can_edit_messages`, `can_delete_messages`, `can_manage_topics` — never `can_promote_members`.
|
||||
- **Anti-abuse**: layered rate limiting (§3); `update_id` uniqueness kills replays; webhook `secret_token` per binding rejects forged updates; per-user per-command-class buckets; flood governors already refuse (not queue) over-budget sends so a poll storm can't wedge the API; poll mirror edit-on-change + render-hash guard; one-open-poll-per-draft unique index stops poll spam gating the same draft; `chat_command_audit` makes every attempt (including rejects) inspectable.
|
||||
- **NEVER runs from chat without confirmation** (and some never from chat at all):
|
||||
- *Confirm-gated*: `/cut`, `/unlock`, `/override on`, `/stop` — CONFIRM/ABORT + dry-run diff + 30 s TTL + requester-only callback + nonce idempotency.
|
||||
- *Console-only, never from chat*: credential/token management, role grants and identity approvals, topic **delete**, policy `dry_run`→enforce flips, retention, workspace settings, and any multi-destination bulk publish that would exceed the broadcast lane (bulk always goes through the Temporal pipeline, never a chat command).
|
||||
- *Structurally impossible from chat*: reading vault contents, cross-workspace data (RLS + no chat surface for it), editing another user's draft.
|
||||
- **Privacy**: anonymous polls store no voter linkage (ballot rows only for named mode); scrub() applies to all audited diffs (existing pattern); control chatter is confined to the locked `#control` topic, and the audit topic is append-only (bot has no delete there operationally).
|
||||
- **Account posture (from scrape-feasibility)**: the multi-account feature targets *our own legitimate accounts* (bots/handles we own). No browser-automation posting, no scraped-account fan-out; any future MTProto admin sidecar (forum-mode enable, topic backfill) stays admin-ops-only, never bulk-posting — that is the documented boundary.
|
||||
|
||||
---
|
||||
|
||||
## 7. Phased build plan
|
||||
|
||||
| Phase | Slice | Deps | Acceptance criteria |
|
||||
|---|---|---|---|
|
||||
| **P0 — spine** | Migrations (§2 tables + RLS + indexes + retention classes), `policy-service` rule CRUD + compile-to-snapshot, `gate_decisions` writer, RBAC permissions (`chatgov.*`), audit events | WP2 vault/RBAC, WP3 drafts | Migrations idempotent + lint-clean; RLS denial suite covers every new table (workspace B sees zero A rows); policy compile is pure + snapshot hash deterministic; `policy.revised` lands in audit chain |
|
||||
| **P1 — E1/E3 gates** | Wire snapshot evaluation into `review→approved` (E1) and the Temporal dispatch worker (E3); blocked queue rows + reason codes; `immediate`/`admin_only` modes live | P0 | `immediate` posts with no gate; `admin_only` blocks until one admin approve; E3 re-check blocks an approval revoked post-approve; partial fan-out observable (A fired, B blocked with codes); no behavior change for existing drafts |
|
||||
| **P2 — polls core (web)** | `poll-service` + open/vote/close/evaluate; WS `kind:"poll"` frames; web chat poll card + vote flow; mixed arms, veto, timebox, min-dwell, anonymity; E2 pass→approve integration with exactly-once outbox | P0, P1 | 2-of-admins poll passes → draft `approved` once under 5 concurrent closing triggers; veto fails immediately; timebox auto-reject fires once; ineligible/self votes stored-not-counted with codes; `quorum_unreachable` eager; draft edit after close blocks E3 (`RESET_ON_CHANGE`) |
|
||||
| **P3 — Telegram gateway** | grammY module, webhook + secret per binding, identity registry + `/register` approval, command router with RBAC + per-class buckets, `chat_command_audit`, non-destructive commands (`/tally`, `/pst`, `/take`, `/lock on`, `/topics`) | P0 | Replay of same `update_id` is a no-op; unregistered identity gets `unauthorized` + `/register`; wrong-permission command rejected with code + audited; `/take`/`/pst` produce real DirectorReceipts via existing `director()`; `/lock on` on locked channel consistent with console |
|
||||
| **P4 — confirm UX + destructive** | CONFIRM/ABORT conversations, nonce idempotency, dry-run diff echo, ephemeral replace (10.3), audit-topic mirror; `/cut`, `/unlock`, `/override`, `/stop` | P3 | `/cut` without confirm never reaches the control plane; confirm panel expires at 30 s; second user's callback on my panel rejected; same nonce can't fire twice; every destructive action mirrored to audit topic with actor + before/after |
|
||||
| **P5 — poll mirror + topics** | Inline-keyboard poll cards (`callback_data ≤64B`), edit-on-change throttled through governors, `poll_mirrors`; `topic_bindings` CRUD, bot create/close/rename topics, per-topic policy precedence, subchat channel keys in live-chat, accounts↔topics assignment | P2, P3 | Vote via Telegram button updates web card ≤3 s and vice versa (same WS spine); card edits throttled ≥2 s and only on change (governor test); topic create returns persisted `message_thread_id`; topic policy overrides account policy (precedence test); forum-mode-off surfaces runbook banner |
|
||||
| **P6 — rules editor + topic screen + blocked queue** | Console UI (§5): destination tree, arm builder, resolved preview, dry-run badge; topic management grid; blocked queue with gap-to-quorum + override-with-reason; mobile vote polish | P1, P2, P5 | Editor save shows exact effective preview matching compiled snapshot hash; dry-run badge blocks nothing but logs `POLICY_DRY_RUN`; blocked post card shows live gap + reason chips; override requires free-text reason and is audited |
|
||||
| **P7 — echo/auto-topic + hardening** | Echo mode (pass→post into same chat/topic as 1-destination fan-out), auto-topic-per-series, `triggerCompose` crosspost hook (WP15), sentiment `sendPoll`, load test (vote storms), secret-scan pass, retention wiring, docs | P5, P6 | Echo post lands in originating topic after pass with `message_id` stored; auto-topic created once per series and closed on archive; 1k votes/min sustained with flat WS queue + no unthrottled mirror edits; secret-scan clean on all gateway paths; docs updated |
|
||||
|
||||
**Dependency order**: P0 → P1 → P2 → {P3 → P4} ∥ P5(P2+P3) → P6 → P7. P3 can start immediately after P0 in parallel with P2 (independent seams: inbound commands vs. poll state machine).
|
||||
|
||||
**Out of scope v1 (explicit)**: MTProto userbot sidecar (forum-mode enable, topic backfill — runbook doc only), weighted votes, OPA/Rego expression policies, Discord/Discourse poll connectors (schema is provider-neutral; adapters later), paid broadcast (`allow_paid_broadcast`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Open items (flagged, decided by default)
|
||||
1. **`of:eligible` rosters** need a membership UI per topic — v1 ships `admins` + `of:observed` + count-only; explicit rosters land with P6 if asked for.
|
||||
2. **Overlay auth for anonymous viewers** voting in browser: v1 restricts overlay poll cards to *display*; voting happens in Telegram/web-app logged-in surface (avoids a new anon-auth surface).
|
||||
3. **Multiple bots per workspace**: supported (per-account tokens = blast-radius isolation + independent rate budgets); one poll may still span destinations on different bots — cards mirror per binding.
|
||||
385
docs/design/2026-10-09-columns-design.md
Normal file
385
docs/design/2026-10-09-columns-design.md
Normal file
|
|
@ -0,0 +1,385 @@
|
|||
# WP3 — Editable Multi-Platform Streams Columns: Design
|
||||
**Status:** decision-complete design (no implementation) · **Author:** glm-5.3 design agent · **Inputs:** `internal-survey.md`, `platform-research.md`, `docs/work-breakdown.md` WP3, current `streams/page.tsx` + `lib/streams-data.ts`.
|
||||
|
||||
---
|
||||
|
||||
## 0. Ground rules (non-negotiable constraints this design obeys)
|
||||
|
||||
1. **No logged-in scraping, no ToS-gray bridges** except where explicitly signed off (Telegram via RSS-Bridge is the only candidate; deferred, not in scope of the base plan).
|
||||
2. **All provider content renders only through `packages/safe-content`** (typed rich-text AST; raw HTML never reaches the client). See §5.
|
||||
3. **Every outbound request goes through the `egress` client** and draws from the **per-host budget** (`app.take_host_budget`) machinery shipped by WP7/WP8 — columns add *callers* of that machinery, not a parallel limiter.
|
||||
4. **Back-compat:** `GET /feeds` (merged all-feeds view, `meta.connections`, cursor shape) keeps working unchanged. Columns are additive.
|
||||
5. **Workspace-scoped everything** (multi-tenant; `workspace_id` in every unique constraint, RLS-checked reads, `draft.view`-class RBAC extended with a `streams.columns.manage` permission — see §5.6).
|
||||
|
||||
---
|
||||
|
||||
## 1. Column types, schema, migration, API surface
|
||||
|
||||
### 1.1 The four column types
|
||||
|
||||
| Type | What it shows | Sources resolved at runtime | Example |
|
||||
|---|---|---|---|
|
||||
| `platform_feed` | One connected account's own home/mentions (today's implicit column, made explicit & editable) | 1 connection × kinds | "Mastodon · @acampada" |
|
||||
| `keyword` | One query across N platforms (each platform adapter shapes the query: full-text, hashtag, video-search) | per-(connection or keyless-source) search job | "acampada26" on Bluesky+Mastodon+YouTube |
|
||||
| `account_list` | A curated list of accounts across platforms (author timelines / RSS) | per-account author-feed job | "Movement voices" = 5 handles × 3 platforms |
|
||||
| `saved_search` | A keyword column created via the existing saved-search UX (localStorage migrated to server) | same as `keyword` | "product" |
|
||||
|
||||
`saved_search` is **not a separate engine** — it is `keyword` with `origin: "saved_search"`. It exists as a persisted type so the UI can label, migrate, and later diverge (e.g. notifications) without another migration. Decision: single `type` enum column with the four values; the keyword engine is shared.
|
||||
|
||||
### 1.2 Persisted schema (workspace-scoped)
|
||||
|
||||
New tables (forward-only numbered SQL migration, additive, no changes to existing tables):
|
||||
|
||||
```sql
|
||||
-- 1. Column definitions
|
||||
CREATE TABLE stream_columns (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id UUID NOT NULL,
|
||||
type TEXT NOT NULL CHECK (type IN ('platform_feed','keyword','account_list','saved_search')),
|
||||
title TEXT NOT NULL, -- user-editable, max 80 chars
|
||||
position INTEGER NOT NULL DEFAULT 0, -- dense rank; reorder rewrites the tail
|
||||
spec JSONB NOT NULL, -- typed per-type, validated in app layer (below)
|
||||
refresh_policy JSONB NOT NULL DEFAULT '{}', -- { "intervalMs"?: number, "paused"?: boolean }
|
||||
created_by UUID NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
deleted_at TIMESTAMPTZ -- soft delete keeps feed_items FK-stable
|
||||
);
|
||||
CREATE UNIQUE INDEX stream_columns_ws_order_uq ON stream_columns (workspace_id, position) WHERE deleted_at IS NULL;
|
||||
CREATE INDEX stream_columns_ws_idx ON stream_columns (workspace_id) WHERE deleted_at IS NULL;
|
||||
|
||||
-- 2. Resolved sources: what the scheduler actually polls for a column
|
||||
CREATE TABLE column_sources (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id UUID NOT NULL,
|
||||
column_id UUID NOT NULL REFERENCES stream_columns(id),
|
||||
-- identity of the thing polled:
|
||||
provider TEXT NOT NULL, -- 'bluesky' | 'mastodon' | 'youtube' | ...
|
||||
connection_id UUID NULL, -- NULL for keyless sources (YouTube RSS, public Bluesky/Mastodon)
|
||||
source_kind TEXT NOT NULL CHECK (source_kind IN ('home','mentions','search','author')),
|
||||
query TEXT NULL, -- platform-adapted query (post-adaptation, what gets sent)
|
||||
actor_ref TEXT NULL, -- canonical account ref for source_kind='author': '{provider}:{handle@host}'
|
||||
state TEXT NOT NULL DEFAULT 'pending'
|
||||
CHECK (state IN ('pending','ready','stale','error','unsupported','rate_limited','budget_paused','disabled')),
|
||||
last_error TEXT NULL,
|
||||
last_polled_at TIMESTAMPTZ NULL,
|
||||
next_poll_at TIMESTAMPTZ NULL,
|
||||
consecutive_failures INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE (workspace_id, column_id, provider, source_kind, COALESCE(connection_id::text,''), COALESCE(query,''), COALESCE(actor_ref,''))
|
||||
);
|
||||
CREATE INDEX column_sources_due_idx ON column_sources (next_poll_at) WHERE state NOT IN ('disabled');
|
||||
CREATE INDEX column_sources_column_idx ON column_sources (column_id);
|
||||
|
||||
-- 3. Ingested items (replaces "cache-only" depth; enables deep pagination + per-column serving)
|
||||
CREATE TABLE feed_items (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id UUID NOT NULL,
|
||||
dedup_key TEXT NOT NULL, -- see §3.3
|
||||
provider TEXT NOT NULL,
|
||||
remote_id TEXT NOT NULL,
|
||||
author_handle TEXT NOT NULL,
|
||||
author_ref TEXT NULL, -- canonical '{provider}:{handle@host}' when resolvable
|
||||
url TEXT NULL, -- provider permalink (http/https only)
|
||||
created_at_remote TIMESTAMPTZ NOT NULL, -- provider timestamp (sort key)
|
||||
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
content JSONB NOT NULL, -- safe-content typed AST (NEVER raw HTML) — §5
|
||||
plain_text TEXT NOT NULL, -- derived flat text (for saved-search client filter & search preview)
|
||||
media JSONB NOT NULL DEFAULT '[]', -- [{kind:'image'|'video', proxyUrl, width?, height?, alt?}] — proxy-minted only
|
||||
metrics JSONB NOT NULL DEFAULT '{}', -- finite numbers only
|
||||
source_id UUID NOT NULL REFERENCES column_sources(id),
|
||||
expires_at TIMESTAMPTZ NULL, -- set for YouTube Data-API-sourced items: fetched_at + 30d (ToS)
|
||||
UNIQUE (workspace_id, dedup_key)
|
||||
);
|
||||
CREATE INDEX feed_items_ws_time_idx ON feed_items (workspace_id, created_at_remote DESC, dedup_key DESC); -- keyset
|
||||
CREATE INDEX feed_items_source_idx ON feed_items (source_id, created_at_remote DESC);
|
||||
CREATE INDEX feed_items_expiry_idx ON feed_items (expires_at) WHERE expires_at IS NOT NULL;
|
||||
|
||||
-- 4. Join: which columns show an item (an item ingested for a source is visible to every column sharing that source)
|
||||
CREATE TABLE feed_item_columns (
|
||||
workspace_id UUID NOT NULL,
|
||||
item_id UUID NOT NULL REFERENCES feed_items(id) ON DELETE CASCADE,
|
||||
column_id UUID NOT NULL REFERENCES stream_columns(id),
|
||||
PRIMARY KEY (item_id, column_id)
|
||||
);
|
||||
|
||||
-- 5. Per-provider budget/spend ledger (X pay-per-use, IG 30-tags/7d, YT search bucket)
|
||||
CREATE TABLE provider_budget_ledger (
|
||||
workspace_id TEXT NOT NULL, -- '' = global (API keys are workspace-owned; see §2.5)
|
||||
provider TEXT NOT NULL,
|
||||
bucket TEXT NOT NULL, -- 'x_post_reads' | 'ig_hashtag_window' | 'yt_search_calls'
|
||||
window_start TIMESTAMPTZ NOT NULL,
|
||||
units_used NUMERIC NOT NULL DEFAULT 0,
|
||||
units_cap NUMERIC NOT NULL,
|
||||
UNIQUE (workspace_id, provider, bucket, window_start)
|
||||
);
|
||||
```
|
||||
|
||||
**Why items are persisted, not just cached:** today's 40-item Redis cache + TTL means cursors die and "Load older" is shallow. Columns are user-curated surfaces; they need stable keyset pagination and cross-column dedup. Redis stays as the hot read-through cache in front of `feed_items`.
|
||||
|
||||
**`spec` JSONB shapes** (validated by a zod-style schema in `packages/core`; invalid spec → `422`, never stored):
|
||||
|
||||
```ts
|
||||
// platform_feed
|
||||
{ connectionId: string, kinds: ("home"|"mentions")[] } // default ["home"]
|
||||
// keyword / saved_search
|
||||
{ query: string, // 1..200 chars, trimmed; no NUL/control chars
|
||||
platforms: string[], // provider ids, e.g. ["bluesky","mastodon","youtube"]
|
||||
matchMode?: "any" } // reserved; v1 = single term set, no boolean ops
|
||||
// account_list
|
||||
{ accounts: Array<{ provider: string, handle: string }> } // 1..50 entries; handle free-text "@user@instance.social" | "@user" | "UC..." for YT
|
||||
```
|
||||
|
||||
`refresh_policy` (clamped server-side): `{ intervalMs?: 60_000..3_600_000, paused?: boolean }`. Defaults per type: `platform_feed` 60 000 (inherits today's effective cadence), `keyword` 120 000, `account_list` 180 000, `saved_search` 300 000. The clamp floor is **never below the per-connection budget-derived interval** — a user setting 60 s on a source whose budget says 300 s gets 300 s and a `meta` hint `effectiveIntervalMs`.
|
||||
|
||||
**Keyless sources:** `column_sources.connection_id` is NULL for YouTube-RSS channels, public Bluesky (`getAuthorFeed`/`searchPosts` unauthenticated), and Mastodon public/RSS reads. These draw from the **per-host budget only** (plus a global keyless concurrency cap of 2 in-flight per host), not from any connection's token bucket. This is what lets an account-list column follow YouTube/Mastodon accounts with **zero connections**.
|
||||
|
||||
### 1.3 Migration approach
|
||||
|
||||
1. **DDL migration** (additive, numbered): the five tables above. No existing table touched.
|
||||
2. **Lazy seeding (no backfill job):** on first `GET /feeds/columns` for a workspace with zero rows, the API inserts one `platform_feed` column per feed-capable visible connection (kinds `["home"]`, title `` `${displayName} · ${handle}` ``) — exactly what `liveColumns()` derives client-side today. Seeding is idempotent under the unique `(workspace_id, position)` index (transaction + `ON CONFLICT DO NOTHING` re-check).
|
||||
3. **Saved-search migration (opt-in, client-driven):** `POST /feeds/columns/import-saved-searches { terms: string[] }` creates `saved_search` columns (platforms = all search-capable connected/keyless providers). The web app calls it once when localStorage `tmbh.saved-searches` is non-empty and the server list lacks them; on success it clears localStorage and dispatches `saved-searches-change`. LocalStorage remains the fallback when the API is unreachable (sample mode) — no behavior regression offline.
|
||||
4. **Rollback story:** columns are additive; disabling the feature flag (`STREAMS_COLUMNS_ENABLED`, default on) falls the UI back to today's derivation path. `/feeds` legacy shape never changes.
|
||||
|
||||
### 1.4 API surface
|
||||
|
||||
All under the existing feeds module, RBAC `draft.view` to read; **new permission `streams.columns.manage`** (granted to Owner/Admin/Editor by default) to write. Group-visibility: a `platform_feed` column referencing a non-visible connection is rejected `403`; keyword/account columns only see what their sources legitimately fetch.
|
||||
|
||||
```
|
||||
GET /feeds/columns → { columns: ColumnView[] } (ordered, with per-source status)
|
||||
POST /feeds/columns → 201 ColumnView (validates spec, resolves column_sources)
|
||||
PATCH /feeds/columns/:id → ColumnView (title, spec, refresh_policy, position-delta)
|
||||
DELETE /feeds/columns/:id → 204 (soft delete; source items expire naturally)
|
||||
POST /feeds/columns/reorder → { columns: ColumnView[] } { orderedIds: string[] } — rewrites positions
|
||||
POST /feeds/columns/import-saved-searches → { columns: ColumnView[] }
|
||||
GET /feeds/columns/:id/items?cursor&limit(1..200,def 40) → { items: FeedItemDTO[], nextCursor, meta: ColumnMeta }
|
||||
POST /feeds/columns/:id/refresh → 202 (shares lease/cooldown/budget machinery)
|
||||
```
|
||||
|
||||
`ColumnView`:
|
||||
```ts
|
||||
{ id, type, title, position, spec, refreshPolicy, effectiveIntervalMs,
|
||||
sources: Array<{ id, provider, connectionId, sourceKind, query, actorRef,
|
||||
state, lastPolledAt, lastError?, handle?, displayName? }>,
|
||||
aggregateState: "ready"|"pending"|"stale"|"error"|"partial"|"empty"|"paused" } // worst-of, §3.5
|
||||
```
|
||||
|
||||
`FeedItemDTO` (column items endpoint; superset of today's `/feeds` item — the UI's `StreamItem` maps 1:1):
|
||||
```ts
|
||||
{ id, remoteId, provider, connectionId: string|null, sourceId,
|
||||
authorHandle, authorRef?, url?, createdAt,
|
||||
content: RichTextAst, // §5 — the ONLY render source
|
||||
plainText, media: MediaRef[], metrics: Record<string,number> }
|
||||
```
|
||||
`ColumnMeta`: `{ sources: [...same as ColumnView.sources], fetchedAt, nextCursor semantics }`.
|
||||
|
||||
**Mapping into `/feeds` serving (back-compat + one merged view):**
|
||||
- `GET /feeds` **unchanged**: merged home-timeline view over capable connections, `meta.connections`, `excludedConnections`, cursor format, `POST /feeds/refresh`. It now reads through the same `feed_items` store (cache-first) — external shape identical.
|
||||
- `GET /feeds` gains **one additive** field: `meta.columns?: { id, title, type, position, aggregateState }[]` (omitted when the feature flag is off). The "All feeds" chip and offline/sample mode keep working off the old shape.
|
||||
- Per-column serving is **only** `GET /feeds/columns/:id/items`. The merged `/feeds` never mixes column-filtered results, so the current switcher ("All feeds" / "Columns" / per-column / per-saved-term) keeps exact semantics.
|
||||
|
||||
**Cursor rule:** every cursor is the existing base64url `{at, key}` position tuple over a **deterministic sort** (`created_at_remote DESC, dedup_key DESC`). Per-column cursors are stable because they page persisted rows, not a Redis snapshot.
|
||||
|
||||
---
|
||||
|
||||
## 2. Feed acquisition strategy per platform
|
||||
|
||||
Order follows the research matrix ("Low effort, $0" → up). **Build order: Bluesky → Mastodon → YouTube → X(paid, gated) → Instagram(hashtag, gated) → TikTok(embed-only, out of scope).**
|
||||
|
||||
### 2.1 Platform decision table
|
||||
|
||||
| Platform | Keyword column | Account-list column | Mechanism | Auth / credential | Budget | Phase |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **Bluesky** | `app.bsky.feed.searchPosts` (`sort=latest`, `q`) | `app.bsky.feed.getAuthorFeed(actor)` | native REST via connector; **poll first**, jetstream later (§2.4) | **optional app password** — created via `createAppPassword`, vaulted per workspace as a *column credential*; unauthenticated mode is best-effort fallback (api.bsky.app works; design never *requires* keyless) | AppView "generous"; per-host bucket default 30 req/10 min/host; keyless concurrency cap 2 | **P2 / P3** |
|
||||
| **Mastodon** | hashtag timeline `timelines/tag/:tag` only (no global search — hard platform fact); non-tag query → provider marked `unsupported` | `accounts/:id/statuses` (public) **or** `@user.rss` (verified keyless) — prefer RSS for keyless follows, `/statuses` when a connection exists | native via existing connector | none for public/RSS; user token (existing connection) for streaming later | 300 req/5 min per account/IP → per-host bucket 10 req/10 s default (egress), per-connection 0.5 RPS cap as today | **P2 / P3** |
|
||||
| **YouTube** | Data API `search.list` (`type=video`, `publishedAfter` = last poll − slack) | **channel RSS** `feeds/videos.xml?channel_id=` (15 latest, keyless, verified) as primary; `playlistItems.list` (1 unit/50) for backfill/deep history | RSS fetcher (new shared `rss-source` reader in ingest, not a full connector) + Data API client | **API key** in vault, workspace-scoped `youtube_data` credential; RSS needs nothing | `search.list` own bucket **100 calls/day** → ledger `yt_search_calls` cap 90/day (10% headroom); `playlistItems` from shared 10k units; RSS = per-host bucket only. **30-day refresh-or-delete** on Data-API items → `expires_at`, attribution required in UI | **P2 (search) / P3 (RSS)** |
|
||||
| **X / Twitter** | `tweets/search/recent` (last 7–10 d) | `users/:id/tweets` | native connector, **behind a pay-per-use gate** | OAuth2 app Bearer; billing = operator's console credits (pay-per-use $0.005/post read; free tier has no reads) | **$ ledger**: `x_post_reads` cap per workspace (default **0 = off**); every read debited; hard stop at cap → source `budget_paused`; platform 3M reads/mo cap noted but irrelevant vs our caps | **P5 (opt-in)** |
|
||||
| **Instagram** | Graph API `ig_hasid_search` → `top_media`+`recent_media` (hashtag columns only) | **not possible** (others' accounts) — UI shows embed-card fallback for known URLs only, never auto-follow | native connector extension | Meta app + App Review; Business/Creator account connection | **30 unique hashtags / rolling 7 d** → ledger `ig_hashtag_window`; source `rate_limited` + "next tag slot frees {time}" when exhausted; `recent_media` ~50-item window → treat as shallow | **P5 (gated on App Review)** |
|
||||
| **TikTok** | none compliant (Research API = academic/nonprofit only; Commercial = ads) | embed-only for known URLs | none — **out of scope**; render link-cards | n/a | n/a | not built |
|
||||
| **Telegram / Discord / Slack** | n/a | n/a (channel-picker columns are WP5 inbox work) | existing `getFeed` home paths unchanged | existing | existing | n/a |
|
||||
| **Threads** | n/a (API is own-account only) | n/a | unchanged | n/a | n/a | n/a |
|
||||
|
||||
**Bridge policy:** no RSS-Bridge/RSSHub/Nitter in any phase of this plan. Telegram public-channel via RSS-Bridge is recorded as a possible later opt-in with legal sign-off; not designed in.
|
||||
|
||||
### 2.2 SDK interface changes (decided)
|
||||
|
||||
Extend `PlatformConnector` (additive, optional methods — existing connectors untouched):
|
||||
|
||||
```ts
|
||||
// keyword search — lifts the existing getFeed kind:"search" branches verbatim
|
||||
search?(connection | null, { query: string, cursor?: string, limit: number }): Promise<Page<FeedItem>>;
|
||||
// author timeline — new
|
||||
getUserFeed?(connection | null, { actorRef: string, cursor?: string, limit: number }): Promise<Page<FeedItem>>;
|
||||
```
|
||||
`connection | null` formalizes **keyless sources** (Bluesky public, Mastodon RSS/public, YouTube RSS). Capability flags added to `ConnectorCapabilities` + manifests: `search: boolean`, `authorFeed: boolean`. These mirror into `/feeds` `meta.connections[].capabilities` (already a string array) so the UI can gate affordances with zero new endpoints. Reddit's degenerate "search = subreddit hot" is **removed** from the search path (no `search()` implementation → `unsupported`), fixing a current semantic lie.
|
||||
|
||||
A small **`rss-source` reader** (XML→FeedItem) lives in the ingest package for YouTube/Mastodon RSS — it is not a `PlatformConnector` because it has no publish/identity surface; it registers per-host budget usage like any egress caller.
|
||||
|
||||
### 2.3 Credential & key residence (vault)
|
||||
|
||||
- **Connection credentials** (Mastodon/Bluesky/IG/X user tokens): unchanged — `vault.open/seal` per `(workspaceId, connectionId)` exactly as `feeds.dependencies.ts` does today.
|
||||
- **Column credentials (new class):** workspace-scoped secrets not tied to a connection — Bluesky app password (optional), YouTube Data API key, X app Bearer. Stored vaulted under `(workspaceId, "columns", provider)`; the poller resolves provider→credential handle at poll time. **Never** in `stream_columns.spec`, never logged, never sent to the client (the API returns only `{ provider, configured: boolean, label }`).
|
||||
- **Keyless sources carry no secret at all** — egress policy + per-host budget is the only control (which is why they're preferred where equal).
|
||||
|
||||
### 2.4 Polling vs streaming
|
||||
|
||||
- **v1 = polling everywhere.** The generalized scheduler (§3.4) with per-source `next_poll_at`, exponential backoff on failure (×2 up to 30 min), jitter ±20%.
|
||||
- **Bluesky Jetstream (P6, optional):** a single workspace-shared subscriber (`collections=app.bsky.feed.post`) maintains a **local keyword index** (FTS table) that backs keyword columns for *Bluesky only*, giving near-real-time + historical depth beyond `searchPosts` page depth. Rules: self-hostable binary or public endpoint, no auth, cursor replay on reconnect, **index rows follow the same safe-content AST + 30-day retention**, and keyword columns **fall back to `searchPosts` polling** whenever the subscriber is down (state `stale`, never `error`). Jetstream is an accelerator, never a dependency.
|
||||
- **Mastodon streaming** (user token + `read:statuses`) is the same pattern later; hashtag polling is good enough for v1.
|
||||
|
||||
### 2.5 Per-host rate budgets (reusing WP7/WP8 machinery)
|
||||
|
||||
No new limiter. The column poller:
|
||||
1. computes the target host from the source's provider+instance (Mastodon instances vary! host = the connection's instance, not `mastodon.social` globally);
|
||||
2. acquires via `egress` → `app.take_host_budget(host_id, 'monitor', n, lease_seconds)` exactly like the live-monitor pollers — so columns, monitor polls, and any other caller of the same instance share one bucket;
|
||||
3. honors `429`/`Retry-After` → cooldown key + source state `rate_limited`;
|
||||
4. keeps the existing per-connection Redis lease/cooldown (`feeds:{ws}:{conn}:…` extended with `:{columnSourceId}`) so **manual refresh cannot bypass budgets** and replicas never double-poll (identical pattern to today's `FeedsService`, keyed finer).
|
||||
|
||||
Per-provider defaults (conservative, env-overridable, `CONNECTOR_RATE_<PROVIDER>_RPS` style retained): Bluesky host 0.2 RPS sustained (keyless 0.05 RPS), Mastodon per-instance 0.1 RPS, YouTube RSS host 0.05 RPS, YouTube API search 90 calls/day ledger, X gated by spend ledger, Instagram by hashtag-window ledger. The **provider-budget ledger** (table 5) is checked *before* the host-budget debit; exhausted → `budget_paused` (distinct from `rate_limited`: no retry storm, requires user action or window rollover).
|
||||
|
||||
---
|
||||
|
||||
## 3. Ingest / search pipeline
|
||||
|
||||
### 3.1 Source resolution (column spec → column_sources)
|
||||
|
||||
On create/update of a column, a **resolution step** (no network for keyword columns; bounded network for account_list) writes `column_sources`:
|
||||
|
||||
- `platform_feed`: 1–2 sources (per kind) on the named connection.
|
||||
- `keyword`: for each `platforms[]` entry → one source per (provider, connection-or-keyless) that **declares the capability** (`search: true` or RSS-search path). Providers in `platforms[]` without capability → **no row**; the API response lists them under `spec.unsupportedPlatforms` so the editor can show "TikTok: not available" without inventing a failing source. Query adaptation happens here, once, per provider:
|
||||
- Bluesky: query as-is (full-text).
|
||||
- Mastodon: extract first `#hashtag` token; if none → this provider is `unsupported` for the column (UI states "Mastodon supports #hashtag queries only").
|
||||
- YouTube: query as-is (`search.list`, `type=video`).
|
||||
- X: query as-is (`search/recent`), only if spend ledger enabled.
|
||||
- Instagram: only if the entire query is a single hashtag; else `unsupported`.
|
||||
- `account_list`: each entry `{provider, handle}` → resolve to `actor_ref` via provider-specific resolution (`resolveHandle` for Bluesky DID, Mastodon `accounts/lookup`, YouTube channel-id via RSS/`channels.list` if key present). Resolution uses the bounded resolver pattern (≤4 in-flight/host, 10-min positive cache / 1-min negative). Unresolvable handle → source row with `state:'error', last_error:'handle_not_found'` — **it does not fail the column**.
|
||||
|
||||
### 3.2 Normalization (into the existing shape, extended)
|
||||
|
||||
Every source type funnels into one `normalize()` (today's `feeds.service.ts` validation retained: finite dates, string remoteId/handle, finite metrics; invalid items dropped + counted in source meta):
|
||||
|
||||
| Provider source | author_handle | content (AST) | media | metrics | url |
|
||||
|---|---|---|---|---|---|
|
||||
| Bluesky post | handle | facets→AST (links, mentions) | embed images/video → **proxy** | likes/reposts/replies | `bsky.app/profile/…/post/…` |
|
||||
| Mastodon status/RSS | `@user@instance` | HTML→AST via allowlist parser (strip scripts/styles/iframes; keep p/br/a/strong/em; links http/https only) | media attachments → proxy | reblogs/favourites/replies | status permalink |
|
||||
| YouTube video/RSS item | channel title | title+description→AST (plain paragraphs; links auto-linked http/https only) | thumbnail → proxy | viewCount/likeCount (API only) | watch URL |
|
||||
| X tweet | handle | entities→AST | media → proxy | likes/reposts/replies | tweet permalink |
|
||||
| IG hashtag media | username | caption→AST | media → proxy | like/comment count | permalink |
|
||||
|
||||
`plain_text` is derived at ingest (AST→text) so saved-search-style client filtering and the column preview keep working. **`media[].proxyUrl` is always our media proxy URL** (WP3 media path) — raw provider CDN URLs never reach the browser.
|
||||
|
||||
### 3.3 Dedup & identity
|
||||
|
||||
- `dedup_key = sha256(workspace_id | provider | canonical_remote_id)` where `canonical_remote_id` = `remoteId` lower-cased and, for handle-bearing providers, `{host}/{remoteId}` with the IDNA-canonical host (same canonicalization rule as WP7 `remote_ref`: lowercase host, punycode, default-port stripped). One object reached via two connections (or a connection + keyless) collapses to one row; the second ingest updates metrics/fetched_at in place (`ON CONFLICT … DO UPDATE` on metrics + fetched_at).
|
||||
- **Cross-provider dedup is explicitly NOT attempted** (no fuzzy matching) — an item is provider-scoped. The UI dedups by `streamItemKey` client-side as today.
|
||||
- `feed_item_columns` is filled by fan-out at ingest: an item lands in every column that shares that `column_source` identity — i.e., columns referencing the same (provider, actor_ref) or (provider, query) share ingested rows, so following the same account in two columns costs **one** poll (mirrors the WP8 "one remote live, three channels, one poller" rule).
|
||||
|
||||
### 3.4 Scheduler (generalizing FeedsService)
|
||||
|
||||
The existing 5 s discovery loop stays; `pollAll()` becomes "poll due `column_sources`" (the legacy per-connection home poll is literally a `platform_feed` source, so old behavior is the degenerate case):
|
||||
- select due sources (`next_poll_at <= now`, `state NOT IN ('disabled')`), `FEED_CONCURRENCY=4` waiter queue retained;
|
||||
- per-source flow: lease → resolve credential (vault or keyless) → budget/ledger check → egress call via connector `getFeed/search/getUserFeed` or RSS reader → normalize → upsert `feed_items` + fan-out join → write hot Redis slice (`feeds:{ws}:{column}:items` head, TTL = interval×1.2+30 s) → update source state/`next_poll_at` (success: interval; failure: backoff ×2 cap 30 min, `consecutive_failures`≥3 → `error`);
|
||||
- `RateLimitError.retryAfterMs` → cooldown, state `rate_limited`; ledger empty → `budget_paused`;
|
||||
- manual `POST /feeds/columns/:id/refresh` → same path (202; cooldown may make it a no-op — response carries `effectiveNextPollAt`).
|
||||
|
||||
### 3.5 Stale / failed / empty behavior in a column
|
||||
|
||||
Per-source `state` (§1.2) aggregates to `aggregateState` shown in the column header:
|
||||
- `ready` — all sources fresh (< 2× effective interval).
|
||||
- `stale` — any source's `last_polled_at` older than 2× interval but data present → header "Updated HH:MM" + amber dot; **old items remain visible** (persisted store — the big UX win over today's TTL'd cache).
|
||||
- `error` — all sources failed ≥3× → red dot + per-source error line (from `last_error`, provider message scrubbed); last-known items stay visible with "showing last data from …".
|
||||
- `partial` — some sources fine, some error/stale → amber; per-source chips in column menu.
|
||||
- `rate_limited` / `budget_paused` — distinct icon + actionable copy ("X reads paused: budget reached — raise cap in settings" / "Instagram hashtag slots full until {time}").
|
||||
- `empty` — source healthy, zero items ever → per-platform empty-state copy ("No public posts match #acampada26 on Mastodon yet").
|
||||
- `unsupported` — never a source row; surfaced in editor only.
|
||||
Deleting the underlying connection → its sources flip `disabled`; column survives with remaining sources; a `platform_feed` column whose connection dies becomes `error` with a reconnect CTA (not auto-deleted).
|
||||
|
||||
**Retention / expiry sweeper:** daily job deletes `feed_items WHERE expires_at < now()` (YouTube Data-API sourced) and everything older than 30 days regardless of source (matches the strictest ToS we're under; RSS items keep 30 d too for uniformity). Deletion is the default; re-polling a live column continuously refreshes the visible window, satisfying "refresh or delete".
|
||||
|
||||
---
|
||||
|
||||
## 4. UI / UX spec (Streams page integration)
|
||||
|
||||
### 4.1 Model change in the web app
|
||||
|
||||
`lib/streams-data.ts` gains `columnViewsToColumns(ColumnView + items)` producing today's `ColumnDef` (key = column id, title, platformId from dominant provider, staleness from `aggregateState`/`lastPolledAt`). The sample/offline path (`MOCK_COLUMN_DEFS` + fixtures) is untouched. Data flow:
|
||||
|
||||
- mount → `GET /feeds/columns` (once, plus after any mutation) → for the visible columns (and lazily the next offscreen ones) `GET /feeds/columns/:id/items` per column, 30 s polling retained per visible column (staggered), merged into `ColumnDef[]`.
|
||||
- "All feeds" chip → unchanged `GET /feeds` merged view. "Columns" chip → column array as above. Per-column chips → single column. Saved-term chips now resolve to their migrated server columns (id `saved:{term}` → real id after migration; localStorage terms still render offline).
|
||||
|
||||
### 4.2 Column editor
|
||||
|
||||
- **"+ Column" button** opens a right-side drawer (desktop) / bottom sheet (mobile): type selector (Platform feed · Keyword · Accounts · Saved search), then type-specific fields:
|
||||
- *Platform feed*: connection picker (visible connections only, feed-capable, capability badge), kinds toggles (Home / Mentions).
|
||||
- *Keyword*: query input with live per-platform feasibility hints as you type ("Bluesky ✓ full-text · Mastodon #hashtag only · YouTube ✓ · TikTok ✗"), platform multi-select chips (search-capable enabled, others visible-but-disabled with reason tooltip), interval select (1/2/5/10/30 min — clamped label shows effective value).
|
||||
- *Accounts*: multi-entry list (paste handles one per line or chip input; provider inferred from handle shape `@x@bsky.social`/`@x@mastodon.social`/`UC…`; `@x` on TikTok/IG → blocked with reason), live resolution status per row (✓ resolved / ✗ not found / ⏳ resolving), 50-entry cap enforced client+server.
|
||||
- *Saved search*: term input; identical to keyword with `origin` label.
|
||||
- **Edit**: column header "⋯" menu → same drawer prefilled; changing spec re-resolves sources (existing items for retained sources persist; removed sources' items age out via retention, no mass delete).
|
||||
- **Reorder**: drag handle on column headers (desktop) / up-down arrows in the columns sheet (mobile) → `POST /reorder`.
|
||||
- **Remove**: "⋯" → Remove (soft delete; confirm if the column has fresh data) and Pause (refresh_policy.paused).
|
||||
- Optimistic UI on create/reorder/remove with server reconciliation; failure → toast + revert.
|
||||
|
||||
### 4.3 States, keyboard, mobile
|
||||
|
||||
- **Empty states:** no columns → existing `EmptyState` + "Add column" CTA. Column with no items → per-source empty copy (§3.5). No connections at all → today's sample-mode fixtures with a "Sample data — connect an account" banner (unchanged).
|
||||
- **Error states:** header dot + per-source detail in the column menu; a "source status" strip lists each source (provider, handle/query, state, last polled, error).
|
||||
- **Sample state:** identical to today (API unreachable / 0 configured feeds → fixtures); the editor still opens but Create is disabled with "offline — changes need the server".
|
||||
- **Keyboard:** j/k flat navigation retained; `c` opens the column editor; `1..9` jumps to column n.
|
||||
- **Mobile:** single-column reading with the existing `SourceChips` bar (All feeds · Columns · each column · saved terms); Columns mode = swipeable pager (one column per screen, `PullToRefresh` triggers the column refresh endpoint); editor as bottom sheet; drag-reorder replaced by arrows. Desktop keeps the multi-column wall.
|
||||
|
||||
### 4.4 Back-compat guarantee
|
||||
|
||||
The source switcher's four modes (all / columns / per-column / saved-term) all survive with identical semantics; when the flag is off or seeding hasn't run, the page behaves exactly as the current build (derived columns from `/feeds` meta). Saved-search localStorage remains the offline fallback.
|
||||
|
||||
---
|
||||
|
||||
## 5. Security design
|
||||
|
||||
1. **safe-content as the only render path.** `FeedItemDTO.content` is the safe-content **typed rich-text AST**, produced **at ingestion** (server-side): Mastodon HTML → AST via the allowlist parser inside the connector/ingest boundary; Bluesky facets, YouTube/RSS text, X entities → AST builders. Raw provider HTML **never** crosses the API boundary; there is no field a client could accidentally render. The web `StreamColumn`/`PostCard` render exclusively `<SafeContent ast={item.content}/>`; the repo lint rule fails on `dangerouslySetInnerHTML` (existing WP3 acceptance) and additionally on any new string prop named `html`.
|
||||
2. **Links:** AST link nodes are http/https-only (other schemes dropped at ingest, not CSS-hidden); unsafe-scheme links render as inert text, not clickable.
|
||||
3. **Media:** `media[].proxyUrl` only — provider CDN URLs are proxied through the media path (CSP `img-src`/`media-src` never include provider origins). No `<iframe>` embeds for IG/TikTok in v1 (an embed-only fallback, if ever added, gets its own sandboxed frame + CSP review — explicitly out of scope here).
|
||||
4. **SSRF:** no user-supplied URL is ever fetched. Account handles and queries go through provider APIs / the bounded resolver, all via `egress` (scheme allowlist, private-range denial, resolve-once pinning). YouTube/Mastodon RSS host URLs are constructed from validated channel-id/handle forms, never pasted URLs.
|
||||
5. **Secrets:** vault-only (§2.3); API responses expose `configured: boolean` at most. Ledger/credential management endpoints are admin-gated.
|
||||
6. **Authz:** read = `draft.view` (+ group visibility for connection-backed sources); write = `streams.columns.manage`; every route workspace-scoped and RLS-checked; import-saved-searches is rate-limited per user (it creates rows).
|
||||
7. **Cost safety:** X/IG gated ledgers with per-workspace caps default-off/pending-review; a keyword column targeting X shows a **pre-flight cost estimate** (posts × $0.005) before creation.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phased build plan (agent-executable slices)
|
||||
|
||||
Tiers: **flash** = small fully-specified slice; **glm** = medium feature slice; **sol** = large/complex or cross-cutting. Phases strictly dependency-ordered; marked slices parallelize.
|
||||
|
||||
### P0 — Foundations (flash ×3, parallel)
|
||||
- **P0a Schema & validation:** migrations for the 5 tables; `spec`/`refresh_policy` zod schemas + clamps in `packages/core`; unit tests for clamps and invalid-spec rejection. **AC:** migration applies on a populated DB; invalid specs rejected `422`; `(workspace_id, position)` unique holds under concurrent inserts.
|
||||
- **P0b SDK contract:** optional `search()` / `getUserFeed()` (`connection | null`), capability flags `search`/`authorFeed`, manifest validation, contract-test fixtures. **AC:** extended contract tests pass; existing connectors compile untouched; capability strings surface in `/feeds` meta.
|
||||
- **P0c FeedItem extension + normalize v2:** extended normalization (author_ref, url, AST content, proxy media refs, expires_at) with per-connector fixtures incl. hostile HTML (script tag, `javascript:` link, iframe → AST contains none). **AC:** malicious fixture yields an AST with zero raw remnants; plain_text derivation stable.
|
||||
|
||||
### P1 — Column CRUD + platform_feed persistence (glm)
|
||||
Tables wired into the API; `GET/POST/PATCH/DELETE /feeds/columns`, `reorder`, `import-saved-searches`; lazy seeding; RBAC `streams.columns.manage`; `GET /feeds/columns/:id/items` keyset-cursored over `feed_items`; scheduler generalization (due-sources loop, per-source lease/cooldown, backoff — legacy home poll becomes the degenerate `platform_feed` source); `/feeds` served cache-first from the store, external shape unchanged + additive `meta.columns`.
|
||||
**AC:** seeded workspace reproduces today's columns exactly (title/key/staleness); CRUD + reorder RBAC-enforced (403 for ungranted roles); manual refresh cannot bypass cooldown (existing test extended to column endpoints); cursor survives a Redis flush; re-running seeding creates no duplicates.
|
||||
|
||||
### P2 — Keyword columns: Bluesky + Mastodon hashtag + YouTube search (glm ×2 + sol + parallel UI glm)
|
||||
- **P2a (glm):** Bluesky `search()` (optional app-password credential, keyless fallback), Mastodon hashtag `search()` (+ explicit `unsupported` for non-tag queries), query-adaptation resolver, keyword column creation flow, per-source states.
|
||||
- **P2b (glm):** YouTube Data-API `search()` client, `yt_search_calls` ledger (cap 90/day, pre-check, `budget_paused`), `publishedAfter` sliding window, `expires_at` = +30 d, attribution line in UI.
|
||||
- **P2c (sol):** ingest fan-out/dedup correctness (two columns sharing a query → one poll, shared rows; one object via two connections → one row), retention sweeper.
|
||||
- **UI slice (glm, parallel after P1):** column editor drawer (keyword type first), per-platform feasibility hints, source-status strip, mobile sheet.
|
||||
**AC:** "acampada26" column on Bluesky+Mastodon+YouTube returns merged, deduped, AST-rendered items; exhausting the API key quota → YouTube source `budget_paused`, others unaffected; a `#tag`-less query shows Mastodon as unsupported (never an error); Playwright malicious-content corpus renders inert through the column UI; YouTube items auto-expire at 30 d.
|
||||
|
||||
### P3 — Account-list columns (glm ×2)
|
||||
- **P3a:** `getUserFeed()` for Bluesky (`getAuthorFeed`) + Mastodon (`/accounts/:id/statuses` with connection, RSS keyless) + YouTube channel RSS via the `rss-source` reader; bounded handle resolver with positive/negative caching.
|
||||
- **P3b:** account-list editor UX (paste list, live resolution, 50-cap), mixed connection/keyless column, connection-loss behavior (`disabled`/reconnect CTA).
|
||||
**AC:** a 50-account column across 3 platforms polls each host within budget (mock server counts requests); unresolvable handle → row error, column healthy; revoking the Mastodon connection flips its sources to `disabled` while Bluesky keyless sources are unaffected; the same account in two columns = one poll.
|
||||
|
||||
### P4 — Saved-search migration + polish (flash + glm)
|
||||
localStorage import endpoint wired into the page; offline fallback preserved; "Layout" stub becomes real (column widths persisted client-side); keyboard `c`/`1..9`; empty/error copy pass.
|
||||
**AC:** a fresh browser with legacy localStorage sees identical saved columns server-side, localStorage cleared, no duplicates on double-run; feature flag off → behavior identical to the pre-columns build.
|
||||
|
||||
### P5 — Gated paid/limited platforms (sol ×2, parallel after P2's ledger exists)
|
||||
- **P5a X:** connector `search()`/`getUserFeed()` vs `search/recent` + `users/:id/tweets`, spend ledger ($/post-read, per-workspace cap default 0=off), pre-flight cost estimate UI, `budget_paused` UX.
|
||||
- **P5b Instagram:** Graph-API hashtag sources (30-tags/7d ledger window math, `top_media`+`recent_media`, shallow-window handling), App-Review notes; account-follow explicitly surfaced as impossible with embed-card fallback design.
|
||||
**AC:** an X column with a $2 cap stops exactly at the cap (mock billing counts reads) and resumes only on explicit cap raise; the 31st unique IG hashtag in a window → `rate_limited` with next-slot time; both default-disabled via env.
|
||||
|
||||
### P6 — Jetstream accelerator + realtime (sol, optional, after P2a)
|
||||
Workspace-shared Jetstream subscriber → local keyword index (same AST/retention); keyword columns prefer the index with `searchPosts` polling fallback; WS push of column item deltas via the existing `RealtimeModule` (versioned, last-value-wins per column).
|
||||
**AC:** killing the subscriber degrades columns to polling (`stale`, not `error`) within one interval; a new matching post appears < 5 s locally; index rows expire at 30 d; no duplicate rows vs the polling path.
|
||||
|
||||
**Dependency chain:** P0 → P1 → P2 → P3 → P4; P5 ∥ after P2; P6 after P2a. Malicious-content Playwright acceptance runs in every phase touching rendering or ingest (P2, P3, P4 at minimum).
|
||||
88
docs/design/2026-10-09-feature-set.md
Normal file
88
docs/design/2026-10-09-feature-set.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# TweetDeck on Steroids — the full feature set
|
||||
|
||||
**A self-hosted, multi-platform broadcast newsroom for collectives: a column wall that watches every platform at once, chats that organize themselves, votes that turn into posts, and a live channel behind it all.**
|
||||
|
||||
TweetDeck died when X locked its doors. We built what should have replaced it — and went much further. Watch everything, decide together, publish everywhere, mix live, and keep an archive no platform can take away. Self-hosted. No algorithm in between. No platform holding your keys.
|
||||
|
||||
---
|
||||
|
||||
## The column wall — every platform in one wall
|
||||
|
||||
Live columns watch all your connected accounts at once: Mastodon, Telegram, Bluesky, PeerTube, Twitch, and more. The whole collective sees the same picture at the same moment.
|
||||
|
||||
Every post is rendered safe (hostile HTML is parsed into a typed format — scripts, tracking pixels and trick links are stripped at ingest and tested against an attack corpus), media is proxied through your server, and duplicates are deduplicated into the archive automatically.
|
||||
|
||||
Columns are becoming **editable**: a column can be a platform feed, a **keyword search across platforms** (one query like "acampada26" pulling from every network that allows it), a curated **list of accounts across platforms**, or a saved search. Columns persist per group, can be reordered, paused, and shared.
|
||||
|
||||
## Collaborative chats
|
||||
|
||||
Chats from every platform bridge into one shared overlay and one web chat — viewers, contributors and moderators in the same room, with flood governors and moderation tools keeping it usable during action.
|
||||
|
||||
The chat is a coordination surface, not just a mirror: it sits next to the live program, next to the file store, next to the drafts. What people say in the chat drives what happens in the system.
|
||||
|
||||
## Voting and decision making — in the chat, automatically
|
||||
|
||||
Run a poll inside any bridged chat: **"post this?"** — and set the rules for what counts as a pass. Configurable per poll:
|
||||
|
||||
- approved by **x of y admins**
|
||||
- approved by **x of y participants**
|
||||
- mixed quorum (admins + participants), veto rights, timeboxes, anonymous or named voting
|
||||
|
||||
Votes come in through Telegram buttons and the web overlay — live tally bars, quorum meters, countdowns. Ineligible votes are stored but never counted; vetoes fail the poll instantly; revotes are supported; everything lands in a tamper-evident audit trail.
|
||||
|
||||
**When the quorum passes, the posting bot fires automatically** — the approved content goes out to the channel, topic, or account the poll was about. No admin phone in the loop. Decisions happen at the speed of the chat, with the full trail preserved.
|
||||
|
||||
## Configurable posting bots — every account, its own rules
|
||||
|
||||
Every connected account carries its own settable rules for what it takes to post:
|
||||
|
||||
- **immediate** — no gate, post as today
|
||||
- **admin-only** — an admin's approval is the whole quorum
|
||||
- **quorum** — configurable thresholds of admins and/or participants
|
||||
- per **topic** and per **content type** overrides, dry-run shadow mode, timeboxes, reset-on-edit
|
||||
|
||||
A rules editor shows the resolved effective policy before you save it — and a blocked queue shows every post waiting on its quorum with a live gap meter. Nothing is ambiguous about why something did or didn't go out.
|
||||
|
||||
## Subchats and topics — structure that follows the work
|
||||
|
||||
Forum topics become **subchats**: working groups inside a channel, created and managed by the bot. Accounts are **assigned to topics** — the press crew on the press topic, the stream crew on the stream topic — and each topic carries its own posting rules and its own view of the store. Topics can be bound to broadcast channels, so a topic isn't just conversation: it's a control surface.
|
||||
|
||||
## Chat-linked files — the store organizes itself
|
||||
|
||||
Files posted into the chat land in the **group's file store automatically** — content-addressed, encrypted, deduplicated (the same clip shared five times is stored once).
|
||||
|
||||
Admins organize: assign items to folders, copy them across subfolder trees, refile by type or date — from **chat commands** (`/ls`, `/find`, `/move`) or from the **mobile file screen** that lives next to the chat, with deep links from bot messages straight into the filtered folder view.
|
||||
|
||||
Bot listings, search across the whole library, and an audit of every move: the file storage stays organized because the people in the chat keep it organized, from their phones.
|
||||
|
||||
## Cross-amplification and retweets — one approval, every platform
|
||||
|
||||
One approved draft fans out to every connected account, native to each platform. Where a platform has native quoting (Mastodon quote posts, Bluesky, Telegram forwards), it's used. Where it doesn't, the system generates a proper attributed quote — author, instance, link back — never a silent copy-paste.
|
||||
|
||||
Amplifying a sister collective's post keeps their names on it by default. Scheduling, per-account timing, and rate awareness come along for free.
|
||||
|
||||
## Live video mixing and channel production
|
||||
|
||||
The **Control Room** runs in the browser: a multiview grid of every feed, TAKE/CUT between preview and program, scene layouts (single, picture-in-picture, two-up, splits, grid), lower-thirds, scrolling chyrons, clocks and logos as overlays, per-source audio mixing with a tally of what's hot.
|
||||
|
||||
**Field streamers** push cameras from a phone straight to your server. The director mixes in the browser; the program goes to your **PeerTube channel**; and when no one is live, the channel keeps broadcasting a scheduled playlist so the station never goes dark.
|
||||
|
||||
**Auto-Director** switches sources by priority rules — a streamer goes down, the next source fades in, the fallback slate catches anything else.
|
||||
|
||||
**Distributed transcode workers:** laptops and GPUs join as render workers over Tailscale — an RTX 3090 laptop is already transcoding real PeerTube jobs with hardware encoding, verified end to end. More workers join the pool as load rises.
|
||||
|
||||
## The backbone
|
||||
|
||||
- **Encrypted vault** for every platform credential — the database never sees a plaintext key.
|
||||
- **Hash-chained audit log** — every editorial action tamper-evident, forever.
|
||||
- **DST-proof scheduling** — the occurrence engine survives clock changes; two scheduled posts that collapse onto the same instant fire exactly once.
|
||||
- **Media sandbox** — every FFmpeg job runs in a per-job jail: unprivileged, no network, resource-capped.
|
||||
- **An archive that can't be purged** — content-addressed, encrypted, deduplicated. Deleted threads stay citable; yesterday's footage is tomorrow's playlist.
|
||||
- **Teams without password-sharing** — roles per account, invite links, people can remove their own access when they move on.
|
||||
- **Self-hosted end to end** — no corporate cloud, no accounts needed to run the system itself.
|
||||
|
||||
---
|
||||
|
||||
## We're looking for testers and intake hackers
|
||||
|
||||
Test the whole flow: post through approvals, run the column wall against your real feeds, mix a stream, break the scheduler across timezones, vote in the chats. And help us on the **hard problem: pulling from the locked platforms** — per-URL fetchers, community-submitted links, bridge services, archive harvesters. If you've built scrapers or run intake pipelines, we want your design input.
|
||||
632
docs/design/2026-10-09-transcode-pool-design.md
Normal file
632
docs/design/2026-10-09-transcode-pool-design.md
Normal file
|
|
@ -0,0 +1,632 @@
|
|||
# NVIDIA GPU Extension + Dynamic Pool Scaling — Design & Research
|
||||
PeerTube transcoding worker cluster · design wave · 2026-10-09
|
||||
|
||||
Companion to the Intel QSV worker implementation landing in
|
||||
`infra/peertube-workers/` (the enrolled worker laptop). This document depends **only on the
|
||||
Intel worker's CONTRACT** (official `@peertube/peertube-runner`, per-worker
|
||||
admission controller, pluggable capability probe, status JSON, drain control,
|
||||
fail-closed hardware rule), never on its internals.
|
||||
|
||||
Empirical anchors in this doc are marked **[E]** and were measured on this
|
||||
machine on 2026-10-09: WSL2 host, 2× NVIDIA GeForce RTX 3090 (Ampere GA102),
|
||||
Windows-side driver 595.58.03 / CUDA 13.2, Debian 12 ffmpeg 5.1.9
|
||||
(`0+deb12u1`, libmfx build), 56-core Xeon E5-2690 v4, 128 GiB RAM. Numbers not
|
||||
marked [E] are from NVIDIA primary documentation (Video Codec SDK 13.x
|
||||
support matrix, NVENC help page, driver release notes, CUDA-on-WSL guide) as
|
||||
recalled; **no web access was available in this session, so treat
|
||||
vendor-doc-derived figures as "verify against the live matrix when the first
|
||||
NVIDIA box is onboarded"** — every one of them is also re-derived at install
|
||||
time by the probe (§5.3), so a stale number here degrades to a warning, never
|
||||
to a wrong production decision.
|
||||
|
||||
---
|
||||
|
||||
## 0. Executive summary
|
||||
|
||||
1. NVENC on GeForce is no longer meaningfully session-capped for our use
|
||||
case: the driver limit is **8 concurrent encode sessions per GPU** since
|
||||
driver branch 536 (mid-2023); professional/datacenter boards have **no
|
||||
driver cap**. Our per-worker concurrency will be bounded by VRAM,
|
||||
NVDEC+filter throughput and bandwidth long before 8 sessions matter —
|
||||
but the cap must still be tracked, because PeerTube fans one job out to
|
||||
many resolution renditions and a 5-rendition job on one GPU *is* 5
|
||||
sessions.
|
||||
2. The hardware rule generalizes cleanly: the same fail-closed contract that
|
||||
keeps QSV from falling back to x264 keeps NVENC from falling back to x264.
|
||||
The only NVIDIA-specific additions are device selection (`-gpu N`),
|
||||
driver/kernel-module installation, session accounting, and VRAM floors.
|
||||
3. **Do not put NVIDIA workers on WSL2.** [E] The API surface exists
|
||||
(`h264_nvenc`/`hevc_nvenc` registered, `libnvidia-encode.so.595.58.03`
|
||||
present) but the GPU is shared with the Windows host, NVML encoder
|
||||
queries are reduced, and there is no `/dev/dri`, no persistence-mode
|
||||
control and no isolation from Windows-side VRAM consumers. On this very
|
||||
box NVENC is currently unusable because two co-tenant Windows processes
|
||||
hold 23.8/24 GiB on each GPU. WSL2 is a dev/probe platform; production
|
||||
NVIDIA workers are native Linux.
|
||||
4. Dynamic pool scaling does **not** require touching PeerTube's scheduler.
|
||||
Runners register once and persist server-side; "joining the pool" is
|
||||
`systemctl start` + admission AVAILABLE, "leaving" is graceful drain +
|
||||
optional `systemctl stop`. The pool manager nudges *which* runners are
|
||||
eligible (and at what concurrency), PeerTube still decides job
|
||||
assignment. Mixed-vendor preference is expressed the same way: selective
|
||||
availability + concurrency, not scheduler surgery.
|
||||
5. The one interface the Intel implementation must get right **now**: the
|
||||
capability probe must be a *mandatory, fail-closed, versioned* plugin
|
||||
interface, and the status JSON must be vendor-generic with a `gpu.vendor`
|
||||
discriminator from day one.
|
||||
|
||||
---
|
||||
|
||||
## 1. NVIDIA capability research (2026)
|
||||
|
||||
### 1.1 NVENC concurrent-session limits
|
||||
|
||||
| GPU class | Driver-imposed concurrent NVENC sessions | Notes |
|
||||
|---|---|---|
|
||||
| GeForce (consumer) | **8** per GPU | Raised 3→5 in 530.41.03/531.29 (2023-03), 5→8 in branch 536 (2023-07). Applies per GPU, across all processes; an ffmpeg process using `-threads` still counts as 1 session per encode context. |
|
||||
| Professional / workstation (RTX A-series, RTX 5000/6000 Ada, RTX Pro Blackwell; ex-Quadro) | **No driver cap** | Bounded only by hardware/VRAM. |
|
||||
| Datacenter media-class (T4, A2, L4, L40, L40S) | **No driver cap** | These are the right datacenter parts for transcoding. |
|
||||
| Datacenter compute-class (A100, H100, H200, B100/B200) | **0 — no NVENC at all** | Compute dies ship no encode engine (and Hopper/A100 have reduced/absent NVDEC). Never buy these for a transcode pool. |
|
||||
|
||||
Design consequences:
|
||||
|
||||
- The cap counts **encode sessions**, not processes and not ffmpeg
|
||||
"engines". A single PeerTube VOD job that produces 4 renditions from one
|
||||
decode can hold 4 encode sessions (PeerTube today typically runs one
|
||||
ffmpeg per rendition, so it holds 4 decode sessions too — see §3.4).
|
||||
- Session pressure must be *measured*, not assumed from the table: NVML
|
||||
exposes `nvmlDeviceGetEncoderCapacity` / encoder-session counts on
|
||||
bare-metal Linux, but **[E]** on WSL2 the capacity query returned a
|
||||
non-success code in our ctypes probe, so the probe design (§5.3) treats
|
||||
NVML as best-effort with a driver-class lookup table and our own
|
||||
accounting as ground truth.
|
||||
|
||||
### 1.2 NVENC encode support matrix by generation
|
||||
|
||||
Generational numbering below is the commonly used NVENC-generation scheme
|
||||
(1=Kepler … 7=Blackwell); NVIDIA's own docs speak of "Turing NVENC",
|
||||
"Ampere NVENC", etc. Quality-relevant jumps: Turing (large HEVC/H.264
|
||||
quality+lookahead gains) and Ada (AV1 + HEVC 4:4:4).
|
||||
|
||||
| NVENC generation (arch) | Example parts | H.264 | HEVC 8-bit | HEVC 10-bit | HEVC B-frames | HEVC 4:4:4 | AV1 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Gen 1 (Kepler) | GTX 680 | ✓ | — | — | — | — | — |
|
||||
| Gen 2 (Maxwell) | GTX 750Ti/970; GM206 (950/960) | ✓ | GM206 only | — | — | — | — |
|
||||
| Gen 3 (Pascal) | GTX 10xx, Quadro P | ✓ | ✓ | ✓ | — | — | — |
|
||||
| Gen 4 (Turing) | GTX 16xx, RTX 20xx, T4 | ✓ | ✓ | ✓ | ✓ | — | — |
|
||||
| Gen 5 (Ampere) | RTX 30xx, A-series | ✓ | ✓ | ✓ | ✓ | ✓ (Pro/A-series) | — |
|
||||
| Gen 6 (Ada) | RTX 40xx, L4, L40/L40S | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| Gen 7 (Blackwell) | RTX 50xx, RTX Pro | ✓ | ✓ | ✓ | ✓ | ✓ + 4:2:2 | ✓ (+UHQ) |
|
||||
|
||||
**[E]** Consistency check on this box (RTX 3090 = Ampere = gen 5):
|
||||
ffmpeg 5.1.9 lists `h264_nvenc` and `hevc_nvenc` and does **not** list
|
||||
`av1_nvenc`; `av1_cuvid` decode **is** listed. Matches the matrix exactly
|
||||
(Ampere: AV1 decode yes, AV1 encode no).
|
||||
|
||||
### 1.3 NVDEC decode support matrix
|
||||
|
||||
| NVDEC generation (arch) | H.264 | HEVC 8/10-bit | VP9 | HEVC 4:4:4 | AV1 |
|
||||
|---|---|---|---|---|---|
|
||||
| Maxwell | ✓ | GM206: 8-bit only | — | — | — |
|
||||
| Pascal | ✓ | ✓ | ✓ | — | — |
|
||||
| Turing | ✓ | ✓ | ✓ 10/12-bit | ✓ | — |
|
||||
| Ampere | ✓ | ✓ | ✓ | ✓ | ✓ 8/10-bit |
|
||||
| Ada | ✓ | ✓ | ✓ | ✓ | ✓ (incl. higher bit depths) |
|
||||
| Blackwell | ✓ | ✓ (+4:2:2) | ✓ | ✓ | ✓ (+4:4:4/12-bit) |
|
||||
|
||||
Practical rule for the pool: **any card we would actually buy (Turing or
|
||||
newer) decodes H.264/HEVC/VP9; AV1 decode needs Ampere+; AV1 *encode* needs
|
||||
Ada+**. An Ampere pool (like this box) can ingest AV1 sources and emit
|
||||
H.264/HEVC, which fits the stated "H.264 primary, HEVC secondary, no AV1
|
||||
output" policy with headroom.
|
||||
|
||||
### 1.4 Linux driver stack
|
||||
|
||||
- **Open vs proprietary kernel modules**: NVIDIA's open kernel modules
|
||||
(`nvidia-open` / open-gpu-kernel-modules) are the **default recommended
|
||||
path for Turing+ since the 560-series**; proprietary remains for
|
||||
pre-Turing and a few feature gaps. All realistic pool hardware (Turing+)
|
||||
→ use open modules. Blackwell effectively requires them.
|
||||
- 2026 driver branches: current feature branches run in the 5xx range;
|
||||
**[E]** this host's Windows-side driver is 595.58.03 (CUDA 13.2). Debian
|
||||
packages: `nvidia-open-kernel-dkms` (or prebuilt `linux-image-amd64` +
|
||||
signed modules), `nvidia-driver-libs`, `nvidia-smi`, `nvidia-utils`;
|
||||
firmware from `non-free-firmware`. Keep driver version **pinned per pool
|
||||
fleet** and record `driver + kernel + module flavor` in the status JSON.
|
||||
- Userspace pieces that matter to us: `libnvidia-encode.so.1` (NVENC),
|
||||
CUDA driver (`libcuda.so.1`) used by ffmpeg's nvenc/cuvid code paths,
|
||||
`libnvidia-ml.so.1` (NVML for nvidia-smi). ffmpeg needs no CUDA toolkit
|
||||
— only the driver libs. (A statically-built ffmpeg with `--enable-ffnvcodec`
|
||||
/ `--enable-cuda-nvcc` is only needed for `scale_npp`; `scale_cuda` is
|
||||
covered by the header-only ffnvcodec path.)
|
||||
- Access control: user in `video` group (or a dedicated worker user with a
|
||||
udev/tagged device ACL); `nvidia-persistenced` (or
|
||||
`NVreg_PreserveVideoMemoryAllocations=1`) to avoid multi-second GPU
|
||||
re-init between jobs on an idle pool box — relevant to dynamic scale-up
|
||||
latency (§3.7).
|
||||
- MIG/compute-mode: leave compute mode `DEFAULT` **[E]**; nothing to
|
||||
configure, but note it in the probe report.
|
||||
|
||||
### 1.5 WSL2: CUDA/NVENC status
|
||||
|
||||
What works [E, this host]:
|
||||
- CUDA user-mode stack present via `/usr/lib/x86_64-linux-gnu/libnvidia-*.so`
|
||||
(WSL-provided, version-matched to the Windows driver).
|
||||
- ffmpeg registers `h264_nvenc`, `hevc_nvenc`, `*_cuvid` decoders and the
|
||||
`cuda` hwaccel; `libnvidia-encode.so.595.58.03` is installed.
|
||||
- `nvidia-smi` works (2 GPUs visible, temps, clocks, throttle counters).
|
||||
|
||||
What breaks it (all [E] on this host):
|
||||
- **No isolation from Windows GPU consumers.** Two Windows processes hold
|
||||
23 856 MiB of each 24 GB GPU → any CUDA context creation fails
|
||||
(`CUDA_ERROR_OUT_OF_MEMORY`), so `h264_nvenc` init fails with *"No capable
|
||||
devices found"* and `h264_cuvid` device setup fails. This is the textbook
|
||||
`GPU_UNAVAILABLE` state, and it is caused by another tenant, not by the
|
||||
driver.
|
||||
- **Reduced NVML**: encoder-capacity/session queries returned non-success
|
||||
in a ctypes probe; `nvidia-smi -q` shows Encoder Stats fields that cannot
|
||||
be validated.
|
||||
- **No `/dev/dri`** [E: directory absent/empty] → no VAAPI/QSV fallback on
|
||||
the same box, no `intel_gpu_top`-style counters, no display-path controls.
|
||||
- No control of persistence mode, ECC, or power limits from inside WSL.
|
||||
|
||||
Verdict: **transcoding in WSL2 is feasible in the "API present, demo works"
|
||||
sense but is not a production worker platform** — the GPU is a shared
|
||||
resource with the Windows host, NVML is limited, and admission signals are
|
||||
unreliable. Use WSL2 to develop the `nvidia-nvenc` probe; run production
|
||||
workers on native Linux (or pass a whole GPU to a VM with VFIO if the box
|
||||
must stay Windows-adjacent).
|
||||
|
||||
### 1.6 FFmpeg flags: the NVIDIA pipeline
|
||||
|
||||
The NVIDIA analogue of `hw decode → hw frames → hw scale → hw encode`:
|
||||
|
||||
```bash
|
||||
# full hardware pipeline, frames never leave the GPU
|
||||
ffmpeg -y -init_hw_device cuda=cu:0 -filter_hw_device cu \
|
||||
-hwaccel cuda -hwaccel_output_format cuda -c:v h264_cuvid -i INPUT \
|
||||
-vf scale_cuda=1280:720:format=yuv420p \
|
||||
-c:v h264_nvenc -preset p5 -tune hq -rc vbr -cq 23 -b:v 0 \
|
||||
-rc-lookahead 20 -spatial_aq 1 -temporal_aq 1 \
|
||||
-bf 3 -b_ref_mode middle \
|
||||
-c:a copy OUTPUT
|
||||
```
|
||||
|
||||
Key flags (PeerTube-runner passes its own argv; these are what the
|
||||
capability probe and any wrapper/profile must emit):
|
||||
|
||||
| Purpose | Flag | Notes |
|
||||
|---|---|---|
|
||||
| HW decode | `-hwaccel cuda -hwaccel_output_format cuda` or explicit `-c:v h264_cuvid`/`hevc_cuvid`/`av1_cuvid` | explicit `*_cuvid` gives named-decoder certainty; `-hwaccel_output_format cuda` keeps frames on-GPU |
|
||||
| HW upload (when input is CPU-decoded) | `-vf hwupload_cuda` | needed for e.g. audio-merge jobs with CPU-side frames |
|
||||
| HW scale | `scale_cuda=w:h` (part of ffmpeg) or `scale_npp` (needs nvcc-built ffmpeg) | `scale_cuda` is sufficient and dependency-light |
|
||||
| Multi-GPU | `-init_hw_device cuda=cu:1`, encoder `-gpu N` | probe must test the *specific* device index it will advertise |
|
||||
| Encode | `-c:v h264_nvenc / hevc_nvenc / av1_nvenc` | `av1_nvenc` exists only on Ada+ builds of ffmpeg ≥ 5.0 |
|
||||
| Quality | `-preset p1..p7` (p7 slowest/best), `-tune hq\|ll\|ull`, `-multipass qres\|fullres` | p4–p5 is the sweet spot for VOD |
|
||||
| Rate control | `-rc vbr -cq N -b:v 0` (quality-targeted VBR) or `-rc constqp` | prefer `-cq` + `-b:v 0` for PeerTube's CRF-style outputs |
|
||||
| Lookahead/AQ | `-rc-lookahead`, `-spatial_aq`, `-temporal_aq`, `-bf`, `-b_ref_mode middle` (Turing+) | the Turing+ quality levers |
|
||||
| 10-bit HEVC | `-profile:v main10 -pix_fmt p010le` | verify per generation (§1.2) |
|
||||
| Session hygiene | `-delay 0`, and **one encode context per session** | sessions are the scarce resource on GeForce (§1.1) |
|
||||
| Verification | run under `nvidia-smi dmon -s u` (enc/dec columns), check encoder-session count delta | this is the no-silent-CPU *evidence*, see §5.4 |
|
||||
|
||||
Minimal VRAM budget per 1080p hw pipeline ≈ 200–400 MiB (decode surfaces +
|
||||
scale + encode lookahead); 4K ≈ 0.8–1.5 GiB. The probe measures real usage
|
||||
per resolution class instead of trusting these numbers.
|
||||
|
||||
---
|
||||
|
||||
## 2. What an NVIDIA worker is (profile)
|
||||
|
||||
Same shape as the Intel worker — official runner, admission controller,
|
||||
status JSON, drain control — differing only in the probe module, deps, and
|
||||
concurrency inputs.
|
||||
|
||||
### 2.1 Differences for `install-peertube-worker.sh`
|
||||
|
||||
| Aspect | Intel QSV worker | NVIDIA worker |
|
||||
|---|---|---|
|
||||
| Kernel/driver deps | `intel-media-va-driver-non-free`, oneVPL runtime, `/dev/dri/renderD128` + `render`/`video` groups | `nvidia-open` kernel module + pinned driver branch, `libnvidia-encode`, `nvidia-smi`, `nvidia-persistenced`, user in `video` group |
|
||||
| Device nodes | `/dev/dri/*` (always present) | `/dev/nvidia*` (created by module; `nvidia-modprobe`/udev) |
|
||||
| Probe module | `intel-qsv` | `nvidia-nvenc` (new module, same interface) |
|
||||
| Config additions | — | `PT_WORKER_GPU_DEVICES=0` (or `0,1`), `PT_WORKER_SESSIONS_RESERVE=1` (headroom sessions), `PT_WORKER_VRAM_FLOOR_MB=1024`, `PT_WORKER_DRIVER_MIN=550` |
|
||||
| Self-test | vainfo + QSV transcode | nvidia-smi gates + per-device NVENC transcode + session-slot test + VRAM floor (§5.3) |
|
||||
| Concurrency inputs | engine util, CPU/RAM | `min(sessions_free, cpu_bound, ram_bound, vram_bound, disk_bound, bw_bound)` |
|
||||
| Runner invocation | as Intel contract | plus `-gpu N` device pinning in the ffmpeg argv and one runner instance per GPU when a box has >1 GPU (see §2.2) |
|
||||
|
||||
Everything else — systemd unit shape, service account, Tailscale posture,
|
||||
registration, VOD-only job types, drain semantics — is inherited unchanged.
|
||||
|
||||
### 2.2 Multi-GPU boxes
|
||||
|
||||
A box with K GPUs runs **K runner instances** (`peertube-worker@gpu0…gpuK-1`
|
||||
template units), each pinned to one device, rather than one runner juggling
|
||||
devices. Reasons: session accounting and VRAM floors are per-device;
|
||||
admission states (GPU_UNAVAILABLE for one card after a co-tenant/ECC event)
|
||||
should not drain healthy siblings; PeerTube sees N independent runners and
|
||||
load-balances them natively. [E: this host has 2× RTX 3090 and would onboard
|
||||
as `…:gpu0` and `…:gpu1`.]
|
||||
|
||||
---
|
||||
|
||||
## 3. Dynamic pool design
|
||||
|
||||
### 3.1 Architecture
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────┐
|
||||
│ pool-manager (new, small, stateless-ish) │
|
||||
│ signals → policy engine → per-worker │
|
||||
│ desired state (AVAILABLE/DRAINED + │
|
||||
│ concurrency hint + job-class eligibility) │
|
||||
└───────┬───────────────────┬────────────────┘
|
||||
read-only admin │ │ control channel (SSH over
|
||||
API polls │ │ Tailscale → pt-worker CLI)
|
||||
PeerTube ◄───────────┘ ▼
|
||||
keeps THE job queue worker-1 (Intel, QSV) worker-2 (NV gpu0) …
|
||||
and does assignment admission controller admission controller
|
||||
+ peertube-runner + peertube-runner
|
||||
```
|
||||
|
||||
The pool manager **never talks to PeerTube as a scheduler**. It only (a)
|
||||
observes queue pressure read-only, (b) starts/stops/drains worker services,
|
||||
(c) sets admission state and concurrency. Assignment remains PeerTube's, via
|
||||
the official runner request/accept cycle. This honors the "do not build a
|
||||
new job scheduler" constraint.
|
||||
|
||||
### 3.2 (a) Load signals
|
||||
|
||||
| Signal | Source | Cadence | Notes |
|
||||
|---|---|---|---|
|
||||
| `pending_depth` | PeerTube admin API: runner-jobs endpoint filtered by state=pending, per job type (vod-web-video / vod-hls / vod-audio-merge) | 15 s | the primary scale-up input |
|
||||
| `oldest_pending_age_s` | same endpoint, `createdAt` of oldest pending job | 15 s | catches stall at low depth (e.g. one 4K job nobody can serve) |
|
||||
| `completed_rate_ewma` | per-worker status JSON `jobs_completed` counters | 30 s | throughput of the currently-active set |
|
||||
| `enc_util`, `dec_util` | NVIDIA: `nvidia-smi --query-gpu=encoder.utilization...`/NVML; Intel: media-engine counters per Intel contract | 10 s | saturation of the active set |
|
||||
| `sessions_active/cap` | our own accounting + NVML where supported | per-job event | GeForce cap awareness (§1.1); NVML unsupported on WSL2 [E] |
|
||||
| `vram_free_mb` | NVML | 10 s | hard eligibility floor |
|
||||
| `recent_download_mbps` / `upload_mbps` (EWMA) | per-job transfer observations (Intel contract already defines these) | per-job | job-size-aware eligibility input |
|
||||
| `realtime_factor_ewma` | completed jobs (encode wall-clock vs duration) | per-job | learned worker weight (§3.3) |
|
||||
| `throttle_reasons`, `temp_c` | NVML throttle-cause bits / thermal zones | 10 s | THERMAL_THROTTLED mapping [E: counters visible on this host] |
|
||||
|
||||
Pressure index (normalized 0..1), computed over *eligible* workers only:
|
||||
|
||||
```
|
||||
P = clamp01( max(
|
||||
pending_depth / (2 * Σ active_capacity_slots),
|
||||
oldest_pending_age_s / 900s
|
||||
))
|
||||
```
|
||||
|
||||
`Σ active_capacity_slots` = Σ over AVAILABLE/BUSY workers of their current
|
||||
concurrency hint. The `2×` makes P=1 mean "queue is two full waves deep".
|
||||
|
||||
### 3.3 (b) Policy engine
|
||||
|
||||
**Pool-worker states** (per worker; these are *manager-side* views layered
|
||||
on the worker's own admission states from the Intel contract):
|
||||
|
||||
| Manager state | Meaning | Transition in |
|
||||
|---|---|---|
|
||||
| `COLD` | powered off / suspended (WoL-able, §3.8) | manual or WoL policy |
|
||||
| `SPARE` | service up, admission DRAINED (registered, idle) | default resting state |
|
||||
| `WARM` | service up, DRAINED, persistence-mode on, probe recently green | pre-warm tier |
|
||||
| `ACTIVE` | admission AVAILABLE at concurrency C | scale-up decision |
|
||||
| `DRAINING` | admission DRAINING, in-flight jobs finishing | scale-down or fault |
|
||||
| `COOLDOWN` | admission DRAINING after repeated job failures | failure policy |
|
||||
|
||||
**Hysteresis** (per-worker and pool-wide):
|
||||
|
||||
- Scale **up** when `P ≥ 0.6` sustained ≥ 90 s (two consecutive 45 s windows)
|
||||
→ promote the highest-weight SPARE/WARM worker to ACTIVE.
|
||||
- Scale **down** when `P ≤ 0.15` sustained ≥ 20 min **and** worker has been
|
||||
idle ≥ 15 min → demote ACTIVE → SPARE (drain, then optional service stop).
|
||||
- Never demote below `min_active = 1` (or 2 once multi-tenant), regardless
|
||||
of idle, to keep probe freshness and WoL-free response.
|
||||
- Promotions are staggered ≥ 30 s apart and probe-gated: a worker may only
|
||||
become ACTIVE if its last probe is < 5 min old and green. This prevents
|
||||
thundering-herd registration wake-ups and GPU re-init spikes.
|
||||
- Failure fast-path bypasses all hysteresis: GPU_UNAVAILABLE, thermal, or a
|
||||
hardware-evidence-failed job → immediate DRAINING.
|
||||
|
||||
**Per-worker learned weights** (used for promotion order and drain order):
|
||||
|
||||
```
|
||||
w_i = EWMA_λ=0.3 over jobs of ( output_pixel_rate_i / class_baseline )
|
||||
shrunk toward class prior: ŵ = (n·w + k·w_prior) / (n + k), k=3
|
||||
```
|
||||
|
||||
`output_pixel_rate` = output pixels/sec during encode phase only (excludes
|
||||
download/upload), bucketed by job class so a worker isn't punished for
|
||||
getting small jobs. Cold-start workers inherit the class prior, so a new
|
||||
box's first promotion comes from queue pressure, not fabricated history.
|
||||
Drain order prefers low `ŵ` *and* low `recent_download_mbps` (cheapest to
|
||||
re-warm).
|
||||
|
||||
**Job-size-aware eligibility.** Job classes from PeerTube job metadata
|
||||
(source resolution × duration, known at pending time):
|
||||
|
||||
| Class | Definition (approx) | Requires |
|
||||
|---|---|---|
|
||||
| S | ≤1080p, ≤10 min | any ACTIVE worker |
|
||||
| M | ≤1080p long, or 1440p | enc_util headroom ≥ 30% |
|
||||
| L | 4K, or 4K-source→1080p | `realtime_factor_ewma ≥ 3`, `recent_download_mbps ≥ 50`, `vram_free ≥ 4 GiB` (NVIDIA) or equivalent QSV gate |
|
||||
| XL | 4K60 long-form | L gates + `ŵ ≥ 1.0` |
|
||||
|
||||
Enforcement is, again, selective availability: when an L/XL job is pending
|
||||
and no one satisfies the gates, the manager *keeps* the small workers at
|
||||
their normal concurrency but does not count them as capacity for that job
|
||||
(PeerTube will still let a small worker take it if it asks — acceptable
|
||||
fallback, not silent CPU fallback, and it still satisfies the hardware
|
||||
rule).
|
||||
|
||||
### 3.4 (c) Scale-up / scale-down mechanics
|
||||
|
||||
The constraint: **PeerTube runner registration is static.** A runner row
|
||||
exists server-side from first `register` (with a registration token) until
|
||||
an admin deletes it; the runner process merely polls. Therefore:
|
||||
|
||||
*Joining the pool* =
|
||||
1. box provisioned + runner registered **once** at install time
|
||||
(`install-peertube-worker.sh --name w2:gpu0 …`),
|
||||
2. service exists but may be stopped or admission-DRAINED,
|
||||
3. manager: `systemctl start peertube-worker@…` (if stopped) then
|
||||
`pt-worker enable` → admission AVAILABLE → runner starts
|
||||
requesting jobs on its normal poll cycle.
|
||||
|
||||
*Leaving the pool* (grace) =
|
||||
1. `pt-worker drain` → admission DRAINING; the controller stops offering
|
||||
the runner for new jobs (per Intel contract: runner paused/disabled, not
|
||||
killed),
|
||||
2. wait for in-flight jobs: `active_jobs → 0`, with a max-drain timeout
|
||||
(e.g. 2× p99 job wall-clock, floor 30 min),
|
||||
3. then either leave at SPARE (service up, DRAINED — cheapest re-join) or
|
||||
`systemctl stop` for overnight-class quiescence, or suspend (§3.8).
|
||||
|
||||
Notes:
|
||||
- Repeated start/stop never touches registration tokens; tokens live only
|
||||
in the install-time secret path on the worker.
|
||||
- Because a drained runner is still *registered*, PeerTube's per-runner
|
||||
job-request rate is unaffected; there is no server-side cleanup needed.
|
||||
- One job → many sessions: PeerTube runs separate ffmpeg instances per
|
||||
output rendition, so a "concurrency 1" job can still consume 4+ NVENC
|
||||
sessions. The admission controller must therefore count **actual encode
|
||||
sessions** (its own children's NVENC contexts), not just jobs, when
|
||||
enforcing `sessions_active < cap` (§1.1, §5.3).
|
||||
|
||||
### 3.5 (d) Mixed-vendor scheduling
|
||||
|
||||
We cannot intercept PeerTube's runner→job assignment, so vendor preference
|
||||
is expressed as *eligibility*, decided at the moment a job class is pending:
|
||||
|
||||
| Job / condition | Prefer | Mechanism |
|
||||
|---|---|---|
|
||||
| AV1 **source** | NVIDIA Ampere+ (NVDEC AV1) or CPU-decode on QSV box | gate QSV workers out of the AV1-decode class via probe capability (`av1_dec`) |
|
||||
| AV1 **output** | NVIDIA **Ada+ only** (`av1_enc`) | same gate; policy says: not enabled in production anyway |
|
||||
| 4K HEVC/H.264 outputs | NVIDIA (throughput, VRAM for lookahead) | weight-based promotion pulls NVENC boxes in first at L/XL classes |
|
||||
| ≤1080p H.264 bulk | whichever has session/enc headroom; QSV is typically *more* power-efficient per 1080p job | neutral; capacity-driven |
|
||||
| Session pressure (NVENC near cap, QSV idle) | QSV | temporarily demote NVENC availability for class S/M |
|
||||
| One vendor fleet-wide drained (driver CVE, bad driver branch) | the other | vendor-level kill switch in manager config: `vendor_disabled: [nvidia]` |
|
||||
|
||||
Probe capability keys that drive this (identical names across vendors):
|
||||
`h264_enc, hevc_enc, av1_enc, h264_dec, hevc_dec, av1_dec, max10bit`,
|
||||
`max_sessions`, `max_width/height`. The Intel probe fills the same keys
|
||||
(i5-1240P: `h264_enc=1, hevc_enc=1, av1_enc=0, av1_dec=1`), which is the
|
||||
whole point of the pluggable module.
|
||||
|
||||
### 3.6 (e) Unified status/metrics schema
|
||||
|
||||
Extends the Intel status JSON with a versioned, vendor-generic `gpu` block.
|
||||
Bump `schema_version`; additive changes only.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"schema_version": 2,
|
||||
"hostname": "w2", "runner_name": "w2:gpu0",
|
||||
"tailscale_ip": "100.x.y.z", "net_path": "DIRECT|DERP",
|
||||
"state": "AVAILABLE", // Intel-contract states, unchanged
|
||||
"registration_state": "registered",
|
||||
"active_jobs": 1, "concurrency": 3, "concurrency_limiters": {
|
||||
"cpu": 4, "ram": 4, "disk": 8, "bw": 3,
|
||||
"gpu_sessions": 3, // NVIDIA only; null on Intel
|
||||
"vram": 3
|
||||
},
|
||||
"gpu": { // NEW: vendor-generic
|
||||
"vendor": "nvidia", // "intel" | "nvidia" | "none"
|
||||
"probe_module": "nvidia-nvenc", // "intel-qsv" | "nvidia-nvenc"
|
||||
"model": "RTX 3090", "generation": "ampere",
|
||||
"nvenc_generation": 5,
|
||||
"driver": "595.58.03", "kernel_module": "open",
|
||||
"capabilities": { "h264_enc": 1, "hevc_enc": 1, "av1_enc": 0,
|
||||
"h264_dec": 1, "hevc_dec": 1, "av1_dec": 1,
|
||||
"max10bit": 1 },
|
||||
"sessions": { "active": 2, "cap": 8,
|
||||
"cap_source": "driver_class_table|nvml|probe" },
|
||||
"devices": [ { "index": 0, "uuid": "GPU-2095…",
|
||||
"vram_total_mb": 24576, "vram_free_mb": 22000,
|
||||
"enc_util_pct": 71, "dec_util_pct": 40,
|
||||
"temp_c": 66, "throttle_reasons": [] } ],
|
||||
"self_test": { "ts": "2026-10-09T14:00:00Z", "ok": true,
|
||||
"evidence": "enc_util_delta=+68%", "detail": "…" }
|
||||
},
|
||||
"hw_evidence_last_job": { "ok": true,
|
||||
"method": "encoder_session_delta>=1", "ts": "…" },
|
||||
"cpu": {"model": "…", "util_pct": 40}, "temps_c": {"cpu": 55},
|
||||
"disk_free_gb": 180,
|
||||
"recent_download_mbps": 92.4, "recent_upload_mbps": 88.1,
|
||||
"jobs": {"completed": 214, "failed": 3, "hw_fallback_prevented": 0},
|
||||
"last_job": {"duration_s": 412, "realtime_factor": 7.1,
|
||||
"class": "M", "output_pixel_rate_mpx_s": 22.6},
|
||||
"manager": {"desired": "ACTIVE", "weight": 1.14, "class_gates": "ok"}
|
||||
}
|
||||
```
|
||||
|
||||
Prometheus mapping is mechanical: one `pt_worker_up`, one
|
||||
`pt_worker_gpu_sessions{vendor,cap_source}`, one
|
||||
`pt_worker_hw_evidence_ok` (the no-silent-CPU invariant as an alertable
|
||||
series), plus re-exports of the Intel contract's series.
|
||||
|
||||
### 3.7 (f) Future: bare-metal autoscaling hooks (WoL)
|
||||
|
||||
Brief by request. If worker boxes are wakeable:
|
||||
|
||||
- Manager keeps `wol_mac`, `wake_budget` (max 2 concurrent wakes, ≥ 3 min
|
||||
apart), and a boot-readiness check = Tailscale SSH + agent status JSON
|
||||
with fresh probe (typical budget 3–6 min including GPU persistence).
|
||||
- Wake policy reuses the scale-up rule with an extra tier: only wake when
|
||||
`P ≥ 0.8` sustained 5 min and all SPARE/WARM workers are ACTIVE
|
||||
(i.e. waking is the last resort, because boot latency is minutes not
|
||||
seconds).
|
||||
- Sleep path = graceful drain (§3.4) then `systemctl suspend`; AC/battery
|
||||
rules from the Intel laptop policy apply first (never sleep a box on
|
||||
battery, never wake one).
|
||||
- Accounting: track `wakes/day` and `woken-minutes-saved` per box; cap wakes
|
||||
(e.g. ≤ 6/day/box) to respect hardware and power-on cycles.
|
||||
|
||||
---
|
||||
|
||||
## 4. Rollout plan
|
||||
|
||||
### 4.1 Change the Intel admission controller NOW (small interface list)
|
||||
|
||||
Five things, all cheap while the code is fresh:
|
||||
|
||||
1. **Versioned status JSON**: add `schema_version: 1` today, and the
|
||||
vendor-generic `gpu` block shape (§3.6) with `"vendor": "intel"` —
|
||||
i.e. `gpu.probe_module`, `gpu.capabilities{h264_enc,hevc_enc,av1_enc,
|
||||
h264_dec,hevc_dec,av1_dec}`, `gpu.model/generation`, per-device array
|
||||
(Intel: one entry, `/dev/dri/renderD128`, `enc_util` from media-engine
|
||||
counters). Avoid leaking QSV-specific keys to the top level.
|
||||
2. **Probe interface contract** (document, even if only one impl exists):
|
||||
`probe() → {ok, capabilities, maxSessions?, devices[], evidence}` and
|
||||
`verifyHardwareDidWork(jobSamples) → {ok, evidence}` — both **fail
|
||||
closed**: any exception, timeout, or absent evidence ⇒ GPU_UNAVAILABLE +
|
||||
drain, never "assume CPU is fine". Selectable via
|
||||
`PT_WORKER_GPU_PROBE=intel-qsv|nvidia-nvenc`.
|
||||
3. **Concurrency resolver** as a pure function of
|
||||
`(probe, cpu, ram, disk, bw, [sessions], [vram])` so NVIDIA is a new
|
||||
input set, not a new control path. Expose which limiter bound the choice
|
||||
(`concurrency_limiters`, §3.6) — this is what the pool manager consumes.
|
||||
4. **Hardware-evidence hook per job**: after each completed job, sample the
|
||||
GPU's encoder utilization delta (Intel: media-engine busy% delta during
|
||||
the job; NVIDIA: NVML encoder-session/util delta) and record
|
||||
`hw_evidence_last_job`. A job that completed without evidence must count
|
||||
as a *failure* for pool purposes even if PeerTube marks it OK — this is
|
||||
the enforcement teeth of the no-silent-CPU rule, and it is vendor-neutral.
|
||||
5. **Job-class metadata in per-job telemetry** (source res/codec, output
|
||||
renditions, wall-clock, realtime factor) so the manager's weights and
|
||||
gates (§3.3) have data from day one on the Intel cluster.
|
||||
|
||||
### 4.2 Onboarding the first NVIDIA box (when one exists)
|
||||
|
||||
1. Pick the platform: **native Linux** (Debian 12/13), Turing or newer;
|
||||
Ada+ if AV1 encode is ever wanted. Reject A100/H100-class compute cards
|
||||
(no NVENC, §1.1).
|
||||
2. Driver install, pinned branch (≥ 550; open kernel modules for Turing+),
|
||||
`nvidia-persistenced`, user in `video`, verify
|
||||
`nvidia-smi` + `libnvidia-encode.so.1` present. Reboot-verify.
|
||||
3. ffmpeg: distro build is fine if it lists `h264_nvenc`/`hevc_nvenc` and
|
||||
`*_cuvid` **[E: Debian 12's ffmpeg 5.1.9 does]**; else pin a static
|
||||
BtbN/johnvansickle-style build ≥ 5.0 with ffnvcodec. Record
|
||||
`ffmpeg --version` in status JSON.
|
||||
4. `install-peertube-worker.sh --name <host>:gpu0 --gpu-probe nvidia-nvenc
|
||||
--gpu-devices 0 …` — same script, new profile (§2.1). Per additional GPU
|
||||
on the box, repeat with `:gpu1`, … (§2.2).
|
||||
5. Self-test gate (must pass before registration): see §5.3 — nvidia-smi
|
||||
gates → per-device NVENC encode probe under `dmon` → session-slot test →
|
||||
VRAM floor. Fail ⇒ abort install with `GPU_UNAVAILABLE`, no runner
|
||||
registration.
|
||||
6. Register runner (VOD job types only, per Intel contract), start service,
|
||||
run one real E2E job, and assert hardware evidence: encoder-session
|
||||
delta ≥ 1 per rendition and enc-util delta > 0 **and** CPU-user-time of
|
||||
the ffmpeg process ≪ CPU-encode baseline (belt and braces).
|
||||
7. Soak: 1 h at target concurrency; watch sessions, VRAM, temps, throttle
|
||||
reasons, throughput; set the box's initial `concurrency` from measured
|
||||
end-to-end throughput, not single-job fps (Intel Phase 9 methodology,
|
||||
unchanged).
|
||||
8. Add to pool manager inventory as `SPARE` with class-prior weight; first
|
||||
promotion happens on the next pressure event.
|
||||
9. Only then consider multi-GPU boxes and WoL tiers (§3.7).
|
||||
|
||||
---
|
||||
|
||||
## 5. Appendices
|
||||
|
||||
### 5.1 Failure-mode → state mapping (NVIDIA additions)
|
||||
|
||||
| Failure | Detection | State | Action |
|
||||
|---|---|---|---|
|
||||
| Driver/module missing or unloaded | nvidia-smi nonzero, no `/dev/nvidia*` | GPU_UNAVAILABLE | refuse work; re-probe 60 s |
|
||||
| VRAM exhausted (co-tenant, leak) [E] | CUDA_ERROR_OUT_OF_MEMORY / "No capable devices found" in ffmpeg init; NVML `vram_free` below floor | GPU_UNAVAILABLE (device-scoped) | drain that device's runner; sibling GPUs unaffected (§2.2) |
|
||||
| Session cap hit | our accounting ≥ cap − reserve | BUSY (capacity) | no new jobs until sessions free; never queue extra ffmpeg per job |
|
||||
| Thermal/power throttle | NVML throttle-cause bits, temp ≥ limit | THERMAL_THROTTLED | drain after in-flight; cooldown |
|
||||
| Encoder produced no HW evidence | `hw_evidence_last_job.ok=false` | GPU_UNAVAILABLE + job counted failed | page; **never** re-run on CPU |
|
||||
| Driver/GPU fallen off bus (Xid errors in dmesg) | dmesg `NVRM: Xid` watch | OFFLINE | drain + host-level alert |
|
||||
| Network DERP/throughput collapse | Intel contract's EWMA + path change | NETWORK_DEGRADED | unchanged from Intel |
|
||||
|
||||
### 5.2 Config surface (NVIDIA profile)
|
||||
|
||||
```
|
||||
PT_WORKER_GPU_PROBE=nvidia-nvenc
|
||||
PT_WORKER_GPU_DEVICES=0 # one runner instance per device
|
||||
PT_WORKER_SESSIONS_CAP=auto # nvml → driver_class_table → 8 (GeForce)
|
||||
PT_WORKER_SESSIONS_RESERVE=1 # keep 1 session headroom
|
||||
PT_WORKER_VRAM_FLOOR_MB=1024
|
||||
PT_WORKER_DRIVER_MIN=550
|
||||
PT_WORKER_FFMPEG_NVENC_PRESET=p5 # map PeerTube quality tier → preset
|
||||
PT_WORKER_PROBE_INTERVAL=300 # full probe; 10 s counters in between
|
||||
```
|
||||
|
||||
### 5.3 `nvidia-nvenc` probe / self-test design (ordered, fail-closed)
|
||||
|
||||
1. **Gate**: `nvidia-smi` exits 0; driver ≥ min; module flavor recorded;
|
||||
`libnvidia-encode.so.1` resolvable; ≥1 device with `vram_free ≥ floor`.
|
||||
2. **Capability enumeration**: per advertised device, from NVML name →
|
||||
generation → capability table (§1.2/§1.3), *then verified empirically*:
|
||||
attempt a 1-second `h264_nvenc` encode on that exact device (`-gpu N`);
|
||||
attempt `hevc_nvenc`; attempt `av1_nvenc` only if generation ≥ Ada.
|
||||
Absence of an encoder in `ffmpeg -encoders` downgrades capability
|
||||
without failing the probe (e.g. distro ffmpeg without av1_nvenc).
|
||||
3. **Pipeline probe**: full hw chain (§1.6) at 1080p on the target device
|
||||
under `nvidia-smi dmon -s u`; require `enc>0` and `dec>0` columns
|
||||
observed during the run, and record VRAM delta as the per-class budget.
|
||||
4. **Session-slot test**: launch K parallel tiny `h264_nvenc` encodes
|
||||
(K = advertised cap) and count how many initialize successfully
|
||||
(`No capable devices found` / init failure = slot unavailable). The
|
||||
largest K′ that succeeds becomes `sessions.cap` with
|
||||
`cap_source=probe`. This is the ground truth the controller trusts —
|
||||
it automatically reflects driver-class caps **and** whatever the
|
||||
platform (e.g. WSL2 [E]) actually allows.
|
||||
5. **Decode matrix**: one decode-only pass per codec family
|
||||
(h264/hevc/av1 `*_cuvid`) on a synthetic stream, asserting `dec>0`.
|
||||
6. **Report**: fill the §3.6 `gpu` block, include `dmon` excerpts and VRAM
|
||||
deltas as `evidence`. Any step erroring or timing out (30 s/step) ⇒
|
||||
`ok=false` ⇒ GPU_UNAVAILABLE. No step may conclude "use CPU instead".
|
||||
|
||||
### 5.4 Enforcing the no-silent-CPU rule on NVIDIA
|
||||
|
||||
Three independent tripwires, any one sufficient to fail a job:
|
||||
|
||||
1. **Argv gate**: the wrapper/runner must be invoking `*_nvenc` for video
|
||||
(grep the spawned argv); PeerTube's own fallback logic must be
|
||||
disabled/refused in config.
|
||||
2. **Evidence gate**: NVML encoder-session count and encoder utilization
|
||||
must increase while the job's ffmpeg runs (sampled ≥ 1 Hz); a completed
|
||||
encode with flat counters = CPU fallback (or a no-op) ⇒ job marked
|
||||
failed, worker GPU_UNAVAILABLE, manager COOLDOWN.
|
||||
3. **Plausibility gate**: ffmpeg CPU user-time for the job must be far
|
||||
below the CPU-encode baseline for that resolution (measured once per
|
||||
box during install); a CPU-encode-shaped CPU profile ⇒ same treatment.
|
||||
|
||||
This is deliberately vendor-neutral (Intel uses media-engine busy% in place
|
||||
of NVML counters), which is exactly why it belongs in the shared admission
|
||||
controller rather than in the probe module.
|
||||
|
||||
### 5.5 Open questions for the Intel implementation (blocking-free)
|
||||
|
||||
- Exact admin endpoint + auth shape for pending runner-jobs depth (manager
|
||||
needs it read-only; a proxy via the worker's own credentials is
|
||||
acceptable if admin scope is unavailable).
|
||||
- Whether the Intel drain implementation pauses the runner process or
|
||||
stops the unit — the manager wants to treat both as DRAINED and needs to
|
||||
know which to expect for start/stop idempotency.
|
||||
- Whether per-rendition ffmpeg invocation is confirmed (drives
|
||||
session-counting assumptions, §3.4).
|
||||
16
docs/design/control-room-scope.md
Normal file
16
docs/design/control-room-scope.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# Control Room — scope statement (user, 2026-10-07)
|
||||
|
||||
The streaming service's job: view several screens and mix. (A separate mobile app is being developed and integrates LATER.)
|
||||
|
||||
## Web (the "simplified OBS")
|
||||
1. **Multiview grid**: watch multiple live feeds simultaneously — layouts 2x1, 2x2, 3x3, 4x4.
|
||||
2. **Audio control**: pick ONE feed's audio, or MIX several audios.
|
||||
3. **Program out**: select what goes out — a single feed, or a mix/composite of feeds.
|
||||
4. **Overlays**: easily editable lower thirds; SCROLLING lower thirds (chyron); logo overlay from SVG input.
|
||||
5. **Playlist fallback**: schedule a playlist that plays on the PeerTube channel when no feeds are live.
|
||||
|
||||
## Mobile (later integration)
|
||||
Simple controls for putting something live — preview→program flow, "go live from preview".
|
||||
|
||||
## Research question
|
||||
Best way to incorporate this OBS-type experience (much simplified) into the existing stack: media-worker (supervisor + ffmpeg relay + signed command bus), live monitor, media-control plane, PeerTube connector.
|
||||
453
docs/design/design-merged.md
Normal file
453
docs/design/design-merged.md
Normal file
|
|
@ -0,0 +1,453 @@
|
|||
# Merged Design — Group Review & Signoff for Posts
|
||||
|
||||
Status: design for implementation (merge of three proposals + user decisions v2, pending audit)
|
||||
Date: 2026-10-07 (v2: connection-scoped approval delegation added per user decision)
|
||||
Inputs: `runs/wp5/proposal-opus.md` (rigor/defense-in-depth), `runs/wp5/proposal-sol.md` (submission lifecycle), `runs/wp5/proposal-glm.md` (minimal surface), spec `runs/wp5/spec-review-workflow.md`, user decisions 2026-10-07
|
||||
|
||||
**Provenance legend**: [OP], [SOL], [GLM] mark which proposal a decision comes from; [ALL] = all three converged; [MERGE] = this document's synthesis.
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive summary — what this design is
|
||||
|
||||
1. **[OP] The foundation is fixed first.** Eight pre-existing defects (§2) are repaired in a prerequisite migration because the review feature multiplies each of them. Most critically: the scheduler currently publishes without checking draft state, and the audit hash does not cover the `diff` field (so "snapshot hash in the audit log" would not be tamper-evident).
|
||||
2. **[ALL] Group roles are a column on `group_members`** — `member | editor | reviewer`, mutually exclusive by construction. The existing epoch trigger fires on UPDATE, so role changes invalidate sessions for free.
|
||||
3. **[GLM+OP] Approvals attach to immutable version rows with a database-computed hash.** `post_versions.content_hash` is a Postgres *generated column* — it can never disagree with the content [GLM]. Votes copy the hash they approve; the scheduler publishes the approved version's bytes, never the mutable draft [OP]. **[AUDIT #24]** The defence order is explicit: §6.6 full-reset is the primary defence; the §4.2 schedule-time hash check is a backstop whose refusal is a 409 `STALE_VERSION` directing the author to re-submit — never a silent second path, never an approved-but-unschedulable draft.
|
||||
4. **[SOL+GLM] The review cycle is the unit of approval.** One `review_cycles` row per (draft, round) pins version, group (nullable + `group_name` snapshot [AUDIT #16]), quorum, deadline, and status. Quorum config (workspace default + group override) is **snapshotted at submit** — config changes never move goalposts mid-review [ALL].
|
||||
5. **[ALL] A single deny is a veto** — cycle closes `denied`, draft → `needs_changes` (replaces today's terminal `rejected`, unlocking resubmit). Deny asks for a reason in the UI (optional but suggested [USER v2]; recorded in thread + audit when given). Any content edit supersedes the cycle; resubmit always starts at 0/N.
|
||||
6. **[GLM] Chat is polling first** (4 s cursor endpoint), over [SOL]'s unified event log: chat messages and system events are rows in one `review_events` table. Schema is transport-agnostic; SSE doorbells [OP] are a later upgrade.
|
||||
7. **[GLM] Deadlocks are refused at the door — with one defined exception [AUDIT #17]**: submit runs an eligibility preflight ("not enough eligible reviewers (0) for quorum (2)") and 400s when quorum ≥ 2 is unreachable; when N=1 and 0 eligible reviewers exist, the cycle is still created, directly in `blocked` (`blocked-at-submit`), so the admin override always has a cycle to act on and quorum 1 never silently self-approves a solo author. A distinct `blocked` state handles mid-review reviewer-pool shrinkage. Admin override is a separate, reason-required action that writes no vote row [SOL+OP].
|
||||
8. **[GLM] `published` is the archive; moderation is flags, not states.** Hide/delete are admin-only soft flags; the scheduler's `scheduled→published` write needs zero changes.
|
||||
9. **[USER v2] Approval authority is per-account (connection), chosen when the account is added.** Every connection declares one of two approval modes: **`system`** — approvals flow through the group/workspace reviewer chain (the default), or **`delegated`** — the account's owner (whoever added/controls it) names the specific people who approve what goes out **from that account**. This scales to hundreds of accounts, each with its own approver pool. Quorum size and admin control remain **group settings** (§3.4).
|
||||
|
||||
New states: `primed` [SOL], `needs_changes` [ALL], `expired` [ALL], `blocked` [GLM]. The state machine is enforced **twice**: CAS UPDATEs in the service and a DB legal-transition trigger (defense in depth) [ALL].
|
||||
|
||||
---
|
||||
|
||||
## 2. Prerequisite fixes (WP5.0) — from [OP], all verified real
|
||||
|
||||
| # | Where | Defect | Fix |
|
||||
|---|---|---|---|
|
||||
| D1 | `apps/scheduler/src/activities/publish.activity.ts:136-141` | `claim()` joins drafts but never checks `d.state`; publishes any pending schedule row's `d.post` | claim requires `state='scheduled'` (plus the pending `scheduled_publications` row); publish `post_versions.post` of `approved_version_id` (post-011) |
|
||||
| D2 | `publish.activity.ts:79` | claim runs outside `try`; "not pending" throw retries forever (no retry limit) | move inside try; use non-retryable application error |
|
||||
| D3 | `001_tenants.sql:92-98` | `audit_hash` excludes `diff` — values recorded in diff are not tamper-evident | migration 010: hash v2 covers diff; verify both versions |
|
||||
| D4 | `publishing.service.ts:136+` | audit written after commit — crash window loses audit rows | same-tx `audit_append` via withTenant client (groups pattern, `groups.service.ts:123-133`) |
|
||||
| D5 | `publishing.controller.ts:113-129` + `rbac.guard.ts:64-74` | `schedule` takes `connectionId` from body; `RequireGroupAccess` only checks `:connectionId` params | validate body connection against group scope via `userCanUseConnection` - kept as the **API-level guard**; 4.1(3) adds the composite FK + seal-time trigger as the DB backstop [SECURITY #9] |
|
||||
| D6 | `realtime/gateway.ts:20-29` | epoch compared against raw bigint string (always "stale"); membership query omits `status='active'` | normalize to text compare; add status filter (kept even though we don't build WS now) |
|
||||
| D7 | `api-client.ts:90` + guard | any epoch bump 401s every user and the client drops the token — no re-issue | `POST /auth/reissue`: re-read live authz, mint fresh token; client retries once on epoch-401 |
|
||||
| D8 | `002_rbac_seed.sql:9,25,35` | `publication.delete` seeded to Owner/Admin/**Publisher**, unenforced | do not reuse it for moderation; new dedicated perms (§3) |
|
||||
|
||||
Migrations: **`010_audit_hash_v2.sql`** (D3: `hash_version` column, `audit_hash_v2` covering diff, advisory-lock append serialization, dual-version `audit_verify_chain`) — full SQL in `proposal-opus.md` §2.2.
|
||||
|
||||
---
|
||||
|
||||
## 3. Roles & permissions (R2)
|
||||
|
||||
### 3.1 Group roles [ALL]
|
||||
```sql
|
||||
ALTER TABLE group_members
|
||||
ADD COLUMN review_role text NOT NULL DEFAULT 'member'
|
||||
CHECK (review_role IN ('member', 'editor', 'reviewer'));
|
||||
```
|
||||
- **member**: author drafts, chat in review threads.
|
||||
- **editor**: edit any group post's content; submit for review; **cannot** approve/deny/publish.
|
||||
- **reviewer**: approve/deny (reason suggested, optional [USER v2]); **cannot** edit content.
|
||||
- Mutually exclusive single column = structural separation of duties. Role changes bump the authz epoch (trigger already fires on UPDATE, `008_groups.sql:79-80`); the service returns a `{token}` replacement for the caller (existing pattern, `groups.service.ts:226-269`) [ALL].
|
||||
- Workspace roles remain the *floor*: any review role still needs `draft.view` via active membership.
|
||||
|
||||
### 3.2 New permissions [MERGE]
|
||||
| permission | granted to | purpose |
|
||||
|---|---|---|
|
||||
| `review.manage` | Owner, Admin | set workspace/group quorum, stale_after |
|
||||
| `approval.override` | Owner, Admin | reason-required override of a blocked/expired cycle (writes no vote) |
|
||||
| `moderation.hide` | Owner, Admin | hide a post (flag) |
|
||||
| `moderation.delete` | Owner, Admin | soft-delete a post (flag) |
|
||||
|
||||
Author exclusion is **structural**: eligibility SQL excludes the author, every row in `draft_contributors` [OP], and (for override) any contributor. Sealed-by also excluded [OP].
|
||||
|
||||
### 3.4 Group settings [USER v2]
|
||||
Groups carry the review settings (writable by `review.manage` holders): **quorum** (`review_approvals_required`, NULL = inherit workspace default) and **admin control** (`review_admin_control`: whether Owner/Admin may exempt their own posts from review or override a stuck cycle in this group).
|
||||
|
||||
### 3.5 Connection approval modes [USER v2 — per-account delegation]
|
||||
Every account (connection) declares how its approvals work, set/changed by the connection's controller (its owner / whoever added it) or workspace admins:
|
||||
- **`system`** (default): the group's reviewer chain approves (§3.1).
|
||||
- **`delegated`**: the controller names a dedicated **approver pool for that account** (`connection_approvers`). Posts targeting this account need signoff from that pool — same quorum N from the group settings, same structural author/contributor exclusion. Group reviewers do **not** count for a delegated account unless also named in its pool.
|
||||
- Scale target: hundreds of connections, each with its own few approvers — "100 of accounts, and there are people per account who actually approve what goes out."
|
||||
- Admin override/exempt still applies per the group's `review_admin_control`, regardless of mode.
|
||||
|
||||
### 3.3 Vote visibility [USER v2 — supersedes the merged blind default]
|
||||
**Fully transparent by default.** Everyone in the group sees the complete vote ledger during review: who approved, who denied, who is yet to vote ("2/3 — Ana ✓, Ben ✓, Chi pending"). Deny reasons (when given) are visible inline in the thread. No server-side identity masking. A `blind_votes` group setting remains available for teams that want blind review later, but transparency is the default posture: [USER] "votes etc super transparent".
|
||||
|
||||
---
|
||||
|
||||
## 4. Data model — `011_review_workflow.sql`
|
||||
|
||||
(Numbering: `009_video.sql` already exists [OP, GLM caught this].)
|
||||
|
||||
### 4.1 Tables
|
||||
|
||||
```sql
|
||||
-- (1) group roles — §3.1
|
||||
-- (2) quorum config [GLM]
|
||||
CREATE TABLE workspace_review_settings (
|
||||
workspace_id uuid PRIMARY KEY REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
approvals_required int NOT NULL DEFAULT 1 CHECK (approvals_required BETWEEN 1 AND 5), -- [AUDIT #23] 0 is refused: no transition closes a required=0 cycle (deadlock-by-config)
|
||||
stale_after_days int NOT NULL DEFAULT 90 CHECK (stale_after_days BETWEEN 1 AND 3650),
|
||||
blind_votes boolean NOT NULL DEFAULT true,
|
||||
updated_by uuid REFERENCES users(id),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
ALTER TABLE groups
|
||||
ADD COLUMN review_approvals_required int CHECK (review_approvals_required BETWEEN 1 AND 5), -- [AUDIT #23] same floor as the workspace default
|
||||
ADD COLUMN review_blind_votes boolean, -- NULL = inherit workspace default
|
||||
ADD COLUMN review_admin_control boolean NOT NULL DEFAULT true; -- [USER v2]
|
||||
-- Precedence resolved ONCE at submit: group row ?? workspace row ?? built-in default(1)
|
||||
-- and snapshotted onto the cycle (§6).
|
||||
|
||||
-- (2b) per-account approval delegation [USER v2]
|
||||
ALTER TABLE connections
|
||||
ADD COLUMN approval_mode text NOT NULL DEFAULT 'system'
|
||||
CHECK (approval_mode IN ('system','delegated'));
|
||||
-- [AUDIT #9/#13] composite-FK target on connections: must exist before any FK that
|
||||
-- references connections(id, workspace_id) (connection_approvers, post_versions)
|
||||
ALTER TABLE connections ADD CONSTRAINT connections_id_ws_uk UNIQUE (id, workspace_id);
|
||||
|
||||
CREATE TABLE connection_approvers (
|
||||
workspace_id uuid NOT NULL,
|
||||
connection_id uuid NOT NULL,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
assigned_by uuid NOT NULL REFERENCES users(id),
|
||||
assigned_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (connection_id, user_id),
|
||||
FOREIGN KEY (connection_id, workspace_id) REFERENCES connections(id, workspace_id) ON DELETE CASCADE
|
||||
);
|
||||
-- Assignments: assigner must be the connection controller or hold connection.manage;
|
||||
-- approver must hold an active workspace membership. Changes bump the authz epoch
|
||||
-- (trigger, mirroring group_members) and revoke the removed approver's live votes.
|
||||
|
||||
-- (2c) [AUDIT #5] composite-FK target on drafts: must exist before any FK that
|
||||
-- references drafts(id, workspace_id) (post_versions, draft_contributors,
|
||||
-- review_cycles, review_events below)
|
||||
ALTER TABLE drafts ADD CONSTRAINT drafts_id_ws_uk UNIQUE (id, workspace_id);
|
||||
|
||||
-- (3) immutable versions [GLM hash mechanics + SOL target pinning]
|
||||
CREATE TABLE post_versions (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
draft_id uuid NOT NULL,
|
||||
version int NOT NULL CHECK (version >= 1),
|
||||
post jsonb NOT NULL,
|
||||
connection_id uuid, -- sealed target; NOT NULL when draft is group-scoped [SOL]
|
||||
content_hash text GENERATED ALWAYS AS (encode(digest(post::text, 'sha256'), 'hex')) STORED,
|
||||
seal_kind text NOT NULL CHECK (seal_kind IN ('submit','edit','reopen','migrated')),
|
||||
sealed_by uuid NOT NULL REFERENCES users(id),
|
||||
sealed_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (draft_id, version),
|
||||
UNIQUE (id, draft_id, workspace_id),
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (connection_id, workspace_id) REFERENCES connections(id, workspace_id) ON DELETE RESTRICT -- [SECURITY #9, AUDIT #13] composite FK: same-workspace target, DB-enforced; RESTRICT so a connection referenced by a sealed version cannot be deleted out from under it (connections are soft-disabled instead)
|
||||
);
|
||||
-- [AUDIT #11] Single contract, stated once: versions are update-immutable and
|
||||
-- cascade-deletable ONLY via `drafts` — the trigger rejects UPDATE outright and
|
||||
-- rejects any DELETE whose root is not the drafts cascade (pg_trigger_depth() check);
|
||||
-- BEFORE INSERT trigger requires connection_id when the draft has group_id [SOL].
|
||||
-- [SECURITY #9] Seal-time guard (BEFORE INSERT - a database invariant; D5 stays as the
|
||||
-- API-level guard): when connection_id IS NOT NULL the connection must belong to the
|
||||
-- SAME workspace (composite FK above) AND be in scope for the draft's group:
|
||||
-- drafts.group_id IS NULL OR EXISTS (SELECT 1 FROM group_connections gc
|
||||
-- WHERE gc.connection_id = NEW.connection_id AND gc.group_id = drafts.group_id).
|
||||
-- Any mismatch raises; a version is never sealed against a cross-tenant or
|
||||
-- out-of-scope connection.
|
||||
|
||||
-- (4) contributors [OP]
|
||||
CREATE TABLE draft_contributors (
|
||||
workspace_id uuid NOT NULL,
|
||||
draft_id uuid NOT NULL,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
first_at timestamptz NOT NULL DEFAULT now(),
|
||||
last_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (draft_id, user_id),
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE CASCADE
|
||||
);
|
||||
-- [AUDIT #10] Append-only and immutable: a BEFORE INSERT/UPDATE/DELETE trigger
|
||||
-- rejects UPDATE and DELETE outright (contributor rows are history; last_at is
|
||||
-- never rewritten), with the sole exception of the drafts cascade
|
||||
-- (pg_trigger_depth() check). INSERT is the only legal mutation and it is
|
||||
-- mandatory: every seal (submit/edit/reopen) inserts rows for the sealer and any
|
||||
-- editor recorded since the last seal, in the same transaction — a post_versions
|
||||
-- seal trigger fails the tx if the sealer has no draft_contributors row.
|
||||
|
||||
-- (5) review cycles [SOL shape, GLM snapshot semantics]
|
||||
CREATE TABLE review_cycles (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
draft_id uuid NOT NULL,
|
||||
group_id uuid, -- [AUDIT #16] nullable: group delete SET NULL, mirroring drafts
|
||||
group_name text, -- [AUDIT #16] group name snapshot at submit: audit trail survives group deletion
|
||||
round int NOT NULL CHECK (round >= 1),
|
||||
version_id uuid NOT NULL,
|
||||
required_approvals smallint NOT NULL CHECK (required_approvals BETWEEN 1 AND 5), -- [AUDIT #23] snapshot inherits the same 1..5 floor
|
||||
policy_source text NOT NULL CHECK (policy_source IN ('group','workspace','default')),
|
||||
approval_source text NOT NULL DEFAULT 'group' CHECK (approval_source IN ('group','connection')), -- [USER v2]
|
||||
blind_votes boolean NOT NULL,
|
||||
publish_at timestamptz, -- declared schedule ("primed to go out")
|
||||
deadline_at timestamptz NOT NULL, -- LEAST(publish_at, submitted_at + stale_after)
|
||||
status text NOT NULL CHECK (status IN
|
||||
('primed','in_review','approved','denied','superseded','expired','blocked','overridden')),
|
||||
submitted_by uuid NOT NULL REFERENCES users(id),
|
||||
submitted_at timestamptz NOT NULL DEFAULT now(),
|
||||
closed_at timestamptz,
|
||||
closed_reason text,
|
||||
UNIQUE (draft_id, round),
|
||||
UNIQUE (id, workspace_id), -- [AUDIT #5] composite-FK target for review_votes / review_overrides / review_events
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (group_id, workspace_id) REFERENCES groups(id, workspace_id)
|
||||
ON DELETE SET NULL (group_id), -- [AUDIT #16] group deletion detaches the cycle; reference kept as history via group_name
|
||||
FOREIGN KEY (version_id, draft_id, workspace_id)
|
||||
REFERENCES post_versions(id, draft_id, workspace_id)
|
||||
);
|
||||
|
||||
-- (6) votes [SOL constraints + OP hash binding]
|
||||
CREATE TABLE review_votes (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
cycle_id uuid NOT NULL,
|
||||
voter_user_id uuid NOT NULL REFERENCES users(id),
|
||||
decision text NOT NULL CHECK (decision IN ('approve','deny')),
|
||||
reason text, -- NOT NULL enforced for deny by CHECK
|
||||
snapshot_sha256 text NOT NULL CHECK (snapshot_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
cast_at timestamptz NOT NULL DEFAULT now(),
|
||||
revoked_at timestamptz, -- reviewer removed from group mid-review
|
||||
revoked_reason text,
|
||||
source text NOT NULL DEFAULT 'live' CHECK (source IN ('live','migrated')),
|
||||
UNIQUE (cycle_id, voter_user_id),
|
||||
CHECK (reason IS NULL OR btrim(reason) <> ''), -- [USER v2] deny reason optional (UI-suggested); never stored blank
|
||||
CHECK (revoked_at IS NULL OR (revoked_reason IS NOT NULL AND btrim(revoked_reason) <> '')),
|
||||
FOREIGN KEY (cycle_id, workspace_id) REFERENCES review_cycles(id, workspace_id)
|
||||
ON DELETE CASCADE -- [AUDIT #1] cycle-scoped history, no independent lifecycle: draft→cycle cascade must not be blocked
|
||||
);
|
||||
-- [AUDIT #11] Same single contract as versions and events: update-immutable, cascade-deletable
|
||||
-- only via `drafts` — the trigger rejects UPDATE of everything except the revoke fields
|
||||
-- (set only by the revocation trigger) and rejects any DELETE whose root is not the
|
||||
-- drafts cascade (pg_trigger_depth() check) [SOL].
|
||||
|
||||
-- (7) overrides [SOL: separate table, one per cycle]
|
||||
CREATE TABLE review_overrides (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL,
|
||||
cycle_id uuid NOT NULL,
|
||||
admin_user_id uuid NOT NULL REFERENCES users(id),
|
||||
reason text NOT NULL CHECK (btrim(reason) <> ''),
|
||||
snapshot_sha256 text NOT NULL CHECK (snapshot_sha256 ~ '^[0-9a-f]{64}$'), -- [AUDIT #19] same format constraint as review_votes
|
||||
at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (cycle_id),
|
||||
FOREIGN KEY (cycle_id, workspace_id) REFERENCES review_cycles(id, workspace_id)
|
||||
ON DELETE CASCADE -- [AUDIT #1] cycle-scoped history, no independent lifecycle: draft→cycle cascade must not be blocked
|
||||
);
|
||||
-- append-only; BEFORE INSERT trigger: admin is Owner/Admin, NOT a contributor, cycle is blocked/expired,
|
||||
-- AND [AUDIT #19] NEW.snapshot_sha256 = (SELECT content_hash FROM post_versions WHERE id = cycle.version_id)
|
||||
-- — an override can only ratify the exact content the cycle pinned. The transition trigger's
|
||||
-- override-row requirement (4.2 enforce_review_transition) is the second layer.
|
||||
|
||||
-- (8) unified event log = chat + system events [SOL]
|
||||
CREATE TABLE review_events (
|
||||
workspace_id uuid NOT NULL,
|
||||
draft_id uuid NOT NULL,
|
||||
seq bigint NOT NULL CHECK (seq > 0), -- gapless per draft, allocated under draft row lock
|
||||
cycle_id uuid,
|
||||
kind text NOT NULL CHECK (kind IN
|
||||
('message','submitted','edited','approved','denied','vote_revoked','quorum_met',
|
||||
'quorum_lost','needs_changes','blocked','unblocked','scheduled','expired',
|
||||
'published','hidden','unhidden','deleted','override')),
|
||||
actor_user_id uuid REFERENCES users(id), -- NULL = system
|
||||
body text, -- required for kind='message'
|
||||
data jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (kind <> 'message' OR (body IS NOT NULL AND btrim(body) <> '')), -- [AUDIT #18] message body must be non-empty
|
||||
CHECK (octet_length(coalesce(body, '')) <= 8192), -- [AUDIT #18] bounded body: 8 KiB hard cap
|
||||
CHECK (octet_length(data::text) <= 8192), -- [AUDIT #18] bounded event payload: 8 KiB hard cap
|
||||
at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (draft_id, seq),
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (cycle_id, workspace_id) REFERENCES review_cycles(id, workspace_id) -- [AUDIT #20, #22] nullable composite FK to review_cycles(id, workspace_id), action consistent with #1/#2; system events carry the cycle they narrate (feasible via the UNIQUE (id, workspace_id) from #5); [AUDIT #22] the cycle-narrating kinds (denied/override/quorum_met/etc.) can never be orphaned from their cycle
|
||||
);
|
||||
-- [AUDIT #11] Same single contract as versions and votes: update-immutable, cascade-deletable
|
||||
-- only via `drafts` — UPDATE rejected; DELETE rejected unless its root is the drafts
|
||||
-- cascade (pg_trigger_depth() check); kind='message' requires non-empty body.
|
||||
-- [AUDIT #18] Size caps are table CHECKs, not conventions: body ≤ 8192 bytes (non-empty when
|
||||
-- kind='message') and data ≤ 8192 bytes — oversize writes are refused by the database.
|
||||
-- [SECURITY #7] review_events policy (RLS IN ADDITION to tenant_isolation; also a
|
||||
-- mandatory service-layer precondition, see 4.2):
|
||||
-- SELECT ... USING: caller holds draft.view AND (drafts.group_id IS NULL OR EXISTS (
|
||||
-- SELECT 1 FROM group_members gm WHERE gm.group_id = drafts.group_id
|
||||
-- AND gm.user_id = current_setting('app.user_id')::uuid AND gm.status = 'active'))
|
||||
-- INSERT of kind='message' WITH CHECK: that same predicate AND an active
|
||||
-- group_members row with review_role IN ('member','editor','reviewer') for
|
||||
-- drafts.group_id; system kinds are inserted only via the SECURITY DEFINER
|
||||
-- service path. Fails closed: no membership => zero rows on read, refused write.
|
||||
|
||||
-- (9) drafts columns
|
||||
ALTER TABLE drafts
|
||||
ADD COLUMN group_id uuid,
|
||||
ADD COLUMN revision int NOT NULL DEFAULT 0, -- optimistic concurrency, bumped on save
|
||||
ADD COLUMN current_version_id uuid,
|
||||
ADD COLUMN approved_version_id uuid,
|
||||
ADD COLUMN event_seq bigint NOT NULL DEFAULT 0,
|
||||
ADD COLUMN hidden_at timestamptz, ADD COLUMN hidden_by uuid REFERENCES users(id),
|
||||
ADD COLUMN hidden_reason text,
|
||||
ADD COLUMN deleted_at timestamptz, ADD COLUMN deleted_by uuid REFERENCES users(id),
|
||||
ADD COLUMN deleted_reason text,
|
||||
ADD CONSTRAINT drafts_group_fk FOREIGN KEY (group_id, workspace_id)
|
||||
REFERENCES groups(id, workspace_id) ON DELETE SET NULL (group_id),
|
||||
ADD CONSTRAINT drafts_curver_fk FOREIGN KEY (current_version_id, id, workspace_id)
|
||||
REFERENCES post_versions(id, draft_id, workspace_id),
|
||||
ADD CONSTRAINT drafts_appver_fk FOREIGN KEY (approved_version_id, id, workspace_id)
|
||||
REFERENCES post_versions(id, draft_id, workspace_id),
|
||||
ADD CONSTRAINT drafts_appver_state CHECK ( -- [AUDIT #21] a live approval pointer exists only in approval-or-later states
|
||||
approved_version_id IS NULL OR state IN ('approved','scheduled','published')),
|
||||
ADD CONSTRAINT drafts_hidden_consistent CHECK (
|
||||
(hidden_at IS NULL AND hidden_by IS NULL AND hidden_reason IS NULL)
|
||||
OR (hidden_at IS NOT NULL AND hidden_by IS NOT NULL AND btrim(coalesce(hidden_reason,'')) <> '')),
|
||||
ADD CONSTRAINT drafts_deleted_consistent CHECK (
|
||||
(deleted_at IS NULL AND deleted_by IS NULL AND deleted_reason IS NULL)
|
||||
OR (deleted_at IS NOT NULL AND deleted_by IS NOT NULL AND btrim(coalesce(deleted_reason,'')) <> ''));
|
||||
|
||||
-- state set: drop 006 CHECK, install
|
||||
CHECK (state IN ('editing','primed','in_review','needs_changes','blocked',
|
||||
'approved','scheduled','published','expired'))
|
||||
```
|
||||
|
||||
Every new table: denormalized `workspace_id` + composite FKs, `reject_workspace_change` trigger, `ENABLE`+`FORCE ROW LEVEL SECURITY` with the exact `tenant_isolation` NULLIF policy, `GRANT … TO sp_app` — shapes copied verbatim from `006`/`008` [ALL].
|
||||
|
||||
### 4.2 Key SQL functions (SECURITY DEFINER, search_path pinned, sp_app-only [OP conventions])
|
||||
|
||||
- **`review_events` access contract (binding)**: enforced as an RLS policy *in addition to* `tenant_isolation` and re-checked as a **mandatory service-layer precondition** on every read/write of the table. **Read** requires `draft.view` **AND** (`drafts.group_id IS NULL` **OR** an active `group_members` row for `drafts.group_id`). **Writes of `kind='message'`** additionally require `review_role IN ('member','editor','reviewer')` in that group. Fails closed: missing membership/permission yields no rows (read) or an exception (write). The `GET /publishing/drafts/:id/events` endpoint inherits exactly this predicate - never a generic `RbacGuard + RLS` passthrough. **[AUDIT #18]** The same endpoint serves reads only through the bounded-page contract of §7 (`limit` default 100 / max 500 with `next_cursor`) — there is no unbounded read of the event log.
|
||||
|
||||
- **Moderation visibility contract (binding) [AUDIT #15]**: every draft read endpoint — list, detail, versions, diffs, events — applies `drafts.hidden_at IS NULL AND drafts.deleted_at IS NULL` unless the caller holds `moderation.hide` (hidden rows) / `moderation.delete` (deleted rows). Enforced as an RLS policy on `drafts` (and, via the parent draft, on `post_versions`/`review_events`) plus a shared repository filter used by every query path — never only a UI/archive filter. Fails closed: a non-admin direct request for a hidden/deleted draft, its versions, diffs or events yields 404/no rows.
|
||||
|
||||
- `review_eligible_reviewers(draft uuid, workspace uuid)` → set of user ids: **[USER v2] if the draft's connection is `delegated`, its `connection_approvers`; else** group role `reviewer` — always ∩ active membership ∩ NOT author ∩ NOT contributor ∩ NOT sealer of current version. **One function used by the service, the vote trigger, and the quorum count** [OP: single source of truth]. **[SECURITY #8]** The body starts with fail-closed tenant validation: ASSERT the target draft's `workspace_id` equals the `workspace` argument (and, when invoked for a session, that `current_setting('app.workspace_id')` IS NOT NULL and equals it); mismatch raises immediately - no rows, no cross-tenant leak.
|
||||
- `review_valid_approval_count(cycle uuid, workspace uuid)` → count of live (not revoked) `approve` votes from eligible reviewers whose `snapshot_sha256` matches the cycle's version hash [SOL]. **[SECURITY #8]** Same contract: the body first asserts the cycle's `workspace_id` equals the `workspace` argument (and matches `current_setting('app.workspace_id')` when session-scoped). The single-UUID signatures are removed; every caller passes both arguments.
|
||||
- `review_vote_eligible()` trigger: refuses votes unless cycle `in_review`, voter eligible (re-checked live), snapshot hash matches [SOL+OP]. **[AUDIT #14]** The cycle-state and live-eligibility checks apply only when `NEW.source = 'live'` (refusal condition is `... AND NEW.source = 'live'`), so `source='migrated'` backfill rows (§11) — inserted into already-`approved` cycles, possibly by a legacy reviewer who is now author/contributor or lacks the reviewer role — bypass them; the snapshot-hash match and all CHECK constraints are still enforced for every row. Only the `011` migration role may insert `source='migrated'` (sp_app inserts with `source <> 'live'` are refused).
|
||||
- `enforce_review_transition()` trigger on drafts: the legal-pairs table in §5; refuses `post`/`group_id` changes outside `editing`/`needs_changes`; refuses `approved` without a closed quorum or an override row **[AUDIT #19]** whose `snapshot_sha256` equals the cycle version's `content_hash` — this override-row requirement is the second layer on top of the 4.1(7) override insert trigger; refuses `scheduled` unless `drafts.post` hash equals the approved version's [SOL] — **[AUDIT #24]** §6.6 full-reset is the primary defence (an edit after approval supersedes the cycle before scheduling can even be attempted); this hash check is the backstop for any gap in it, and its refusal surfaces as 409 `STALE_VERSION` directing the user to re-submit — no approved-but-unschedulable state without a recovery transition. **[AUDIT #12]** `→ expired` is legal only from the pre-publication set `primed|in_review|blocked|approved` — the trigger refuses `scheduled → expired` and `published → expired`. **[SECURITY #9]** It also refuses `scheduled` unless the approved version's `connection_id` satisfies the 4.1(3) seal-time guard (same workspace, in scope for `drafts.group_id`) - re-validated at the transition, not only at seal time.
|
||||
- `assign_event_seq()` trigger: `event_seq := event_seq + 1` under the draft row lock; gapless per draft [OP]. **[AUDIT #26] Global lock order** (binding for every path that touches review state): authz rows (`group_members` / `connection_approvers`) → draft row → cycle row → votes/events. The vote path takes the draft row `FOR UPDATE` (seq + quorum + deadline check) *before* the cycle row `FOR UPDATE` and only then inserts the vote/event; the membership-change path takes authz rows → draft → cycle → votes/events in the same order. No path may lock a cycle row before its draft row.
|
||||
- `review_due_expiries()` sweeper discovery, mirroring `publishing_due_schedules()` (`006:79-85`) [OP]. **[SECURITY #8]** It takes no row argument, so it runs strictly under the caller's tenant context: every candidate cycle must satisfy `workspace_id = current_setting('app.workspace_id')::uuid`; a NULL/empty setting selects zero rows (fail-closed), never all tenants.
|
||||
- Vote revocation on membership/role change: trigger on `group_members` UPDATE/DELETE revokes that user's live votes on open cycles in the group (sets `revoked_at/reason`), then recount — quorum already met stays met [OP, fixed lock order: authz rows → draft → cycle → votes/events — the [AUDIT #26] global order above, identical to the vote path].
|
||||
|
||||
---
|
||||
|
||||
## 5. State machine (deliverable 3)
|
||||
|
||||
| from → to | fired by | guard |
|
||||
|---|---|---|
|
||||
| editing → primed | author/editor stages (choose group + connection + optional publish_at) | group has ≥ N eligible reviewers (**preflight, 400 if not**) — solo exception [AUDIT #17]: N=1 with 0 eligible reviewers still primes (workspace `approval.grant` holders are the fallback pool, and if that is empty too, submit lands `blocked-at-submit`) [GLM] |
|
||||
| needs_changes → primed | author re-stages after edits | same preflight |
|
||||
| expired → primed | author re-primes with new schedule | same preflight |
|
||||
| primed → in_review | submit (seals version n, opens cycle) | version sealed; policy snapshotted; deadline = LEAST(publish_at, now+stale_after); same tx inserts `draft_contributors` rows for the sealer + any editor since the last seal [AUDIT #10, seal trigger enforces] | **[AUDIT #17]** if N=1 and eligibility is 0, the cycle is created directly in `blocked` (`blocked-at-submit`) instead of `in_review` — a cycle always exists for the admin override |
|
||||
| in_review → approved | quorum_met (Nth valid approve, same tx) | row lock + `clock_timestamp() < deadline_at` else → expired [ALL] |
|
||||
| in_review → needs_changes | deny (veto) | reason optional, UI prompts for it [USER v2] |
|
||||
| in_review → editing | author withdraws for edits | cycle superseded |
|
||||
| in_review → blocked | recount after reviewer removal | eligibility < required [GLM] |
|
||||
| approved → scheduled | quorum tx (auto, if publish_at future) | same tx inserts `scheduled_publications`, starts workflow with startDelay [ALL] |
|
||||
| approved → editing | late edit | pending schedule cancelled in same tx; cycle superseded; loud dismissal event; **[AUDIT #21]** `approved_version_id = NULL` is set in the same tx as the supersede — between supersede and the next approval no reader (D1's claim check included) can dereference a stale approval |
|
||||
| scheduled → approved | cancelSchedule (existing) | unchanged from 006 |
|
||||
| scheduled → published | scheduler activity | **D1 fix: claim requires state='scheduled' AND a pending `scheduled_publications` row, and publishes `post_versions.post` of `approved_version_id`** |
|
||||
| published → (none) | archive = filtered read of published | no state change [GLM] |
|
||||
| primed, in_review, blocked, approved → expired | sweeper/vote-time check when deadline passes | pre-publication (deadline-bearing) states only — the explicit set `primed|in_review|blocked|approved → expired`; `scheduled → expired` and `published → expired` are refused by the trigger [AUDIT #12]; no remote publish; notification hook |
|
||||
| blocked → in_review | reviewer added (auto-recount) | eligibility ≥ required again [GLM] |
|
||||
| blocked/expired → approved | admin override | reason mandatory; not a contributor; cycle closed `overridden` [SOL+OP] |
|
||||
|
||||
Service returns 400/403/409 with precise codes (`REVIEW_EXPIRED`, `QUORUM_UNREACHABLE`, `STALE_VERSION`, `VOTE_INELIGIBLE`); the DB trigger is the backstop, not the error surface [SOL].
|
||||
|
||||
---
|
||||
|
||||
## 6. Quorum engine semantics (deliverable 5)
|
||||
|
||||
1. **Config**: `groups.review_approvals_required ?? workspace_review_settings.approvals_required ?? 1`, resolved once at submit, stored on the cycle with `policy_source`. Default `1` preserves today's behavior for solo workspaces **as a `blocked-at-submit` cycle, never a self-approval [AUDIT #17]**: with N=1 and 0 eligible reviewers the cycle is created in `blocked` and is resolved by a fallback reviewer (below) or the admin override. `≥2` turns on structural four-eyes [GLM]. Admin-only setters (`review.manage`). **[AUDIT #23]** `required` is bounded to 1–5 at every layer (workspace §4.1(2), group §4.1(2), cycle snapshot §4.1(5)): `required=0` has no closing rule — no transition fires `quorum_met` without a vote and submit-time auto-approve is deliberately not defined — so it is refused by CHECK at the door, exactly the deadlock-by-config class §6.8 exists to refuse.
|
||||
2. **Parallel any-N-of-eligible** [ALL]. No tiers (noted as future). **Eligibility pool [USER v2]**: if the draft's target connection is `delegated`, the pool is its `connection_approvers` (minus author/contributors); otherwise the group's `reviewer`s. **Solo fallback [AUDIT #17]**: when that pool is empty, workspace `approval.grant` holders (Owner/Admin/Approver, minus author/contributors) become eligible; if they are empty too, submit lands `blocked-at-submit` with the admin override as the only exit. Resolved at submit and snapshotted on the cycle (`approval_source`); mode changes affect future cycles only.
|
||||
3. **Vote rules**: one live vote per reviewer per cycle (`UNIQUE`); decision immutable after cast — a reviewer may not flip; they may deny later if they first approved? **No: one vote total, immutable.** To change course, the deny path is a *new* cycle after withdrawal — keep it strict [MERGE, strictest of the three].
|
||||
4. **Deny = veto** [ALL]: cycle closes `denied`, draft `needs_changes` [USER v2: deny reason **optional but suggested** — the UI asks "why?" and records it in chat + audit when given; this supersedes the spec's mandatory-reason rule]. Sibling approvals remain as history; count for nothing (cycle-scoped tally).
|
||||
5. **Staleness**: every vote copies the cycle's version hash; the vote trigger rejects a hash mismatch (409 `STALE_VERSION` — can't happen given full-reset, backstop only) [SOL].
|
||||
6. **Edit = full reset** [ALL]: any content/target change after seal supersedes the cycle, cancels any pending schedule in the same tx, seals version n+1 on resubmit, and emits a `edited` event listing dismissed approvers with a diff link (loud dismissal [OP]). **[AUDIT #10] Same-tx contributor contract (binding)**: every seal (`submit`/`edit`/`reopen`) inserts `draft_contributors` rows for the sealer and for any editor recorded since the last seal, in the seal transaction itself; the `post_versions` seal trigger fails the transaction if the sealer is absent — eligibility can never miss a contributor, and existing rows are immutable (§4.1(4)).
|
||||
7. **Scheduled-time race** [ALL]: quorum handler, holding the draft row lock, checks `clock_timestamp() < publish_at`; if true → schedule in same tx; else → `expired`. Submissions require `publish_at ≥ now() + 120 s` (min lead). Correctness never depends on sweeper latency.
|
||||
8. **Deadlock** [GLM]: preflight at priming (§5) + `blocked` state on mid-review shrinkage + override exit (§5). `blocked ≠ needs_changes`: "edit the post" is wrong advice when the problem is staffing.
|
||||
|
||||
---
|
||||
|
||||
## 7. Review chat & delivery (deliverable 6)
|
||||
|
||||
- **One table, both kinds** [SOL]: `review_events` — `message` rows are human chat; everything else is system events rendered inline (submitted, votes **fully attributed** — transparent ledger, reasons shown when given [USER v2] — state changes, overrides, moderation).
|
||||
- **Transport = polling** [GLM]: `GET /publishing/drafts/:id/events?after=<seq>&limit=<n>` while a detail view is open (4 s); queue lists refresh on action or 15 s. Every poll rides the existing epoch-fenced `RbacGuard` + RLS, i.e. the 4.2 `review_events` policy (read = `draft.view` AND (ungrouped draft OR active `group_members` row for `drafts.group_id`); posting a `message` additionally requires `review_role IN ('member','editor','reviewer')` in that group) — revocation latency = poll interval, no new credential surface (EventSource can't send our Bearer header).
|
||||
- **Bounded pages [AUDIT #18]**: `limit` defaults to 100 and is clamped to 1–500; each response returns at most `limit` events plus `next_cursor` = the `seq` of the last row returned (omitted when the page is not full). Clients loop `?after=<next_cursor>` until no cursor is present, so a 4 s poll — or a backlog catch-up — always returns a bounded page (≤ 500 events), never an unbounded response.
|
||||
- **Ordering is structural**: gapless `seq` allocated under the draft row lock; the cursor is a total order; lost/duplicated polls are harmless (`?after=`).
|
||||
- **Upgrade path**: SSE doorbells + `LISTEN/NOTIFY` [OP] later; identical `after=<seq>` query, no schema change.
|
||||
- Approve/Deny buttons live in the chat surface (R5): approve = one click; deny = reason modal (required).
|
||||
|
||||
---
|
||||
|
||||
## 8. UI plan (deliverable 7)
|
||||
|
||||
1. **Review queue** (replaces the `/approvals` stub): virtualized column list (reuse `components/streams/column.tsx` pattern, `@tanstack/react-virtual`, j/k navigation) of drafts in `primed/in_review/needs_changes/blocked/expired`; state chips (`STATE_TONE` extended); quorum progress ("2/3 ✓"); filters: needs-my-review, mine, needs-changes, expiring soon, blocked.
|
||||
2. **Post detail** (builds on Compose patterns): rendered post preview + version diff (v n-1 → n), side chat/event stream with vote buttons, quorum tracker, schedule info, contributor list.
|
||||
3. **Compose additions**: "Prime for review" action (group + connection + optional schedule), replacing the current self-approve chain when quorum ≥ 2.
|
||||
4. **Archive**: filtered read of `published` (chat history preserved, read-only), admin sees hidden posts with a badge + unhide/delete actions.
|
||||
5. **Team/group screens**: `review_role` selector on group members; **group settings** (quorum, admin control) [USER v2].
|
||||
6. **Connections screen additions [USER v2]**: per-account approval settings — mode picker (`system` vs `delegated to named approvers`), approver pool editor for delegated accounts, shown on the connection detail; the queue groups/filters by account so a delegated approver sees "awaiting my signoff" across the accounts they own.
|
||||
|
||||
---
|
||||
|
||||
## 9. Edge cases (deliverable 8) — each with its resolution
|
||||
|
||||
1. **Reviewer leaves mid-review** → votes revoked by trigger (reason), recount; met quorums stay met; unmet → `blocked` [OP lock order: authz rows → draft → cycle → votes/events — global order [AUDIT #26], shared with the vote path so a concurrent vote cannot invert draft/cycle locks].
|
||||
2. **Deadline race (vote vs expiry)** → both take `FOR UPDATE` on the draft; vote-time expiry check; loser sees `REVIEW_EXPIRED` [ALL].
|
||||
3. **Sole reviewer is the author** → with quorum ≥ 2: preflight 400 at priming. With quorum 1: prime/submit succeeds — the pool falls back to workspace `approval.grant` holders [AUDIT #17], and if that pool is empty too the cycle is created `blocked-at-submit` awaiting admin override; never enters an unresolvable state [GLM].
|
||||
4. **Admin who contributed tries to override** → refused structurally (contributors table) [OP].
|
||||
5. **Quorum met after publish_at** → `expired`, nothing scheduled, author notified, re-prime path [ALL].
|
||||
6. **Concurrent Nth and (N+1)th vote** → both under row lock; second lands as history after cycle closed (no double schedule: `UNIQUE(draft_id)` on scheduled_publications backstop).
|
||||
7. **Epoch bump mid-vote** (role change logs reviewer out) → `POST /auth/reissue` + client single retry (D7 fix) [OP].
|
||||
8. **Group deleted while cycles open** → `ON DELETE SET NULL (group_id)` frees the draft **and sets `review_cycles.group_id` to NULL the same way [AUDIT #16]** — each cycle keeps its `group_name` snapshot so the audit trail stays readable, and superseding a cycle preserves the reference as history rather than deleting it; open cycles superseded by group-delete guard trigger; audit event [OP].
|
||||
9. **Workspace owner cleanup deletes drafts** → cascades flow (versions, cycles, votes, events) under the single §4.1 contract — **[AUDIT #1]** `review_votes` and `review_overrides` reference `review_cycles` with `ON DELETE CASCADE`, so drafts → cycles → votes/overrides is never blocked by a NO ACTION FK: events/versions/votes are update-immutable and cascade-deletable **only via `drafts`**; direct DELETEs are refused by the trigger [GLM, AUDIT #11].
|
||||
10. **Hidden post with open cycle** → hide is metadata only; cycle continues (admins act with full information) [MERGE].
|
||||
11. **Connection approval mode switched mid-review** [USER v2] → the cycle snapshotted `approval_source` at submit; open cycles keep their pool, new cycles use the new mode. Mode change emits a thread event for visibility.
|
||||
12. **Delegated pool shrinks below N** (approver removed/leaves) → same `blocked` handling as group reviewers: recount trigger revokes their live votes, `blocked` if quorum unreachable, admin override or pool restore resolves.
|
||||
13. **Preflight for delegated accounts** counts the connection pool, not group reviewers: "Account X needs 2 approvals; pool has 1" at priming.
|
||||
|
||||
---
|
||||
|
||||
## 10. Audit events (deliverable 9) — all through `audit_append`, same-tx (D4 fixed)
|
||||
|
||||
`review.primed`, `review.submitted` (version hash, policy snapshot, approval_source), `review.vote_cast` (decision; reason when provided; snapshot hash), `review.vote_revoked`, `review.quorum_met`, `review.denied`, `review.cycle_superseded`, `review.blocked`, `review.unblocked`, `review.overridden` (admin, reason), `review.expired`, `review.hidden`/`unhidden`/`deleted` (moderation, reason), `review.policy_changed`, `review.connection_delegation_changed` (mode/pool changes [USER v2]).
|
||||
Scheduler keeps `publication.published`/`failed` as `sp_scheduler`. Hash v2 (010) makes every diff value — snapshot hashes, reasons, policy before/after — tamper-evident [OP].
|
||||
|
||||
---
|
||||
|
||||
## 11. Migration & backfill (deliverable: Q9) — `011`, per-tenant `set_config` before guard triggers
|
||||
|
||||
| legacy state | new state | cycle | votes |
|
||||
|---|---|---|---|
|
||||
| editing | editing | none | none |
|
||||
| in_review | needs_changes | none (author re-primes) | none |
|
||||
| rejected | needs_changes | none | none |
|
||||
| approved/scheduled/published | unchanged | backfilled cycle `required=1, status=approved`, v1 from current post | one synthetic `approve` vote (source='migrated') by legacy `reviewer_user_id` where present — exempt from the live cycle-state/eligibility checks of `review_vote_eligible()` via its `NEW.source = 'live'` guard (§4.2), hash still verified [AUDIT #14] |
|
||||
| (none) | — | — | legacy deny reasons recovered from `audit_log.diff->>'reason'` on `draft.rejected`, else migration note |
|
||||
|
||||
`reviewer_user_id` becomes read-only/deprecated; dropped in a later `012` once web stops reading it. Legacy self-approved drafts are kept as history and counted in a `review.migrated` audit row [OP].
|
||||
|
||||
---
|
||||
|
||||
## 12. Phased implementation (deliverable 10)
|
||||
|
||||
- **WP5.0 — Foundation fixes (010)**: D1–D8. Gate: existing suites green + new regression tests for claim-state-check and audit hash v2.
|
||||
- **WP5.1 — Schema (011)**: tables, triggers, functions, RLS, backfill. Gate: migration up/down on dev DB; `audit_verify_chain()` true post-backfill.
|
||||
- **WP5.2 — Service + API**: priming/submit/vote/override/moderation endpoints, quorum engine, event log writes; every draft read endpoint (list, detail, versions, diffs, events) applies the §4.2 moderation visibility contract. Gate: integration tests (RLS, transitions, quorum, races) on live Postgres, including admin vs non-admin reads of hidden/deleted drafts across all five read endpoints.
|
||||
- **WP5.3 — Scheduler contract**: claim eligibility + approved-version publishing + expiry sweeper. Gate:Temporal tests incl. deadline race.
|
||||
- **WP5.4 — Web**: queue, detail+chat, compose changes, archive, role/quorum admin. Gate: E2E on real API; visual QA.
|
||||
- **WP5.5 — Hardening**: lock-order verification, load test on event seq allocation, docs.
|
||||
|
||||
## 13. Test plan sketch (deliverable 11) — the 10 that catch the worst
|
||||
|
||||
1. Claim refuses any draft not in `state='scheduled'` with a pending `scheduled_publications` row, and publishes `post_versions.post` of `approved_version_id` (D1). 2. Audit verify passes with v1+v2 rows. 3. Author/contributor vote → 403. 4. Deny without reason succeeds; deny with reason records it in thread + audit [USER v2]. 5. Nth vote under lock schedules exactly once. 6. Vote at deadline → expired, not scheduled. 7. Edit after approval cancels schedule + supersedes cycle same-tx. 8. Reviewer removal revokes votes; met quorum survives. 9. Preflight blocks solo-author quorum-2 priming. 10. Migration backfill: verify chain + counts + resubmit path from legacy rejected. 11. Delegated account: group reviewer's vote → 403; pool member's vote counts [USER v2]. 12. Mode switch mid-review: open cycle keeps its pool [USER v2]. 13. Delegated pool shrink → blocked + override path [USER v2]. 14. **Transition-table test [AUDIT #12]**: `enforce_review_transition()` refuses `scheduled → expired` and `published → expired`; `→ expired` succeeds from each of `primed`, `in_review`, `blocked`, `approved`. 15. **Seal/contributor test [AUDIT #10]**: a seal transaction that does not insert a `draft_contributors` row for the sealer fails; UPDATE/DELETE of an existing contributor row fails outside the drafts cascade. 16. **Edit-after-approval-then-schedule [AUDIT #24]**: both paths — (a) primary: an edit after approval supersedes the cycle and cancels the pending schedule in the same tx (draft back to `editing`, `approved_version_id = NULL`); (b) backstop: `approved → scheduled` while the `drafts.post` hash no longer equals the approved version's is refused with 409 `STALE_VERSION` and the UI directs the author to re-submit. 17. **Draft-delete cascade [AUDIT #1]**: under the real `011` migration, deleting a draft that has cast votes (live and revoked) and an `overridden` cycle with a `review_overrides` row succeeds and removes its cycles, votes, overrides, events and versions; a direct DELETE of a vote/override row is still refused.
|
||||
|
||||
## 14. Confidence & risks
|
||||
|
||||
- **Highest confidence**: schema shape, single-veto deny, full-reset staleness, cycle-snapshotted quorum (all three proposals converged independently).
|
||||
- **Least sure**: (a) gapless `seq` allocation under extreme concurrency — measure contention in WP5.5; (b) blind-vote redaction completeness (audit page leak accepted, documented); (c) D7 reissue flow client behavior across all screens.
|
||||
- **Prototype first**: WP5.0 + WP5.1 on a copy of dev data, run tests 1, 5, 6, 8 — everything whose failure would be silent.
|
||||
987
docs/design/proposal-glm.md
Normal file
987
docs/design/proposal-glm.md
Normal file
|
|
@ -0,0 +1,987 @@
|
|||
# Proposal — Group Review & Signoff for Posts ("Editorial Workflow")
|
||||
|
||||
Lane: `glm` · Date: 2026-10-07 · Spec: `runs/wp5/spec-review-workflow.md`
|
||||
Status: design only — nothing here is implemented.
|
||||
|
||||
---
|
||||
|
||||
## 0. Files this proposal is grounded in
|
||||
|
||||
Everything below cites these repo files (paths relative to repo root):
|
||||
|
||||
| File | What it establishes |
|
||||
| --- | --- |
|
||||
| `apps/api/migrations/006_publishing.sql` | current 6-state CHECK (`:7-8`), single `reviewer_user_id` (`:9`), `UNIQUE(draft_id)` on schedules (`:24`), FORCE RLS + `tenant_isolation` policies (`:52-66`), `sp_app` grants (`:68`), SECURITY DEFINER sweeper fns (`:72-89`) |
|
||||
| `apps/api/src/publishing/publishing.service.ts` | `DraftState` union (`:8`), submit/approve/reject (`:179-205`), schedule (`:212-247`), CAS `transition()` helper `WHERE id=$1 AND state=$4` (`:275-292`), the "FK checks bypass RLS" tenancy note (`:217-219`) |
|
||||
| `apps/scheduler/src/activities/publish.activity.ts` | scheduler owns `scheduled→published` (`:182`), claims via `publishing_schedule_workspace()` (`:160-176`), writes audit as `sp_scheduler` (`:184-199`), tenant-tx pattern `inTenant()` (`:228-245`) |
|
||||
| `apps/api/migrations/008_groups.sql` | `group_members` has **no role column** (`:27-39`), `UNIQUE(group_id,user_id)` (`:34`), composite tenant-pinning FKs (`:38-39`), epoch bump on any membership mutation (`:79-80`), `groups UNIQUE(id, workspace_id)` (`:21`) |
|
||||
| `apps/api/src/groups/groups.service.ts` | same-tx audit helper (`:100-110`), `{token}` replacement-session pattern on epoch-bumping mutations (`:168-200`), guard-side `userCanUseConnection` reading live role from DB (`:425-455`) |
|
||||
| `apps/api/migrations/002_rbac_seed.sql` | 10 roles; `approval.grant` on Owner/Admin/Approver (`:8, :24, :49`), `approval.request` on Owner/Admin/Publisher, workspace `Editor` role has `draft.edit` and **no** approval power (`:39-43`) |
|
||||
| `apps/api/src/rbac/rbac.guard.ts` | epoch fail-closed check (`:45-47`), permission lookup inside `withTenant` (`:57-62`), `RequireGroupAccess` precedent for guard-side group checks (`:64-74`) |
|
||||
| `apps/api/src/audit/audit.service.ts` + `apps/api/migrations/001_tenants.sql` | `scrub()` (`:11`), `AuditEvent` (`:20-27`), `audit_append` SECURITY DEFINER + `fn_assert_tenant` (`001:85-111`), `audit_hash` (`001:74-83`), `reject_workspace_change` (`001:141-150`), `bump_authz_epoch` (`001:160-168`), fail-closed NULLIF RLS (`001:172-190`) |
|
||||
| `apps/api/src/auth/auth.service.ts` / `session.ts` | `issueSessionInTenant` (`:72-84`), `epochValid` (`:50-52`) |
|
||||
| `apps/api/src/db/client.ts` | `withTenant` sets `app.workspace_id` per tx (`:18-44`) |
|
||||
| `apps/web/src/app/(main)/compose/page.tsx` | `STATE_TONE` badge map (`:35-42`), the create→submit→approve→schedule chain incl. self-approve (`:119-142`), preview cards (`:244-262`), "Recent drafts" list (`:255-263`) |
|
||||
| `apps/web/src/components/streams/column.tsx` + `streams/page.tsx` | virtualizer config (`column.tsx:26-34`), j/k keyboard model with input-guard (`streams/page.tsx:56-70`) |
|
||||
| `apps/web/src/app/(main)/inbox/page.tsx` | two-pane chat visual pattern (`:52-120`) |
|
||||
| `apps/web/src/app/(main)/approvals/page.tsx` | 10-line stub this feature replaces |
|
||||
| `apps/api/src/realtime/gateway.ts` | authz **port only** — no transport, no topics, no client (`:1-45`) |
|
||||
| `apps/api/migrations/009_video.sql` | takes number `009`; this proposal's migration is **`010_review_workflow.sql`** |
|
||||
| `apps/api/src/db/migrate.ts` | each migration file runs in one tx, exactly once (`:33-41`) |
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive summary — the five sharpest decisions
|
||||
|
||||
1. **Group roles are a column on `group_members`** (`role ∈ member|editor|reviewer`), not a new table and not workspace roles. The existing epoch trigger (`008_groups.sql:79-80`) already fires on UPDATE, so role changes invalidate sessions for free and the `{token}` re-issue pattern (`groups.service.ts:168-200`) extends verbatim. One role per user per group is a *feature*: letting one person be both editor and reviewer in the same group would re-open the maker-checker hole we are structurally closing.
|
||||
|
||||
2. **Approvals attach to immutable version rows, never to the draft.** `post_versions` is append-only (UPDATE blocked by trigger), carries a `content_hash` generated column (`encode(digest(post::text,'sha256'),'hex')`, same crypto as `audit_hash` in `001:74-83`), and a review cycle pins one version. Any edit — before, during, or after review — closes the cycle as `superseded` and requires a fresh quorum. There is no state in which content is mutable while approvals are live.
|
||||
|
||||
3. **Quorum is snapshotted per review cycle**, resolved as `groups.review_approvals_required ?? workspace_review_settings.approvals_required ?? 1` at submit time. Mid-review config changes never retroactively move the goalposts; only future cycles pick them up. Default `1` preserves today's behavior exactly (including the solo-workspace self-approve flow at `compose/page.tsx:128-131`); `≥2` turns on structural four-eyes, at which point author votes are refused server-side with a dedicated 403.
|
||||
|
||||
4. **Chat is polling with a per-thread monotonic cursor**, not WS/SSE. `realtime/gateway.ts` is an unwired port; EventSource cannot send the Bearer token `api-client.ts` stores in localStorage; and every poll rides the existing epoch-fenced `RbacGuard` (`rbac.guard.ts:45-47`) for free, so an admin removing a reviewer mid-thread fails the *next poll* closed. The data model (`review_messages` with `UNIQUE(draft_id, ord)`) is transport-agnostic so WS can later reuse the identical "events since ord" query.
|
||||
|
||||
5. **`published` is the archive; hide and delete are flags, not states.** The scheduler's `scheduled→published` write (`publish.activity.ts:182`) needs zero changes, the archive screen is a filtered read of `state='published'`, and moderation (`hidden_at`, `deleted_at`) is admin-only (`moderation.hide`, `moderation.delete` granted to Owner/Admin only), soft, and hash-chain-audited — the chat record survives deletion because review history is a compliance artifact.
|
||||
|
||||
The five new machine states are `needs_changes` (replaces terminal `rejected`, unlocks resubmit), `expired`, `blocked`, plus reuse of `editing/in_review/approved/scheduled/published`; the state machine is enforced **twice** — CAS UPDATEs in `publishing.service.ts` style and a new DB legal-transition trigger (defense in depth, per spec §3).
|
||||
|
||||
---
|
||||
|
||||
## 2. Answers to the ten open questions (spec §5)
|
||||
|
||||
**Q1 — Group-scoped role storage: column on `group_members` vs `group_member_roles` table vs reusing workspace roles?**
|
||||
**Decision: `group_members.role text NOT NULL DEFAULT 'member' CHECK (role IN ('member','editor','reviewer'))`.**
|
||||
Reusing workspace roles is disqualified by construction: `memberships` has `UNIQUE(workspace_id,user_id)` (`001_tenants.sql:30`), i.e. exactly one role per user per workspace, and the seed's `Editor`/`Approver` are workspace-wide — a group-scoped Approver would approve everything. A separate `group_member_roles` table buys multi-role-per-group, which we explicitly do not want (editor+reviewer in one group = the same person edits and approves; that is the conflict four-eyes exists to prevent) at the cost of a second cascade path, a second epoch trigger, and a join in every eligibility check. The column reuses the table `groups.service.ts` already owns, and — decisively — `trg_group_member_bump_epoch` (`008_groups.sql:79-80`) fires on `UPDATE`, so a role change already invalidates every live session; the service just has to return the caller's replacement token exactly like `addMember`/`removeMember` do today (`groups.service.ts:168-200`). *Future work:* if per-group role delegation ("reviewer for text, not for media") ever lands, promote to a table then; the CHECK constraint is the only thing to widen.
|
||||
|
||||
**Q2 — Snapshot model: full content copies per version vs content-hash + immutable version rows?**
|
||||
**Decision: immutable version rows that contain the full post AND a stored generated hash — `post_versions(post jsonb, content_hash text GENERATED ALWAYS AS (encode(digest(post::text,'sha256'),'hex')) STORED)`.**
|
||||
These are usually presented as opposites; they shouldn't be. A hash alone can't be rendered or diffed, so the reviewer's "approve exactly what you see" surface (R4) would need to reconstruct content from somewhere — the moving target we're eliminating. Full copies make the version row the single source for preview, diff, and publication, and posts are tiny (`drafts.post jsonb`, `006_publishing.sql:6`), so copy cost is irrelevant. Making the hash a *generated* column (pgcrypto is already loaded, `001:3`, and `digest` is what `audit_hash` uses, `001:74-83`) means the hash can never disagree with the content — no TS-side canonicalization bugs, no `as any` json wrangling. Rows are append-only via a `BEFORE UPDATE` rejection trigger; `UNIQUE(draft_id, version)` gives stable version numbers for the UI ("v3 of 4"). *Future work:* content-addressed dedup (`UNIQUE(workspace_id, content_hash)`) if cross-post reuse ever matters.
|
||||
|
||||
**Q3 — Where does quorum config live and what's the precedence?**
|
||||
**Decision: workspace default in a new 1:1 `workspace_review_settings` table (`approvals_required int NOT NULL DEFAULT 1 CHECK (… BETWEEN 1 AND 5)`), overridable per group by a nullable `groups.review_approvals_required`; precedence at **submit** time is `groups.review_approvals_required ?? workspace_review_settings.approvals_required ?? 1`, and the resolved value is **snapshotted onto the review cycle** (`review_cycles.required_approvals`).**
|
||||
The snapshot is the load-bearing part: a quorum rule that can change under an open review either silently promotes a post past a threshold the admin just raised, or strands it — both feel like tampering to reviewers. Snapshotting makes each cycle a closed contract ("2 approvals of version 3") and makes config changes purely forward-looking. Both setters are admin-only via `review.manage` (granted to Owner/Admin — matching `workspace.members.manage` holders in `002:7, :23`, satisfying R3's "who sets it"). A separate settings table (rather than a column on `workspaces`) keeps the migration additive and lets RLS/policy/grants copy the standard shape; `groups` already has an `ALTER` in this migration anyway. *Future work:* per-connection quorum (spec mentions it) is deliberately out — drafts target one connection but reviews are editorial, not per-channel.
|
||||
|
||||
**Q4 — Chat transport: WS vs SSE vs polling?**
|
||||
**Decision: HTTP polling of a cursor endpoint, `GET /publishing/drafts/:id/messages?after=<ord>`, 4s while a detail view is open; queue lists refresh on action or every 15s.**
|
||||
Cost/complexity first: WS requires building the entire missing transport — `realtime/gateway.ts` is an authorization port with no socket server, no topic registry, no client (`gateway.ts:1-45`) — plus heartbeat/reconnect, plus an epoch-aware re-auth dance (any group membership change anywhere in the workspace invalidates the token the socket authenticated with, `008:79-80`, so sockets need mid-flight re-authentication or mass disconnect-reconnect). SSE is cheaper but `EventSource` cannot set an `Authorization` header, and this app authenticates every request with a Bearer token from localStorage (`api-client.ts:57-59`) — we'd have to mint query-param or cookie tokens, a new credential surface. Polling reuses `RbacGuard` + tenant RLS + the epoch check verbatim on every request, so revocation latency equals poll interval, and review-thread volume (a handful of messages per post) makes the cost negligible. Ordering is guaranteed structurally, not by transport: `ord` is assigned per-thread under a row lock on the draft (`UNIQUE(draft_id, ord)` backstop), so the cursor is a total order regardless of transport. *Future work:* the WS gateway should reuse `authorizeSubscription` plus the identical `after=<ord>` query — the data model was chosen so transport can swap without schema change.
|
||||
|
||||
**Q5 — Deny semantics: 1-deny-returns-to-author vs majority-deny vs configurable; what happens to other cast approvals?**
|
||||
**Decision: single deny (reason mandatory) is a veto — cycle closes `denied`, draft → `needs_changes`. Other cast approvals are preserved immutably as history but belong to the closed cycle, so they count toward nothing on resubmit.**
|
||||
A reviewer empowered to approve is empowered to block; anything weaker (majority-deny) lets a post publish over an explicit, reasoned objection — the exact failure four-eyes exists to prevent — and configurable deny modes multiply the state space for no safety gain. Preserving rather than deleting sibling approvals matters for the record ("two had approved, one objected, here's why") and costs nothing because quorum is always evaluated *within a cycle*: `review_votes.cycle_id` scopes every tally, so "revoking" approvals is unnecessary — the next cycle simply starts at zero. The deny reason is mandatory at the API boundary (`400` if blank — same `requireString` semantics as `http/validation.ts:23-28`) and lands in both the thread (system event, R5) and the audit chain (R8). *Future work:* a group-level `deny_threshold` config if some workspace ever wants consensus-style review.
|
||||
|
||||
**Q6 — Re-review after edit: full reset vs dismiss-only-changed-scope?**
|
||||
**Decision: full reset. Any content mutation after submission closes the cycle (`superseded`) and requires a new version, a new cycle, and a fresh quorum.**
|
||||
Partial dismissal requires knowing *which part* of a post each approval covered; there is no field-level approval scope in this system (nor in Hootsuite/Sprout per the research digest — both lock the whole post). Full reset is the only rule that makes "approve a frozen snapshot" structurally true: there is never a moment where `drafts.post` is mutable and an approval is live, which means the reviewer's diff view can be trusted without checking timestamps. It's also the cheapest to reason about under concurrency (one immutable version per cycle; no merge of partial validity). The cost — re-getting N approvals after a typo fix — is real, so the UI shows the version diff in the re-approval request ("v3→v4: 1 line changed") to make re-approval a two-second decision. *Future work:* "minor edit" carve-outs (e.g. schedule-only changes don't reset — which this design already grants: `publish_at` lives on the cycle, not the content, so rescheduling without quorum is impossible by the same rule; only content resets).
|
||||
|
||||
**Q7 — Scheduled-time race: quorum completes at T-1min — auto-publish or manual re-confirm?**
|
||||
**Decision: auto-publish. The quorum-met handler creates the `scheduled_publications` row and starts the workflow iff `publish_at > now()`; if quorum lands after `publish_at`, the draft goes to `expired` and nothing is ever scheduled.**
|
||||
The content approved is bit-identical to what publishes (frozen version), the approvals are valid, and the schedule was declared up front ("primed to go out", user req #1) — a re-confirm step at T-1min injects a human bottleneck precisely when humans are scarcest, and its failure mode is a missed post. The Temporal path already handles near-term starts (`startDelay = max(0, …)`, `publishing.service.ts:99-101`) and the durable-row sweeper (`publishing_due_schedules()`, `006:79-87`) already covers a missed workflow start, so no new machinery is needed for the happy path. The guard is the ordering rule: **schedules are only ever created for future times**, which makes "approved too late" equivalent to "not approved in time" — the single `expired` state covers both R1 cases (deadline passed, and stale-beyond-90-days). The race between a last-instant vote and the expiry sweeper is serialized on the draft row (both take `FOR UPDATE`; see §9). *Future work:* per-workspace "re-confirm window" config for teams that want the brake.
|
||||
|
||||
**Q8 — Author is also the only reviewer in a tiny group — deadlock handling?**
|
||||
**Decision: refuse at the door, then a named exit. Submit runs a preflight — count of *currently eligible non-author reviewers* ≥ `required_approvals` — and returns `400 not enough eligible reviewers (0) for quorum (2)` instead of entering a state that can never resolve. If the pool shrinks mid-review (reviewer removed/demoted), the recount trigger moves the draft to `blocked`; Owner/Admin can resolve `blocked` via the override endpoint (mandatory reason, logged as override) or by adding reviewers, which auto-reopens the cycle.**
|
||||
A deadlock you can enter is a bug; a deadlock you're told about at submit time is a configuration error with a clear message. The preflight is cheap (one count over memberships/group_members) and converts the worst failure mode into the best UX. `blocked` is a distinct state — not `needs_changes` — because "needs changes" tells the author to edit, when the actual problem is staffing; conflating them sends authors rewriting posts that were fine. The override path (R2: "Admin/Owner retain override power, logged as override") is the emergency brake for both deadlock and genuine urgency, and it is *not* a vote: it writes no `review_votes` row, it closes the cycle with `closed_reason='admin_override'` and a dedicated audit action, so quorum metrics can't be inflated by admin rescues. *Future work:* vacation delegation (a reviewer temporarily acting for another) — the group-role mechanism extends naturally.
|
||||
|
||||
**Q9 — Migration path for existing single-step approvals (`reviewer_user_id` backfill)?**
|
||||
**Decision: migrate in `010` — every existing draft gets a `post_versions` v1 row from its current `post`; drafts with `reviewer_user_id` get a backfilled cycle (`required_approvals=1`) plus one synthetic vote (`approve`, or `deny` whose reason is recovered from the audit chain via `audit_log.diff->>'reason'` on the matching `draft.rejected` event, falling back to a migration note); terminal `rejected` rows become `needs_changes`. The `reviewer_user_id` column is then dead: `010` stops writing it, keeps it readable for one release, and a later `012` drops it once `draftFromRow` (`publishing.service.ts:315-317`) and `api-types.ts:70` stop referencing it.**
|
||||
Dropping in the same migration that the running API still reads from would couple deploy order brittlely; the repo ships API and migration together (`migrate.ts` runs on boot) but the web bundle may lag by a session lifetime, so a deprecation window is the honest cost. Recovering deny reasons from `audit_log` is possible because rejects already write `{reason}` into the diff (`publishing.service.ts:197-205`) and the chain is queryable by `target_id` (`audit.query.ts:30-45`) — this keeps the mandatory-reason invariant true even for migrated rows (synthetic fallback text where history is missing). Historical intermediate versions are unrecoverable (`updateDraft` overwrote `post` in place, `:160-176`), so v1 = content at migration time; noted rather than hidden. *Future work:* none needed — `012` drop is the only follow-up.
|
||||
|
||||
**Q10 — Should reviewers see who else approved before them, or vote blind?**
|
||||
**Decision: blind until the cycle closes. While a cycle is `open`, the API returns only `{approvals: n, required: m, mine: …}` and system events in the thread are identity-masked ("an approval was recorded · 2/3"); on close (`approved`/`denied`/`superseded`) the full vote ledger with identities is revealed. Holders of `review.override` (Owner/Admin) see identities live.**
|
||||
Independence is the point of parallel review — the research digest flags rubber-stamping and anchoring as the failure modes of visible-first voting, and quorum metrics (approvals per reviewer, cycle time) only stay meaningful if early votes don't herd late ones. Masking is enforced server-side at the read endpoint (the `review_messages` row keeps the actor — the audit record must — but the serializer redacts `event_data.actorUserId` while `status='open'`), never trusted to the client. Admins see live because moderation is their job (spotting the rubber-stamper *during* review, not after). On close everything is revealed: the permanent record is fully transparent, which is what distinguishes this from secret balloting. *Future work:* per-group `blind_votes=false` toggle if a team insists on open voting.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model (deliverable 2) — `010_review_workflow.sql`
|
||||
|
||||
One file, one transaction (`migrate.ts:33-41`). Ordering inside the file matters and is annotated. Conventions copied everywhere: denormalized `workspace_id`, composite tenant-pinning FKs where a parent exists (`008:38-39` pattern), `reject_workspace_change` trigger, `ENABLE`+`FORCE ROW LEVEL SECURITY`, the exact `tenant_isolation` NULLIF policy (`001:176-179` / `006:58-66`), and `GRANT … TO sp_app` (`003`).
|
||||
|
||||
```sql
|
||||
-- 010: group review & signoff — post versions, review cycles, quorum votes,
|
||||
-- per-post chat, moderation flags. Extends 006 (publishing) and 008 (groups).
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (1) Group roles. Column, not a table (Q1): trg_group_member_bump_epoch
|
||||
-- (008:79-80) already fires on UPDATE, so role changes invalidate sessions
|
||||
-- and the service returns a replacement token (groups.service.ts:168-200).
|
||||
-- ---------------------------------------------------------------------------
|
||||
ALTER TABLE group_members
|
||||
ADD COLUMN role text NOT NULL DEFAULT 'member'
|
||||
CHECK (role IN ('member', 'editor', 'reviewer'));
|
||||
|
||||
COMMENT ON COLUMN group_members.role IS
|
||||
'member = author/chat only; editor = edit group posts, cannot approve/publish; reviewer = approve/deny with reason';
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (2) Quorum config (Q3). 1:1 with workspace -> same RLS shape as every table.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE workspace_review_settings (
|
||||
workspace_id uuid PRIMARY KEY REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
approvals_required int NOT NULL DEFAULT 1 CHECK (approvals_required BETWEEN 1 AND 5),
|
||||
stale_after_days int NOT NULL DEFAULT 90 CHECK (stale_after_days BETWEEN 1 AND 3650),
|
||||
updated_by uuid REFERENCES users(id),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
ALTER TABLE groups
|
||||
ADD COLUMN review_approvals_required int CHECK (review_approvals_required BETWEEN 1 AND 5);
|
||||
-- NULL = inherit workspace default. Resolved once, at submit, onto the cycle.
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (3) Immutable post versions (Q2, R4). Append-only; hash is GENERATED so it
|
||||
-- can never disagree with the content (pgcrypto digest, as in audit_hash
|
||||
-- 001:74-83).
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE post_versions (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL REFERENCES drafts(id) ON DELETE CASCADE,
|
||||
version int NOT NULL CHECK (version >= 1),
|
||||
post jsonb NOT NULL,
|
||||
content_hash text GENERATED ALWAYS AS (encode(digest(post::text, 'sha256'), 'hex')) STORED,
|
||||
created_by uuid NOT NULL REFERENCES users(id),
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (draft_id, version)
|
||||
);
|
||||
|
||||
CREATE INDEX post_versions_draft ON post_versions (draft_id, version DESC);
|
||||
|
||||
-- Append-only guard (new helper; workspace-delete cascades still work because
|
||||
-- only UPDATE is blocked, DELETE flows through ON DELETE CASCADE).
|
||||
CREATE FUNCTION reject_mutation() RETURNS trigger
|
||||
LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
RAISE EXCEPTION '% rows are append-only', TG_TABLE_NAME USING ERRCODE = '23514';
|
||||
END $$;
|
||||
|
||||
CREATE TRIGGER trg_post_versions_immutable BEFORE UPDATE ON post_versions
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_mutation();
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (4) Drafts: group scoping, version pointer, moderation flags (R7).
|
||||
-- Composite FK pins the draft's group to this workspace (008 pattern);
|
||||
-- NO ACTION (not SET NULL) because SET NULL would null workspace_id,
|
||||
-- violating NOT NULL + trg_draft_ws_immutable (006:41-42). Consequence:
|
||||
-- a group with posts cannot be deleted (deleteGroup gains a guard).
|
||||
-- ---------------------------------------------------------------------------
|
||||
ALTER TABLE drafts
|
||||
ADD COLUMN group_id uuid,
|
||||
ADD COLUMN current_version_id uuid REFERENCES post_versions(id),
|
||||
ADD COLUMN hidden_at timestamptz,
|
||||
ADD COLUMN hidden_by uuid REFERENCES users(id),
|
||||
ADD COLUMN deleted_at timestamptz,
|
||||
ADD COLUMN deleted_by uuid REFERENCES users(id),
|
||||
ADD CONSTRAINT drafts_group_fk
|
||||
FOREIGN KEY (group_id, workspace_id) REFERENCES groups(id, workspace_id);
|
||||
|
||||
CREATE INDEX drafts_queue ON drafts (workspace_id, state, updated_at DESC)
|
||||
WHERE deleted_at IS NULL;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (5) Review cycles: one per (draft, submission). Pins the version and the
|
||||
-- quorum snapshot (Q3). At most one open cycle per draft.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE review_cycles (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL REFERENCES drafts(id) ON DELETE CASCADE,
|
||||
version_id uuid NOT NULL REFERENCES post_versions(id),
|
||||
cycle int NOT NULL CHECK (cycle >= 1),
|
||||
required_approvals int NOT NULL CHECK (required_approvals BETWEEN 1 AND 5),
|
||||
status text NOT NULL DEFAULT 'open'
|
||||
CHECK (status IN ('open', 'approved', 'denied', 'superseded', 'expired', 'blocked')),
|
||||
publish_at timestamptz, -- "primed to go out": intent declared at submit
|
||||
connection_id uuid REFERENCES connections(id),
|
||||
opened_by uuid NOT NULL REFERENCES users(id),
|
||||
closed_at timestamptz,
|
||||
closed_reason text, -- 'quorum' | 'deny' | 'superseded_by_edit' | 'expired' | 'pool_below_quorum' | 'admin_override'
|
||||
UNIQUE (draft_id, cycle)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX review_cycles_one_open ON review_cycles (draft_id) WHERE status = 'open';
|
||||
CREATE INDEX review_cycles_open_by_age ON review_cycles (workspace_id, created_at) -- stale sweep
|
||||
WHERE status = 'open';
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (6) Votes (Q5, R6 in spec's R3). Immutable (one live vote per reviewer per
|
||||
-- cycle, no vote changes). Deny reason mandatory, approve reason optional —
|
||||
-- expressed as a CHECK so the DB enforces it too, not just the DTO parser.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE review_votes (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
cycle_id uuid NOT NULL REFERENCES review_cycles(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL REFERENCES drafts(id) ON DELETE CASCADE, -- denormalized for queue queries
|
||||
voter_user_id uuid NOT NULL REFERENCES users(id),
|
||||
vote text NOT NULL CHECK (vote IN ('approve', 'deny')),
|
||||
reason text CHECK (vote = 'approve' OR btrim(reason) <> ''),
|
||||
snapshot_hash text NOT NULL, -- copy of the version hash the voter saw
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (cycle_id, voter_user_id) -- one live vote per reviewer per submission
|
||||
);
|
||||
|
||||
CREATE INDEX review_votes_cycle ON review_votes (cycle_id, vote);
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (7) Review chat (R5): messages + system events in one append-only thread.
|
||||
-- `ord` is per-thread, assigned under the draft row lock -> total order,
|
||||
-- transport-agnostic cursor (Q4).
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE review_messages (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL REFERENCES drafts(id) ON DELETE CASCADE,
|
||||
ord int NOT NULL,
|
||||
kind text NOT NULL CHECK (kind IN ('message', 'system')),
|
||||
author_user_id uuid REFERENCES users(id), -- NULL for system events
|
||||
body text NOT NULL CHECK (btrim(body) <> ''),
|
||||
event text, -- system only: submitted|vote|state|…
|
||||
event_data jsonb NOT NULL DEFAULT '{}',
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (draft_id, ord)
|
||||
);
|
||||
|
||||
CREATE INDEX review_messages_thread ON review_messages (draft_id, ord DESC);
|
||||
|
||||
CREATE TRIGGER trg_review_messages_immutable BEFORE UPDATE ON review_messages
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_mutation();
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (8) Notification hook (R9): transactional outbox; consumers come later.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE review_notifications (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL REFERENCES drafts(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id), -- intended recipient
|
||||
kind text NOT NULL, -- 'review_requested'|'denied'|'quorum_met'|'expired'|'blocked'|'mentioned'
|
||||
payload jsonb NOT NULL DEFAULT '{}',
|
||||
consumed_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX review_notifications_inbox ON review_notifications (workspace_id, user_id)
|
||||
WHERE consumed_at IS NULL;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (9) Backfill BEFORE touching the state CHECK (the old CHECK only allows the
|
||||
-- six legacy values, so 'needs_changes' must be written after dropping it).
|
||||
-- Order: drop legacy CHECK -> rewrite 'rejected' -> add new CHECK.
|
||||
-- (Trg_draft_ws_immutable from 006:41-42 is unaffected: workspace_id
|
||||
-- never changes here.)
|
||||
-- ---------------------------------------------------------------------------
|
||||
ALTER TABLE drafts DROP CONSTRAINT drafts_state_check;
|
||||
|
||||
UPDATE drafts SET state = 'needs_changes' WHERE state = 'rejected';
|
||||
|
||||
ALTER TABLE drafts ADD CONSTRAINT drafts_state_check
|
||||
CHECK (state IN ('editing', 'in_review', 'needs_changes', 'approved',
|
||||
'scheduled', 'published', 'expired', 'blocked'));
|
||||
|
||||
-- v1 for every draft: content as it exists at migration time (intermediate
|
||||
-- versions were overwritten in place by updateDraft, publishing.service.ts:160-176).
|
||||
INSERT INTO post_versions (workspace_id, draft_id, version, post, created_by, created_at)
|
||||
SELECT d.workspace_id, d.id, 1, d.post, d.author_user_id, d.created_at
|
||||
FROM drafts d;
|
||||
|
||||
UPDATE drafts d SET current_version_id = pv.id
|
||||
FROM post_versions pv WHERE pv.draft_id = d.id AND pv.version = 1;
|
||||
|
||||
-- Legacy single-step approvals (Q9): one cycle + one synthetic vote each.
|
||||
-- opened_by has no historical value; the author is the least-wrong actor and
|
||||
-- closed_reason marks the row as migrated.
|
||||
INSERT INTO review_cycles
|
||||
(workspace_id, draft_id, version_id, cycle, required_approvals, status,
|
||||
publish_at, connection_id, opened_by, closed_at, closed_reason)
|
||||
SELECT d.workspace_id, d.id, pv.id, 1, 1,
|
||||
CASE d.state
|
||||
WHEN 'published' THEN 'approved'
|
||||
WHEN 'scheduled' THEN 'approved'
|
||||
WHEN 'approved' THEN 'approved'
|
||||
WHEN 'needs_changes' THEN 'denied' -- migrated from 'rejected'
|
||||
ELSE 'open' -- in_review: keeps working, one vote finishes it
|
||||
END,
|
||||
NULL, NULL, d.author_user_id,
|
||||
CASE WHEN d.state IN ('published','scheduled','approved','needs_changes')
|
||||
THEN d.updated_at END,
|
||||
'migrated_from_single_step'
|
||||
FROM drafts d
|
||||
JOIN post_versions pv ON pv.draft_id = d.id AND pv.version = 1
|
||||
WHERE d.reviewer_user_id IS NOT NULL OR d.state = 'in_review';
|
||||
|
||||
INSERT INTO review_votes
|
||||
(workspace_id, cycle_id, draft_id, voter_user_id, vote, reason, snapshot_hash, created_at)
|
||||
SELECT c.workspace_id, c.id, c.draft_id, d.reviewer_user_id,
|
||||
CASE WHEN d.state = 'needs_changes' THEN 'deny' ELSE 'approve' END,
|
||||
CASE WHEN d.state = 'needs_changes' THEN
|
||||
COALESCE(a.reason, 'migrated from legacy rejection (no reason recorded)')
|
||||
ELSE NULL END,
|
||||
pv.content_hash, d.updated_at
|
||||
FROM drafts d
|
||||
JOIN review_cycles c ON c.draft_id = d.id AND c.cycle = 1
|
||||
JOIN post_versions pv ON pv.id = c.version_id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT (diff ->> 'reason') AS reason
|
||||
FROM audit_log
|
||||
WHERE workspace_id = d.workspace_id AND target_id = d.id::text
|
||||
AND action = 'draft.rejected'
|
||||
ORDER BY seq DESC LIMIT 1
|
||||
) a ON true
|
||||
WHERE d.reviewer_user_id IS NOT NULL;
|
||||
|
||||
-- reviewer_user_id is now dead: 010 never writes it, 012 drops it
|
||||
-- (publishing.service.ts:315-317 and api-types.ts:70 stop reading first).
|
||||
COMMENT ON COLUMN drafts.reviewer_user_id IS
|
||||
'DEPRECATED: superseded by review_votes in 010; dropped in 012';
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (10) Defense in depth (spec §3): the state machine lives in the service AND
|
||||
-- in the DB. Same CAS shape as publishing.service.ts:283-288 stays valid;
|
||||
-- the scheduler's scheduled->published write (publish.activity.ts:182)
|
||||
-- is an allowed edge.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE FUNCTION drafts_assert_transition() RETURNS trigger
|
||||
LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF OLD.state = NEW.state THEN RETURN NEW; END IF;
|
||||
IF NOT (
|
||||
(OLD.state = 'editing' AND NEW.state IN ('in_review'))
|
||||
OR (OLD.state = 'in_review' AND NEW.state IN ('approved', 'scheduled', 'needs_changes',
|
||||
'expired', 'blocked', 'editing'))
|
||||
OR (OLD.state = 'needs_changes' AND NEW.state IN ('in_review'))
|
||||
OR (OLD.state = 'approved' AND NEW.state IN ('scheduled', 'needs_changes'))
|
||||
OR (OLD.state = 'scheduled' AND NEW.state IN ('approved', 'published', 'needs_changes'))
|
||||
OR (OLD.state = 'expired' AND NEW.state IN ('in_review'))
|
||||
OR (OLD.state = 'blocked' AND NEW.state IN ('in_review', 'needs_changes'))
|
||||
) THEN
|
||||
RAISE EXCEPTION 'illegal draft transition % -> %', OLD.state, NEW.state
|
||||
USING ERRCODE = '23514';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
|
||||
CREATE TRIGGER trg_drafts_legal_transition BEFORE UPDATE OF state ON drafts
|
||||
FOR EACH ROW EXECUTE FUNCTION drafts_assert_transition();
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (11) Live-eligibility recount (R8): reviewer removed / role changed ->
|
||||
-- re-tally open cycles of that group's drafts. Pool below quorum =>
|
||||
-- blocked; pool restored => reopened. Runs as the mutating user inside
|
||||
-- the tenant tx, so RLS and audit_append's fn_assert_tenant both hold.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE FUNCTION review_recount_group() RETURNS trigger
|
||||
LANGUAGE plpgsql AS $$
|
||||
DECLARE r record; v_pool int; v_tally int;
|
||||
BEGIN
|
||||
WITH affected AS (
|
||||
SELECT DISTINCT d.id AS draft_id
|
||||
FROM drafts d
|
||||
WHERE d.group_id = COALESCE(NEW.group_id, OLD.group_id)
|
||||
AND d.state IN ('in_review', 'blocked')
|
||||
)
|
||||
SELECT c.id, c.draft_id, c.required_approvals, d.author_user_id
|
||||
INTO r
|
||||
FROM review_cycles c JOIN drafts d ON d.id = c.draft_id
|
||||
WHERE c.status IN ('open', 'blocked')
|
||||
AND EXISTS (SELECT 1 FROM affected a WHERE a.draft_id = c.draft_id)
|
||||
LIMIT 1;
|
||||
WHILE r IS NOT NULL LOOP
|
||||
SELECT count(DISTINCT u.id) INTO v_pool -- currently-eligible non-author reviewers
|
||||
FROM (SELECT m.user_id AS id
|
||||
FROM memberships m JOIN role_permissions rp ON rp.role = m.role
|
||||
WHERE m.workspace_id = (SELECT workspace_id FROM drafts WHERE id = r.draft_id)
|
||||
AND m.status = 'active' AND rp.permission = 'review.vote'
|
||||
UNION
|
||||
SELECT gm.user_id FROM group_members gm
|
||||
WHERE gm.group_id = (SELECT group_id FROM drafts WHERE id = r.draft_id)
|
||||
AND gm.role = 'reviewer') u
|
||||
WHERE u.id <> r.author_user_id;
|
||||
|
||||
SELECT count(*) INTO v_tally -- live tally: votes JOIN current eligibility
|
||||
FROM review_votes v
|
||||
WHERE v.cycle_id = r.id AND v.vote = 'approve'
|
||||
AND EXISTS (SELECT 1 FROM memberships m JOIN role_permissions rp ON rp.role = m.role
|
||||
WHERE m.user_id = v.voter_user_id AND m.status = 'active'
|
||||
AND rp.permission = 'review.vote'
|
||||
UNION ALL
|
||||
SELECT 1 FROM group_members gm
|
||||
WHERE gm.user_id = v.voter_user_id AND gm.role = 'reviewer')
|
||||
AND v.voter_user_id <> r.author_user_id;
|
||||
|
||||
IF v_pool < r.required_approvals THEN
|
||||
UPDATE review_cycles SET status = 'blocked', closed_reason = 'pool_below_quorum'
|
||||
WHERE id = r.id AND status = 'open';
|
||||
UPDATE drafts SET state = 'blocked', updated_at = now() WHERE id = r.draft_id AND state = 'in_review';
|
||||
ELSE
|
||||
UPDATE review_cycles SET status = 'open', closed_reason = NULL
|
||||
WHERE id = r.id AND status = 'blocked'; -- pool restored: reopen
|
||||
UPDATE drafts SET state = 'in_review', updated_at = now() WHERE id = r.draft_id AND state = 'blocked';
|
||||
END IF;
|
||||
|
||||
SELECT NULL INTO r; -- loop replaced by set-based form in impl;
|
||||
END LOOP; -- sketch shows intent, see §6.4
|
||||
RETURN COALESCE(NEW, OLD);
|
||||
END $$;
|
||||
|
||||
CREATE TRIGGER trg_group_role_recount
|
||||
AFTER INSERT OR UPDATE OF role ON group_members
|
||||
FOR EACH ROW EXECUTE FUNCTION review_recount_group();
|
||||
-- DELETE is covered by the existing cascade path: removing a membership rows
|
||||
-- deletes group_members (008:39 FK), whose own AFTER trigger runs the recount.
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (12) Expiry sweeper for the scheduler (R1). Same shape as
|
||||
-- publishing_due_schedules (006:79-87): STABLE, SECURITY DEFINER, no
|
||||
-- tenant context needed, ids only, revoked from PUBLIC, granted to sp_app.
|
||||
-- Covers both deadline-passed and stale (stale_after_days).
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE FUNCTION review_due_expiries() RETURNS TABLE(draft_id uuid, cycle_id uuid, kind text)
|
||||
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$
|
||||
SELECT c.draft_id, c.id,
|
||||
CASE WHEN c.publish_at IS NOT NULL AND c.publish_at <= now()
|
||||
THEN 'deadline' ELSE 'stale' END
|
||||
FROM review_cycles c
|
||||
JOIN workspace_review_settings s ON s.workspace_id = c.workspace_id
|
||||
WHERE c.status = 'open'
|
||||
AND ( (c.publish_at IS NOT NULL AND c.publish_at <= now())
|
||||
OR (c.publish_at IS NULL AND c.created_at < now() - (s.stale_after_days || ' days')::interval) )
|
||||
$$;
|
||||
|
||||
REVOKE ALL ON FUNCTION review_due_expiries() FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION review_due_expiries() TO sp_app;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (13) Tenancy: the standard block, verbatim from 006/008, per new table.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TRIGGER trg_wrs_ws_immutable BEFORE UPDATE ON workspace_review_settings
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_post_versions_ws_immutable BEFORE UPDATE ON post_versions
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_cycles_ws_immutable BEFORE UPDATE ON review_cycles
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_votes_ws_immutable BEFORE UPDATE ON review_votes
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_messages_ws_immutable BEFORE UPDATE ON review_messages
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_notifications_ws_immutable BEFORE UPDATE ON review_notifications
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
|
||||
ALTER TABLE workspace_review_settings ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE post_versions ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_cycles ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_votes ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_messages ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_notifications ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE workspace_review_settings FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE post_versions FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_cycles FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_votes FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_messages FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_notifications FORCE ROW LEVEL SECURITY;
|
||||
|
||||
CREATE POLICY tenant_isolation ON workspace_review_settings
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON post_versions
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_cycles
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_votes
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_messages
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_notifications
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON workspace_review_settings, post_versions,
|
||||
review_cycles, review_votes, review_messages, review_notifications TO sp_app;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- (14) Permission seeds — purely additive; no existing role loses anything
|
||||
-- (R2: extend, don't break the 10-role contract).
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO role_permissions (role, permission)
|
||||
SELECT r, p FROM unnest(ARRAY['Owner', 'Admin']) AS r,
|
||||
unnest(ARRAY['review.vote', 'review.manage', 'review.override',
|
||||
'moderation.hide', 'moderation.delete']) AS p
|
||||
ON CONFLICT DO NOTHING;
|
||||
|
||||
INSERT INTO role_permissions (role, permission)
|
||||
SELECT 'Approver', 'review.vote' ON CONFLICT DO NOTHING;
|
||||
```
|
||||
|
||||
### 3.1 Why these shapes and not others
|
||||
|
||||
- **`review_cycles` pins `version_id` and snapshots `required_approvals`, `publish_at`, `connection_id`.** The cycle is the unit of trust: "2 approvals of hash X by time T through connection C". Everything the quorum-met handler needs is on one row, so no config read can race the transition.
|
||||
- **`review_votes.snapshot_hash`** duplicates `post_versions.content_hash` so each vote is a self-contained attestation — the audit chain (`audit_log.diff`) and the vote ledger independently prove what was approved, without a join.
|
||||
- **`review_messages` merges human messages and system events** (R5 "system events rendered inline") instead of two tables: one cursor, one ordering, one render list; `kind` + nullable `author_user_id` + `event` distinguish them.
|
||||
- **Moderation is flags, not states** (§1): `hidden_at`/`deleted_by` coexist with any machine state, the legal-transition trigger stays a pure function of state, and the scheduler needs no second transition after publish.
|
||||
- **`drafts_queue` is a partial index** (`WHERE deleted_at IS NULL`) — soft-deleted rows are excluded from the hot path but retained for the audit record.
|
||||
|
||||
---
|
||||
|
||||
## 4. State machine (deliverable 3)
|
||||
|
||||
States after `010`: `editing · in_review · needs_changes · approved · scheduled · published · expired · blocked`.
|
||||
`rejected` is gone (migrated to `needs_changes`, which is resubmittable — closing the "no resubmit path" gap, spec §2.1).
|
||||
|
||||
| # | From | To | Fired by | Guard / effect |
|
||||
|---|------|----|----------|----------------|
|
||||
| 1 | `editing` | `in_review` | author, group **editor**, or `approval.request` holder | `POST /drafts/:id/submit {publishAt?, connectionId?}`. Freezes a new `post_versions` row, opens cycle 1..n with snapshotted quorum. **Preflight:** eligible non-author reviewers ≥ required (else `400`). If `connectionId` given: RLS-scoped existence + group-scope check (the `publishing.service.ts:217-219` pattern). Audit `review.submitted`. |
|
||||
| 2 | `in_review` | `needs_changes` | any eligible **reviewer**, deny vote | Reason mandatory (`400` if blank). Cycle → `denied`; sibling approvals preserved as history, count for nothing (Q5). System event + audit `review.denied`. |
|
||||
| 3 | `in_review` | `needs_changes` | author/group editor, **edit during review** | New version row; cycle → `superseded` (`closed_reason='superseded_by_edit'`). Full reset (Q6). Audit `review.cycle_superseded`. |
|
||||
| 4 | `in_review` | `approved` | quorum engine (N live approvals ≥ required), no `publish_at` | Cycle → `approved`; draft awaits manual schedule (existing `approved→scheduled` path unchanged). Audit `review.quorum_met`. |
|
||||
| 5 | `in_review` | `scheduled` | quorum engine, `publish_at` **and** `publish_at > now()` | Creates `scheduled_publications` + starts workflow (`startScheduledPublish`, `publishing.service.ts:83-113`). Audit `review.quorum_met` + `publication.scheduled`. |
|
||||
| 6 | `in_review` | `expired` | quorum engine with `publish_at <= now()`, **or** sweeper | Nothing is ever scheduled late (Q7). Audit `review.expired`. |
|
||||
| 7 | `in_review` | `blocked` | recount trigger (pool < quorum) | Reviewer removed/demoted mid-review (Q8). Audit `review.blocked`. |
|
||||
| 8 | `in_review` | `editing` | author withdraws | Cycle → `superseded`. Audit `review.withdrawn`. |
|
||||
| 9 | `needs_changes` | `in_review` | author or group editor resubmits | New version + new cycle (fresh quorum). Optionally new `publishAt`/`connectionId`. This is the resubmit path that does not exist today. |
|
||||
| 10 | `approved` | `scheduled` | `publication.schedule` holder | Existing `schedule()` unchanged (`publishing.service.ts:212-247`). |
|
||||
| 11 | `approved` | `needs_changes` | author/editor edit after approval | Post-approval edit re-triggers the chain (spec §3 lock-after-approve). Cycle already closed; new cycle on resubmit. |
|
||||
| 12 | `scheduled` | `approved` | `publication.schedule` holder | Existing `cancelSchedule()` unchanged. |
|
||||
| 13 | `scheduled` | `needs_changes` | edit, or admin override-deny | Transactionally deletes the pending `scheduled_publications` row first (else `UNIQUE(draft_id)`, `006:24`, blocks re-scheduling). |
|
||||
| 14 | `scheduled` | `published` | **scheduler only** (`sp_scheduler`) | Unchanged: `publish.activity.ts:182`. Archive = this state (§1). |
|
||||
| 15 | `expired` | `in_review` | author re-primes | New cycle; must supply a future `publishAt` if it wants scheduling. R1's "re-prime after re-approval". |
|
||||
| 16 | `blocked` | `in_review` | recount trigger (pool restored) | Adding/re-promoting a reviewer reopens automatically; no human step needed. |
|
||||
| 17 | `blocked` | `needs_changes` | admin override-deny, or author withdraws | Override: mandatory reason, `review.override_denied` audit. |
|
||||
| 18 | `blocked` | `approved`/`scheduled` | admin override-approve (`review.override`) | Emergency brake (Q8). Not a vote: no `review_votes` row; `closed_reason='admin_override'`. |
|
||||
|
||||
Who **cannot** fire things (structural, server-side):
|
||||
|
||||
- The **author can never vote** on their own post when `required_approvals ≥ 2` — `403 ForbiddenException("reviewers cannot vote on their own post")` from the service, *in addition to* eligibility being excluded from the tally SQL. At `required_approvals = 1` self-approval is permitted (legacy/self-service mode) and stamped `selfApproved: true` in the audit diff — see §6.3.
|
||||
- Group **editors** cannot vote (their group role grants edit, never `reviewer`), cannot schedule (`publication.schedule` is not granted to them), and workspace role `Editor` already lacks `approval.grant` (`002:39-43`).
|
||||
- Group **reviewers** cannot edit content (edit path checks author ∨ group-editor ∨ `draft.edit`).
|
||||
- Nobody transitions into `published` except the scheduler; nobody transitions out of it (terminal; moderation is flags).
|
||||
|
||||
Every transition above is (a) a CAS `UPDATE … WHERE id=$1 AND state=$from` exactly like `transition()` (`publishing.service.ts:283-288`), **and** (b) legal per `trg_drafts_legal_transition` — the DB is the second enforcement layer even against direct SQL.
|
||||
|
||||
---
|
||||
|
||||
## 5. RBAC (deliverable 4)
|
||||
|
||||
### 5.1 New permissions (additive seeds only)
|
||||
|
||||
| Permission | Granted to | Meaning |
|
||||
|---|---|---|
|
||||
| `review.vote` | Owner, Admin, Approver | Workspace-level reviewer. Guard gate for the vote endpoint; the service then re-checks author-exclusion and group scope. |
|
||||
| `review.manage` | Owner, Admin | Set workspace/group quorum, resolve `blocked`, view live vote identities. (Same holders as `workspace.members.manage`, `002:7/:23` — admin-only per R3.) |
|
||||
| `review.override` | Owner, Admin | Force approve/deny with mandatory reason; logged as override, never as a vote. |
|
||||
| `moderation.hide` | Owner, Admin | Hide/unhide any post (R7, admin-only). |
|
||||
| `moderation.delete` | Owner, Admin | Soft-delete any post (R7, admin-only). |
|
||||
|
||||
No existing role loses a permission; `approval.request`/`approval.grant` keep their seed rows (`002:8/:24/:35/:49`) — `approval.grant` becomes the *workspace-wide* form of review power (holders are eligible voters everywhere), which is exactly the "extend, don't break" reading of R2. The legacy `/approve` and `/reject` endpoints are superseded by `/votes` + `/override` and removed in WP-C (their guard permission `approval.grant` still exists for compatibility with any external caller during the deprecation window).
|
||||
|
||||
### 5.2 Group-role mechanism
|
||||
|
||||
`group_members.role` (§3 step 1). Effective capabilities for a draft with `group_id G`:
|
||||
|
||||
| Capability | Satisfied by |
|
||||
|---|---|
|
||||
| Author | any member of G (or any `draft.create` holder for ungrouped drafts) |
|
||||
| Edit content | author ∨ `group_members.role='editor'` in G ∨ workspace `draft.edit` |
|
||||
| Submit / resubmit | author ∨ group editor ∨ `approval.request` |
|
||||
| Vote | (`review.vote` via workspace role) ∨ (`group_members.role='reviewer'` in G) — **and** ≠ author (at N≥2) |
|
||||
| Schedule | `publication.schedule` (unchanged; group editors never get it) |
|
||||
| Moderate | `moderation.hide`/`.delete` (Owner/Admin only) |
|
||||
|
||||
Guard-side checks are service functions on the tenant client, mirroring `userCanUseConnection` (`groups.service.ts:425-455`) — role read from the DB, never from the token:
|
||||
|
||||
```ts
|
||||
// apps/api/src/review/review.eligibility.ts (sketch)
|
||||
export interface VoteEligibility {
|
||||
eligible: boolean;
|
||||
reason?: "not_group_reviewer" | "author" | "not_in_review";
|
||||
groupRole: "member" | "editor" | "reviewer" | null;
|
||||
}
|
||||
|
||||
export async function voteEligibility(
|
||||
client: PoolClient,
|
||||
workspaceId: string,
|
||||
userId: string,
|
||||
draft: { id: string; groupId: string | null; authorUserId: string; state: DraftState },
|
||||
requiredApprovals: number,
|
||||
): Promise<VoteEligibility> {
|
||||
const res = await client.query(
|
||||
`SELECT
|
||||
EXISTS (SELECT 1 FROM role_permissions rp
|
||||
JOIN memberships m ON m.role = rp.role
|
||||
AND m.workspace_id = $1 AND m.user_id = $2
|
||||
AND m.status = 'active'
|
||||
WHERE rp.permission = 'review.vote') AS workspace_reviewer,
|
||||
(SELECT gm.role FROM group_members gm
|
||||
WHERE gm.group_id = $3 AND gm.user_id = $2) AS group_role`,
|
||||
[workspaceId, userId, draft.groupId],
|
||||
);
|
||||
const row = requiredRow(res.rows[0], "eligibility");
|
||||
const byRole = row.group_role === "reviewer";
|
||||
if (!byRole && row.workspace_reviewer !== true) {
|
||||
return { eligible: false, reason: "not_group_reviewer", groupRole: null };
|
||||
}
|
||||
// Structural self-approval exclusion (four-eyes). At quorum 1 the legacy
|
||||
// self-service mode is preserved (compose/page.tsx:128-131) and flagged.
|
||||
if (userId === draft.authorUserId && requiredApprovals >= 2) {
|
||||
return { eligible: false, reason: "author", groupRole: row.group_role };
|
||||
}
|
||||
if (draft.state !== "in_review") return { eligible: false, reason: "not_in_review", groupRole: row.group_role };
|
||||
return { eligible: true, groupRole: row.group_role };
|
||||
}
|
||||
```
|
||||
|
||||
`requiredRow` follows `publishing.service.ts:297-300`. No `as any`; `group_role` is `null` for ungrouped drafts and the SQL handles `$3 = NULL` (no row → NULL).
|
||||
|
||||
### 5.3 Self-approval exclusion & admin override
|
||||
|
||||
- Self-approval: enforced in `voteEligibility` (above) **and** excluded from the tally SQL (§6.2) **and** impossible to satisfy at N≥2 even for Owner — three independent layers.
|
||||
- Override: `POST /publishing/drafts/:id/override {decision: 'approve'|'deny', reason}` — `@RequirePermission("review.override")`, reason mandatory. Effects per §4 rows 17–18. Writes **no** `review_votes` row (overrides must not pollute quorum metrics), writes audit `review.override_granted`/`review.override_denied` with the reason and the actor, and a masked-then-revealed system event.
|
||||
|
||||
### 5.4 Epoch interplay
|
||||
|
||||
- `group_members.role` UPDATE fires `trg_group_member_bump_epoch` (`008:79-80`) → every session dies → `PATCH /groups/:id/members/:userId` returns `{token}` minted in-tx via `issueSessionInTenant` (`auth.service.ts:72-84`), same contract as `addMember`/`removeMember` (`groups.service.ts:168-200`).
|
||||
- Group **deletion** must gain a guard in `deleteGroup`: `drafts_group_fk` is `NO ACTION`, so deleting a group with posts raises FK — surface it as `409 group still has posts; move or delete them first` instead of a 500.
|
||||
- The recount trigger (§3 step 11) runs inside the same tx, so "reviewer demoted" and "cycle re-tallied" commit atomically — R8's "votes revoked/re-counted" is one transaction, not a eventual-consistency hope.
|
||||
|
||||
---
|
||||
|
||||
## 6. Quorum engine (deliverable 5)
|
||||
|
||||
### 6.1 Vote lifecycle
|
||||
|
||||
1. **One live vote per reviewer per submission**: `UNIQUE(cycle_id, voter_user_id)`; a second attempt returns `409 ConflictException("already voted in this cycle")`.
|
||||
2. **Votes are immutable** (no vote changes — `review_votes` has no UPDATE path and `reject_mutation`-style immutability is the norm here). Justification: a vote attests a hash; a changed attestation rewrites history the hash-chained audit already recorded. A reviewer who reconsidered can comment; a deny closes the cycle anyway; a genuine misfire is an admin override. *Future work:* "retract my approval while cycle open" if teams demand it.
|
||||
3. **Deny = veto** (Q5): closes the cycle `denied`, draft → `needs_changes`, reason mandatory end-to-end (DTO `requireString`, DB `CHECK (vote = 'approve' OR btrim(reason) <> '')`).
|
||||
4. **Eligibility is re-checked live at every evaluation** (spec §3 four-eyes): the tally JOINs *current* membership/role state, so a removed reviewer's historical vote stops counting without being deleted.
|
||||
|
||||
### 6.2 Tally (live, cycle-scoped, author-excluded)
|
||||
|
||||
```sql
|
||||
-- inside castVote(), same tx, after inserting the vote
|
||||
WITH eligible AS (
|
||||
SELECT m.user_id AS id
|
||||
FROM memberships m JOIN role_permissions rp ON rp.role = m.role
|
||||
WHERE m.workspace_id = $1 AND m.status = 'active' AND rp.permission = 'review.vote'
|
||||
UNION
|
||||
SELECT gm.user_id FROM group_members gm
|
||||
WHERE gm.group_id = $2 AND gm.role = 'reviewer'
|
||||
)
|
||||
SELECT count(*)::int AS approvals
|
||||
FROM review_votes v JOIN eligible e ON e.id = v.voter_user_id
|
||||
WHERE v.cycle_id = $3 AND v.vote = 'approve'
|
||||
AND ($4 < 2 OR v.voter_user_id <> $5) -- author exclusion applies only at quorum >= 2
|
||||
```
|
||||
|
||||
`$4` = `required_approvals` snapshot, `$5` = author. At quorum 1 the author's own (permitted, flagged) vote counts — that is today's behavior, preserved.
|
||||
|
||||
### 6.3 `castVote` service sketch (repo style: strict TS, DI like `PUBLISHING_DEPENDENCIES`)
|
||||
|
||||
```ts
|
||||
export interface CastVoteInput {
|
||||
vote: "approve" | "deny";
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export interface CastVoteResult {
|
||||
draftState: DraftState;
|
||||
cycleStatus: ReviewCycleStatus;
|
||||
tally: { approvals: number; required: number };
|
||||
}
|
||||
|
||||
async castVote(workspaceId: string, actor: Actor, draftId: string, input: CastVoteInput): Promise<CastVoteResult> {
|
||||
if (input.vote === "deny" && (input.reason ?? "").trim() === "") {
|
||||
throw new BadRequestException("deny requires a non-empty reason");
|
||||
}
|
||||
return withTenant(workspaceId, async (client) => {
|
||||
await client.query("SELECT id FROM drafts WHERE id = $1 FOR UPDATE", [draftId]); // serialize vs sweeper/other voters
|
||||
const draft = await lockDraft(client, draftId); // 404 / RLS-scoped
|
||||
const cycle = await openCycle(client, draftId); // 409 if not open
|
||||
const eligibility = await voteEligibility(client, workspaceId, actor.userId, draft, cycle.requiredApprovals);
|
||||
if (!eligibility.eligible) {
|
||||
if (eligibility.reason === "author") throw new ForbiddenException("reviewers cannot vote on their own post");
|
||||
throw new ForbiddenException("you are not a reviewer for this post");
|
||||
}
|
||||
const version = await versionRow(client, cycle.versionId);
|
||||
try {
|
||||
await client.query(
|
||||
`INSERT INTO review_votes (workspace_id, cycle_id, draft_id, voter_user_id, vote, reason, snapshot_hash)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
|
||||
[workspaceId, cycle.id, draftId, actor.userId, input.vote, input.reason ?? null, version.contentHash],
|
||||
);
|
||||
} catch (err) {
|
||||
if (pgCode(err) === UNIQUE_VIOLATION) throw new ConflictException("already voted in this cycle");
|
||||
throw err;
|
||||
}
|
||||
const tally = await liveTally(client, workspaceId, draft.groupId, cycle.id, cycle.requiredApprovals, draft.authorUserId);
|
||||
|
||||
if (input.vote === "deny") {
|
||||
await this.closeCycle(client, cycle, "denied", "deny");
|
||||
await this.transitionDraft(client, draftId, "in_review", "needs_changes");
|
||||
} else if (tally.approvals >= cycle.requiredApprovals) {
|
||||
await this.meetQuorum(client, draft, cycle, tally);
|
||||
}
|
||||
// Same-tx audit + system message (groups.service.ts:100-110 pattern): a vote
|
||||
// can never land without its audit entry.
|
||||
await audit(client, workspaceId, {
|
||||
actorUserId: actor.userId, actorRole: actor.role,
|
||||
action: input.vote === "deny" ? "review.denied" : "review.vote_cast",
|
||||
targetType: "draft", targetId: draftId,
|
||||
diff: { cycle: cycle.cycle, vote: input.vote, ...(input.reason === undefined ? {} : { reason: input.reason }),
|
||||
snapshotHash: version.contentHash, approvals: tally.approvals, required: cycle.requiredApprovals,
|
||||
...(actor.userId === draft.authorUserId ? { selfApproved: true } : {}) },
|
||||
});
|
||||
await systemMessage(client, workspaceId, draftId, "vote",
|
||||
{ masked: true, vote: input.vote, approvals: tally.approvals, required: cycle.requiredApprovals });
|
||||
return { draftState: await draftState(client, draftId), cycleStatus: await cycleStatus(client, cycle.id), tally };
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### 6.4 `meetQuorum` — the race-condition core (Q7)
|
||||
|
||||
```ts
|
||||
private async meetQuorum(client: PoolClient, draft: DraftRow, cycle: CycleRow, tally: Tally): Promise<void> {
|
||||
if (draft.hiddenAt !== null || draft.deletedAt !== null) {
|
||||
await this.closeCycle(client, cycle, "blocked", "hidden_at_quorum");
|
||||
await this.transitionDraft(client, draft.id, "in_review", "blocked");
|
||||
return; // hidden posts never publish; an admin resolves via override/delete
|
||||
}
|
||||
if (cycle.publishAt !== null) {
|
||||
if (cycle.publishAt.getTime() <= Date.now()) {
|
||||
await this.closeCycle(client, cycle, "expired", "deadline"); // too late: never schedule late
|
||||
await this.transitionDraft(client, draft.id, "in_review", "expired");
|
||||
return;
|
||||
}
|
||||
const scheduled = await this.insertSchedule(client, workspaceIdOf(draft), draft.id, cycle.connectionId, cycle.publishAt);
|
||||
await this.closeCycle(client, cycle, "approved", "quorum");
|
||||
await this.transitionDraft(client, draft.id, "in_review", "scheduled"); // edge 5 of §4
|
||||
await this.workflowStarter.startScheduledPublish(scheduled.idempotencyKey, cycle.publishAt).catch(() => undefined);
|
||||
return; // sweeper publishing_due_schedules() (006:79-87) re-starts if Temporal was down
|
||||
}
|
||||
await this.closeCycle(client, cycle, "approved", "quorum");
|
||||
await this.transitionDraft(client, draft.id, "in_review", "approved"); // edge 4: manual schedule later
|
||||
}
|
||||
```
|
||||
|
||||
Race resolutions:
|
||||
|
||||
- **Two reviewers approve concurrently.** Both take `SELECT … FOR UPDATE` on the drafts row before touching the cycle; the second transaction blocks, then re-reads the cycle as already closed → its `openCycle` returns `409 cycle closed` *after* its vote insert… which would leak a vote onto a closed cycle. Fix (implementation note): the vote insert and the cycle re-read happen under the same row lock, and `closeCycle` is itself a CAS (`UPDATE review_cycles SET status=… WHERE id=$1 AND status='open'`); the loser's insert committed first in lock order, so the winner's tally **included** it — i.e. the "loser" is whoever didn't win the `closeCycle` CAS, and their vote is exactly what made quorum. The invariant to test (§12, test 4): exactly one `approved`/`scheduled` transition and at most one `scheduled_publications` row per draft (`UNIQUE(draft_id)`, `006:24`, is the backstop).
|
||||
- **Quorum vote vs expiry sweeper at T±ε.** Both paths lock the draft row; whichever wins sets the state, the other's CAS (`WHERE state='in_review'`) matches zero rows and no-ops (the `throwTransitionError` pattern from `publishing.service.ts:294-302` turns it into a clean "already handled").
|
||||
- **Quorum met at T-1min.** `publish_at > now()` → schedule created, `startDelay ≈ 60s` (`publishing.service.ts:99-101`), publishes on time. Auto, no re-confirm (Q7).
|
||||
- **Quorum met at T+1min.** `expired` (above). Nothing publishes late, ever.
|
||||
- **Sweeper shape.** `review_due_expiries()` (§3 step 12) + a scheduler activity `reviewExpire()` that mirrors `listDue`/`publishScheduled` (`publish.activity.ts:160-245`): look up workspace via a small SECURITY DEFINER helper (or return `workspace_id` from the sweeper fn), open `inTenant`, close cycle + transition + `audit_append(…, NULL, 'sp_scheduler', 'review.expired', …)` — the exact actor convention of `publication.published` (`publish.activity.ts:184-190`). Lazy double-check at quorum time (above) covers clock skew between API and scheduler.
|
||||
|
||||
### 6.5 Submit preflight (Q8)
|
||||
|
||||
```sql
|
||||
-- count of currently eligible non-author reviewers for this draft's scope
|
||||
WITH eligible AS ( /* same CTE as §6.2 */ )
|
||||
SELECT count(*)::int FROM eligible WHERE id <> $author
|
||||
```
|
||||
If `< requiredApprovals` → `400 BadRequestException("not enough eligible reviewers (0) for quorum (2) — add reviewers to the group or lower its quorum")`. The draft stays `editing`; nothing half-created. (At `required_approvals = 1` no preflight — solo workspaces keep today's self-serve flow.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Review chat (deliverable 6)
|
||||
|
||||
### 7.1 Data model
|
||||
|
||||
`review_messages` (§3 step 7): one append-only thread per draft, human messages (`kind='message'`, `author_user_id` set) and system events (`kind='system'`, `event` + `event_data`) interleaved on one `ord` sequence. Posting:
|
||||
|
||||
```ts
|
||||
async postMessage(workspaceId: string, actor: Actor, draftId: string, body: string): Promise<ReviewMessage> {
|
||||
const clean = body.trim();
|
||||
if (clean === "") throw new BadRequestException("message must be a non-empty string");
|
||||
return withTenant(workspaceId, async (client) => {
|
||||
await client.query("SELECT id FROM drafts WHERE id = $1 FOR UPDATE", [draftId]); // ord assignment is serialized
|
||||
const res = await client.query(
|
||||
`INSERT INTO review_messages (workspace_id, draft_id, ord, kind, author_user_id, body)
|
||||
SELECT $1, $2, coalesce(max(ord), 0) + 1, 'message', $3, $4
|
||||
FROM review_messages WHERE draft_id = $2
|
||||
RETURNING *`,
|
||||
[workspaceId, draftId, actor.userId, clean],
|
||||
);
|
||||
return messageFromRow(requiredRow(res.rows[0], "review message")); // UNIQUE(draft_id, ord) is the backstop
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Who can post: any group member of the draft's group, any workspace member for ungrouped drafts (gate: `draft.view` at the guard + membership check in service, mirroring `RequireGroupAccess`'s split, `rbac.guard.ts:64-74`). Chat is **not** audit-chained (R8 requires transitions and votes; messages are content, preserved immutably by `reject_mutation` — their integrity story is append-only + RLS, not the hash chain). Reasons are content, not secrets: `scrub()` (`audit.service.ts:11-17`) would not redact them anyway (no secret-key match).
|
||||
|
||||
### 7.2 System event types
|
||||
|
||||
| `event` | Emitted on | `event_data` (masked while cycle open, Q10) |
|
||||
|---|---|---|
|
||||
| `submitted` | cycle opened (v, cycle n, required, publishAt?) | `{version, cycle, required, publishAt}` |
|
||||
| `vote` | vote cast | `{vote, approvals, required}` — actor identity added on cycle close |
|
||||
| `denied` | cycle closed `denied` | `{reason, by}` (identity revealed at close — the author must see who/why) |
|
||||
| `quorum_met` | cycle closed `approved` | `{approvals, required, scheduledAt?}` |
|
||||
| `cycle_superseded` | edit during/after review | `{fromVersion, toVersion}` |
|
||||
| `state` | any other draft transition (expired, blocked, scheduled, published, hidden, deleted) | `{from, to}` |
|
||||
| `override` | admin override | `{decision, reason, by}` |
|
||||
|
||||
Read endpoint masks identity server-side while `status='open'` (Q10): the row keeps `event_data.actorUserId`, the serializer drops it unless the cycle is closed or the caller holds `review.override`.
|
||||
|
||||
### 7.3 Transport (Q4 decision recap)
|
||||
|
||||
`GET /publishing/drafts/:id/messages?after=<ord>&limit=100` — cursor pagination by `ord` (validate with `parseLimit`, `http/validation.ts:57-63`), polled at 4s while the detail pane is open; the queue polls at 15s or refetches on any local action. Every request passes `RbacGuard` (session → epoch → `draft.view`) and tenant RLS, so a reviewer removed mid-thread fails closed on the next poll with `401 authorization epoch stale` (`rbac.guard.ts:46-47`) and the client re-authenticates — no long-lived authorization to revoke. *Future work:* WS gateway reusing `authorizeSubscription` (`gateway.ts:12-45`) + a `review.<draftId>` topic check + the identical `after=<ord>` replay query.
|
||||
|
||||
---
|
||||
|
||||
## 8. UI plan (deliverable 7)
|
||||
|
||||
### 8.1 `/approvals` → the working-posts queue (replaces the stub, `approvals/page.tsx`)
|
||||
|
||||
Two-pane layout (the Inbox shape, `inbox/page.tsx:52-120`):
|
||||
|
||||
- **Left: queue list** — virtualized with the exact `column.tsx:26-34` config (`useVirtualizer`, `estimateSize: () => 110`, `overscan: 8`, `scrollToIndex` on selection). Row: state chip (`Badge`, tones below), author, first line of text, **quorum progress chip `2/3 ✓`** (live tally from `GET /publishing/queue`), relative deadline when `publish_at` is near.
|
||||
- **Filters** (R6): `needs my review` (default; eligibility computed server-side per caller), `needs changes`, `expiring soon` (`publish_at < now()+24h`), `all`. Server-side filter params keep the virtualizer honest.
|
||||
- **Right: post detail** — three stacked zones:
|
||||
1. **Rendered preview** — reuse the compose preview card markup (`compose/page.tsx:244-262`) against the pinned version's `post`.
|
||||
2. **Version diff** — `v(n-1) → v(n)` selector; reuse the audit page's diff viewer (audit page, 236 lines, has the component) over the two `post` jsonbs.
|
||||
3. **Chat + vote panel** — Inbox bubble thread (`inbox/page.tsx:82-104`) + reply box (`:105-118`) with **Approve / Deny…** buttons inline (R5). Deny opens a mandatory-reason textarea; submit disabled while blank (mirrors `requireString` server-side). Buttons render only when the server says the caller is eligible (`canVote` flag on the queue/detail payload — never inferred client-side).
|
||||
- Schedule strip: `publish_at`, connection label, "expiring in 3h" warning.
|
||||
- **Keyboard model** (Streams pattern, `streams/page.tsx:56-70`, incl. the INPUT/TEXTAREA/contentEditable guard): `j/k` move, `Enter` open, `a` approve, `d` deny (focuses reason box), `c` focus chat, `e` toggle diff, `Esc` back to list.
|
||||
|
||||
### 8.2 `/archive`
|
||||
|
||||
Read-only list of `state='published'` (§1: archive *is* published), same virtualized list, chat history preserved and readable (record requirement, R7). Admin affordances gated on `can("moderation.hide")` / `can("moderation.delete")`: **Hide** (members stop seeing it; admins see a `hidden` danger Badge), **Delete** (soft; disappears for everyone, remains in audit). Moderation scope (R7 decision): **any state**, not just archive — an embarrassing draft should be removable before it publishes; hiding a `scheduled` post transactionally cancels the pending schedule first.
|
||||
|
||||
### 8.3 Compose changes (`compose/page.tsx`)
|
||||
|
||||
- **Group picker** when the author belongs to groups (`GET /groups/mine` exists, `groups.controller.ts:57-61`); ungrouped = workspace review.
|
||||
- The current auto-approve chain (`compose/page.tsx:119-142`) becomes: create → **submit** (`{publishAt, connectionId}` = "primed"), stop. The self-approve shortcut survives only in the `required_approvals=1` workspace via the vote endpoint; the button label changes to "Submit for review (2 approvals)" using the resolved quorum so expectations are set before submit.
|
||||
- `STATE_TONE` (`:35-42`) gains: `needs_changes: "warn"`, `expired: "danger"`, `blocked: "danger"`.
|
||||
- "Recent drafts" sidebar (`:255-263`) links queue-ward; sidebar badge (`sidebar.tsx:13` already hints "2") wires to the `needs my review` count.
|
||||
|
||||
### 8.4 Client plumbing
|
||||
|
||||
- `api-types.ts`: `DraftState` widened (mirrors `publishing.service.ts:8`), plus `ReviewCycleSummary`, `ReviewVoteSummary`, `ReviewMessage`, `QueueItem`. `reviewerUserId` (`:70`) marked deprecated (dropped with the column in 012).
|
||||
- `useApiResource` (`use-api.ts`) gains an optional `refetchIntervalMs` (non-breaking, default off) for the two poll rates; everything still flows through `api`/`apiFetch` so auth, 401 redirect, and offline fallback stay uniform (`api-client.ts:57-59, :100-106`).
|
||||
- Offline mode: queue/detail ship fixture data like every other screen (`DataSource: "sample"`, `use-api.ts:16-20`).
|
||||
|
||||
---
|
||||
|
||||
## 9. Edge cases — the ugly ones (deliverable 8)
|
||||
|
||||
| # | Case | Concrete resolution |
|
||||
|---|---|---|
|
||||
| 1 | **Reviewer removed from group mid-review** | `group_members` DELETE cascades from `memberships` (FK, `008:39`) → `trg_group_role_recount` re-tallies: their historical vote stops counting (live-eligibility tally, §6.2); if pool < required → cycle `blocked`, draft `blocked` (same tx); the epoch bump (`008:79-80`) kills their session, so their in-flight client gets `401` on the next call — no stale vote can arrive after the removal commits. |
|
||||
| 2 | **Reviewer demoted to `member`/`editor`** | Same as #1 via the `UPDATE OF role` branch; the demote endpoint returns `{token}` (epoch re-issue pattern). |
|
||||
| 3 | **Reviewer added while `blocked`** | Recount sees pool ≥ required → cycle back to `open`, draft back to `in_review`, system event `state`. No human unblock step. |
|
||||
| 4 | **Deadline race: quorum at T-1min** | Auto-publish (Q7): schedule created (`publish_at > now()`), `startDelay ≈ 60s`, sweeper backstops a missed Temporal start (`006:79-87`). |
|
||||
| 5 | **Quorum at T+1min / never** | `expired` via the same `meetQuorum` branch or the `review_due_expiries()` sweep (covers both deadline and `stale_after_days` staleness). Author re-primes: `expired → in_review` with a new cycle and a fresh `publishAt`. |
|
||||
| 6 | **Vote tx races the expiry sweeper** | Both lock the draft row (`FOR UPDATE`); loser's CAS `WHERE state='in_review'` matches zero rows and no-ops (`throwTransitionError` pattern, `publishing.service.ts:294-302`). Test 7 in §12. |
|
||||
| 7 | **Deadlock: author is the only reviewer, quorum 2** | Submit preflight `400` with an actionable message (Q8); if it happens later anyway → `blocked` + admin `review.override` (mandatory reason) or add-a-reviewer. |
|
||||
| 8 | **Epoch churn mid-vote** | Vote is one tx; the guard's epoch check (`rbac.guard.ts:45-47`) runs *before* it, so a membership change between guard and commit is bounded by: the tally re-checks eligibility *inside* the tx, and the recount trigger already re-tallied — worst case the vote lands and immediately doesn't count, which is the correct outcome, not corruption. |
|
||||
| 9 | **Edit during review** | New version; cycle `superseded`; sibling votes preserved-but-not-counted; draft `needs_changes`; resubmit = new cycle, fresh quorum (Q6). Editors *can* edit in review — that's their job — the invariant is "no mutable content while approvals are live", not "no edits". |
|
||||
| 10 | **Edit after scheduling** | `scheduled → needs_changes` transactionally deletes the pending `scheduled_publications` row first (else `UNIQUE(draft_id)`, `006:24`); the started Temporal workflow finds no pending row and its claim fails cleanly (`publish.activity.ts:137-158` "not pending"). |
|
||||
| 11 | **Connection deleted between submit and quorum** | `meetQuorum`'s `insertSchedule` catches FK violation → cycle `blocked` (`closed_reason='connection_unavailable'`), audit `review.blocked`; admin overrides or author re-primes. (Same FK-bypasses-RLS caveat as `publishing.service.ts:217-219`: the *submit-time* check is an RLS-scoped read, the quorum-time failure is the backstop.) |
|
||||
| 12 | **Hidden post reaches quorum** | `meetQuorum` refuses to schedule: cycle `blocked` (`hidden_at_quorum`); an admin unhides (recount does not auto-reopen — hiding is a moderation decision, so the exit is override or unhide+resubmit). |
|
||||
| 13 | **Group deleted with in-flight posts** | `drafts_group_fk` is `NO ACTION` → FK error; `deleteGroup` surfaces `409 group still has posts`. (SET NULL was impossible: it would null `workspace_id`, violating NOT NULL + `trg_draft_ws_immutable`, `006:41-42`.) |
|
||||
| 14 | **Author leaves the workspace mid-review** | `memberships` cascade removes their group rows → recount (pool unchanged unless they were also a reviewer); draft and cycle persist (`author_user_id` has no cascade, `006:6`); the queue shows the (still existing) user record; `needs_changes` posts by a gone author are exited via admin override-deny or an editor resubmits (group editors may resubmit). |
|
||||
| 15 | **Duplicate/late vote on a closed cycle** | `openCycle` → `409 cycle closed`; the UI's vote panel is disabled by the server-computed `canVote` flag anyway. |
|
||||
| 16 | **Two messages concurrently** | `ord` assigned under the draft row lock; `UNIQUE(draft_id, ord)` is the backstop; clients reconcile by re-pulling from `after=<lastOrd>`. |
|
||||
| 17 | **Migration over existing data** | Backfills are plain INSERT/UPDATE inside the migration tx (`migrate.ts:33-41`); the state rewrite runs *after* dropping the legacy CHECK and *before* adding the new one (§3 step 9); `rejected` rows all become `needs_changes` so no row violates the new CHECK. |
|
||||
|
||||
---
|
||||
|
||||
## 10. Audit & security (deliverable 9)
|
||||
|
||||
### 10.1 Event list (all through `audit_append` — `001:85-111` — same-tx for decisions, per `groups.service.ts:100-110`)
|
||||
|
||||
| Action | Actor | When |
|
||||
|---|---|---|
|
||||
| `review.submitted` | submitter | cycle opened (diff: version, cycle, required, snapshotHash, publishAt?, connectionId?) |
|
||||
| `review.vote_cast` | voter | approve vote (diff: vote, snapshotHash, approvals/required, `selfApproved?`) |
|
||||
| `review.denied` | voter | deny vote (diff: reason — mandatory) |
|
||||
| `review.quorum_met` | last voter | cycle closed approved (diff: approvals, scheduled?) |
|
||||
| `review.cycle_superseded` | editor | edit during/after review (diff: from/to version hashes) |
|
||||
| `review.withdrawn` | author | in_review → editing |
|
||||
| `review.expired` | `sp_scheduler` (actor NULL) | sweep or late quorum |
|
||||
| `review.blocked` / `review.reopened` | trigger (actor: mutating admin) or `sp_scheduler` | pool below/restored |
|
||||
| `review.override_granted` / `review.override_denied` | Owner/Admin | override (reason mandatory; **never** a `review_votes` row) |
|
||||
| `review.settings_changed` | Owner/Admin | quorum/staleness config (diff old→new) |
|
||||
| `review.group_role_changed` | Owner/Admin | member→editor/reviewer (diff old→new) |
|
||||
| `post.hidden` / `post.unhidden` / `post.deleted` | Owner/Admin | moderation (diff: state at time, reason) |
|
||||
| `draft.created/updated/submitted`, `publication.scheduled/schedule_cancelled/published/failed` | existing | unchanged from `publishing.service.ts` / `publish.activity.ts:184-199` |
|
||||
|
||||
Every state transition and every vote is hash-chained (R8). The chain stays verifiable with the existing `audit_verify_chain` (`001:113-126`) — new events are just new `action` strings; no schema change to `audit_log`. `snapshotHash` in vote/submit diffs ties the chain to `post_versions.content_hash`, so tampering with either side is detectable.
|
||||
|
||||
### 10.2 Security notes
|
||||
|
||||
- **Tenancy**: every new table has FORCE RLS + the verbatim `tenant_isolation` NULLIF policy + `reject_workspace_change` + `sp_app` grants (§3 step 13). Cross-tenant reads return zero rows even for the app role (`003` rationale).
|
||||
- **Enforcement order**: `RbacGuard` (session → epoch → permission) → service-level group-scope/eligibility checks on the tenant client (the `userCanUseConnection` precedent) → DB CHECKs/triggers as the last line. Three layers for the two structural rules that matter most: no self-approval at N≥2, no illegal transition.
|
||||
- **Soft delete / hide**: flags only; reads filter `deleted_at IS NULL` in the service (RLS stays pure tenancy — a simple predicate per repo convention, `001:172-179`). Audit rows outlive the rows they describe (`target_id` is text, no FK).
|
||||
- **Blind-vote masking is server-side** (§7.2): the client never receives identities it shouldn't, so it can't leak them.
|
||||
- **No new transport = no new attack surface**: polling endpoints are ordinary guarded routes; cursors are validated (`parseLimit`); `ord` is an int, not a raw SQL fragment.
|
||||
- **Reasons are not secrets** and intentionally survive in audit diffs (`scrub()` only matches secret-ish keys, `audit.service.ts:11-17`); visibility of the chain remains `audit.view`-gated (Approver/Analyst/Owner/Admin, `002`).
|
||||
|
||||
---
|
||||
|
||||
## 11. Phased implementation plan (deliverable 10)
|
||||
|
||||
Each WP ships behind the previous one's test gate; migrations are sequential (`010` here; `011` sweeper indexes if needed; `012` column drop).
|
||||
|
||||
| WP | Scope | Test gate |
|
||||
|---|---|---|
|
||||
| **WP-A: schema** | `010_review_workflow.sql` (all of §3 incl. backfill + transition trigger + seeds); `publishing.service.ts` state union widened; legacy `/approve`,`/reject` kept but routed through the new tables for `required=1`. | Migration applies on a seeded DB; tenancy integration test (mirror `db/tenancy.integration.test.ts`) proves zero cross-tenant rows on all six new tables; `audit_verify_chain` true after backfill; illegal-transition trigger rejects `published→in_review` via direct SQL. |
|
||||
| **WP-B: roles & config** | `PATCH /groups/:id/members/:userId {role}` (+ `{token}`), `GET/PUT /workspace/review-settings`, `PATCH /groups/:id` quorum override, `deleteGroup` post-guard; `voteEligibility`. | Role change bumps epoch (session dies, response carries fresh token); config writes are `review.manage`-only (403 for Publisher); preflight rejects quorum-2 submit in a 1-reviewer group. |
|
||||
| **WP-C: quorum engine** | `submit` (freeze+preflight+cycle), `POST /drafts/:id/votes`, `/override`, recount trigger hardening (set-based), `meetQuorum` + auto-schedule; same-tx audit + system messages. | Tests 1–5 of §12 green; concurrency test (two racing final votes) yields exactly one transition and ≤1 schedule row. |
|
||||
| **WP-D: chat & read models** | `review_messages` endpoints (post + cursor read + masking), `GET /publishing/queue` filters, `GET /publishing/drafts/:id` detail (version, tally, canVote, schedule), `GET /publishing/archive`. | Masking test (open cycle hides identities from non-override holders, closed reveals); cursor pagination stable under concurrent posts; RLS on all reads. |
|
||||
| **WP-E: web** | `/approvals` queue + detail (virtualizer, keyboard, vote panel, diff), `/archive` with moderation actions, Compose group picker + submit-only flow + tone map, `useApiResource` interval option, `api-types.ts`. | Keyboard model works with input-guard; vote buttons gated on server `canVote`; offline fixtures render; no `as any` (tsc strict). |
|
||||
| **WP-F: sweeper & moderation** | `review_due_expiries()` + scheduler `reviewExpire` activity + `notifications` outbox writes on submit/deny/quorum/expiry/block; hide/unhide/soft-delete endpoints (schedule-cancel-first). | Tests 6–10 of §12 green; hidden scheduled post never publishes; outbox rows written in the same tx as their transition. |
|
||||
| **WP-G (later)** | `012` drops `drafts.reviewer_user_id` once web stopped reading it. | Removal of dead code only. |
|
||||
|
||||
---
|
||||
|
||||
## 12. Test plan sketch (deliverable 11)
|
||||
|
||||
1. **Tenancy wall**: as `sp_app` with tenant A context, read `review_votes`, `post_versions`, `review_cycles`, `review_messages`, `review_notifications`, `workspace_review_settings` for a tenant-B id → zero rows; a B-context write is rejected by `WITH CHECK`. (Mirror `db/tenancy.integration.test.ts`.)
|
||||
2. **Structural no-self-approval**: quorum-2 workspace, Owner authors and tries to vote → `403 reviewers cannot vote on their own post`; quorum-1 workspace, same action succeeds and the audit diff carries `selfApproved: true`.
|
||||
3. **Deny semantics**: deny without reason → `400`; with reason → cycle `denied`, draft `needs_changes`, sibling approve-votes still present in `review_votes` but the next cycle's tally starts at 0; the reason appears in the thread event and the audit chain.
|
||||
4. **Concurrent final votes**: two reviewers approve simultaneously (two txs) → exactly one `approved`/`scheduled` transition, one closed cycle, ≤1 `scheduled_publications` row (`UNIQUE(draft_id)` never violated), both votes persisted, chain verifiable.
|
||||
5. **Edit-during-review reset**: editor edits an `in_review` draft → new `post_versions` row, old cycle `superseded`, draft `needs_changes`; resubmit opens cycle n+1 with `required` re-snapshotted (change workspace config in between and prove the old cycle's snapshot held).
|
||||
6. **Reviewer removed mid-review**: remove the only remaining eligible reviewer of a quorum-2 cycle → same-tx `blocked` + recount; their old vote no longer counts (tally query with them excluded); re-add a reviewer → auto `in_review`; the removed user's session fails with `401` (stale epoch).
|
||||
7. **Deadline race both ways**: (a) quorum at `publish_at-60s` → scheduled and published; (b) quorum at `publish_at+60s` → `expired`, and `publishing_due_schedules()` returns nothing for the draft (nothing was ever created); (c) sweeper and a last-instant vote race → one outcome, no double transition.
|
||||
8. **Admin override**: override-approve a `blocked` draft with reason → `approved`/`scheduled` and **no** `review_votes` row; audit shows `review.override_granted`; override-deny on a `scheduled` draft cancels the pending schedule row in the same tx.
|
||||
9. **Chain & snapshot integrity**: run a full lifecycle (create → submit → vote ×2 → schedule → publish → hide) then `audit_verify_chain(workspace)` = true; every transition/vote has an event; every vote's `snapshot_hash` equals the approved `post_versions.content_hash`.
|
||||
10. **Immutability & DB-level state machine**: `UPDATE post_versions …`, `UPDATE review_messages …` → error (append-only); `UPDATE drafts SET workspace_id=…` → `23514` (existing trigger); direct-SQL `published → in_review` → `illegal draft transition` from `trg_drafts_legal_transition`, proving enforcement survives a service bug.
|
||||
|
||||
---
|
||||
|
||||
## 13. Requirements traceability (R1–R9)
|
||||
|
||||
- **R1** — states + resubmit + expired: §4 (rows 2/9/15 close the terminal-`rejected` gap; rows 6/15 + sweeper cover expiry); archived = `published` (§1).
|
||||
- **R2** — group roles, editor/reviewer split, structural no-self-approval, admin override: §5.
|
||||
- **R3** — quorum config location/precedence/who, parallel any-N, one immutable vote, 1-deny, auto-transition + auto-schedule: §3 step 2, §6.
|
||||
- **R4** — frozen snapshots, hash in audit, reviewer sees rendered post + diff: §3 steps 3/5/6, §8.1.
|
||||
- **R5** — per-post thread with system events, vote buttons in chat, transport decision: §7.
|
||||
- **R6** — queue UI, quorum chips, filters, j/k: §8.1.
|
||||
- **R7** — archive view, hide/delete admin-only with new permissions, soft-delete choice, scope = all states: §1, §3 step 4, §8.2, §9 edge case 12.
|
||||
- **R8** — RbacGuard, FORCE RLS, `010`, chained events, epoch re-token + vote re-count: §3, §5.4, §10.
|
||||
- **R9** — no external notifications built; outbox hook designed (§3 step 8, WP-F); no mobile/per-platform/AI scope anywhere.
|
||||
|
||||
---
|
||||
|
||||
## 14. Confidence & risks
|
||||
|
||||
**Least sure about:**
|
||||
|
||||
1. **The recount trigger as written.** `review_recount_group()` is sketched with a loop for readability; the implementation should be set-based (one UPDATE over affected open cycles) and must be load-tested against the epoch trigger firing in the same statement. The subtle risk is trigger ordering inside `groups` mutations: the recount runs *after* `bump_authz_epoch`'s row update but in the same tx, which is what we want, but cascaded deletes (workspace deletion) hit it too — the `blocked` write during a workspace teardown is wasted work at best. Prototype should validate: remove-a-reviewer → blocked, re-add → reopened, all inside the admin's original request.
|
||||
2. **The `FOR UPDATE`-on-draft serialization point.** I've made the draft row the single mutex for votes, quorum evaluation, expiry, edits, and `ord` assignment. That's correct but coarse: a hot post with N reviewers serializes fully. If contention shows up, the fix is a dedicated advisory lock or moving the mutex to `review_cycles` — but only measure first; review volume is human-scale.
|
||||
3. **Blind voting + system events interplay.** Masking identity in `event_data` server-side is straightforward, but the *denied* case reveals the denier to the author immediately (deliberate — the author needs the reason's context). If a workspace wants anonymous denials too, that's a product decision hiding in what is currently an implementation detail; flag it before WP-C.
|
||||
4. **Quorum-1 self-approve carve-out.** Preserving the solo-workspace flow (`compose/page.tsx:128-131`) means the four-eyes guarantee is only real at `≥2`. I've labeled it (UI shows the resolved quorum at submit; audit stamps `selfApproved`), but a stricter reading of R2 would force `≥2` everywhere and break single-user workspaces outright. This is the decision most likely to get overturned in review — it's cheap to flip either way before WP-A.
|
||||
5. **`publish_at`/`connection_id` living on the cycle.** It makes "primed to go out" atomic with submission, but it also means a deadline can expire a review that was otherwise progressing (by design, R1). The alternative — deadline as a soft warning until scheduling — contradicts the spec, so I didn't take it, but the UX copy around `expiring soon` matters more than it looks.
|
||||
|
||||
**What a prototype should validate first (in order):**
|
||||
|
||||
1. Migration `010` on a copy of production-shaped data: backfill row counts, `rejected→needs_changes` rewrite, chain still verifies, old API paths keep working against the new tables (WP-A gate).
|
||||
2. The concurrency core: two racing final votes + a racing sweeper — one transition, one schedule row, no lost votes (§12 tests 4 and 7). Everything else in the design is single-transaction logic; this is the only genuinely concurrent part.
|
||||
3. Epoch/recount interaction: demote a reviewer with an open cycle and watch session invalidation, re-tally, and `blocked` land atomically (§12 test 6).
|
||||
4. Polling ergonomics: 4s cursor polls on a real thread — server load is a non-issue, but check the UI's perceived liveness and the 401→re-auth→resume path after a mid-thread membership change.
|
||||
|
||||
---
|
||||
|
||||
*End of proposal — `runs/wp5/proposal-glm.md`. Design only; no repo files were modified.*
|
||||
1667
docs/design/proposal-opus.md
Normal file
1667
docs/design/proposal-opus.md
Normal file
File diff suppressed because it is too large
Load diff
588
docs/design/proposal-sol.md
Normal file
588
docs/design/proposal-sol.md
Normal file
|
|
@ -0,0 +1,588 @@
|
|||
# Group Review & Signoff for Posts — design proposal
|
||||
|
||||
Status: proposal only. Migration names, endpoints, and types below are proposed, not present code. The current implementation has a single reviewer field and a direct in_review → approved transition, with no vote records [apps/api/migrations/006_publishing.sql:2-12; apps/api/src/publishing/publishing.service.ts:179-205].
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
1. A post belongs to exactly one group and one target connection for a review cycle. The existing scheduler already binds one draft to one connection and enforces UNIQUE(draft_id), while Compose expands multiple targets into multiple drafts [apps/api/migrations/006_publishing.sql:14-25; apps/web/src/app/(main)/compose/page.tsx:114-139]. Preserve that unit of approval.
|
||||
2. Group members get one explicit editorial role: member, editor, or reviewer. Editors edit and submit; reviewers vote but cannot edit. Workspace Owner/Admin can manage policy, moderate, and use a reasoned override. A post's author can never cast an approval or override it.
|
||||
3. Every edit creates a full immutable version. Submission pins one version, target, deadline, and quorum. Editing an in-review or approved post starts a new cycle; old votes remain historical and count for no later submission. This replaces the mutable post plus reviewer_user_id as the authority [apps/api/migrations/006_publishing.sql:6-11].
|
||||
4. Any one denial with a trimmed nonempty reason sends the submission to needs_changes. Parallel distinct approvals automatically satisfy a frozen quorum of 2 or 3; the final vote schedules the already specified publication if its deadline is still in the future. A missed deadline expires, with no remote publish.
|
||||
5. The review thread is an ordered database event log delivered by SSE. HTTP handles writes. This uses the existing session/epoch gate pattern without pretending the current realtime gateway is a transport: it is an authorization function only [apps/api/src/realtime/gateway.ts:3-31].
|
||||
|
||||
### The ten decisions requested in spec §5
|
||||
|
||||
**Q1 — Store one role on group_members.** Add review_role text with member/editor/reviewer values and default member. A single role makes editor and reviewer mutually exclusive for a group post, and reuses the existing unique group_id,user_id membership and epoch trigger [apps/api/migrations/008_groups.sql:27-38,75-80]. A separate roles table could support combined duties later but adds a second authorization surface and makes the stated separation easier to violate. Workspace roles remain intact as route admission and administrator authority, not substitutes for a group role; memberships currently has one role per workspace [apps/api/migrations/001_tenants.sql:20-32].
|
||||
|
||||
**Q2 — Store full immutable content per version, with a hash.** A hash alone cannot reproduce the exact rendered content or show a trustworthy diff after drafts.post changes. draft_versions therefore holds the complete normalized DraftPost and SHA-256 of canonical JSON plus target ID; review_submissions points to one version. This follows the current JSONB post model [apps/api/migrations/006_publishing.sql:2-11] and gives the renderer an immutable payload. Deduplicated blobs may be future storage work.
|
||||
|
||||
**Q3 — Workspace default 2, group override 2 or 3, frozen on submit.** Owner/Admin sets workspaces.review_quorum_default (2 or 3) and optional groups.review_quorum_override. Effective value is COALESCE(group override, workspace default), copied into each submission. This lets a workspace start safely with one setting while sensitive groups require three; later setting changes affect only future submissions. The current groups table has no review policy [apps/api/migrations/008_groups.sql:17-25]. Per-connection policy is future work because a reviewed draft already has one group and target.
|
||||
|
||||
**Q4 — SSE for thread delivery, HTTP for mutations.** Use one EventSource connection for the selected post, with replay from a per-draft sequence and a short database polling loop; queue counts refresh by HTTP. SSE is cheaper to wire than a bidirectional WS gateway for a read-heavy thread, while preserving order through a row-locked per-draft sequence. The existing gateway only authorizes stream.view subscriptions and has no transport or topic implementation [apps/api/src/realtime/gateway.ts:3-31]. WS fanout can replace SSE later without changing the event schema.
|
||||
|
||||
**Q5 — One denial closes the round.** Denial requires a reason and moves the draft to needs_changes immediately. Already cast approvals remain immutable historical votes, displayed in the thread, but do not count in a resubmission. This is a clear author-facing repair path and avoids publishing content one qualified reviewer explicitly objected to. Majority denial can be a future policy if product evidence calls for it; today's reject reason is optional and only stored in an audit diff [apps/api/src/publishing/publishing.controller.ts:30-32,172-177; apps/api/src/publishing/publishing.service.ts:197-204].
|
||||
|
||||
**Q6 — Full reset on edit.** Editing during review or after approval first supersedes the submission, then moves to editing, creates a version, and requires resubmit. Reviewers see a diff from the prior submitted version. Retaining votes for allegedly unchanged fields would need a defensible semantic diff of text, media, visibility, target, and provider metadata; the current post is one JSONB value [apps/api/src/publishing/publishing.service.ts:10-16]. Partial dismissal is future work.
|
||||
|
||||
**Q7 — Auto-schedule if quorum lands before the deadline.** The final vote uses database clock time while holding the draft row lock. If clock_timestamp() is before publish_at, create the pending scheduled_publications row in the same transaction; at or after the time, expire and require a new deadline plus fresh submission. A T−1 minute quorum therefore schedules automatically, with no manual reconfirmation. The current Temporal starter can accept startDelay and its pending row is durable [apps/api/src/publishing/publishing.service.ts:82-106,216-245].
|
||||
|
||||
**Q8 — Deadlock requires another person.** A sole reviewer who authored the draft cannot vote. A different Owner/Admin may use a reasoned override, which records a separate override action and does not masquerade as a reviewer vote; a sole human admin author also cannot override. The UI offers Invite reviewer / Ask another admin, never a self-approve button. This preserves the four-eyes rule in spec §3 while retaining administrator rescue for small groups.
|
||||
|
||||
**Q9 — Grandfather completed old approvals; re-prime unfinished ones.** Backfill version 1 for every existing draft, retain reviewer_user_id as historical metadata only, and record a migration provenance audit event per changed draft. Existing approved/scheduled/published rows remain valid legacy outcomes; do not fabricate multiple votes from one reviewer. Existing in_review and rejected rows become needs_changes and require group assignment, an editor, and a fresh submission. This is honest about the one-reviewer current model [apps/api/migrations/006_publishing.sql:7-10; apps/api/src/publishing/publishing.service.ts:188-205]. The old direct approve/reject routes become compatibility aliases to vote actions only for a valid group submission, never direct state setters.
|
||||
|
||||
**Q10 — Show prior votes and identities.** Reviewers see who has approved, denied, or had a vote revoked before voting. The thread already needs system approval events; hiding names there would be inconsistent and would obscure duplicate/reassignment handling. Mitigate peer pressure by making the rendered snapshot and diff the visual focus and requiring an explicit individual confirmation. Blind voting can be tested later as an organization policy, but the audit trail remains attributed.
|
||||
|
||||
## 2. Data model and migration 009 sketch
|
||||
|
||||
### 2.1 Relational shape
|
||||
|
||||
The SQL below is the intended shape for apps/api/migrations/009_group_review.sql. It follows direct workspace_id predicates, FORCE RLS, immutability triggers, composite group FKs, and sp_app grants from 006 and 008 [apps/api/migrations/006_publishing.sql:44-68; apps/api/migrations/008_groups.sql:14-15,66-104]. Run as a migration transaction; backfill existing rows before enabling the final state guard. Values for content_sha256 are calculated from jsonb::text and the target UUID by the database function shown below, rather than trusting client bytes.
|
||||
|
||||
BEGIN;
|
||||
ALTER TABLE workspaces ADD COLUMN review_quorum_default smallint NOT NULL DEFAULT 2
|
||||
CHECK (review_quorum_default IN (2, 3));
|
||||
ALTER TABLE groups ADD COLUMN review_quorum_override smallint
|
||||
CHECK (review_quorum_override IN (2, 3));
|
||||
ALTER TABLE group_members ADD COLUMN review_role text NOT NULL DEFAULT 'member'
|
||||
CHECK (review_role IN ('member', 'editor', 'reviewer'));
|
||||
-- The existing group_members AFTER UPDATE trigger bumps authz_epoch for role changes.
|
||||
ALTER TABLE drafts ADD COLUMN group_id uuid;
|
||||
ALTER TABLE drafts ADD COLUMN review_round integer NOT NULL DEFAULT 0 CHECK (review_round >= 0);
|
||||
ALTER TABLE drafts ADD COLUMN review_event_seq bigint NOT NULL DEFAULT 0 CHECK (review_event_seq >= 0);
|
||||
ALTER TABLE drafts ADD COLUMN hidden_at timestamptz;
|
||||
ALTER TABLE drafts ADD COLUMN hidden_by uuid REFERENCES users(id);
|
||||
ALTER TABLE drafts ADD COLUMN deleted_at timestamptz;
|
||||
ALTER TABLE drafts ADD COLUMN deleted_by uuid REFERENCES users(id);
|
||||
ALTER TABLE drafts ADD CONSTRAINT drafts_id_workspace_key UNIQUE (id, workspace_id);
|
||||
ALTER TABLE drafts ADD CONSTRAINT drafts_group_workspace_fk
|
||||
FOREIGN KEY (group_id, workspace_id) REFERENCES groups(id, workspace_id) ON DELETE RESTRICT;
|
||||
ALTER TABLE drafts DROP CONSTRAINT drafts_state_check; -- resolve actual generated name in migration
|
||||
ALTER TABLE drafts ADD CONSTRAINT drafts_state_check CHECK (state IN
|
||||
('editing','primed','in_review','needs_changes','approved','scheduled',
|
||||
'published','archived','expired'));
|
||||
ALTER TABLE drafts ADD CONSTRAINT drafts_deleted_hidden_check
|
||||
CHECK (deleted_at IS NULL OR hidden_at IS NOT NULL);
|
||||
ALTER TABLE scheduled_publications ADD COLUMN claim_eligibility_checked_at timestamptz;
|
||||
|
||||
CREATE TABLE draft_versions (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL,
|
||||
version_no integer NOT NULL CHECK (version_no > 0),
|
||||
post jsonb NOT NULL,
|
||||
connection_id uuid,
|
||||
content_sha256 text NOT NULL CHECK (content_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
made_by uuid REFERENCES users(id),
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (draft_id, version_no), UNIQUE (id, draft_id, workspace_id),
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE RESTRICT,
|
||||
FOREIGN KEY (connection_id, workspace_id) REFERENCES connections(id, workspace_id) ON DELETE RESTRICT
|
||||
);
|
||||
CREATE TABLE review_submissions (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL,
|
||||
group_id uuid NOT NULL,
|
||||
round integer NOT NULL CHECK (round > 0),
|
||||
version_id uuid NOT NULL,
|
||||
required_approvals smallint NOT NULL CHECK (required_approvals IN (2, 3)),
|
||||
publish_at timestamptz,
|
||||
status text NOT NULL CHECK (status IN
|
||||
('primed','in_review','approved','denied','superseded','expired')),
|
||||
submitted_by uuid NOT NULL REFERENCES users(id),
|
||||
submitted_at timestamptz NOT NULL DEFAULT now(),
|
||||
closed_at timestamptz,
|
||||
UNIQUE (draft_id, round), UNIQUE (id, workspace_id),
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE RESTRICT,
|
||||
FOREIGN KEY (group_id, workspace_id) REFERENCES groups(id, workspace_id) ON DELETE RESTRICT,
|
||||
FOREIGN KEY (version_id, draft_id, workspace_id)
|
||||
REFERENCES draft_versions(id, draft_id, workspace_id) ON DELETE RESTRICT
|
||||
);
|
||||
CREATE TABLE review_votes (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
submission_id uuid NOT NULL,
|
||||
voter_user_id uuid NOT NULL REFERENCES users(id),
|
||||
decision text NOT NULL CHECK (decision IN ('approve','deny')),
|
||||
reason text,
|
||||
snapshot_sha256 text NOT NULL CHECK (snapshot_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
cast_at timestamptz NOT NULL DEFAULT now(),
|
||||
revoked_at timestamptz,
|
||||
revoked_reason text,
|
||||
UNIQUE (submission_id, voter_user_id),
|
||||
CHECK (decision <> 'deny' OR (reason IS NOT NULL AND btrim(reason) <> '')),
|
||||
CHECK (revoked_at IS NULL OR (revoked_reason IS NOT NULL AND btrim(revoked_reason) <> '')),
|
||||
FOREIGN KEY (submission_id, workspace_id)
|
||||
REFERENCES review_submissions(id, workspace_id) ON DELETE RESTRICT
|
||||
);
|
||||
CREATE TABLE review_overrides (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
submission_id uuid NOT NULL,
|
||||
admin_user_id uuid NOT NULL REFERENCES users(id),
|
||||
reason text NOT NULL CHECK (btrim(reason) <> ''),
|
||||
snapshot_sha256 text NOT NULL CHECK (snapshot_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (submission_id),
|
||||
FOREIGN KEY (submission_id, workspace_id)
|
||||
REFERENCES review_submissions(id, workspace_id) ON DELETE RESTRICT
|
||||
);
|
||||
CREATE TABLE review_events (
|
||||
workspace_id uuid NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
draft_id uuid NOT NULL,
|
||||
seq bigint NOT NULL CHECK (seq > 0),
|
||||
submission_id uuid,
|
||||
kind text NOT NULL CHECK (kind IN
|
||||
('message','submitted','edited','approved','denied','vote_revoked',
|
||||
'quorum_met','quorum_lost','needs_changes','scheduled','expired','published','archived',
|
||||
'hidden','unhidden','deleted','override')),
|
||||
actor_user_id uuid REFERENCES users(id),
|
||||
body text,
|
||||
data jsonb NOT NULL DEFAULT '{}',
|
||||
at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (draft_id, seq),
|
||||
FOREIGN KEY (draft_id, workspace_id) REFERENCES drafts(id, workspace_id) ON DELETE RESTRICT,
|
||||
FOREIGN KEY (submission_id, workspace_id)
|
||||
REFERENCES review_submissions(id, workspace_id) ON DELETE RESTRICT,
|
||||
CHECK (kind <> 'message' OR (body IS NOT NULL AND btrim(body) <> ''))
|
||||
);
|
||||
CREATE INDEX drafts_review_queue ON drafts (workspace_id, group_id, state, updated_at DESC, id)
|
||||
WHERE deleted_at IS NULL;
|
||||
CREATE INDEX review_submissions_due ON review_submissions (publish_at, id)
|
||||
WHERE status = 'in_review' AND publish_at IS NOT NULL;
|
||||
CREATE INDEX review_votes_live ON review_votes (submission_id, voter_user_id)
|
||||
WHERE revoked_at IS NULL AND decision = 'approve';
|
||||
CREATE INDEX review_events_replay ON review_events (workspace_id, draft_id, seq);
|
||||
|
||||
CREATE FUNCTION review_content_hash(p_post jsonb, p_connection_id uuid) RETURNS text
|
||||
LANGUAGE sql IMMUTABLE AS $$
|
||||
SELECT encode(digest(p_post::text || '|' || coalesce(p_connection_id::text, 'unassigned'), 'sha256'), 'hex')
|
||||
$$;
|
||||
REVOKE ALL ON FUNCTION review_content_hash(jsonb, uuid) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION review_content_hash(jsonb, uuid) TO sp_app;
|
||||
|
||||
CREATE FUNCTION review_version_sealed() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF TG_OP <> 'INSERT' THEN
|
||||
RAISE EXCEPTION 'draft versions are immutable' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
IF NEW.content_sha256 <> review_content_hash(NEW.post, NEW.connection_id) THEN
|
||||
RAISE EXCEPTION 'version hash mismatch' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
IF NEW.connection_id IS NULL AND EXISTS
|
||||
(SELECT 1 FROM drafts WHERE id = NEW.draft_id AND group_id IS NOT NULL) THEN
|
||||
RAISE EXCEPTION 'review version needs a target connection' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER trg_draft_version_sealed BEFORE INSERT OR UPDATE OR DELETE ON draft_versions
|
||||
FOR EACH ROW EXECUTE FUNCTION review_version_sealed();
|
||||
CREATE FUNCTION review_vote_frozen() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF TG_OP = 'DELETE' OR
|
||||
(TG_OP = 'UPDATE' AND ((NEW.id, NEW.workspace_id, NEW.cast_at,
|
||||
NEW.decision, NEW.reason, NEW.snapshot_sha256,
|
||||
NEW.voter_user_id, NEW.submission_id) IS DISTINCT FROM
|
||||
(OLD.id, OLD.workspace_id, OLD.cast_at,
|
||||
OLD.decision, OLD.reason, OLD.snapshot_sha256,
|
||||
OLD.voter_user_id, OLD.submission_id) OR
|
||||
OLD.revoked_at IS NOT NULL OR NEW.revoked_at IS NULL)) THEN
|
||||
RAISE EXCEPTION 'vote decision is immutable' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
RETURN COALESCE(NEW, OLD);
|
||||
END $$;
|
||||
CREATE TRIGGER trg_review_vote_frozen BEFORE UPDATE OR DELETE ON review_votes
|
||||
FOR EACH ROW EXECUTE FUNCTION review_vote_frozen();
|
||||
CREATE FUNCTION review_reject_update_delete() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
RAISE EXCEPTION 'review evidence is append-only' USING ERRCODE = '23514';
|
||||
END $$;
|
||||
CREATE TRIGGER trg_review_override_sealed BEFORE UPDATE OR DELETE ON review_overrides
|
||||
FOR EACH ROW EXECUTE FUNCTION review_reject_update_delete();
|
||||
CREATE TRIGGER trg_review_event_sealed BEFORE UPDATE OR DELETE ON review_events
|
||||
FOR EACH ROW EXECUTE FUNCTION review_reject_update_delete();
|
||||
CREATE TRIGGER trg_draft_version_ws_immutable BEFORE UPDATE ON draft_versions
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_submission_ws_immutable BEFORE UPDATE ON review_submissions
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_vote_ws_immutable BEFORE UPDATE ON review_votes
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_override_ws_immutable BEFORE UPDATE ON review_overrides
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
CREATE TRIGGER trg_review_event_ws_immutable BEFORE UPDATE ON review_events
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_workspace_change();
|
||||
ALTER TABLE draft_versions ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_submissions ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_votes ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_overrides ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_events ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE draft_versions FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_submissions FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_votes FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_overrides FORCE ROW LEVEL SECURITY;
|
||||
ALTER TABLE review_events FORCE ROW LEVEL SECURITY;
|
||||
CREATE POLICY tenant_isolation ON draft_versions
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_submissions
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_votes
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_overrides
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
CREATE POLICY tenant_isolation ON review_events
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
WITH CHECK (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid);
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON draft_versions, review_submissions,
|
||||
review_votes, review_overrides, review_events TO sp_app;
|
||||
COMMIT;
|
||||
|
||||
The ALTER that drops the old state check must discover its generated PostgreSQL name during migration, or 006 should first be inspected with pg_constraint. All new mutable tenant tables have the same trigger, FORCE RLS policy, and grants as 006. draft_versions is append-only despite UPDATE/DELETE grants because its trigger rejects those operations. A later migration may narrow grants after existing runtime role assumptions are checked. Current audit_append is SECURITY DEFINER and asserts app.workspace_id, so new services invoke it on their withTenant client [apps/api/migrations/001_tenants.sql:72-86,100-121; apps/api/src/db/client.ts:11-33].
|
||||
|
||||
### 2.2 Additional database guards and backfill
|
||||
|
||||
009 also installs a drafts BEFORE UPDATE trigger named enforce_review_transition. It rejects any state pair outside the table in §3; rejects post/group changes except while entering or in editing; rejects approved unless the latest active submission has valid distinct approvals at its frozen required_approvals or a non-author admin override row; rejects scheduled unless the row's version hash still matches drafts.post; and rejects published unless a same-draft schedule is done. Its SQL predicate for a valid approval is a count of review_votes joined to active memberships and group_members(review_role='reviewer'), excluding revoked_at, the draft author, and any vote whose snapshot_sha256 differs from the submission version. A legacy row with group_id IS NULL is exempt only for approved/scheduled/published → later legacy states, and cannot enter a new review without group assignment. This is the database half of the state machine; service-side checks return useful 400/403/409 errors. The scheduler currently updates a draft to published without checking the affected row count, so that call must assert one row and roll back on failure [apps/scheduler/src/activities/publish.activity.ts:158-190].
|
||||
|
||||
The security-critical predicate and trigger shape are:
|
||||
|
||||
CREATE FUNCTION review_valid_approval_count(p_submission uuid) RETURNS integer
|
||||
LANGUAGE sql STABLE AS $$
|
||||
SELECT count(DISTINCT v.voter_user_id)::integer
|
||||
FROM review_votes v
|
||||
JOIN review_submissions s ON s.id = v.submission_id
|
||||
JOIN draft_versions ver ON ver.id = s.version_id
|
||||
JOIN drafts d ON d.id = s.draft_id
|
||||
JOIN memberships m ON m.workspace_id = s.workspace_id
|
||||
AND m.user_id = v.voter_user_id AND m.status = 'active'
|
||||
JOIN group_members gm ON gm.group_id = s.group_id
|
||||
AND gm.user_id = v.voter_user_id AND gm.review_role = 'reviewer'
|
||||
WHERE s.id = p_submission AND v.decision = 'approve'
|
||||
AND v.revoked_at IS NULL AND v.voter_user_id <> d.author_user_id
|
||||
AND v.snapshot_sha256 = ver.content_sha256
|
||||
$$;
|
||||
REVOKE ALL ON FUNCTION review_valid_approval_count(uuid) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION review_valid_approval_count(uuid) TO sp_app;
|
||||
CREATE FUNCTION review_vote_eligible() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
DECLARE v_author uuid; v_hash text; v_state text; v_group uuid;
|
||||
BEGIN
|
||||
SELECT d.author_user_id, ver.content_sha256, d.state, s.group_id
|
||||
INTO v_author, v_hash, v_state, v_group
|
||||
FROM review_submissions s JOIN drafts d ON d.id = s.draft_id
|
||||
JOIN draft_versions ver ON ver.id = s.version_id
|
||||
WHERE s.id = NEW.submission_id AND s.workspace_id = NEW.workspace_id;
|
||||
IF v_state <> 'in_review' OR v_author = NEW.voter_user_id OR
|
||||
v_hash IS DISTINCT FROM NEW.snapshot_sha256 OR
|
||||
NOT EXISTS (SELECT 1 FROM memberships m JOIN group_members gm
|
||||
ON gm.workspace_id = m.workspace_id AND gm.user_id = m.user_id
|
||||
WHERE m.workspace_id = NEW.workspace_id AND m.user_id = NEW.voter_user_id
|
||||
AND m.status = 'active' AND gm.group_id = v_group
|
||||
AND gm.review_role = 'reviewer') THEN
|
||||
RAISE EXCEPTION 'ineligible or stale review vote' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER trg_review_vote_eligible BEFORE INSERT ON review_votes
|
||||
FOR EACH ROW EXECUTE FUNCTION review_vote_eligible();
|
||||
CREATE FUNCTION review_override_eligible() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
DECLARE v_author uuid; v_hash text; v_state text;
|
||||
BEGIN
|
||||
SELECT d.author_user_id, ver.content_sha256, d.state
|
||||
INTO v_author, v_hash, v_state
|
||||
FROM review_submissions s JOIN drafts d ON d.id = s.draft_id
|
||||
JOIN draft_versions ver ON ver.id = s.version_id
|
||||
WHERE s.id = NEW.submission_id AND s.workspace_id = NEW.workspace_id;
|
||||
IF v_state <> 'in_review' OR v_author = NEW.admin_user_id OR
|
||||
v_hash IS DISTINCT FROM NEW.snapshot_sha256 OR
|
||||
NOT EXISTS (SELECT 1 FROM memberships m WHERE m.workspace_id = NEW.workspace_id
|
||||
AND m.user_id = NEW.admin_user_id AND m.status = 'active'
|
||||
AND m.role IN ('Owner','Admin')) THEN
|
||||
RAISE EXCEPTION 'ineligible or stale review override' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER trg_review_override_eligible BEFORE INSERT ON review_overrides
|
||||
FOR EACH ROW EXECUTE FUNCTION review_override_eligible();
|
||||
|
||||
The final transition trigger uses the exact allowed (OLD.state,NEW.state) pairs in §3. Its critical branch is:
|
||||
|
||||
CREATE FUNCTION enforce_review_transition() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
DECLARE v_round review_submissions%ROWTYPE; v_hash text; v_override boolean;
|
||||
BEGIN
|
||||
IF OLD.review_round > 0 AND NEW.group_id IS DISTINCT FROM OLD.group_id THEN
|
||||
RAISE EXCEPTION 'review group is sealed' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
IF NEW.post IS DISTINCT FROM OLD.post AND NEW.state <> 'editing' THEN
|
||||
RAISE EXCEPTION 'post content is sealed outside editing' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
IF NEW.state IS DISTINCT FROM OLD.state AND NOT (
|
||||
(OLD.state, NEW.state) IN (
|
||||
('editing','primed'), ('needs_changes','primed'), ('expired','primed'),
|
||||
('primed','in_review'), ('in_review','needs_changes'),
|
||||
('in_review','approved'), ('in_review','expired'),
|
||||
('in_review','editing'), ('approved','editing'),
|
||||
('approved','scheduled'), ('approved','expired'),
|
||||
('approved','in_review'), ('scheduled','in_review'),
|
||||
('approved','needs_changes'), ('scheduled','needs_changes'),
|
||||
('scheduled','approved'), ('scheduled','editing'),
|
||||
('scheduled','published'), ('published','archived'))
|
||||
) THEN
|
||||
RAISE EXCEPTION 'illegal review transition' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
IF NEW.group_id IS NOT NULL AND NEW.state IN ('approved','scheduled') THEN
|
||||
SELECT s, ver.content_sha256 INTO v_round, v_hash
|
||||
FROM review_submissions s JOIN draft_versions ver ON ver.id = s.version_id
|
||||
WHERE s.draft_id = NEW.id AND s.round = NEW.review_round;
|
||||
IF NOT FOUND OR v_round.status <> 'approved' OR
|
||||
review_content_hash(NEW.post, (SELECT connection_id FROM draft_versions
|
||||
WHERE id = v_round.version_id)) IS DISTINCT FROM v_hash THEN
|
||||
RAISE EXCEPTION 'approved snapshot missing or changed' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
SELECT EXISTS (SELECT 1 FROM review_overrides o
|
||||
JOIN memberships m ON m.workspace_id = o.workspace_id
|
||||
AND m.user_id = o.admin_user_id AND m.status = 'active'
|
||||
AND m.role IN ('Owner','Admin')
|
||||
WHERE o.submission_id = v_round.id AND o.snapshot_sha256 = v_hash
|
||||
AND o.admin_user_id <> OLD.author_user_id) INTO v_override;
|
||||
IF review_valid_approval_count(v_round.id) < v_round.required_approvals
|
||||
AND NOT v_override THEN
|
||||
RAISE EXCEPTION 'review quorum missing' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
END IF;
|
||||
IF NEW.state = 'published' AND OLD.state = 'scheduled' AND NOT EXISTS
|
||||
(SELECT 1 FROM scheduled_publications sp WHERE sp.draft_id = NEW.id
|
||||
AND sp.state = 'done' AND
|
||||
(NEW.group_id IS NULL OR sp.claim_eligibility_checked_at IS NOT NULL)) THEN
|
||||
RAISE EXCEPTION 'publication not complete' USING ERRCODE = '23514';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER trg_enforce_review_transition BEFORE UPDATE ON drafts
|
||||
FOR EACH ROW EXECUTE FUNCTION enforce_review_transition();
|
||||
|
||||
This branch is supplemented by a priming guard that requires a current group/version/connection and by a submission trigger that rejects UPDATE of draft_id, group_id, round, version_id, required_approvals, publish_at, submitted_by, or submitted_at; only status and closed_at may change along the status sequence in §3. Legacy group_id NULL rows are allowed only for their grandfathered approved/scheduled/published transitions. The service writes submission status approved before updating draft state. At claim, the scheduler sets claim_eligibility_checked_at only after its current quorum/hash/target check, then enters in_flight before calling the provider. A later reviewer removal cannot undo that external call; completion validates the claim proof rather than recounting live membership. The scheduler writes schedule state done before publishing, matching its current update order [apps/scheduler/src/activities/publish.activity.ts:177-184].
|
||||
|
||||
Backfill one draft_versions row for each draft, taking its current post and its target from providerMetadata.connectionId when present. Existing drafts with no unambiguous target keep NULL connection_id in that legacy version and must choose a target before a new submission; review_version_sealed refuses a NULL target for group drafts. This avoids inventing a target for old multi-target drafts; Compose's saved draft path currently stores targetConnectionIds while the publish path stores connectionId [apps/web/src/app/(main)/compose/page.tsx:108-110,125-127]. On rollout, convert rejected/in_review to needs_changes with a migration audit event; leave approved/scheduled/published grandfathered. Background migration archives already published rows with a distinct publication.archived audit entry. Keep reviewer_user_id read-only historical metadata; never derive quorum from it.
|
||||
|
||||
The scheduler's existing publishing_due_schedules() returns pending rows [apps/api/migrations/006_publishing.sql:79-89]. Add narrow SECURITY DEFINER functions, matching publishing_schedule_workspace's restricted grants:
|
||||
|
||||
CREATE FUNCTION review_due_deadlines() RETURNS TABLE(submission_id uuid)
|
||||
LANGUAGE sql VOLATILE SECURITY DEFINER SET search_path = public AS $$
|
||||
SELECT id FROM review_submissions
|
||||
WHERE status = 'in_review' AND publish_at <= clock_timestamp()
|
||||
ORDER BY publish_at, id
|
||||
$$;
|
||||
CREATE FUNCTION review_submission_workspace(p_submission_id uuid) RETURNS uuid
|
||||
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$
|
||||
SELECT workspace_id FROM review_submissions WHERE id = p_submission_id
|
||||
$$;
|
||||
REVOKE ALL ON FUNCTION review_due_deadlines() FROM PUBLIC;
|
||||
REVOKE ALL ON FUNCTION review_submission_workspace(uuid) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION review_due_deadlines(),
|
||||
review_submission_workspace(uuid) TO sp_app;
|
||||
|
||||
A periodic worker uses those functions, enters withTenant, locks the draft, and expires an in_review round whose deadline is at or before DB clock. No pending scheduled_publications row exists until quorum. This avoids making an unapproved post visible to the existing sweeper.
|
||||
|
||||
## 3. State machine
|
||||
|
||||
Primed is a short, durable state after an editor seals the version and before the server opens review; the same request normally advances primed → in_review. Keeping it visible allows recovery of a transaction interrupted between an external workflow start and the durable submission (Temporal start remains after commit). Published is a short successful result, then archived in the same completion transaction; both transitions get events. The current state CHECK and TS union must be extended together [apps/api/migrations/006_publishing.sql:7-8; apps/api/src/publishing/publishing.service.ts:8; apps/web/src/lib/api-types.ts:64].
|
||||
|
||||
| From → to | Initiator | Required guard and action |
|
||||
| --- | --- | --- |
|
||||
| create → editing | group editor or Owner/Admin | Group and target connection exist in same workspace; group owns connection; create version 1. |
|
||||
| editing/needs_changes/expired → primed | editor or Owner/Admin | Lock draft; valid target; nonempty content; at least N eligible non-author reviewers or a declared admin escalation; future deadline if supplied; increment round, snapshot post/target/quorum/deadline. |
|
||||
| primed → in_review | service | Same transaction or idempotent recovery; event and audit; no votes yet. |
|
||||
| in_review → needs_changes | eligible reviewer | Deny with nonempty reason; close submission denied. |
|
||||
| in_review → approved | service | Distinct currently eligible non-author approvals reach frozen quorum before deadline, or non-author admin override with reason. |
|
||||
| in_review → expired | deadline worker or final vote | DB clock >= publish_at without quorum; close round, no schedule row. |
|
||||
| in_review/approved → editing | editor or Owner/Admin | Supersede round, retain history, clear schedule intent; approved has no active schedule. |
|
||||
| approved → scheduled | service or Publisher/Admin | Recount current votes/override; version/target unchanged; connection still assigned; pending schedule insert. If no desired time, authorized Publisher/Admin chooses publish time after approval. |
|
||||
| approved → expired | service | Stored desired publish_at passed before scheduling. |
|
||||
| scheduled → approved | authorized scheduler-canceller | Only pending schedule, cancel/delete row as current service does [apps/api/src/publishing/publishing.service.ts:248-272]. |
|
||||
| approved/scheduled → in_review | membership reconciliation | Quorum lost before remote claim; cancel pending schedule, reopen same sealed round, and preserve remaining votes. |
|
||||
| in_review/approved/scheduled → needs_changes | connection/group reconciliation | Target is no longer valid; cancel pending schedule and require editor to choose a valid target and resubmit. |
|
||||
| scheduled → editing | editor or Owner/Admin | First cancel pending schedule, then supersede; reject if in_flight/done. |
|
||||
| scheduled → published | scheduler | Claim pending only when due, current quorum and hash still valid; remote success; idempotency key matches. |
|
||||
| published → archived | scheduler | Same completion transaction, preserving publication row and review history. |
|
||||
| any nondeleted → same state, hidden/unhidden | Owner/Admin | Admin moderation; hide applies to working and archive views. |
|
||||
| any except in_flight scheduled → tombstoned | Owner/Admin | Admin soft delete with mandatory reason; pending schedule cancelled; no physical cascade. |
|
||||
|
||||
No direct editing → approved, needs_changes → approved, or expired → scheduled path exists. A denied post re-enters review only via a new version/submission, even if text is unchanged. A schedule request at a timestamp already passed is rejected; immediate publication is represented by publish_at=clock_timestamp() only after approval. Existing schedule is CAS on approved and inserts one scheduled_publications row [apps/api/src/publishing/publishing.service.ts:216-235].
|
||||
|
||||
## 4. RBAC and API contract
|
||||
|
||||
The guard currently checks signed session, live workspace authz_epoch, then workspace role_permissions inside withTenant [apps/api/src/rbac/rbac.guard.ts:33-62]. Keep that order. Add RequireReviewGroupAccess(action) as a second decorator, analogous to RequireGroupAccess for connectionId [apps/api/src/rbac/rbac.guard.ts:8-19,64-74]. It reads :draftId or :groupId and performs a tenant-scoped draft/group/active membership check, returning 404 for invisible group drafts. Every write service repeats the check after SELECT ... FOR UPDATE because epoch or group membership may change between guard and mutation.
|
||||
|
||||
Exact permission additions to role_permissions (no new workspace role names):
|
||||
|
||||
INSERT INTO role_permissions(role, permission)
|
||||
SELECT r, p FROM unnest(ARRAY[
|
||||
'Owner','Admin','Publisher','Editor','Approver','Analyst','Moderator',
|
||||
'VideoPublisher','StreamOperator','BroadcastOperator']) AS r
|
||||
CROSS JOIN unnest(ARRAY[
|
||||
'review.view','review.comment','review.submit','review.edit','review.vote']) AS p
|
||||
ON CONFLICT DO NOTHING;
|
||||
INSERT INTO role_permissions(role, permission)
|
||||
SELECT r, p FROM unnest(ARRAY['Owner','Admin']) AS r
|
||||
CROSS JOIN unnest(ARRAY[
|
||||
'review.config.manage','review.override','post.hide','post.delete']) AS p
|
||||
ON CONFLICT DO NOTHING;
|
||||
|
||||
These route permissions admit an active workspace member to the group-role decision; they do not independently grant editorial actions. This is necessary because group reviewer may currently have any one of the ten workspace roles, while approval.grant today belongs only to Owner/Admin/Approver and approval.request to Owner/Admin/Publisher [apps/api/migrations/002_rbac_seed.sql:4-50]. Keep existing approval.grant/approval.request for legacy callers, but new group review routes require the review.* permission plus exact group role. Group member may view/comment; editor may view/comment/create/edit/submit but cannot vote or schedule/publish; reviewer may view/comment/vote but cannot change content; Owner/Admin may manage policy, hide/delete, and override with a reason. Publisher workspace role alone cannot bypass group role or quorum. The existing publication.schedule permission is still required for manual scheduling after approval; automatic scheduling is a server action caused by the authorized final vote.
|
||||
|
||||
Proposed Nest controller shape (each handler also validates DTOs and uses the service transaction):
|
||||
|
||||
@Get('groups/:groupId/review') @RequirePermission('review.view') @RequireReviewGroupAccess('view')
|
||||
listReviewQueue(...): Promise<ReviewQueuePage>;
|
||||
@Post('publishing/drafts/:draftId/prime') @RequirePermission('review.submit') @RequireReviewGroupAccess('submit')
|
||||
prime(...): Promise<ReviewDetail>;
|
||||
@Post('publishing/drafts/:draftId/votes') @RequirePermission('review.vote') @RequireReviewGroupAccess('vote')
|
||||
vote(@Body() dto: { submissionId: string; versionId: string;
|
||||
snapshotSha256: string; decision: 'approve' | 'deny'; reason?: string }): Promise<ReviewDetail>;
|
||||
@Post('publishing/drafts/:draftId/override') @RequirePermission('review.override')
|
||||
override(@Body() dto: { submissionId: string; reason: string }): Promise<ReviewDetail>;
|
||||
@Post('publishing/drafts/:draftId/messages') @RequirePermission('review.comment') @RequireReviewGroupAccess('comment')
|
||||
comment(@Body() dto: { body: string }): Promise<ReviewEvent>;
|
||||
@Patch('publishing/drafts/:draftId/moderation') @RequirePermission('post.hide')
|
||||
setHidden(@Body() dto: { hidden: boolean; reason: string }): Promise<ReviewDetail>;
|
||||
@Delete('publishing/drafts/:draftId') @RequirePermission('post.delete')
|
||||
tombstone(@Body() dto: { reason: string }): Promise<void>;
|
||||
|
||||
The new vote endpoint accepts submission/version/hash as optimistic preconditions and returns 409 on mismatch; the server derives actor and role from its session, never request JSON. Self approval is checked in the service AND in a DB vote guard: NEW.voter_user_id <> drafts.author_user_id; the override table/record uses the same exclusion. Reusing the direct current approve endpoint without this change would bypass quorum [apps/api/src/publishing/publishing.controller.ts:91-110]. Group role/policy mutations already bump the workspace epoch through group_members AFTER UPDATE; group settings and connection assignment changes need these additional triggers:
|
||||
|
||||
CREATE TRIGGER trg_group_review_policy_epoch
|
||||
AFTER UPDATE OF review_quorum_override ON groups
|
||||
FOR EACH ROW WHEN (OLD.review_quorum_override IS DISTINCT FROM NEW.review_quorum_override)
|
||||
EXECUTE FUNCTION bump_authz_epoch();
|
||||
CREATE TRIGGER trg_group_connection_review_epoch
|
||||
AFTER INSERT OR DELETE ON group_connections
|
||||
FOR EACH ROW EXECUTE FUNCTION bump_authz_epoch();
|
||||
CREATE FUNCTION bump_review_default_epoch() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NEW.review_quorum_default IS DISTINCT FROM OLD.review_quorum_default THEN
|
||||
NEW.authz_epoch := OLD.authz_epoch + 1;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER trg_workspace_review_default_epoch
|
||||
BEFORE UPDATE OF review_quorum_default ON workspaces
|
||||
FOR EACH ROW EXECUTE FUNCTION bump_review_default_epoch();
|
||||
|
||||
These mutations mint the caller a replacement token inside the transaction, matching GroupsService [apps/api/src/groups/groups.service.ts:12-18,226-267].
|
||||
|
||||
## 5. Quorum engine and integrity
|
||||
|
||||
One submission has one vote per reviewer enforced by UNIQUE(submission_id,voter_user_id). Votes cannot change within that submission; a mistaken decision is handled by an editor resubmission, with a reason and clear history. A denial closes the round immediately. Quorum is COUNT(DISTINCT voter_user_id) among approvals that are unrevoked, from active workspace and reviewer group memberships, excluding the author and matching the sealed hash. The round's required_approvals never changes. Config edits affect the next round. The repository already uses CAS transitions, so the quorum transaction locks the draft row and then writes vote, count, state, review event, and audit entry atomically [apps/api/src/publishing/publishing.service.ts:275-292]. All review votes, role changes, and scheduler claims lock the workspace row first, then draft, then schedule, so an epoch-changing membership mutation and a remote claim have a defined serialization point. A claim that wins the lock may publish; a removal that wins first forces the claim to recheck and fail.
|
||||
|
||||
type VoteInput = {
|
||||
submissionId: string; versionId: string; snapshotSha256: string;
|
||||
decision: 'approve' | 'deny'; reason?: string;
|
||||
};
|
||||
async function castVote(client: PoolClient, draftId: string, actor: Actor, input: VoteInput) {
|
||||
const draft = await lockDraft(client, draftId); // SELECT ... FOR UPDATE
|
||||
assertLiveEpochAndGroupReviewer(client, draft, actor);
|
||||
assert(draft.authorUserId !== actor.userId, 'SELF_APPROVAL_FORBIDDEN');
|
||||
assert(draft.state === 'in_review', 'REVIEW_CLOSED');
|
||||
const round = await activeSubmission(client, draft);
|
||||
assert(round.id === input.submissionId && round.versionId === input.versionId &&
|
||||
round.contentSha256 === input.snapshotSha256, 'STALE_SNAPSHOT');
|
||||
if (round.publishAt !== null && await dbNow(client) >= round.publishAt) return expire(client, draft, round);
|
||||
if (input.decision === 'deny') assert(input.reason?.trim(), 'DENIAL_REASON_REQUIRED');
|
||||
await insertVoteAndEventAndAudit(client, draft, round, actor, input);
|
||||
if (input.decision === 'deny') return closeNeedsChanges(client, draft, round);
|
||||
if (await validApprovalCount(client, round) >= round.requiredApprovals) {
|
||||
await markApprovedAndAudit(client, draft, round);
|
||||
if (round.publishAt !== null) await scheduleIfFuture(client, draft, round);
|
||||
}
|
||||
return readReviewDetail(client, draft.id);
|
||||
}
|
||||
|
||||
The hash covers all JSONB post fields and the target connection. The reviewer detail response uses draft_versions.post, never drafts.post, to render the proposed publication; it includes previous submitted version and a structured diff for text, media IDs, visibility, target, and provider metadata. The current scheduler reads d.post at claim time [apps/scheduler/src/activities/publish.activity.ts:133-145]; change that query to join the approved submission/version and use its post. Also require d.state='scheduled', sp.publish_at <= clock_timestamp(), a current quorum/override, and the approved version hash at claim. This is the final guard against edits, role removals, and early Temporal wakeups. Claims must be idempotent and use the pending→in_flight state as today [apps/scheduler/src/activities/publish.activity.ts:133-155].
|
||||
|
||||
The review_overrides table in §2 carries a nonempty reason and immutable snapshot hash. It is valid only while the admin retains active Owner/Admin status, is not the author, and the hash matches. It satisfies the approval gate but is labeled Override, never 2/3. One valid denial closes the round before an override can be applied; admin must return the post to an editor for a new submission. A second different admin can rescue a one-reviewer group; no override can waive the self-approval exclusion.
|
||||
|
||||
## 6. Review chat and delivery
|
||||
|
||||
review_events is the durable, per-post thread. Message bodies are plain text (escape on render), length limited by API to 4,000 characters, and not editable or deletable by ordinary members. System events carry structured IDs, version numbers, hashes, old/new state, reason, and quorum counts in data; do not put secrets or access tokens in it. The current audit scrubber redacts keys matching secret/token/credential, so event-to-audit diff construction must use the same scrub helper [apps/api/src/audit/audit.service.ts:9-18]. Events are written with the state mutation in the same tenant transaction. Increment drafts.review_event_seq under the draft row lock, then insert (draft_id,seq); this serializes commit order per draft and makes Last-Event-ID safe.
|
||||
|
||||
SSE endpoint GET /publishing/drafts/:draftId/events uses @RequirePermission('review.view') and group access. It sends rows after a validated integer cursor, then polls for new rows every 2 seconds, heartbeat every 15 seconds, and rechecks live epoch/group access at least each heartbeat; on change it sends auth_revoked and closes. Reconnect fetches the gap with Last-Event-ID. For native EventSource bearer-token limits, use fetch streaming with Authorization: Bearer rather than cookie or token in URL. The server caps replay pages and asks the client to resync detail when cursor is too old. SSE can lag but cannot grant action rights: every vote/comment still crosses RbacGuard and the transaction check. The existing realtime authz port compares epoch but assumes a numeric DB epoch; the new SSE path should normalize bigint as RbacGuard does [apps/api/src/realtime/gateway.ts:19-23; apps/api/src/rbac/rbac.guard.ts:42-47].
|
||||
|
||||
System event type shape:
|
||||
|
||||
type ReviewEventKind = 'message' | 'submitted' | 'edited' | 'approved' |
|
||||
'denied' | 'vote_revoked' | 'quorum_met' | 'scheduled' | 'expired' |
|
||||
'published' | 'archived' | 'hidden' | 'unhidden' | 'deleted' | 'override';
|
||||
interface ReviewEvent { draftId: string; seq: number; kind: ReviewEventKind;
|
||||
actorUserId: string | null; submissionId: string | null;
|
||||
body: string | null; data: Record<string, unknown>; at: string; }
|
||||
|
||||
## 7. UI plan
|
||||
|
||||
Replace the ten-line Approvals stub with a queue/detail surface; Compose remains the editor entry point. Compose already has tiptap, target chips, visibility, a schedule field, and state badges [apps/web/src/app/(main)/compose/page.tsx:65-88,151-220,247-265]. Remove its current create→submit→self-approve→schedule loop for group posts, since it permits an author with approval.grant to approve the post they just created [apps/web/src/app/(main)/compose/page.tsx:114-139]. Creation requires a group and exactly one target per draft; multiple targets produce linked sibling drafts, each independently reviewed. Submit opens the sealed review detail. The schedule field becomes a deadline/desired publish time included in the submission, explicitly labeled; if empty, approved posts await an authorized scheduler.
|
||||
|
||||
The queue uses cursor pagination (updated_at DESC,id DESC) and virtual rows via the existing StreamColumn useVirtualizer pattern [apps/web/src/components/streams/column.tsx:28-39,56-78]. Each row shows group, target, preview, state, required/current count, author, due time, and an expired/needs_changes label. Filters: needs my review (eligible, not author, no vote), created by me, needs changes, expiring soon, approved awaiting schedule, and archive. Use j/k and arrow navigation, Enter to open, and preserve focus/scroll on return, following Streams' existing keyboard scheme [apps/web/src/app/(main)/streams/page.tsx:77-80]. Do not bind shortcuts while focus is in Tiptap, an input, or a chat composer.
|
||||
|
||||
Detail has the immutable rendered post preview, target/visibility/schedule, version selector and diff, quorum progress with named votes, then the thread with Approve and Deny+reason buttons for eligible reviewers. The buttons submit the displayed submission/version/hash preconditions, disable while pending, and handle 409 by refreshing detail. An editor sees Edit and Resubmit; a reviewer sees no content edit control. Admin controls Hide/Unhide and Delete appear on working and archive posts. Hidden posts are omitted from member queue/archive/search, but admins see a Hidden badge and full record. Archive is read-only for content and comments, with preserved review history and publication link; admin moderation remains available. The current Published state is already a draft state but has no archive UI [apps/api/migrations/006_publishing.sql:7-8; apps/web/src/app/(main)/compose/page.tsx:35-42].
|
||||
|
||||
## 8. Edge cases and exact resolutions
|
||||
|
||||
| Case | Resolution |
|
||||
| --- | --- |
|
||||
| Reviewer leaves group or loses reviewer role before quorum | Existing group_members change bumps epoch; a same-transaction reconciliation marks their votes revoked_at/reason, emits vote_revoked and audit entries, then recalculates count. New tokens are minted for the mutator; stale sessions get 401 [apps/api/migrations/008_groups.sql:75-80; apps/api/src/groups/groups.service.ts:226-267]. |
|
||||
| Workspace membership is removed/suspended | Membership epoch trigger invalidates sessions [apps/api/migrations/001_tenants.sql:180-192]. Team mutation must call the same reconciliation; claim and every vote independently join active membership, so a missed callback cannot publish. |
|
||||
| Quorum had been met, but reviewer leaves before pending schedule claims | Reconciliation cancels a pending schedule, returns scheduled/approved to in_review, emits quorum_lost and audit, and alerts the queue. If claim already set in_flight, its pre-call gate wins; once remote call began, treat completion as published and log the exact eligibility change timestamp. No attempt to pretend a remote publish can be rolled back. |
|
||||
| Final vote and deadline worker race | Both lock drafts row. DB clock is checked after lock. Vote first before deadline may schedule; worker later sees approved/scheduled and does nothing. Worker first at/after deadline expires; vote gets REVIEW_CLOSED. |
|
||||
| Final vote at T−1 minute | Auto-schedule pending row for T, start Temporal after commit. If Temporal wakes early, claim rejects until publish_at; sweeper picks it up when due. |
|
||||
| Quorum arrives at/after deadline | Expire instead of approve. Editor chooses a new future time, creates a new round, and all reviewers vote anew. No auto-publish of stale content. |
|
||||
| Author is sole reviewer | Prime is blocked with insufficient independent reviewers, unless a different Owner/Admin is available for reasoned override. The author, even as Owner/Admin, cannot vote/override. |
|
||||
| Reviewer goes on vacation | Admin changes group reviewer assignment or adds another group member with reviewer role; epoch invalidates old sessions, pending votes from removed reviewers revoke, and the current round can continue if enough eligible people remain. Delegation is an explicit role change, not token sharing. |
|
||||
| Editor changes text/media/visibility/target during review | Action supersedes round and its votes; new immutable version and diff are required. Existing vote requests fail 409. |
|
||||
| Editor changes approved post | Reopen to editing; if pending scheduled first cancel it atomically. In_flight/done cannot be reopened; create a new draft instead. |
|
||||
| Duplicate vote or concurrent final votes | Unique submission/reviewer plus draft row lock yields one live count and one schedule. Duplicate returns 409 with current detail; retries with an idempotency request key may return prior result. |
|
||||
| Epoch changes after guard but before vote | Transaction reads epoch again while holding draft lock and checks active group/role; mismatch fails 401 with no vote. |
|
||||
| Group/connection unassigned mid-review | Prime/vote/schedule recheck assignment. On unassignment, active round becomes needs_changes and pending schedule cancels; editor chooses a valid target and resubmits. Existing group_connections mutations currently audit assignment but do not bump epoch, so 009 adds an epoch trigger on that table too [apps/api/src/groups/groups.service.ts:271-314]. |
|
||||
| Group deleted with working drafts | New draft group FK uses ON DELETE RESTRICT, so admin must tombstone/reassign drafts first; this deliberately prevents review evidence cascading with group deletion, unlike current group children [apps/api/migrations/008_groups.sql:27-60]. |
|
||||
| Hidden or deleted post has a pending schedule | Hide is internal only and does not cancel publication by itself; UI explicitly warns that scheduling continues. Delete cancels pending schedule and tombstones; reject delete if in_flight, and on published archive tombstone only the internal record. Remote provider deletion is a separate future integration. |
|
||||
| Scheduler failure after remote call | Preserve current idempotency_key and publication lookup, reconcile by remote result, and never automatically retry with a new key; audit publication.failed and show retry state. Current activity first checks publication by key [apps/scheduler/src/activities/publish.activity.ts:75-90,123-130]. |
|
||||
| Old rejected/in_review row | Migration moves to needs_changes and demands group/target selection before prime; never count reviewer_user_id as a vote. |
|
||||
|
||||
The same-transaction reconciliation is a change to GroupsService, TeamService, and group connection mutation paths, not a background-only cleanup. DB transition and claim guards remain fail closed even if an application path forgets to call it. For direct SQL/cascades, add an AFTER DELETE/UPDATE trigger on group_members and memberships that invokes a narrow review_reconcile_member(workspace_id,user_id,group_id) function under tenant context; it updates affected vote revocations and pending draft states and appends audit events. Because a cascade can delete the parent before the child trigger runs, the function uses group_members' denormalized OLD.workspace_id and OLD.group_id, following the existing epoch design [apps/api/migrations/008_groups.sql:4-12].
|
||||
|
||||
## 9. Audit, moderation, and security
|
||||
|
||||
Every state transition, vote, revocation, override, config/role change, message, hide/unhide, and tombstone gets audit_append in the SAME tenant transaction as the corresponding rows, as GroupsService already does [apps/api/src/groups/groups.service.ts:12-13,123-133]. Current PublishingService appends audit after the mutation commits, which leaves a gap; migrate its review paths to same-tx audit [apps/api/src/publishing/publishing.service.ts:127-138,307-322]. Scheduler already appends publication.published in its completion transaction with actor_role sp_scheduler [apps/scheduler/src/activities/publish.activity.ts:158-190]. Never call audit_append_system from a tenant workflow; the DB distinguishes tenant/system by app.workspace_id [apps/api/migrations/001_tenants.sql:88-121,123-144].
|
||||
|
||||
Event names: review.primed, review.submitted, review.version_created, review.vote_approved, review.vote_denied, review.vote_revoked, review.quorum_met, review.quorum_lost, review.expired, review.needs_changes, review.overridden, publication.scheduled, publication.schedule_cancelled, publication.published, publication.archived, post.hidden, post.unhidden, post.deleted, review.message_added, review.policy_changed, review.role_changed. Audit diff includes draftId, groupId, round, versionId, contentSha256, target connection ID, frozen quorum, deadline, old/new state, actor and decision/reason when relevant. audit_hash currently chains actor/action/target/time, while diff is stored but not part of the hash input [apps/api/migrations/001_tenants.sql:55-67,92-121]. Prototype should verify whether regulatory expectations require diff integrity; if so, use a separately versioned audit hash migration and verifier, not a silent change to existing hashes.
|
||||
|
||||
Moderation is workspace Owner/Admin only, irrespective of group role; new post.hide/post.delete permissions are seeded only to those two roles. Hide toggles hidden_at/hidden_by, keeps all rows and remote content intact, and excludes the post from non-admin list/detail/event endpoints; an admin sees a badge. Soft delete sets deleted_at/deleted_by, hidden_at, reason in audit, cancels pending work, and retains versions, votes, thread, and publication rows for verification. The DELETE HTTP verb therefore means tombstone, not a SQL DELETE. A published post may remain remote, so the UI says Remove from this workspace and does not promise deletion at the provider. Archive remains read-only for normal group members. RLS isolates workspaces; group visibility is an explicit guard/service condition because workspace RLS alone cannot hide another group's draft [apps/api/migrations/006_publishing.sql:58-66; apps/api/migrations/008_groups.sql:91-104].
|
||||
|
||||
No API accepts caller-supplied actor role, vote tally, or audit hash. FK checks can bypass RLS, so draft/group/connection lookups must be RLS-scoped before insert, as the current schedule method explains [apps/api/src/publishing/publishing.service.ts:216-220]. The hash and preview are bound to the version; responses use optimistic preconditions. Read endpoints must filter hidden/deleted before loading events. Admin reads are still tenant-scoped.
|
||||
|
||||
## 10. Phased implementation plan and gates
|
||||
|
||||
1. **WP1 — Migration and compatibility.** Add 009 schema, constraints, RLS, role seed additions, backfill/version hash, and transition guard. Gate: migrate a fixture DB containing all six old states, confirm no fabricated votes, FORCE RLS isolation, valid legacy scheduling, and audit chain verification.
|
||||
2. **WP2 — Group policy and authz.** Group role/config endpoints, epoch triggers and replacement tokens, draft group/connection scope, guard decorator. Gate: role matrix, self-approval DB rejection, cross-group 404, stale token 401, and cascade reconciliation.
|
||||
3. **WP3 — Snapshot/quorum services.** Prime, edit/resubmit, votes, denial, override, transactional event+audit, version diff API. Gate: concurrent last votes create one schedule; denial reason and stale version fail; no author vote.
|
||||
4. **WP4 — Scheduler/deadlines.** Due deadline worker, claim eligibility/hash/time check, automatic schedule, archive on completion, failure/idempotency behavior. Gate: deadline/claim race and remote call exactly once under retries.
|
||||
5. **WP5 — Queue/detail/archive UI and SSE.** Compose handoff, virtualized queue, keyboard model, thread replay, buttons, moderation. Gate: keyboard and focus checks, 409 refresh, hidden visibility, SSE reconnect/epoch revocation. Email/external notification hook can emit an outbox row for future workers, without sending in this phase (spec R9).
|
||||
|
||||
## 11. Ten regression tests
|
||||
|
||||
1. **Self approval:** author with group reviewer role and Owner permission cannot vote or override; another eligible reviewer can.
|
||||
2. **Two final votes race:** both requests serialize; one state transition, one scheduled_publications row, two votes, valid audit chain.
|
||||
3. **Denial reason:** blank/whitespace denied at DTO and DB; valid denial closes round and previous approvals cannot count on resubmit.
|
||||
4. **Edit staleness:** after a reviewer loads version V, editor edits/resubmits V+1; vote with V/hash fails 409, and V remains viewable.
|
||||
5. **Role removal and epoch:** group reviewer removal revokes pending vote, reduces count, invalidates old token, and blocks schedule/claim.
|
||||
6. **Deadline serialization:** vote at T−1s schedules; vote at/after T expires; worker and final vote in parallel never publish an unapproved post.
|
||||
7. **Scheduler claim:** early Temporal wake, stale hash, missing quorum, and unassigned connection all fail before connector.publish; retry by same key does not duplicate a successful remote post.
|
||||
8. **Tenant and group isolation:** FORCE RLS with missing/foreign app.workspace_id returns zero rows; cross-group member gets 404 on detail/SSE and cannot vote via guessed IDs.
|
||||
9. **Migration:** old approved/scheduled/published remain grandfathered; old in_review/rejected become needs_changes; reviewer_user_id creates no fake vote; audit_verify_chain stays true.
|
||||
10. **Moderation/archive:** ordinary group member cannot hide/delete; admin tombstone cancels pending schedule, preserves audit/thread; published completion creates archived state and hides internal posts from member lists.
|
||||
|
||||
The live-DB tests should follow the existing groups integration pattern for RLS/FKs/epoch and the scheduler repository's idempotency behavior, rather than only mock SQL calls [apps/api/src/groups/groups.integration.test.ts:217-245; apps/scheduler/src/activities/publish.activity.ts:75-90].
|
||||
|
||||
## Confidence & risks
|
||||
|
||||
The strongest risk is the boundary between a pending schedule, an in_flight remote call, and a reviewer role removal: once a provider call starts, local revocation cannot retract it. Prototype that race first with a controllable connector and assert the claim gate's exact lock order. The second risk is migration of old drafts without a unique target; inspect real providerMetadata shapes before finalizing the nullable backfill window. The third is throughput of per-post SSE polling and audit append contention; measure with many open review tabs before choosing polling intervals or a broker. The existing audit chain does not hash diff content [apps/api/migrations/001_tenants.sql:92-121], so a separate audit-hardening decision is needed if signoff evidence must be tamper-evident down to the snapshot hash and denial reason.
|
||||
153
docs/design/spec-review-workflow.md
Normal file
153
docs/design/spec-review-workflow.md
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
# Spec — Group Review & Signoff for Posts ("Editorial Workflow")
|
||||
|
||||
Status: DRAFT for proposals
|
||||
Owner: product owner
|
||||
Date: 2026-10-07
|
||||
Inputs: user requirements, codebase inventory (explore agent), background research (Hootsuite/Sprout approval features, four-eyes/maker-checker compliance patterns)
|
||||
|
||||
## 1. What the user asked for (verbatim intent)
|
||||
|
||||
1. Posts can be **primed to go out**, then reviewed and edited by others in the group.
|
||||
2. Roles: **editors** (can edit, can NOT post/approve) and **reviewers** (give the OK).
|
||||
3. System configurable so **2–3 people sign off** on a post before it publishes.
|
||||
4. Reviewers can **approve, or deny with reasoning** (reason mandatory).
|
||||
5. A **parallel chat** per post with voting/approval buttons.
|
||||
6. Easy **scrolling through existing working posts** (the in-review queue).
|
||||
7. When posts publish, they are **archived**.
|
||||
8. Ability to make our own posts **invisible or delete** them — **admin-only** features.
|
||||
9. Think through the whole system; spec the needs.
|
||||
|
||||
## 2. Current system (facts from codebase inventory)
|
||||
|
||||
### 2.1 Publishing state machine
|
||||
- States in DB CHECK (`migrations/006_publishing.sql:7-8`): `editing, in_review, approved, rejected, scheduled, published`. TS mirrors: `publishing.service.ts:8`, web `api-types.ts:64`.
|
||||
- Transitions are guarded CAS UPDATEs (`WHERE id=$1 AND state=$4`) via `transition()` helper (`publishing.service.ts:275-292`):
|
||||
- `editing→in_review` submitForReview (perm `approval.request`)
|
||||
- `in_review→approved` approve (perm `approval.grant`, stamps `reviewer_user_id`)
|
||||
- `in_review→rejected` reject (perm `approval.grant`; reason goes ONLY into audit diff, not the draft row)
|
||||
- `approved→scheduled` schedule (inserts `scheduled_publications`, perm `publication.schedule`)
|
||||
- `scheduled→approved` cancelSchedule
|
||||
- `scheduled→published` by scheduler activity (`publish.activity.ts:181-184`)
|
||||
- **Approval record: NONE.** Only `drafts.reviewer_user_id` (single reviewer, last-writer-wins) + audit events. No quorum structure exists.
|
||||
- **`rejected` is terminal** — there is NO resubmit path today. This feature must add one.
|
||||
- Temporal: one workflow `scheduledPublish(idempotencyKey)` proxying one activity, **no signals/queries**. API starts it fire-and-forget with `startDelay`; durable `scheduled_publications` row (pending/in_flight/done/failed) is source of truth; sweeper `publishing_due_schedules()` re-starts missed workflows.
|
||||
|
||||
### 2.2 Data model
|
||||
- `drafts(id, workspace_id, author_user_id, post jsonb, state, reviewer_user_id, created_at, updated_at)` — post payload = `{text, mediaAssetIds[], visibility, scheduledAt?, providerMetadata}`. **No soft-delete, no hidden flag, no delete endpoint at all.**
|
||||
- `scheduled_publications` — `UNIQUE(draft_id)` (one schedule per draft). `publications` — outcome rows (remote_id, idempotency_key).
|
||||
- All tables: FORCE RLS tenant_isolation on `app.workspace_id` + workspace-immutability triggers + `sp_app` grants. **Drafts are workspace-scoped only — no group column.**
|
||||
|
||||
### 2.3 RBAC
|
||||
- 10 roles seeded (`002_rbac_seed.sql`): Owner, Admin, Publisher, Editor, Approver, Analyst, Moderator, VideoPublisher, StreamOperator, BroadcastOperator.
|
||||
- `approval.grant`: only Owner/Admin/Approver. `approval.request`: Owner/Admin/Publisher.
|
||||
- Enforcement: `RbacGuard` + `@RequirePermission()` decorator; guard verifies HMAC session → workspace `authz_epoch` (fail-closed 401 on mismatch) → permission via `role_permissions JOIN memberships(status='active')` inside `withTenant`.
|
||||
- **Roles are per-workspace only** (`memberships.role`, UNIQUE(workspace_id,user_id) — one role per user per workspace). **`group_members` has NO role column** — no per-group roles exist.
|
||||
|
||||
### 2.4 Groups
|
||||
- Tables (`008_groups.sql`): groups, group_members (no role), group_connections, group_assets. Denormalized workspace_id + composite FKs prevent cross-tenant groups.
|
||||
- Epoch: any group_members change fires `bump_authz_epoch()`; sessions carry authzEpoch; guard compares live. Membership mutations mint a replacement token in the same tx (`{token}` response pattern).
|
||||
- Connection scoping: `getVisibleConnections` — Owner/Admin see all; others see union of group connections via their memberships. Guard-side `userCanUseConnection` reads role from DB.
|
||||
|
||||
### 2.5 Realtime & chat
|
||||
- **No WS transport exists.** `realtime/gateway.ts` is an authz port only (`authorizeSubscription` checks session, workspace, epoch, `stream.view`). No topics, no client code.
|
||||
- Web Inbox is a fixtures-only two-pane chat UI (conversation list + thread bubbles + reply box). Reusable as visual pattern, no backend.
|
||||
|
||||
### 2.6 Frontend
|
||||
- Compose (`compose/page.tsx`): tiptap editor, target chips, visibility, approval select, schedule; `publish()` chains `create→submit→[approve]→[schedule]` gated on `can()`; "Recent drafts" sidebar with STATE_TONE badges is the only post list today.
|
||||
- `/approvals` is a 10-line stub; `/calendar` stub.
|
||||
- Streams columns (`components/streams/column.tsx`): `@tanstack/react-virtual` useVirtualizer, j/k keyboard nav — reusable pattern for the working-posts queue.
|
||||
- Audit page (236 lines) with diff viewer exists.
|
||||
|
||||
### 2.7 Audit
|
||||
- `audit_log` hash-chained per workspace (`audit_hash = sha256(prev|actor|action|target_id|epoch)`), SECURITY DEFINER append fns, `audit_verify_chain`. Services write via `appendTenantAudit` (after-commit) or same-tx `SELECT audit_append(...)`. Scheduler writes with `actor_role='sp_scheduler'`.
|
||||
|
||||
## 3. Background research — patterns to draw from
|
||||
|
||||
### Social tools (Hootsuite, Sprout)
|
||||
- Workflow shapes: **linear, tiered (multi-level), parallel, conditional**. Sprout steps support **Any vs All** sign-off ("any 1 of step's approvers" / "all must approve").
|
||||
- **Post-locking after approval**: any post-approval edit re-triggers the chain. Hootsuite: "Once approved, the post is locked; changes send it back through approval."
|
||||
- **Expired queue**: if not approved by scheduledAt, publish FAILS, post moves to Expired (author re-schedules + resubmits). 90-day auto-reject of stale pending items.
|
||||
- Rejected → author edits → resubmits. Rejection notes surface to author.
|
||||
- **Approval Activity** panel: every edit (who/when) + threaded internal comments + current step; approvers can edit-before-approve; tag members; notifications (digest + hourly reminders).
|
||||
- Approvals inbox filters: Assigned-to-me, Created-by-me, Rejected, Expired.
|
||||
- Teams of approvers (not individuals) to avoid bottlenecks; duplicate-reviewer skip.
|
||||
|
||||
### Four-eyes / maker-checker (compliance-grade)
|
||||
- **Self-approval must be structurally impossible** — server-side check `approver ≠ author`, dedicated error, not policy.
|
||||
- **Approve a frozen snapshot, not a moving target**: submit seals content (hash); edits after submit = new submission + new approval cycle; prior snapshots stay in audit trail. Reviewer sees the real diff.
|
||||
- **Deny requires non-empty reason** (empty reason refused); approvals/denials recorded with actor+timestamp+snapshot hash.
|
||||
- Eligibility re-checked at every transition (not just submit).
|
||||
- Parallel double-approval (Signavio): parallel split/join, either order, automatic gateway; enforce distinct reviewers at assignment.
|
||||
- Edge cases to model explicitly: reviewer leaves mid-review (votes revoked? reassigned?), vacation delegation, emergency admin override (logged as override), conflict discovered late.
|
||||
- State machine enforced in BOTH app logic and DB triggers (defense in depth).
|
||||
- Metrics: approval cycle time, approvals per reviewer (rubber-stamp detection).
|
||||
|
||||
## 4. Requirements (normative for proposals)
|
||||
|
||||
### R1 Lifecycle
|
||||
- New/extended states covering: draft/editing → **primed/submitted for review** → in-review → approved (quorum met) → scheduled → published → **archived**; deny → **needs-changes** (back to author/editor with reasons); **resubmit** path (currently missing — `rejected` is terminal today).
|
||||
- If a scheduled time passes without quorum → **expired** state (post does NOT publish; author notified; re-prime after re-approval).
|
||||
|
||||
### R2 Roles & permissions
|
||||
- **Editor**: can edit group posts (content), cannot approve, cannot publish.
|
||||
- **Reviewer**: can approve/deny with mandatory reason, cannot edit content.
|
||||
- Roles must be assignable **per group** (group-scoped), coexisting with workspace roles. Author can never approve their own post (structural, server-side). Admin/Owner retain override power (logged as override).
|
||||
- Existing `approval.grant`/`approval.request` permissions are the workspace-level seeds; extend, don't break, current 10-role model.
|
||||
|
||||
### R3 Quorum signoff
|
||||
- Configurable **N approvals required** (2–3 typical) — decide where config lives (workspace default + per-group override? per-connection?) and who sets it (admin-only).
|
||||
- Parallel (any-N-of-eligible) semantics by default; justify if you add tiered/sequential mode.
|
||||
- Vote rules: one live vote per reviewer per submission; vote change allowed while in-review (with new reason) or immutable — pick and justify. Deny = single vote sends to needs-changes. Quorum met → auto-transition approved (→ scheduling continues if schedule set).
|
||||
|
||||
### R4 Staleness & integrity
|
||||
- Content edits during review must NOT silently invalidate approvals: pick a model (frozen snapshot + new cycle on edit, vs edit-with-approval-dismissal + diff shown) and justify. Snapshot hash in audit log.
|
||||
- Reviewer must always see exactly what they're approving (rendered post + diff since last review).
|
||||
|
||||
### R5 Review chat
|
||||
- Per-post thread: messages from group members + **system events** (submitted, edits, approvals/denials with reasons, state changes) rendered inline.
|
||||
- **Approve / Deny+reason buttons in the chat surface** for eligible reviewers.
|
||||
- Realtime delivery: no WS transport exists today — propose transport (WS gateway wiring, SSE, or polling) with epoch/authz integration; justify choice.
|
||||
|
||||
### R6 Working-posts queue (UI)
|
||||
- Scrollable list of in-flight posts: state chips, quorum progress (e.g. "2/3 ✓"), filters (needs-my-review, needs-changes, expiring soon), keyboard-friendly (j/k pattern from Streams).
|
||||
- Post detail: rendered post preview + version diff + chat/vote panel + schedule info.
|
||||
|
||||
### R7 Archive & moderation
|
||||
- Published posts land in an **archive view** (read-only, chat history preserved for record).
|
||||
- **Hide** (invisible to members; badge for admins) and **delete** (hard or soft — justify) of posts — **space-admin-only**, new permissions, hash-chained audit events, epoch-safe.
|
||||
- Decide moderation scope: archive-only or also drafts/working posts.
|
||||
|
||||
### R8 Security & audit (mandatory repo patterns)
|
||||
- Everything through existing RbacGuard/`@RequirePermission`, FORCE RLS + tenant isolation on new tables, sequential migration `009_*`, hash-chained audit events for every state transition and vote, epoch re-token where membership/role changes affect pending reviews (reviewer removed from group → their pending votes must be revoked/re-counted).
|
||||
|
||||
### R9 Non-goals (this phase)
|
||||
- Email/external notifications (design the hook, don't build), mobile-specific UI, per-platform conditional routing, AI-assisted review.
|
||||
|
||||
## 5. Open design questions (proposals must answer explicitly)
|
||||
|
||||
1. Group-scoped role storage: column on `group_members` vs separate `group_member_roles` table vs reusing workspace roles — tradeoffs?
|
||||
2. Snapshot model: full content copies per version vs content-hash + immutable version rows?
|
||||
3. Where does quorum config live and what's the precedence (workspace default → group override)?
|
||||
4. Chat transport: WS (wire the existing authz port) vs SSE vs polling — cost/complexity/ordering guarantees?
|
||||
5. Deny semantics: 1-deny-returns-to-author vs majority-deny vs configurable; what happens to other cast approvals on deny?
|
||||
6. Re-review after edit: full reset vs dismiss-only-changed-scope (recommend one, note the other as future)?
|
||||
7. Scheduled-time race: quorum completes at T-1min to scheduledAt — auto-publish or require manual re-confirm?
|
||||
8. Author is also the only reviewer in a tiny group — deadlock handling (admin override path)?
|
||||
9. Migration path for existing single-step approvals (`reviewer_user_id` backfill)?
|
||||
10. Should reviewers see who else approved before them (peer pressure) or vote blind until quorum?
|
||||
|
||||
## 6. Proposal deliverable format (each lane produces `runs/wp5/proposal-<lane>.md`)
|
||||
|
||||
1. Executive summary — the 5 sharpest decisions you make.
|
||||
2. Data model — migration SQL sketch (009+, following repo conventions: FORCE RLS, tenant isolation, immutability triggers, sp_app grants, audit fns reuse).
|
||||
3. State machine — full state/transition table with guards and who can fire each.
|
||||
4. RBAC — new permissions, group-role mechanism, self-approval exclusion, admin override.
|
||||
5. Quorum engine — vote lifecycle, staleness/snapshot semantics, deny handling, race conditions.
|
||||
6. Review chat — data model, event types, transport decision, authz integration.
|
||||
7. UI plan — screens (queue, detail, archive), reuse of existing components, keyboard model.
|
||||
8. Edge cases — the ugly ones (reviewer leaves, deadline race, deadlock, epoch churn mid-vote).
|
||||
9. Audit & security — event list, chain integration, RLS notes.
|
||||
10. Phased implementation plan — WP-sized chunks with test gates.
|
||||
11. Test plan sketch — the 10 tests that would catch the worst regressions.
|
||||
|
||||
Constraints: NestJS + Postgres 16 + Temporal patterns as in repo; TS strict, no `as any`; follow existing naming/style; do not break the 10-role seed contract; proposals are designs — do NOT implement.
|
||||
540
docs/design/streaming-architecture.md
Normal file
540
docs/design/streaming-architecture.md
Normal file
|
|
@ -0,0 +1,540 @@
|
|||
# Streaming Architecture — Control Room, Overlays, Fallback, Chat, Telegram Ingest
|
||||
|
||||
Status: decision document (synthesis of four WP4 research reports). No code was changed while writing it.
|
||||
Date: 2026-10-07
|
||||
Inputs:
|
||||
- **[flash]** `runs/wp4/control-room-research-flash.md`
|
||||
- **[glm]** `runs/wp4/control-room-research-glm.md`
|
||||
- **[as]** (integration research, kept outside the repository)
|
||||
- **[tg]** `runs/wp4/telegram-amplify-research.md` (WP15 middle)
|
||||
- **[scope]** `runs/wp4/control-room-scope.md`; **[dm]** `docs/design/design-merged.md`; **[wb]** `docs/work-breakdown.md` WP9–WP13, WP15; **[prop]** `docs/proposal.md` §4.4/§4.5
|
||||
|
||||
**Provenance legend.** `[flash]`, `[glm]`, `[as]`, `[tg]` mark the report a decision comes from. `[both]` means flash and glm agree. `[SYN]` marks a decision this document makes where neither report settled the question, or where the two disagree. Each `[SYN]` explains why it wins.
|
||||
|
||||
---
|
||||
|
||||
## 0. Decisions at a glance
|
||||
|
||||
| # | Question | Decision | Source |
|
||||
|---|---|---|---|
|
||||
| D1 | Where is program composited? | **On the server**, in the media worker's FFmpeg compositor. The browser is a control surface and a preview consumer only. | [both] |
|
||||
| D2 | One FFmpeg process per channel, or split processes? | **Split**: ingest×N → compositor (replaceable) → **frame gate** (clock master) → persistent encoder → push relays. A graph change starts a new compositor (A/B swap). The encoder and the RTMP session are never restarted. | [SYN] over [flash]'s one-process invariant |
|
||||
| D3 | Multiview | Every tile plays the worker's **preview tap**, never the remote source. 2x1/2x2 tiles are live LL-HLS. 3x3/4x4 tiles are thumbnails, with live LL-HLS only on the PGM tile and the focused tile. Hard cap of **4 live decodes on desktop and 1 on mobile**. | [both]; caps [SYN] |
|
||||
| D4 | Audio | **Monitor mix** runs in the browser (one shared `AudioContext`, gain per tile, same-origin preview audio). **Program mix** runs in the compositor (`amix` with weights from desired state, gain in 3 dB steps). | [both]; steps [glm] |
|
||||
| D5 | Overlay rendering | **Pre-rendered assets.** The API renders an SVG template to a PNG with alpha. The worker composites it with `overlay`. A chyron is one wide PNG strip scrolled by an `overlay` x-expression. FFmpeg never sees user text. | [flash] over [glm]'s `drawtext`+`textfile` |
|
||||
| D6 | Playlist fallback | **On the worker.** A playlist is an ordinary ingest kind feeding a SLATE/PLAYLIST scene into the same encoder. PeerTube runs as `permanentLive`. PeerTube-native playlists are only the programming source of truth. | [both]; "Tier-2 PeerTube-only mode" from [glm] deferred |
|
||||
| D7 | Auto Director | **Rules only**: a priority ladder over monitor states, plus dwell time per source class, ending in the fallback scene. Automation writes the same buses through the same transaction as a human TAKE. | [both]; dwell-by-class [glm] |
|
||||
| D8 | PGM/PST transitions | v1 ships **CUT**. FADE follows behind a gate, as a transition graph swapped twice (old+new with `xfade`, then new only). The transition is part of desired state, not an imperative command. | [SYN] from [flash] Q6 + [glm] Q6 |
|
||||
| D9 | Chat transport to the app | **WebSocket**: subscribe/ack, auth sent in the first frame, `lastSeq` resume from a server ring buffer. REST cursor polling (`?after=seq`) is the fallback. **No FCM.** | [as]; polling fallback aligned with [dm] §7 |
|
||||
| D10 | Telegram "join and amplify" | Bot-as-channel-admin webhook → `archived_feed_items` → review draft → existing `amplify()`. MTProto stays a flagged Phase 2 and is not in this plan. | [tg] |
|
||||
|
||||
---
|
||||
|
||||
## 1. THE composition decision
|
||||
|
||||
### 1.1 Options
|
||||
|
||||
- **(A) Server-side FFmpeg compositor.** Runs inside `apps/media-worker` under the supervisor and receives commands over the existing Ed25519 signed-snapshot bus.
|
||||
- **(B) Browser-composited program** (the StreamYard model). The browser draws onto a canvas, then `captureStream()`/WebCodecs → WebRTC/WHIP upstream → the worker transcodes to RTMP.
|
||||
|
||||
### 1.2 Evidence
|
||||
|
||||
| Criterion | (A) server compositor | (B) browser compositor | Source |
|
||||
|---|---|---|---|
|
||||
| **Operator laptop sleeps, tab crashes, Wi-Fi drops** | Program keeps running. The supervisor, output lease and reconcile loop already live on the worker. | **Program ends.** For a self-hosted product there is no cloud fallback, so the operator's machine becomes a single point of failure for a 24/7 channel. | [flash] Q3.1, [glm] Q3 |
|
||||
| **24/7 premise** (scope item 5: a playlist plays when nothing is live) | Fallback is just another scene, and the encoder never idles. | Unattended hours have no browser to run in, so a second, server-side path is needed anyway. | [glm] Q3 "disqualifying on its own" |
|
||||
| **Mobile (later)** | The phone is a pure control-plane client sending desired-state commands. No media runs on the device. | `HTMLMediaElement.captureStream` is `false` on Safari. WebCodecs `AudioEncoder` arrives only in Safari 26. Firefox Android has no `VideoEncoder`. | [flash] Q3 (BCD), [glm] Q3 (BCD) |
|
||||
| **Signed command bus** | Fits 1:1. A TAKE writes a bus row, bumps `desiredVersion`, writes an outbox row and publishes a signed full snapshot. The worker converges idempotently. The fence, seen-set, reconcile and assignment epochs all carry over unchanged. | Needs a second "what is program" authority in the browser, a WHIP ingest trust boundary, and a WebRTC signalling path outside egress mediation, secret handling and the reconcile gate. | [flash] Q3.2, [glm] Q3 "Fit with the signed-command worker" |
|
||||
| **Latency at the viewer** | Adds about one encode generation. | Can be under a second from glass to RTMP. | Both say **PeerTube's live→HLS chain dominates** (latency modes 1/2/3), so the gap mostly disappears at the viewer. [flash] Q3.4, Q5 |
|
||||
| **CPU placement** | Server decodes and encodes. This is already priced in encoder units by WP9 admission. | Moves CPU to the operator machine. | [glm] Q3 |
|
||||
| **Security** | Secrets, sandbox and egress pinning stay server-side, where the proposal already solved them. | The browser must decode every source at full quality, which breaks the "operator never pulls the remote source" rule. | [glm] Q3, [prop] §4.4 previews |
|
||||
|
||||
### 1.3 Decision — (A), server-side compositor [both]
|
||||
|
||||
The worker composes. The browser only watches previews and sends desired-state commands [glm]. A WebRTC/WHIP low-latency path, if ever needed, enters as **one more source kind** feeding the server compositor. Composition stays server-side either way [flash].
|
||||
|
||||
**Dissenting view (recorded).** Option (B) is the right answer for *interactive call-in shows*: sub-second guest interaction, zero server encode cost, and WYSIWYG editing on the operator's screen. StreamYard/Restream ship it (their internals are not verified in either report). We reject it because our product is a continuous Virtual Channel, not a studio session. If call-in guests become a requirement, the shape is vMix Call's [glm]: browsers *contribute* sources over WHIP, and the mix still happens on our worker.
|
||||
|
||||
### 1.4 Where the two reports disagree inside (A): process topology [SYN]
|
||||
|
||||
- **[flash]** proposes the invariant "**one FFmpeg process per channel**", with every TAKE, layout or overlay change rebuilding the graph.
|
||||
- **[glm]** and **[prop] §4.4** use separate ingest / compositor / persistent encoder / pusher processes: "scene switch = graph rewiring… encoder and pushers are never restarted".
|
||||
|
||||
**Decision: split topology, plus a frame gate.** FFmpeg cannot rewire a running filtergraph. Runtime commands only change parameters of filters that already exist. So in flash's single process, *every* scene change restarts the encoder and the RTMP publish. That breaks WP9's "output that does not stop" and the WP11/WP12 continuity acceptance. flash's own Risk 2 ("a bad rebuild drops frames or kills program") is the symptom. The split topology moves the rebuild away from the encoder:
|
||||
|
||||
```
|
||||
ingest_i (normalize → raw video + PCM over pipes, per source; also emits preview taps)
|
||||
│ (N inputs, plus a playlist ingest and overlay asset inputs)
|
||||
▼
|
||||
compositor[A] ── active graph for desiredVersion v
|
||||
compositor[B] ── started for v+1, warmed until its first frame is ready
|
||||
│ rawvideo (yuv420p, fixed frame size) + s16le 48 kHz (fixed bytes per frame)
|
||||
▼
|
||||
FRAME GATE (clock master: paced at the profile fps; on compositor stall it repeats the
|
||||
last frame and writes silence; switches A→B at a frame boundary once B
|
||||
has produced K frames; then SIGTERMs A)
|
||||
▼
|
||||
encoder (persistent; reads gate pipes; never restarted by scene changes)
|
||||
▼
|
||||
push relay ×M (WP9/WP13: lease-bound key over fd, pinned destination)
|
||||
```
|
||||
|
||||
- **Why the gate is the clock master.** Raw video and PCM have fixed byte sizes per frame, so the gate needs no demuxer and no timestamp parsing. Frames are counted, and the encoder sees a perfectly regular input with no gaps. Sync between audio and video holds by construction.
|
||||
- **Cost.** About 93 MB/s of raw 1080p30 through one more pipe hop, plus a brief doubled compositor CPU during each swap. The doubled CPU is admission-accounted as a transient burst inside the worker's 20 % failover reserve.
|
||||
- **Spike gate (first task of S2).** A gate written in Node with preallocated ring buffers must sustain 1080p60 at under 5 % of a core with no GC pauses above one frame interval. If it fails, the gate becomes a small static Rust binary behind the same pipe contract. The WP9 GStreamer decision gate ([wb] WP9) is folded into this spike: if A/B swap with the gate is not frame-continuous across 1 000 swaps, escalate to GStreamer `input-selector`.
|
||||
|
||||
flash's argument for a single process was simplicity and keeping "WP11's continuity risk class" in one place. The gate keeps that risk in one place, the gate, while making continuity a property of the design instead of a hope about rebuild speed.
|
||||
|
||||
---
|
||||
|
||||
## 2. Multiview + audio spec
|
||||
|
||||
### 2.1 What a tile plays [both]
|
||||
|
||||
Tiles never touch remote HLS. Each active ingest emits one **shared preview tap** ([prop] §4.4):
|
||||
- a JPEG thumbnail, every 2 s by default (configurable 1–10 s);
|
||||
- an on-demand **360p / ≤500 kbps LL-HLS** rendition;
|
||||
- **[SYN] an on-demand audio-only rendition** (AAC 64 kbps LL-HLS). Thumbnails carry no audio, so without it a 4x4 operator could not hear a thumbnail tile.
|
||||
|
||||
These are served from the authenticated preview endpoint with `ETag`, single-flight fill, and an LRU of 6 segments or fewer per source. The endpoint is **same-origin with correct CORS**. This is load-bearing: a cross-origin `MediaElementAudioSourceNode` outputs silence per the Web Audio spec [flash] Q2 and [glm] Q2.
|
||||
|
||||
The **program tap** is the same mechanism applied to the gate output (one extra tap per channel). The PGM tile always shows what the encoder actually receives.
|
||||
|
||||
### 2.2 Grid tiers and the decode budget
|
||||
|
||||
No browser documents a hard limit on concurrent decoders; both reports flag this as **not verified**. The binding constraints are pixel throughput, per-tab MSE memory and mobile hardware-decoder sessions [glm] Q1. Budget: **live decodes ≤ 4 on desktop, ≤ 1 on mobile** [SYN]. This is flash's "never more than 4" plus glm's mobile "thumbnails + one live tile".
|
||||
|
||||
| Layout | Desktop tiles | Live (LL-HLS 360p) tiles | Mobile |
|
||||
|---|---|---|---|
|
||||
| 2x1 | 2 | both live | PGM live, other thumbnail |
|
||||
| 2x2 | 4 | all 4 live | PGM live, others thumbnail |
|
||||
| 3x3 | 9 | **PGM + focused** live; the other 7 are thumbnails | 1 live (PGM or focused) |
|
||||
| 4x4 | 16 | **PGM + focused** live; the other 14 are thumbnails | 1 live (PGM or focused) |
|
||||
|
||||
Additional rules:
|
||||
- **"Upgrade more tiles"** is an explicit user action, capped at 4 total, with a capability warning [glm].
|
||||
- **Thumbnail cadence**: 2 s by default and 1 s for the PST-selected tile. Off-screen tiles stop polling (`IntersectionObserver` + `visibilitychange`) [prop].
|
||||
- **hls.js config per tile** [glm] Q1:
|
||||
- `lowLatencyMode: true`;
|
||||
- `maxBufferLength: 4`, `maxMaxBufferLength: 8`, `backBufferLength: 4` (the default `Infinity` is wrong for a multiview);
|
||||
- `liveSyncDurationCount: 2`.
|
||||
- **Player selection** [flash] Q1: if `Hls.isSupported()`, use MSE. Otherwise, if `ManagedMediaSource` exists and `canPlayType(m3u8)`, use native HLS (Safari/iOS 17.1+). Otherwise the client is thumbnail-only.
|
||||
- **Autoplay**: every tile boots `muted` + `playsinline`. One "Arm audio" gesture unlocks the shared `AudioContext` [flash] Q1.
|
||||
- **Tile chrome**: monitor badge (`LIVE/UNSTABLE/OFFLINE/…` from the live-monitor WS fan-out), tally border (red = PGM, green = PST), and a tile-state label ("thumbnail · 2 s", "live preview").
|
||||
- **Tally reflects only state stamped with `observed_version`.** It is never optimistic [glm] Q6, Risk 4.
|
||||
- **PST rendering** [SYN] (adopting [glm] Q3's browser *preview* compositor):
|
||||
- The PST monitor is rendered **client-side** onto a canvas, from the preview taps of the slots in the PST scene plus the scene's overlay SVGs. These are the same SVGs the server rasterizes (§4), so the look matches.
|
||||
- The worker does **not** composite a second (preview) program. That would double compositor CPU for every channel.
|
||||
- Caveat shown in the UI: "PST is a look-ahead; PGM is the truth".
|
||||
|
||||
### 2.3 Audio: two mixes, two owners [both]
|
||||
|
||||
**Monitor mix** (what the operator hears; browser):
|
||||
- **Graph**: one `AudioContext` per page. Each audible tile's media element → `MediaElementAudioSourceNode` → its own `GainNode` → master `GainNode` → destination [glm] Q2.
|
||||
- **Modes**:
|
||||
- *Follow program* (default): plays the program tap's audio, which is exactly what viewers hear, delayed by roughly the tap latency.
|
||||
- *Solo source X*: plays that source's audio rendition.
|
||||
- *Custom monitor mix*: up to 4 sources with independent faders. These are continuous UI faders, because the mix is local.
|
||||
- **Bounds**: never more than 4 audio renditions plus the program tap. Thumbnails gain audio by attaching the audio-only rendition (§2.1).
|
||||
- **Privacy**: per operator and never sent to the server, so no command traffic.
|
||||
- **Mobile**: the monitor collapses to "the one tile you are watching" [glm].
|
||||
|
||||
**Program mix** (what viewers hear; compositor):
|
||||
- **Per-slot audio intent**: `scene_slots.audio_mode ∈ off|on|primary`, with at most one `primary` per scene (partial unique index, [prop] §4.4). The default is primary-only, meaning one feed's audio. "Mix N feeds" means several `on` slots.
|
||||
- **Graph**: `volume` per input → `amix inputs=N duration=longest dropout_transition=2 normalize=1` [glm] Q2. A dead source fades out instead of cutting.
|
||||
- **Gain** is desired state, in **3 dB steps from −24 to 0 dB**. No continuous program fader, because continuous faders would turn the signed bus into a UI-rate control channel [glm] Q2.
|
||||
- **Applying a change**: a program gain change bumps `desiredVersion` and triggers an A/B compositor swap (§1.4). It is inaudible because the gate splices at a frame boundary.
|
||||
- **Deferred optimization**: `amix`/`volume` runtime commands via `azmq` on an `ipc://` socket in job scratch [flash] Q2. **Not verified**: whether our pinned FFmpeg has libzmq, and whether it is compatible with the seccomp profile.
|
||||
|
||||
---
|
||||
|
||||
## 3. Chat unification contract (Opticast ↔ Tactical Media Broadcast Hub)
|
||||
|
||||
### 3.1 Roles [as] §2.1
|
||||
|
||||
**Tactical Media Broadcast Hub is the chat aggregation authority.** Connectors ingest server-side into a normalized stream. A new `chat-hub` module persists, dedupes, sequences and fans out. Opticast, the web Control Room and the Tactical Media Broadcast Hub mobile web shell are all **thin authenticated realtime clients**. The phone never holds Twitch/YouTube/PeerTube credentials.
|
||||
|
||||
The privacy pledge in Opticast's README ("no network access beyond the configured server") remains true. The integration is opt-in and off by default, and the only endpoint is a user-configured, self-hosted Tactical Media Broadcast Hub.
|
||||
|
||||
### 3.2 Ingestion (platform → `LiveChatMessage`) [as] §2.3
|
||||
|
||||
| Platform | Read | Reply | Moderate | Mechanism / governance |
|
||||
|---|---|---|---|---|
|
||||
| Twitch | ✅ | ✅ | delete, timeout, ban | EventSub WS `channel.chat.message` plus delete/clear events. Helix `POST /chat/messages` for replies; Helix moderation and ban endpoints. **No replay after a WS drop.** The hub ring buffer covers clients; the platform gap is accepted and marked with a `system` gap message. Token bucket of 100 per 30 s (elevated). |
|
||||
| YouTube | ✅ | ✅ | delete, ban | `liveChatMessages.list` long-poll honouring `pollingIntervalMillis`, **one shared poller per broadcast**. Quota ledger: list = 1 unit, insert/delete/ban = 50, 10 000 per day by default. Quota exhaustion is a first-class `paused:quota` state. |
|
||||
| PeerTube + livechat plugin, C2S on | ✅ | ✅ | MUC moderation | Server-side XMPP client joins as the streamer with the livechat long-term token, which is **"not officially supported"** per the plugin docs. |
|
||||
| PeerTube, embed only | read-only | ❌ | ❌ | Read-only embed pane, web only. |
|
||||
| PeerTube, no plugin | comments | comment reply | ✅ | Existing `getComments` on the connector, labelled "comments". |
|
||||
| Mastodon | mentions | status reply | delete own | Streaming `/api/v1/streaming` (user), `source: "social-reply"`. |
|
||||
| Mock livestream | ✅ | ✅ | ✅ | Contract and E2E harness. |
|
||||
|
||||
**Capability probing.** Each connection carries a probed capability row, with `pt-chat: xmpp-c2s | readonly-embed | comments-only | none` for PeerTube.
|
||||
|
||||
**Storage.** `live_chat_messages` is partitioned monthly per the [wb] retention row. `live_chat_seen_events` uses `ON CONFLICT DO NOTHING`, mirroring `monitor_seen_events`. Dedup key: `(workspace_id, provider, provider_message_id)`. **Ordering**: a per-live monotonic `seq` assigned at hub ingress. Provider timestamps are display-only, because clocks skew [as] §2.4.
|
||||
|
||||
**Local display filters.** Hide `!commands`, hide bots, mute user, word filter. These run once, hub-side, so web and mobile inherit them [as] §2.8.
|
||||
|
||||
### 3.3 Transport decision [SYN] (from [as] §2.5 + [dm] §7)
|
||||
|
||||
| Option | Verdict |
|
||||
|---|---|
|
||||
| **WebSocket** (subscribe/ack + resume) | **Primary.** Chat volume is far beyond review-chat volume. OkHttp WS on Android can authenticate in the first frame. The existing `authorizeSubscription` port is the gate. |
|
||||
| SSE | Rejected. `EventSource` cannot send our Bearer header ([dm] §7 rationale), and it offers no upstream channel for acks. |
|
||||
| REST cursor polling | **Fallback** for constrained clients and for resync after `RESYNC_REQUIRED`. It uses the same `after=seq` semantics as [dm] `review_events`, so one cursor discipline applies across the codebase. |
|
||||
| FCM push | **Rejected** for v1, because it breaks Opticast's privacy pledge. A later opt-in UnifiedPush or self-hosted web-push is possible. |
|
||||
|
||||
**WS lifetime on the phone.** The socket stays open only while the chat sheet is open, or while `StreamingService`, the existing camera/mic foreground service, is live. There is no background socket when idle. Backoff from 1 s to 8 s mirrors `ReconnectController` [as] §2.5, Risk 4.
|
||||
|
||||
**Fan-out.** The hub publishes to Redis `chat:{workspaceId}:{liveId}`. The gateway subscribes only to authorized and viewed topics, using WP11 fan-out rules: bounded per-socket queues, `RESYNC_REQUIRED` on overflow. **Prerequisite:** the API-side `RespControlRedis` has PUBLISH but not SUBSCRIBE, so a subscribe-capable client must land first [as] §2.2.
|
||||
|
||||
### 3.4 Device pairing [SYN]
|
||||
|
||||
Opticast authenticates with a **device token**, not a user session cookie.
|
||||
1. The web UI calls `POST /devices/pairings` and shows a QR code containing `{serverUrl, pairingCode}` (TTL 5 min, single use).
|
||||
2. The app redeems the code with `POST /devices/pairings/redeem` and receives a device token. The token is bound to `(workspaceId, userId)`, scoped to `livechat.view|send|moderate` + `stream.view`, epoch-gated, and revocable from the web UI.
|
||||
3. The app stores the token in `EncryptedSecretStore` [as] §4 Phase 2.
|
||||
|
||||
### 3.5 Endpoint shapes (both apps implement)
|
||||
|
||||
Routes follow repo convention: no `/workspaces/{wid}` prefix, because the workspace comes from the session or device token; compare `@Controller("publishing/drafts")`. [as] Phase 0 shapes are adapted accordingly.
|
||||
|
||||
```
|
||||
REST (Tactical Media Broadcast Hub API; RbacGuard + RLS; Idempotency-Key on POSTs)
|
||||
GET /live-chat/lives → { items: LiveSummary[] } # stream.view
|
||||
GET /live-chat/lives/{liveId}/messages?after=<seq>&limit=<1..500, default 100>&platform=<p>
|
||||
→ { items: ChatMessage[], nextCursor?: number } # livechat.view
|
||||
POST /live-chat/lives/{liveId}/messages
|
||||
body { text (1..500 chars), targets: "all" | connectionId[], replyTo?: messageId }
|
||||
→ 202 { message: ChatMessage, delivery: Delivery[] } # livechat.send
|
||||
POST /live-chat/lives/{liveId}/messages/{messageId}/moderation
|
||||
body { action: "delete"|"timeout"|"ban", durationSeconds?: 1..1209600 }
|
||||
→ 202 { delivery: Delivery[] } (audited via audit_append) # livechat.moderate
|
||||
GET /live-chat/lives/{liveId}/capabilities
|
||||
→ [{ connectionId, provider, read, reply, moderate, state: "ok"|"paused:quota"|"paused:rate"|"degraded", reason? }]
|
||||
POST /devices/pairings → { pairingCode, qrPayload, expiresAt } # session user
|
||||
POST /devices/pairings/redeem body { pairingCode, deviceName } → { deviceToken, workspaceId }
|
||||
DELETE /devices/{deviceId} (revoke; bumps the device's epoch)
|
||||
|
||||
Delivery = { connectionId, provider, status: "queued"|"sent"|"failed"|"unsupported", error?: string }
|
||||
|
||||
WS /realtime (one socket, multiplexed topics; also used by the Control Room)
|
||||
C→S { op:"auth", token } # first frame; never in the URL
|
||||
S→C { op:"ready" } | { op:"close", reason }
|
||||
C→S { op:"subscribe", id, topic:"livechat", liveId, lastSeq? }
|
||||
C→S { op:"subscribe", id, topic:"channel", channelId } # Control Room: buses/monitor/observed
|
||||
S→C { op:"ack", id, fromSeq } | { op:"nack", id, reason } # reason ∈ authorizeSubscription reasons
|
||||
S→C { op:"msg", topic, seq, type, payload }
|
||||
type ∈ chat.message | chat.moderated | chat.delivery | chat.gap | stream.state | capability
|
||||
S→C { op:"resync", topic } # client re-fetches via REST ?after=
|
||||
C→S { op:"unsubscribe", id } | { op:"ping" } S→C { op:"pong" }
|
||||
|
||||
ChatMessage = { seq, id, liveStreamId, connectionId, provider, providerMessageId,
|
||||
author: { handle, displayName?, avatarUrl?, badges?: string[], color },
|
||||
text, sentAt, receivedAt, isModerated, replyTo?, source: "live-chat"|"social-reply"|"system" }
|
||||
```
|
||||
|
||||
Contract rules:
|
||||
- **Resume.** `lastSeq` replays from a ring buffer of the last 500 messages per live, held in Redis. If `lastSeq` is older than the ring, the server sends `resync` and the client pages via REST.
|
||||
- **Epoch.** An authz-epoch bump (role change, device revoke) closes every socket of that principal within 5 s, matching the WP11 revocation SLO.
|
||||
- **Reply semantics.** The composer target defaults to "All capable". An incapable target is refused client-side with its reason. Delivery ticks per target reconcile from `chat.delivery` events, following Restream's pattern [as] §2.7.
|
||||
- **Identity.** No cross-platform identity fusion. The author colour is a hash of `provider:handle` [as] §2.6.
|
||||
- **Licensing.** Opticast is GPLv3 and the Tactical Media Broadcast Hub licence is **not verified**. The relationship stays API-only, with no shared code [as] Risk 5.
|
||||
|
||||
### 3.6 Opticast UX (for the client slice) [as] §3
|
||||
|
||||
- Chat is a bottom sheet on `LiveScreen` with snap points at 40/70/100 %. It is not a third route.
|
||||
- One unified list with platform badges, filter chips `All · Twitch · YouTube · PeerTube · Mentions`, and swipe moderation (left = delete, right = ban; long-press for timeout).
|
||||
- A "↓ N new" pill appears when scrolled up.
|
||||
- A `ChatViewModel` sits beside `StreamViewModel`. Streaming code is untouched.
|
||||
- **Opticast as a camera source** for the Control Room needs a push ingest kind (`rtmp-push`/`srt-listen`) that Tactical Media Broadcast Hub lacks today (`source.kind` is `"hls"` only). It is **out of scope for this plan** and listed as a follow-on in §6.4.
|
||||
|
||||
---
|
||||
|
||||
## 4. Overlay system
|
||||
|
||||
### 4.1 The disagreement
|
||||
|
||||
- **[glm] `drawtext` + `textfile` + `reload=1`.** Text edits land without a graph restart, and scrolling uses `x = main_w − mod(t·speed, main_w + text_w)`. [prop] §4.4 also mentions "overlay text → written to a `textfile`".
|
||||
- **[flash] pre-rendered assets.** SVG template → PNG with alpha (via resvg/sharp) → `overlay`. The chyron is one wide strip scrolled by an `overlay` x-expression. FFmpeg never sees text.
|
||||
|
||||
### 4.2 Decision — pre-rendered assets [flash], with glm's no-restart property recovered by the A/B swap [SYN]
|
||||
|
||||
Why pre-rendered assets win:
|
||||
1. **Same pixels in PST and PGM.** The browser PST look-ahead (§2.2) renders the *same SVG* the server rasterizes. With `drawtext`, the browser would have to imitate FreeType layout and would never match.
|
||||
2. **Brand-quality typography.** The geometric design language (#FF1610, solid disk mark, rounded/angled boxes) is trivial in SVG and painful in `drawtext` box parameters.
|
||||
3. **Smaller attack surface inside the sandbox.** `drawtext` needs freetype, harfbuzz, fontconfig and fribidi in the pinned FFmpeg, plus font files mounted into every compositor. [glm] Risk 3 itself names "font bombs". Rasterizing once in an API-side sandboxed job keeps fonts and SVG parsing out of the 24/7 process.
|
||||
4. **Cost and determinism.** A static PNG overlay is cheaper than per-frame glyph rendering. [glm] flags `drawtext` scroll CPU at 1080p60 as a risk, and its own fallback is "render the band into a strip", which is flash's design. PNGs are also golden-frame deterministic.
|
||||
5. **glm's advantage is no longer needed.** glm's argument for `drawtext` (an edit without a graph restart) matters only if restarts are visible. With the frame gate (§1.4), a graph swap is frame-continuous, so an overlay edit is just another `desiredVersion`.
|
||||
|
||||
`drawtext` survives in one narrow role: a **server-generated clock/countdown** using `expansion=normal` with `localtime`. The format string is a typed enum, never user text.
|
||||
|
||||
### 4.3 Pipeline
|
||||
|
||||
```
|
||||
Operator edits a lower third (web/mobile)
|
||||
│ POST /production/overlays/{overlayId} { templateId, fields: {title ≤ 80, subtitle ≤ 120}, style enum }
|
||||
▼
|
||||
API validation: length, Unicode category allowlist, no control characters → overlay_revisions row
|
||||
▼
|
||||
render job (sandboxed WP4 probe class): SVG template (server-owned, typed field slots, bundled fonts)
|
||||
→ resvg (no scripts, no external fetch) → PNG with alpha at output resolution
|
||||
→ StorageService (sha256 key, dedup) → asset id
|
||||
▼
|
||||
1 tx: scene overlay slot → new assetId + desired_version++ + outbox row (same path as a TAKE)
|
||||
▼
|
||||
signed snapshot → worker → compositor B built with the new overlay input → gate swap at a frame boundary
|
||||
```
|
||||
|
||||
Overlay kinds:
|
||||
- **Lower third**: PNG composited with `overlay=x:y`. There is no timed `enable=` expression. Toggling is a slot flag in desired state, so the next graph simply includes or omits it.
|
||||
- **Chyron**: render the *whole ticker strip* (all items with separators) as one wide PNG. The height of the band is fixed. Width is capped at **16 384 px** [SYN]; longer text is split into a carousel of strips. Scroll with `overlay=x='W-mod(t*S\,w+W)':y=Y`, where speed S is a typed integer of 40–400 px/s.
|
||||
- A chyron text edit renders a new strip and swaps graphs. The scroll position restarts, and that is accepted.
|
||||
- **Logo from an SVG input**: a tenant upload goes through the WP4 sandboxed probe and is rasterized once per asset version at 1×/2× sizes. Position and size are typed values (corner enum + margin, or explicit numbers).
|
||||
|
||||
Security gates:
|
||||
- **Injection.** Overlay text never reaches argv or a graph, because the graph references `assetId` only. The injection tests from [wb] WP12 apply to the render job too: SVG text nodes are XML-escaped, and the template engine is typed field-slot substitution, not string concatenation.
|
||||
- **Worker storage access.** The worker fetches assets through presigned GETs scoped to its assignment ([prop] §4.5 object-storage tenant ACLs).
|
||||
|
||||
**Not verified:**
|
||||
- resvg and sharp APIs. flash fetched the resvg README; glm did not verify either. **Pick resvg-js.**
|
||||
- Whether the image2 demuxer re-reads a file on `-loop 1`. flash and glm do not discuss it. It is **no longer needed**, because edits go through the swap.
|
||||
|
||||
---
|
||||
|
||||
## 5. Playlist fallback + Auto Director
|
||||
|
||||
### 5.1 Fallback venue [both]
|
||||
|
||||
**Decision: worker-side fallback.** The PeerTube live is `permanentLive: true`, so the channel URL stays live forever. "Nothing is live" means the director puts the **FALLBACK scene** on program.
|
||||
- No live/VOD flapping, no RTMP reconnect, and no change of PeerTube latency mode. Overlays and program audio stay consistent [flash] Q5, [glm] Q5.
|
||||
- There is no PeerTube API to play a playlist into a live [flash] Q5 (verified absence). Switching a channel between live and VOD relies on unverified viewer-UI behaviour [glm] Q5.
|
||||
- `permanentLive` and `saveReplay` are mutually exclusive (verified 400 error). Fallback channels therefore run **without replay**, and the channel config UI says so [flash] Risk 5.
|
||||
|
||||
**Deferred.** [glm]'s **Tier-2 mode** ("live windows + PeerTube playlist page", where the worker stops pushing when dark) is a cost-saving option for channels that do not need continuity. It is **not in this plan**. Revisit it after we have measured the 24/7 encoder cost.
|
||||
|
||||
**Mechanics** [SYN] (fits the WP9 sandbox: FFmpeg reads only pipes):
|
||||
- **A playlist is a source kind** (`kind: "playlist"`). A worker-side **asset feeder** walks the ordered asset list:
|
||||
- assets are presigned GETs through the ingest egress relay;
|
||||
- they are piped one after another into a normal ingest process (`-re`, normalize stage);
|
||||
- the compositor sees a continuous input like any other source.
|
||||
- Asset boundaries are hidden by the ingest normalize stage plus the gate (PTS regenerated). Concat-protocol tricks are not used; they are banned by the WP9 protocol rules.
|
||||
- **Assets are pre-normalized at upload** to the channel profile by the existing WP4 video pipeline. Loops never transcode. Pricing therefore follows the [glm] "normalize-on-ingest, not in the loop" note.
|
||||
- **Programming source of truth.** The FALLBACK scene's playlist is either a local asset list or **mirrored from a PeerTube channel playlist** through the connector (`GET /video-playlists/{id}/videos`), downloaded to S3 and normalized [flash] Q5.
|
||||
- **Slate floor.** A static slate (logo + "back soon" + optional clock) is generated by the overlay renderer. It is always available, independent of asset availability. When the playlist ingest is unhealthy (asset missing, decode error), the scene degrades to the slate automatically.
|
||||
|
||||
### 5.2 Auto Director rules [both]
|
||||
|
||||
The evaluation context and plumbing already exist and are not redesigned here:
|
||||
- **Monitor states**: the reducer emits `OFFLINE|PROBING|LIVE|UNSTABLE|ERROR`, with STARTING/ENDING/UNKNOWN added in the DB. Hysteresis: `liveConfirmations=2`, `offlineConfirmations=3`, `noMediaMs=10 s`, `minHoldMs=2 s`, `cooldownMs=10 s` (`apps/api/src/live-monitor/reducer.ts`).
|
||||
- **Evaluation plumbing**: WP10's 250 ms debounce, one evaluation in flight per channel, and `continue-as-new`.
|
||||
|
||||
Rule ladder, evaluated top-down. The first match writes program.
|
||||
|
||||
1. **Program lock / operator override** (four WP10 release modes). Automation never undoes a human TAKE while an override or lock holds [both].
|
||||
2. **Media-health failover**: if the program source's media-health tier reports failure, the debounce is bypassed. Fail over to the next eligible source, otherwise to FALLBACK. This works during an API outage (WP10).
|
||||
3. **Scheduled windows**: WP13 schedules can force a scene inside a window.
|
||||
4. **Priority list**: the highest-priority source in `LIVE` wins. A source already on program that turns `UNSTABLE` keeps program until `noMediaMs` plus confirmations expire.
|
||||
5. **Terminal**: the FALLBACK scene (playlist, else slate).
|
||||
|
||||
Anti-flap at the director level [glm] Q7:
|
||||
- **Dwell time by source class.** Defaults: a live source holds at least 15 s; FALLBACK holds at least 60 s before a newly LIVE source may preempt it, unless that source's priority is "interrupt". This complements, and does not duplicate, monitor hysteresis.
|
||||
- **Cooldown** of 10 s per source after it leaves program before it may return automatically.
|
||||
- Out of scope: "real direction" such as audio-activity speaker switching or content-aware cuts. Future work adds it as new *conditions* in the rule `condition` jsonb, not as a new engine [both].
|
||||
|
||||
**Monitor-state integration in the Control Room.** The UI renders:
|
||||
- AUTO/MANUAL/LOCKED status;
|
||||
- for each rule, a "**would fire now**" indicator computed from the same monitor snapshot, so operators see *why* the director acted [flash] Q7;
|
||||
- a dwell countdown on the program tile;
|
||||
- **RETURN TO AUTO**, which re-arms the ladder.
|
||||
|
||||
Every automated action uses the same bus transaction as a human TAKE: one `desired_version` bump and one outbox row. Automation has no special-cased path [glm] Q7.
|
||||
|
||||
### 5.3 PGM/PST control set (web now, same commands on mobile later) [both]
|
||||
|
||||
The v1 control set:
|
||||
- **PST select** (tile or scene);
|
||||
- **CUT** (TAKE, `transition: {type:"cut"}`);
|
||||
- **overlay on/off** for each overlay slot (armed on PST, live after TAKE);
|
||||
- **program audio** primary pick and gain steps;
|
||||
- **monitor audio** (local only);
|
||||
- **LOCK**;
|
||||
- **RETURN TO AUTO**.
|
||||
|
||||
Layout picker, position swap and PIP all operate on the **PST scene** and reach program only through TAKE [flash] Q6.
|
||||
|
||||
FADE (`{type:"fade", ms: 300|500|1000|2000}`) is added in S4 behind the continuity gate (D8). If a fade graph fails to start within 1 s, the worker degrades to CUT and reports `TRANSITION_DEGRADED` [flash] Risk 2.
|
||||
|
||||
Every command carries an `Idempotency-Key` and is checked against `director.*`. The target is under 300 ms from command to observed (WP11).
|
||||
|
||||
Mobile "go live from preview" is the same PST select followed by TAKE. It needs no new server surface.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phased build plan
|
||||
|
||||
Six dispatchable slices. Migration numbers are **pre-assigned** so parallel slices do not collide. `011` stays reserved for the review workflow ([dm] §4), and `012` exists.
|
||||
|
||||
Default gate for every slice: `pnpm typecheck && pnpm test` green from the repo root (turbo), plus the slice-specific gates below. Each slice writes `runs/<wp>/<slice>-report.md`.
|
||||
|
||||
```
|
||||
┌── S2 worker compositor ──┐
|
||||
S1 ──────┤ ├── S4 fallback + director + FADE
|
||||
└── S3 control room ───────┘
|
||||
S5 chat hub + contract (independent; starts day 1) ── S5b Opticast client (anarchostreamer repo, after S5 contract freeze)
|
||||
S6 telegram WP15 middle (independent; starts day 1)
|
||||
```
|
||||
|
||||
**Parallelism:** S1, S5 and S6 can start together. S2 and S3 run in parallel once S1 lands. S3 develops against a fake worker that echoes realized versions. S4 needs both S2 and S3.
|
||||
|
||||
### S1 — Desired-state v2 + production schema (small; blocks S2/S3)
|
||||
|
||||
- **Scope:**
|
||||
- Extend `DesiredChannel`/`DesiredSpec` from a single `source:{kind:"hls"}` to `{ scenes, buses, sources[], overlays[], transition, audio }`. Old v1 snapshots must still validate.
|
||||
- Keep the strict key allowlist and secret rejection.
|
||||
- Add migration `013_production.sql`: `production_scenes`, `scene_slots` (layout enum, geometry CHECKs, `audio_mode` with the one-primary partial unique index, `gain_db` in −24…0 in steps of 3), `production_buses` (preview|program, unique per channel), `overlay_slots`/`overlay_revisions` (by asset id), and `source.kind` gaining `playlist`.
|
||||
- All tables use channel-scoped composite FKs for the same-channel invariant, plus RLS.
|
||||
- **Files:** `apps/api/src/media-control/spec.ts`, `wire.ts`, `apps/media-worker/src/types.ts`, `apps/api/migrations/013_production.sql`, plus tests.
|
||||
- **Gates:**
|
||||
- Old snapshots round-trip.
|
||||
- The [wb] WP12 same-channel invariant tests fail at the DB constraint.
|
||||
- Injection payloads in labels (`;[,'`, `movie=`) are rejected by the type layer.
|
||||
- The RLS denial suite covers the new tables.
|
||||
|
||||
### S2 — Worker compositor + frame gate + taps (XL, highest risk; after S1, ∥ S3)
|
||||
|
||||
- **Scope:**
|
||||
- The **frame-gate spike** comes first (§1.4 gate: Node vs Rust; 1 000-swap continuity).
|
||||
- `MediaCommandBuilder` graph builders emit `xstack` (2x1/2x2/3x3/4x4 plus custom), PIP `overlay`, overlay asset inputs, chyron x-expression, and `volume`+`amix`, all from typed params only.
|
||||
- Ingest×N normalize stage; compositor A/B lifecycle in the supervisor; persistent encoder fed by the gate; playlist feeder ingest.
|
||||
- Preview taps: JPEG, 360p LL-HLS, audio-only, and the program tap.
|
||||
- The supervisor/bus/fence stay **untouched** apart from the meaning of a RUNNING snapshot [glm] Phase 1.
|
||||
- **Files:** `apps/media-worker/src/compositor/*` (new), `src/gate/*` (new), `src/relay/ffmpeg.ts` (generalized), `src/supervisor/index.ts` (A/B lifecycle), `src/preview/*` (new).
|
||||
- **Gates:**
|
||||
- Golden-frame hashes for each layout.
|
||||
- **Zero gap at the encoder input** across 1 000 scripted scene, overlay and gain swaps (frame counter plus PCM continuity check).
|
||||
- A 4-input 1080p30 CPU budget test on the reference host.
|
||||
- `kill -9` of the compositor mid-swap leaves the encoder and RTMP session up.
|
||||
- The ≥8 h soak is wired into the nightly run (not blocking merge).
|
||||
- Existing WP9 sandbox, bus and secret-leak suites stay green.
|
||||
|
||||
### S3 — Control plane + Control Room UI + overlay pipeline (XL; after S1, ∥ S2)
|
||||
|
||||
- **Scope:**
|
||||
- **Bus writes**: TAKE/CUT/PST-select/lock/override in `media-control.service.ts`. One tx = bus write + `desired_version++` + outbox, with `Idempotency-Key` and `director.*`.
|
||||
- **Preview endpoint**: ETag, single-flight fill, same-origin, signed short TTL.
|
||||
- **Realtime gateway**:
|
||||
- wire `/realtime` for real, including the **subscribe-capable Redis client**, which is shared with S5;
|
||||
- **S3 owns the gateway** and S5 builds on it;
|
||||
- if S5 starts first, S5 lands the gateway and S3 rebases.
|
||||
- **Web**: multiview tiers per §2.2, Web Audio monitor per §2.3, PST canvas look-ahead, tally from observed state only, overlay editor.
|
||||
- **Overlay pipeline**: SVG templates → resvg-js render job → StorageService → asset id → desired-state bump.
|
||||
- **Files:** `apps/api/src/media-control/*`, `apps/api/src/realtime/*`, `apps/api/src/production/*` (new: scenes, overlays, preview), `apps/web/src/app/control-room/*`, `apps/web/src/components/control-room/*`.
|
||||
- **Gates:**
|
||||
- Idempotency matrix: duplicate, concurrent duplicate, crash mid-tx.
|
||||
- Command → observed under 300 ms against the fake worker.
|
||||
- Playwright E2E: PST → CUT → tally flips only on observed state.
|
||||
- Grid tier caps enforced: a 5th live tile is refused.
|
||||
- Overlay injection tests through the render job.
|
||||
- Revocation evicts the operator within 5 s.
|
||||
|
||||
### S4 — Fallback scene + Auto Director v1 + FADE (L; after S2 + S3)
|
||||
|
||||
- **Scope:**
|
||||
- FALLBACK scene: playlist source mirrored from a PeerTube channel playlist via `packages/connector-peertube`, plus the slate floor.
|
||||
- `permanentLive` wiring and the replay conflict surfaced in channel config.
|
||||
- Rule ladder per §5.2 with dwell-by-class and cooldown, consuming live-monitor outbox events (WP10 intake rules).
|
||||
- "Would fire now" indicators and RETURN TO AUTO.
|
||||
- FADE via the two-swap transition graph, degrading to CUT.
|
||||
- **Files:** `apps/api/src/director/*` (new), `apps/scheduler/*` (Temporal evaluator workflow), `packages/connector-peertube/src/*` (playlist mirror), `apps/media-worker/src/compositor/transition.ts`, and the Control Room AUTO panel.
|
||||
- **Gates:**
|
||||
- [wb] WP10 matrix: 0/1/2/3+ live, high-priority interrupt, TAKE not undone, lock stability, flap damping, failover without output kill, API-outage scenario.
|
||||
- Event-storm test (50 sources at 10 Hz → ≤4 evaluations/s).
|
||||
- Playlist asset removed mid-loop → slate within 2 s with no output gap.
|
||||
- FADE continuity holds across 200 fades.
|
||||
|
||||
### S5 — Chat hub + unified contract + web chat panel (L; independent)
|
||||
|
||||
- **Scope:**
|
||||
- **Phase 0 contract** (§3.5) frozen in `packages/core` as types and JSON schema.
|
||||
- Migration `014_live_chat.sql`: `live_chat_messages` partitioned monthly, `live_chat_seen_events`, `devices`, `device_pairings`.
|
||||
- `apps/api/src/live-chat/` (new):
|
||||
- Twitch EventSub WS and YouTube shared-poller ingestion;
|
||||
- seq assignment and ring buffer;
|
||||
- send governors (token buckets, YT quota ledger);
|
||||
- reply fan-out and moderation fan-out with per-target `Delivery`;
|
||||
- display filters.
|
||||
- Twitch connector gains Helix send/moderate.
|
||||
- Device pairing.
|
||||
- Web chat panel in the Control Room.
|
||||
- PeerTube XMPP adapter = **S5 phase 2**. Ship capability-probe + read-only-embed + comments first.
|
||||
- **Files:** `apps/api/src/live-chat/*`, `apps/api/src/devices/*`, `apps/api/src/realtime/*` (shared with S3, see S3), `packages/connector-twitch/src/index.ts`, `packages/connector-mock-livestream/*`, `apps/web/src/components/live-chat/*`, `apps/api/migrations/014_live_chat.sql`.
|
||||
- **Gates:**
|
||||
- Mock-livestream E2E: ingest → hub → Redis → WS → client, including `lastSeq` resume and the `resync` path.
|
||||
- Dedupe: the same provider id twice gives one row.
|
||||
- Authz-epoch bump closes sockets within 5 s.
|
||||
- The YT quota ledger refuses a send below budget and surfaces `paused:quota`.
|
||||
- Delivery `unsupported` per target never fails the batch.
|
||||
- Retention sweep test per the [wb] cross-cutting row.
|
||||
- **S5b — Opticast client** (separate repo `anarchostreamer`; dispatch after the S5 contract freeze):
|
||||
- OkHttp (its only new network dependency), a pairing settings screen, `ChatViewModel` + bottom sheet, WS lifetime tied to `StreamingService`, the composer with target pill, and swipe moderation.
|
||||
- Gate: `testDebugUnitTest assembleDebug` green, plus a contract test against a recorded S5 WS transcript.
|
||||
|
||||
### S6 — WP15 Telegram middle (M; independent)
|
||||
|
||||
- **Scope** [tg] §5 Phase 1:
|
||||
- Migration `015_archive.sql`: `feed_subscriptions` (`{type:"telegram_channel", connectionId, channelId}`) and `archived_feed_items` (RLS; dedup on `(workspace_id, remote_ref=chat_id:message_id)` and content hash).
|
||||
- Channel-post normalizer: `media_group_id` album debounce of 2 s, `edited_channel_post` → revision, `forward_origin` and `author_signature` kept as attribution.
|
||||
- `my_chat_member` → subscription health ("bot removed").
|
||||
- Media harvest via `getFile` into StorageService. Files over 20 MB are recorded as "Bot API cannot fetch", not failed.
|
||||
- "Create draft from archived item" → the existing review states in `publishing.service.ts`.
|
||||
- Approval calls the existing `amplify()`. Protected content shows "amplify unavailable".
|
||||
- Rights note in the UI, and attribution on by default.
|
||||
- No MTProto.
|
||||
- **Dependency note.** The draft binding targets today's `editing → in_review → approved` states. When the review workflow (`011`, [dm] WP5.1–5.2) lands, the archived-item draft enters the review cycle unchanged, because it is just a draft. Amplifying others' content must **never auto-publish** [tg] §2.3.
|
||||
- **Files:** `apps/api/src/archive/*` (new), `packages/connector-telegram/src/index.ts` (normalizer hook only), `apps/api/src/publishing/*` (draft-from-item), `apps/web/src/app/archive/*`, `apps/api/migrations/015_archive.sql`.
|
||||
- **Gates:**
|
||||
- Webhook fixture → one archived row for a 3-photo album.
|
||||
- The same post via two subscriptions → one row.
|
||||
- The same image → one blob.
|
||||
- Tenant B cannot read tenant A's rows despite a shared blob.
|
||||
- A protected-content forward surfaces `ConnectorError`.
|
||||
- E2E: post → archive → draft → approve → `forwardMessage` on the mock.
|
||||
|
||||
### 6.4 Not in this plan (follow-ons)
|
||||
|
||||
- Opticast as a camera **source**: an `rtmp-push`/`srt-listen` ingest kind with per-channel ingest keys.
|
||||
- WHIP guest contribution.
|
||||
- Tier-2 PeerTube-playlist mode.
|
||||
- MTProto grabber ([tg] Phase 2: flagged, ToS disclosure).
|
||||
- Chat relay bot.
|
||||
- Runtime `azmq` gain control.
|
||||
- Mobile Control Room screens. These reuse the S3 endpoints and need no new server surface.
|
||||
|
||||
---
|
||||
|
||||
## 7. Top risks and mitigations
|
||||
|
||||
| # | Risk | Likelihood / impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| 1 | **The frame gate / A/B swap is not frame-continuous**, or costs too much CPU. Every TAKE, overlay and gain change depends on it. | Medium / critical | It is the first task of S2 and gates the whole slice: 1 000-swap continuity plus a CPU budget; a Rust gate if Node fails; GStreamer `input-selector` as the escalation path (the WP9 decision gate). The transient doubled-compositor CPU is reserved inside the 20 % failover headroom. |
|
||||
| 2 | **Multiview expectations versus browser physics.** Users expect 16 live tiles. Decode limits are undocumented ([both]: not verified), and iPhone MSE exists only from 17.1. | High / medium | Thumbnail-default tiers with a hard cap of 4 (1 on mobile) and explicit tile-state copy. Run a device-matrix test before any marketing claim. Tiles never decode remote sources. |
|
||||
| 3 | **Worker CPU at scale.** A 1080p30 N-input grid plus overlays and `amix`, plus an always-on fallback encoder 24/7 for every channel. | High / high | Input caps per profile in admission. Playlist assets pre-normalized, never transcoded in the loop. FALLBACK priced in capacity units, not treated as free. Tier-2 "dark" mode as the documented future cost lever. CI CPU-budget test in S2. |
|
||||
| 4 | **Chat platform fragility**: Twitch has no EventSub replay and harsh rate enforcement; the YouTube quota is 10k/day; the PeerTube livechat plugin varies by instance and its bot token is unsupported. | High / medium | Ring-buffer resume plus `chat.gap` markers. Token buckets and the quota ledger with `paused:*` states. Capability probe with tiered PeerTube degradation (C2S → read-only embed → comments). Request a YouTube quota increase for production. |
|
||||
| 5 | **PeerTube coupling**: `permanentLive` excludes replay, Small latency disables P2P, and live↔VOD switching is unmeasured. | Medium / high | Worker-side FALLBACK from day one. Replay and latency mode are explicit channel config with in-UI consequences. Measure live-end→reconnect on a test instance before promising Tier-2. |
|
||||
| 6 | **Overlay and asset attack surface**: SVG parsing, fonts, render-job resource exhaustion. | Medium / high | Server-owned templates with typed field slots. resvg with no scripts and no external fetch, run in the WP4 sandbox with time and memory limits. Rendered PNGs are content-addressed and cached. FFmpeg only ever sees asset ids. Injection tests in S1 and S3. |
|
||||
| 7 | **Telegram amplification ethics and ToS**: re-posting others' content, protected channels, the 20 MB bot cap. | Certain / medium | A human review gate is mandatory. Attribution is on by default. Protected content and oversize media are surfaced honestly. MTProto stays out of this plan. |
|
||||
| 8 | **Opticast privacy pledge erosion**: pressure to add FCM, analytics or a cloud relay. | Medium / medium | The contract offers no push and only the self-hosted endpoint. The integration is opt-in and off by default. Any push must be UnifiedPush/self-hosted and opt-in. The relationship stays API-only, so the unresolved GPLv3 question remains moot. |
|
||||
|
||||
---
|
||||
|
||||
## Appendix — where the two control-room reports disagree
|
||||
|
||||
| Topic | [flash] | [glm] | Pick | Why |
|
||||
|---|---|---|---|---|
|
||||
| Process topology | One FFmpeg per channel | Split ingest/compositor/encoder/pusher ([prop]) | **Split + frame gate** | FFmpeg cannot rewire graphs, so a single process restarts the RTMP session on every change. |
|
||||
| Overlay text | Pre-rendered PNG (SVG→resvg) | `drawtext` + `textfile` `reload` | **PNG** | PST/PGM pixel parity, brand typography, fonts kept out of the 24/7 sandbox, cheaper CPU. glm's no-restart edge is covered by the A/B swap. |
|
||||
| Live tile cap at 3x3/4x4 | Focused + PST live (≤4 total) | 1–2 live (focus + PGM) | **PGM + focused live, cap 4 on explicit upgrade** | The PGM tile must be real, so its truth comes from the program tap. PST is a client-side look-ahead, not a decoded stream. |
|
||||
| Fallback modes | Worker-side only | Worker-side Tier 1 + PeerTube-playlist Tier 2 | **Tier 1 now, Tier 2 deferred** | Tier 2 depends on unverified PeerTube UI behaviour; it stays as a later cost lever. |
|
||||
| Program gain changes | `amix` runtime commands | Desired-state bump in coarse 3 dB steps | **3 dB steps via swap; runtime commands later** | `azmq` availability is unverified, and swaps are already continuous. |
|
||||
| PST rendering | Not specified (implied worker) | Browser preview compositor (canvas) | **Browser canvas look-ahead** | Avoids a second server composite for each channel. |
|
||||
| Transitions | CUT plus FADE presets; fall back to CUT | CUT plus 300 ms fade default | **CUT in v1; FADE in S4, degrading to CUT** | FADE needs a two-swap transition graph and must be proven continuous first. |
|
||||
151
docs/engineering-guide.md
Normal file
151
docs/engineering-guide.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Tactical Media Broadcast Hub — Engineering Guide
|
||||
|
||||
**Covers:** everything built as of `499c9fe` (WP0–WP3-in-flight), plus Level-2 systems as marked *(planned)*. Companion docs: [admin-guide.md](./admin-guide.md) · [user-guide.md](./user-guide.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. System overview
|
||||
|
||||
Tactical Media Broadcast Hub is a self-hosted, multi-tenant social-media management platform (Level 1: post & schedule) growing into a video/livestream master-control system (Level 2, planned). Three planes:
|
||||
|
||||
```
|
||||
CONTROL PLANE MEDIA PLANE (planned, WP9+)
|
||||
apps/web (Next.js UI) media-worker: ingest → compositor →
|
||||
apps/api (NestJS) persistent encoder → outputs
|
||||
apps/scheduler (Temporal) FFmpeg-first; never through the API
|
||||
PostgreSQL · Redis · Temporal
|
||||
STORAGE PLANE
|
||||
S3-compatible object storage (MinIO dev / any S3 prod)
|
||||
ALL media client-side encrypted; content-addressed
|
||||
```
|
||||
|
||||
Hard rules enforced in code: media bytes never transit API processes; provider branching outside connector packages fails lint; passwords never transit the system (OAuth tokens / bot tokens only); every tenant table is RLS-isolated with `FORCE ROW LEVEL SECURITY`.
|
||||
|
||||
## 2. Repository layout
|
||||
|
||||
```
|
||||
apps/
|
||||
web/ Next.js 14 App Router frontend (all screens, mock-registry layer)
|
||||
api/ NestJS control plane (health, auth, rbac, vault, audit, publishing, storage)
|
||||
scheduler/ Temporal worker (publish workflow + activities)
|
||||
media-worker/ health-only stub (WP9 fills it)
|
||||
packages/
|
||||
core/ normalized domain types (permissions, connections, publishing, VC family)
|
||||
connector-sdk/ PlatformConnector interface, manifest, registry, HTTP client, contract kits
|
||||
connector-example/ connector-mock-social/ connector-mock-video/ connector-mock-livestream/
|
||||
connector-mastodon/ connector-telegram/ (WP3, in flight)
|
||||
testing/ live-source simulator (dod-flap scenario) + mock RTMP sink
|
||||
infra/ docker-compose (postgres:5433, redis, temporal, minio, otel, prometheus, grafana)
|
||||
docs/ this documentation set
|
||||
```
|
||||
|
||||
Monorepo: pnpm workspaces + Turborepo. `pnpm lint|typecheck|build|test` at root runs all 13+ packages.
|
||||
|
||||
## 3. Identity, tenancy & security (WP2)
|
||||
|
||||
### 3.1 Database roles — two-pool discipline
|
||||
- **Owner pool** (`SUPER_DATABASE_URL`, the postgres superuser in dev): runs migrations, provisions. Never serves requests.
|
||||
- **App pool** (`DATABASE_URL`, role `sp_app`): neither superuser nor table owner → `FORCE ROW LEVEL SECURITY` binds for real. Created by migration `003_app_role.sql` with table/sequence grants + default privileges.
|
||||
|
||||
### 3.2 Row-level security
|
||||
Every tenant table (memberships, connections, credentials, audit_log, drafts, publications, scheduled_publications) has:
|
||||
```sql
|
||||
USING (workspace_id = NULLIF(current_setting('app.workspace_id', true), '')::uuid)
|
||||
```
|
||||
`NULLIF` matters: pooled sessions carry an empty-string GUC placeholder after `SET LOCAL` commits; without it the cast throws instead of failing closed. Tenant context is set per-transaction via `withTenant(workspaceId, fn)` (`apps/api/src/db/client.ts`) using `set_config(..., true)` — transaction-local, cannot leak.
|
||||
|
||||
### 3.3 SECURITY DEFINER discipline (audit finding #3)
|
||||
Cross-tenant functions take an explicit `p_workspace_id` and call `fn_assert_tenant()`, which asserts the parameter against the session's tenant context. No unguarded `EXECUTE` grants for `sp_app`.
|
||||
|
||||
### 3.4 Continuous authorization (audit finding #2)
|
||||
Any membership insert/update/delete fires `bump_authz_epoch()` — `COALESCE(NEW.workspace_id, OLD.workspace_id)` (OLD is NULL on INSERT; the original bug silently no-oped). Sessions embed the epoch at issue; `RbacGuard` compares against the live `workspaces.authz_epoch` and fails closed (`401, epoch stale`) on mismatch. Revocation propagates to live sessions without waiting for re-login.
|
||||
|
||||
### 3.5 Sessions & OIDC
|
||||
HMAC-SHA256 signed tokens (`apps/api/src/auth/session.ts`), 12h TTL, constant-time verify. OIDC: generic code flow + PKCE S256, discovery-document driven (`auth/oidc.ts`) — Authentik/Keycloak/any compliant issuer via `OIDC_*` env.
|
||||
|
||||
### 3.6 Credential vault
|
||||
Envelope encryption (`vault/vault.ts`): per-workspace DEK (AES-256-GCM) wrapped by master KEK (`SUPERPOSTER_KEK`, 32B base64; production-required). Rotation re-wraps; old DEK dropped → pre-rotation ciphertext unreadable (tested). Connectors receive opaque `CredentialRef` handles via `CredentialAccessor` — never ciphertext, never raw tokens.
|
||||
|
||||
### 3.7 Audit log
|
||||
Hash-chained per branch (`audit_append` for tenant rows, `audit_append_system` for workspaceless events). Branch selection is by **tenant-context presence, never by role** (findings #1/#6 — `sp_scheduler` with tenant context appends to the tenant chain). `audit_verify_chain(workspace)` recomputes hashes — tampering is detectable (tested). Secret-shaped keys are scrubbed to `[REDACTED]` before append (`audit/audit.service.ts`).
|
||||
|
||||
### 3.8 Storage encryption (user mandate)
|
||||
`storage/storage.ts` — client-side envelope: body = `SPENC1 | iv(12) | KEK-wrapped CEK(48) | AES-256-GCM ciphertext+tag`. `S3_KEK` required in production. SSE-at-rest header set additionally. Keys are the **plaintext** sha256 (`sha256/<hash>.<ext>`), so put-dedup survives; `getObject` re-verifies the hash → storage tampering throws. Cross-workspace media dedup is storage-level only; row visibility stays tenant-scoped.
|
||||
|
||||
## 4. Connector architecture (WP1)
|
||||
|
||||
### 4.1 The contract
|
||||
`PlatformConnector` (`connector-sdk/src/types.ts`): `manifest()`, `authorize()`, `getIdentity()`, optional `publish/uploadMedia/uploadVideo/getFeed/getComments/reply/getMessages/amplify/unamplify/subscribeWebhooks/handleWebhook/...`. Every list method returns `Page<T, cursor>`.
|
||||
|
||||
### 4.2 Manifest & capabilities
|
||||
`ConnectorManifest` declares `id`, `displayName`, `capabilities` (30+ booleans incl. `amplifyRepost/amplifyQuote/amplifyShare`), `connectionFields` (e.g. Mastodon `instanceUrl`, Telegram `botToken` secret), `oauth` flavor. **The manifest validator rejects password fields outright.** UI renders strictly from capabilities — an undeclared capability renders nothing, never a grayed-out lie.
|
||||
|
||||
### 4.3 Registry & the no-branching rule
|
||||
`ConnectorRegistry.register/get` — unknown provider throws `UnknownProviderError`. The ESLint rule `no-restricted-syntax` (root `eslint.config.mjs`) fails any provider string literal outside `packages/connector-*/`, `packages/testing/`, and `apps/web/src/lib/` (the mock-registry boundary standing in for the real registry until the API wires up — same discipline, documented in the config).
|
||||
|
||||
### 4.4 HTTP discipline
|
||||
`RateLimitedHttpClient`: per-host token bucket (default 1 rps; Mastodon 0.5), exponential backoff with jitter on 429/5xx, `Retry-After` honored.
|
||||
|
||||
### 4.5 Amplify truth table (verified per platform, 2026)
|
||||
| Platform | Repost | Quote | Share | Notes |
|
||||
|---|---|---|---|---|
|
||||
| Mastodon | ✅ reblog | ✅ `quoted_status_id` (≥4.5) | — | native scheduling via `scheduled_at` |
|
||||
| Telegram | — | — | ✅ forward/copy | `has_protected_content` blocks |
|
||||
| Facebook | ⚠️ no native reshare API | — | link-share approximation | manifest marks degraded |
|
||||
| Instagram | ❌ | ❌ | own-media stories only | API exposes no amplify |
|
||||
| Bluesky / Threads / LinkedIn | ✅ / ✅ / ✅(reshare w/ commentary) | ✅ / ✅ / — | — | Level 1b wave |
|
||||
| Reddit | — | — | crosspost (stable-but-undocumented) + vote | no vote automation ever |
|
||||
| X | ✅ (metered) | ❌ self-serve (Apr 2026) | — | deferred; pay-per-post model when added |
|
||||
|
||||
### 4.6 Contract test kits
|
||||
`connectorContractTests / videoConnectorContractTests / livestreamConnectorContractTests` — capability-aware: declared capability without implementing method = FAIL; methods behind undeclared capabilities must be absent. Kits caught two real violations during development.
|
||||
|
||||
## 5. Publishing pipeline (WP3, sol-delivered & verified)
|
||||
|
||||
- **State machine** (`publishing.service.ts`): `editing → in_review → approved → scheduled → published` (+ `rejected`, `cancelled`). Every transition permission-checked (`approval.request` / `approval.grant` / `publication.schedule`) and audited.
|
||||
- **Durability** (`006_publishing.sql`): `scheduled_publications` carries a `UNIQUE idempotency_key`; the DB row commits first; if Temporal is unreachable the row stays `pending` and the worker's startup sweep picks it up.
|
||||
- **Temporal** (`scheduler/src/workflows/publish.workflow.ts`): `scheduledPublish(id)` activity (5-min timeout) claims pending-only, resolves the connector via registry, publishes, records the publication, idempotent on re-run.
|
||||
|
||||
## 6. Frontend (apps/web)
|
||||
|
||||
Tailwind v4 `@theme` tokens (dark-first OKLCH, `@theme inline` brand hook for tenant theming), hand-rolled primitives, cmdk palette, TanStack Virtual streams (500+ items/column, scroll-into-view), Tiptap compose with per-platform char counters/previews. **Streams flagship:** capability-gated amplify bars, keyboard `j/k` navigate, `r` repost, `q` quote. All data from `src/lib/` mock-registry (fixtures are seeded-deterministic). QA: `apps/web/scripts/visual-qa.mjs` — 14 states, 0 console errors; byte-diff confirms every interaction renders.
|
||||
|
||||
## 7. Testing strategy
|
||||
|
||||
| Suite | What it proves | Runs |
|
||||
|---|---|---|
|
||||
| connector contract kits | connectors don't lie about capabilities | CI, always |
|
||||
| pure units (vault, session, oidc, scrub, storage-crypto, key-scheme) | crypto/format invariants incl. tamper detection | CI, always |
|
||||
| `tenancy.integration.test.ts` | RLS isolation, tenant-branch audit, epoch bump, chain tamper, immutability triggers | needs `DATABASE_URL` (sp_app) + `SUPER_DATABASE_URL` (owner), else skips |
|
||||
| `storage.integration.test.ts` | S3 roundtrip, dedup, presign | needs `S3_ENDPOINT`, else skips |
|
||||
| visual-qa (playwright) | screens render, interactions fire, console clean | local/CI with chromium |
|
||||
|
||||
Live-DB invocation (from repo root, postgres on :5433):
|
||||
```
|
||||
DATABASE_URL='postgresql://sp_app:sp_app_dev@localhost:5433/tmbh' \
|
||||
SUPER_DATABASE_URL='postgresql://tmbh:tmbh_dev@localhost:5433/tmbh' \
|
||||
pnpm --filter @tactical-media-broadcast-hub/api test
|
||||
```
|
||||
|
||||
## 8. Observability & ops surface
|
||||
|
||||
`/healthz` (liveness), `/readyz` (TCP probes: postgres/redis/temporal with per-component status). OTel OTLP tracing skeleton (disabled-safe without collector). Structured pino logs with credential-key redaction. Grafana (3001) provisioned against Prometheus (9090). Full metrics rollout is WP14.
|
||||
|
||||
## 9. Level 2 systems *(planned — designs in proposal.md §4.4)*
|
||||
|
||||
Live Source Monitor (adaptive polling, anti-flapping, per-host budgets); media engine (persistent encoder, graph-rewire switching, GStreamer decision gate); Auto Director (policies/rules/override/program-lock); control room (Preview/Program, TAKE/CUT); archive/grabber/crosspost (S3-mandated, encrypted, content-deduped).
|
||||
|
||||
## 10. Key file map
|
||||
|
||||
| Concern | Path |
|
||||
|---|---|
|
||||
| Tenant context / pools | `apps/api/src/db/client.ts` |
|
||||
| Migrations 001–006 | `apps/api/migrations/` |
|
||||
| Vault / storage crypto | `apps/api/src/vault/vault.ts`, `apps/api/src/storage/storage.ts` |
|
||||
| Sessions / OIDC | `apps/api/src/auth/` |
|
||||
| RBAC guard | `apps/api/src/rbac/rbac.guard.ts` |
|
||||
| Audit chain | `migrations/001_tenants.sql` (SQL) + `apps/api/src/audit/` |
|
||||
| Publishing | `apps/api/src/publishing/`, `apps/scheduler/src/` |
|
||||
| Connector SDK | `packages/connector-sdk/src/` |
|
||||
| Frontend mock-registry | `apps/web/src/lib/` |
|
||||
| Lint: no-provider-branching | `eslint.config.mjs` |
|
||||
BIN
docs/manual/screenshots/desktop-analytics.png
Normal file
BIN
docs/manual/screenshots/desktop-analytics.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
docs/manual/screenshots/desktop-approvals.png
Normal file
BIN
docs/manual/screenshots/desktop-approvals.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
docs/manual/screenshots/desktop-audit-diff-open.png
Normal file
BIN
docs/manual/screenshots/desktop-audit-diff-open.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
BIN
docs/manual/screenshots/desktop-audit-sample.png
Normal file
BIN
docs/manual/screenshots/desktop-audit-sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
BIN
docs/manual/screenshots/desktop-audit.png
Normal file
BIN
docs/manual/screenshots/desktop-audit.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 111 KiB |
BIN
docs/manual/screenshots/desktop-calendar.png
Normal file
BIN
docs/manual/screenshots/desktop-calendar.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
docs/manual/screenshots/desktop-command-palette.png
Normal file
BIN
docs/manual/screenshots/desktop-command-palette.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
BIN
docs/manual/screenshots/desktop-compose-draft-saved.png
Normal file
BIN
docs/manual/screenshots/desktop-compose-draft-saved.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
BIN
docs/manual/screenshots/desktop-compose-empty.png
Normal file
BIN
docs/manual/screenshots/desktop-compose-empty.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
BIN
docs/manual/screenshots/desktop-compose-sample.png
Normal file
BIN
docs/manual/screenshots/desktop-compose-sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
BIN
docs/manual/screenshots/desktop-compose-typed.png
Normal file
BIN
docs/manual/screenshots/desktop-compose-typed.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
docs/manual/screenshots/desktop-connections-capacity-popover.png
Normal file
BIN
docs/manual/screenshots/desktop-connections-capacity-popover.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 91 KiB |
BIN
docs/manual/screenshots/desktop-connections-live-empty.png
Normal file
BIN
docs/manual/screenshots/desktop-connections-live-empty.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
BIN
docs/manual/screenshots/desktop-connections-sample.png
Normal file
BIN
docs/manual/screenshots/desktop-connections-sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
docs/manual/screenshots/desktop-inbox.png
Normal file
BIN
docs/manual/screenshots/desktop-inbox.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 87 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-connect-simulated.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-connect-simulated.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-connect.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-connect.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-done.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-done.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-invite.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-invite.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 48 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-register-filled.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-register-filled.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-register.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-register.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-sample.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-signed-in.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-signed-in.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-signin-error.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-signin-error.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
BIN
docs/manual/screenshots/desktop-onboarding-signin.png
Normal file
BIN
docs/manual/screenshots/desktop-onboarding-signin.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
BIN
docs/manual/screenshots/desktop-settings.png
Normal file
BIN
docs/manual/screenshots/desktop-settings.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
docs/manual/screenshots/desktop-streams-quote-modal.png
Normal file
BIN
docs/manual/screenshots/desktop-streams-quote-modal.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 182 KiB |
BIN
docs/manual/screenshots/desktop-streams-repost-toast.png
Normal file
BIN
docs/manual/screenshots/desktop-streams-repost-toast.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 204 KiB |
BIN
docs/manual/screenshots/desktop-streams-selected.png
Normal file
BIN
docs/manual/screenshots/desktop-streams-selected.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 199 KiB |
BIN
docs/manual/screenshots/desktop-streams.png
Normal file
BIN
docs/manual/screenshots/desktop-streams.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 206 KiB |
BIN
docs/manual/screenshots/desktop-team-sample.png
Normal file
BIN
docs/manual/screenshots/desktop-team-sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
BIN
docs/manual/screenshots/desktop-team.png
Normal file
BIN
docs/manual/screenshots/desktop-team.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 90 KiB |
BIN
docs/manual/screenshots/desktop-video-after-delete.png
Normal file
BIN
docs/manual/screenshots/desktop-video-after-delete.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 60 KiB |
BIN
docs/manual/screenshots/desktop-video-after-refresh.png
Normal file
BIN
docs/manual/screenshots/desktop-video-after-refresh.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 60 KiB |
BIN
docs/manual/screenshots/desktop-video-delete-confirm.png
Normal file
BIN
docs/manual/screenshots/desktop-video-delete-confirm.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
BIN
docs/manual/screenshots/desktop-video-detail-drawer.png
Normal file
BIN
docs/manual/screenshots/desktop-video-detail-drawer.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 89 KiB |
BIN
docs/manual/screenshots/desktop-video-empty.png
Normal file
BIN
docs/manual/screenshots/desktop-video-empty.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
docs/manual/screenshots/desktop-video-library.png
Normal file
BIN
docs/manual/screenshots/desktop-video-library.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
BIN
docs/manual/screenshots/desktop-video-sample.png
Normal file
BIN
docs/manual/screenshots/desktop-video-sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
docs/manual/screenshots/mobile-analytics.png
Normal file
BIN
docs/manual/screenshots/mobile-analytics.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
BIN
docs/manual/screenshots/mobile-approvals.png
Normal file
BIN
docs/manual/screenshots/mobile-approvals.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
BIN
docs/manual/screenshots/mobile-audit.png
Normal file
BIN
docs/manual/screenshots/mobile-audit.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
BIN
docs/manual/screenshots/mobile-calendar.png
Normal file
BIN
docs/manual/screenshots/mobile-calendar.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
BIN
docs/manual/screenshots/mobile-compose.png
Normal file
BIN
docs/manual/screenshots/mobile-compose.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
BIN
docs/manual/screenshots/mobile-connections.png
Normal file
BIN
docs/manual/screenshots/mobile-connections.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 139 KiB |
BIN
docs/manual/screenshots/mobile-inbox.png
Normal file
BIN
docs/manual/screenshots/mobile-inbox.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
BIN
docs/manual/screenshots/mobile-onboarding-register.png
Normal file
BIN
docs/manual/screenshots/mobile-onboarding-register.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 132 KiB |
BIN
docs/manual/screenshots/mobile-onboarding-signin.png
Normal file
BIN
docs/manual/screenshots/mobile-onboarding-signin.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 89 KiB |
BIN
docs/manual/screenshots/mobile-onboarding.png
Normal file
BIN
docs/manual/screenshots/mobile-onboarding.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 94 KiB |
BIN
docs/manual/screenshots/mobile-settings.png
Normal file
BIN
docs/manual/screenshots/mobile-settings.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
BIN
docs/manual/screenshots/mobile-streams.png
Normal file
BIN
docs/manual/screenshots/mobile-streams.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 170 KiB |
BIN
docs/manual/screenshots/mobile-team.png
Normal file
BIN
docs/manual/screenshots/mobile-team.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 154 KiB |
BIN
docs/manual/screenshots/mobile-video.png
Normal file
BIN
docs/manual/screenshots/mobile-video.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 141 KiB |
BIN
docs/manual/user-manual-v2.pdf
Normal file
BIN
docs/manual/user-manual-v2.pdf
Normal file
Binary file not shown.
41
docs/matrix-setup.md
Normal file
41
docs/matrix-setup.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# Matrix Setup — Tactical Media Broadcast Hub team chat
|
||||
|
||||
OccupyMedia storage phase one adds [group attachment/archive contracts](occupymedia-storage.md)
|
||||
and an [independent files/docs scaffold](../infra/occupymedia/files/README.md).
|
||||
Live `occ-chat` remains conduwuit v0.5.0 with its existing local state/media;
|
||||
no volume or image switch is part of this phase. Future Tuwunel native R2 media
|
||||
requires a separate rehearsal and authorized upgrade. Archive mirroring is
|
||||
restricted to explicitly mapped **unencrypted archive rooms**; encrypted rooms
|
||||
are excluded. Collaborative ODT editing uses oCIS/Collabora WOPI, not Matrix.
|
||||
|
||||
Matrix (conduwuit homeserver) is the **team/internal** chat plane. Stream viewer chat stays on the S5 hub (`docs/design/streaming-architecture.md` §3). Division of labor:
|
||||
|
||||
| Plane | Tech | Who it serves |
|
||||
|---|---|---|
|
||||
| Team chat, DMs, bridged Telegram channels | Matrix (conduwuit + bridges) | editors, reviewers, admins |
|
||||
| Stream viewer chat | S5 hub → /realtime WS | Opticast app + control room |
|
||||
| Review threads (per-post) | Tactical Media Broadcast Hub native (WP5.1) | editorial workflow |
|
||||
|
||||
## Deploy
|
||||
1. `cp infra/matrix/.env.example infra/matrix/.env` — set `MATRIX_SERVER_NAME` (permanent!) and a registration token
|
||||
2. Point your reverse proxy at :6167 with HTTPS (federation requires valid TLS)
|
||||
3. `docker compose -f infra/matrix/docker-compose.yml up -d`
|
||||
4. First account: temporarily set `MATRIX_ALLOW_REGISTRATION=true`, register the admin via any Matrix client (Element), then lock it back. Share the token for invites.
|
||||
|
||||
## Rich text
|
||||
Conduwuit supports formatted bodies natively — clients (Element et al.) give Markdown, media, reactions, edits, replies, threads. **As good as Telegram** out of the box, plus E2E encryption.
|
||||
|
||||
## Telegram bridge (feeds WP15 — join channels & amplify)
|
||||
`mautrix-telegram` runs alongside. One-time setup:
|
||||
1. `docker compose run --rm mautrix-telegram` — first boot writes `/data/config.yaml`; set homeserver + bridge admin
|
||||
2. In Matrix, DM `@telegrambot:your-server`, send `login` + the phone/code → your Telegram account is bridged
|
||||
3. Telegram channels you join appear as Matrix rooms; posts arrive as rich messages. The WP15 grabber (phase 2) reads those rooms → archive → review → amplify — no custom MTProto client needed, and the bridge is the battle-tested path
|
||||
4. Two-way: replying in the Matrix room posts to Telegram (subject to the same rights/attribution rules as S6)
|
||||
|
||||
## Branding
|
||||
Element Web (if self-hosted as the client) accepts custom CSS — the same propagation rule as PeerTube: repo is the source of truth, a sync script pushes (accent #FF1610 on black, solid disk mark) after any brand change. Native clients keep their own themes; brand consistency applies to web surfaces.
|
||||
|
||||
## Tactical Media Broadcast Hub integration points (follow-ups)
|
||||
- **Inbox screen**: swap fixtures for Matrix client-sdk (matrix-js-sdk) — DMs + team rooms render in-app
|
||||
- **Review notifications**: WP5.1 posts approval-needed events into a team room
|
||||
- **WP15**: grabber consumes bridged Telegram rooms (above)
|
||||
51
docs/oauth-connect.md
Normal file
51
docs/oauth-connect.md
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# Connecting platform accounts
|
||||
|
||||
The API resolves authorization behavior through connector methods and manifests. It currently registers Mastodon, PeerTube, and Telegram. Other providers are not offered as connectable until a connector implements the same authorization surface.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Environment variable | Meaning |
|
||||
| --- | --- |
|
||||
| `APP_URL` | Web origin used for the final `/connections?connect=success` or `failure` redirect. Default `http://localhost:3000`; set `http://localhost:3001` when running the repository's usual API-on-3000/web-on-3001 setup. |
|
||||
| `API_URL` | Public API base. Default `http://localhost:3000`. |
|
||||
| `CONNECTIONS_CALLBACK_URL` | Optional exact callback URI override; default `${API_URL}/connections/callback`. Register this exact URI with configured provider applications. |
|
||||
| `CORS_ORIGINS` | Comma-separated permitted browser origins. Include the web origin; cookie credentials are enabled only for these explicit origins. |
|
||||
| `REDIS_URL` | Pending authorization store; default `redis://localhost:6379`. Production should use an authenticated `rediss://` URL. |
|
||||
| `SUPERPOSTER_KEK` | Production-required 32-byte base64 envelope encryption key. Keep it stable across replicas/restarts and secure its backups. The existing development key fallback is unsuitable for production. |
|
||||
| `MASTODON_INSTANCE_URL` | Instance origin to which the configured Mastodon client belongs. |
|
||||
| `MASTODON_CLIENT_ID`, `MASTODON_CLIENT_SECRET` | Optional pre-registered Mastodon application credentials, used only for `MASTODON_INSTANCE_URL`. |
|
||||
| `PEERTUBE_INSTANCE_URL` | Instance origin to which the configured PeerTube bridge client belongs. |
|
||||
| `PEERTUBE_CLIENT_ID`, `PEERTUBE_CLIENT_SECRET` | Optional credentials for that instance. Otherwise the connector discovers the local client through `/api/v1/oauth-clients/local`. |
|
||||
| `PEERTUBE_AUTHORIZE_PATH`, `PEERTUBE_TOKEN_PATH` | Required to enable a PeerTube redirect bridge. Both must be paths beginning with `/`, without a second leading slash, on the selected instance origin. |
|
||||
|
||||
Use HTTPS and the same browser site for the web and API (for example `app.example.org` and `api.example.org`), or proxy the API under the web origin. The callback binding cookie is HttpOnly, SameSite=Lax, and Secure in production. For development, consistently use `localhost` or consistently use `127.0.0.1` for both web and API; mixing the two breaks browser cookie delivery.
|
||||
|
||||
### Mastodon
|
||||
|
||||
Without a configured client for the selected instance, the connector registers an application using `POST /api/v1/apps`, with the exact callback URI and `read write:statuses write:media` scopes. This is the supported path for arbitrary self-hosted instances; there is no universal client secret shared by all instances. Reconnect reuses that connection's registered client. Both legs use S256 PKCE. [Mastodon's OAuth documentation](https://docs.joinmastodon.org/methods/oauth/) describes the authorization-code grant and PKCE support (Mastodon 4.3+).
|
||||
|
||||
### PeerTube limitation and bridge contract
|
||||
|
||||
Stock PeerTube documents local OAuth client discovery followed by a **password grant**, rather than an authorization-code consent flow. The connect API never accepts account passwords, so the default PeerTube connect button is disabled with an explanation. See the [official PeerTube authentication guide](https://docs.joinpeertube.org/api/rest-getting-started).
|
||||
|
||||
An instance administrator can enable a compatible OAuth bridge, then configure the two paths above. Its authorize endpoint must preserve `state`, register/validate the redirect URI, support `response_type=code` and S256 PKCE, and redirect back with a one-use code. Its token endpoint must validate `client_id`, `client_secret`, `redirect_uri`, `code`, and `code_verifier`, returning **actual PeerTube API** `access_token` and `refresh_token` values. Identity, refresh, and revoke then use PeerTube's native `/api/v1/users/me`, `/api/v1/users/token`, and `/api/v1/users/revoke-token` endpoints. These settings do not add an authorization-code grant to a stock instance, and an OIDC identity token alone is insufficient. The mock integration test proves this adapter contract; no third-party bridge is included in this change.
|
||||
|
||||
### Telegram
|
||||
|
||||
Create a bot in BotFather and enter its bot token. The connector validates it with `getMe`, then stores the token through the same encrypted vault envelope as OAuth credentials. Optionally supply a chat/channel ID (`@channel` or a numeric chat ID); add the bot to the target and grant any necessary posting rights. Identity health confirms token validity, not permission to publish to every chat. Reconnect accepts a replacement bot token for the same bot and rotates the credential row in place. Telegram's Bot API cannot revoke its own bot token: Remove clears local authorization; revoke/rotate the token through BotFather when needed.
|
||||
|
||||
## API and storage behavior
|
||||
|
||||
- `GET /connections/providers` returns available connectors, connect support, required instance/target fields, limitations, and real manifest capabilities.
|
||||
- `POST /connections/authorize` accepts `{ provider, instanceUrl? }`, validates connection-management permission, returns `{ redirectUrl }`, and sets the callback binding cookie.
|
||||
- `GET /connections/callback` is intentionally outside the bearer guard. It requires the random state and matching browser/workspace binding, rechecks the initiating user's current membership/epoch/permission, exchanges the code, validates identity, and redirects with a fixed success/failure result. Provider error messages never reach the redirect or API logs.
|
||||
- `POST /connections/telegram/bot-token` accepts `{ token, channelId?, connectionId? }`. `connectionId` rotates the existing bot connection instead of creating another.
|
||||
- `POST /connections/:connectionId/reconnect` reauthorizes the existing account and instance. A different provider identity is rejected; failed consent/exchange leaves the previous credential intact.
|
||||
- `DELETE /connections/:connectionId` locks the connection, revokes supported provider credentials, and deletes the connection and cascaded credential row. It also removes local schedules/publication records required by existing restrictive foreign keys; published provider posts remain. The UI asks for confirmation with this consequence. If a provider is unavailable, local removal is retained for retry; already-invalid credentials can be removed.
|
||||
- `GET /connections` probes provider identity: `healthy`, `needs_credentials`, or `unavailable`, and returns capability names without credentials. Automatic token refreshes are encrypted and persisted with a compare-and-swap fence against concurrent reconnects.
|
||||
|
||||
Pending authorizations are random 256-bit state keys with a ten-minute Redis TTL. The Redis value is encrypted, including app secrets, PKCE verifier, initiating session context, and redirect configuration. Atomic compare-and-delete permits only one callback across replicas. Credentials use a persistent AES-256-GCM envelope in the existing `credentials.ciphertext` column, with its DEK wrapped by the existing KEK and authenticated tenant/handle binding. No database migration or dependency addition is needed. Audit mutations and credential writes share a tenant transaction; audits contain only the provider identifier/action, never credentials. Bot request URLs and callback URLs are excluded from HTTP tracing, and API structured logs redact token fields and request bodies/query/cookie/authorization data.
|
||||
|
||||
## Verification
|
||||
|
||||
`apps/api/src/connections/connections.integration.test.ts` runs real Nest routes, real connector auth/identity/revoke code, a PKCE-checking mock authorization server, TTL-faithful pending storage, and encrypted connection storage through in-process HTTP. It needs no open TCP ports, PostgreSQL, or Redis. Persistence unit tests independently exercise the real SQL repository's parameterization, tenant context, revocation ordering, and ciphertext-only audit writes. Existing live database integration suites still require their documented database environment variables.
|
||||
142
docs/occupymedia-storage.md
Normal file
142
docs/occupymedia-storage.md
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
# OccupyMedia shared files, documents and pads — phase one
|
||||
|
||||
**Implemented: contracts, config validation and independent compose scaffolding.**
|
||||
No services, Matrix upgrade, archive bot, index API, database migration or UI have
|
||||
been deployed/implemented by this phase. Existing API/media-worker storage paths
|
||||
and deployment edits are preserved. [Deployment steps and resources](../infra/occupymedia/files/README.md)
|
||||
are the next operator runbook. Design input: `runs/wp4/fileshare-research.md`.
|
||||
|
||||
## Storage ownership
|
||||
|
||||
| Payload bucket | Exclusive owner | Status |
|
||||
|---|---|---|
|
||||
| `occupymedia-matrix-media` | Future Tuwunel native S3 media | Existing bucket; live conduwuit still local |
|
||||
| `occupymedia-files-blobs` | oCIS S3NG, working files/docs and pad exports | Proposed new bucket; inventory/provision first |
|
||||
| `occupymedia-media-encrypted` | Tactical Media Broadcast Hub media/library, existing envelope format | Existing configuration; preserve KEK |
|
||||
| `occupymedia-peertube-videos` | PeerTube video objects | Existing dedicated bucket |
|
||||
| `occupymedia-peertube-playlists` | PeerTube playlists | Existing dedicated bucket |
|
||||
|
||||
Verify actual inventory; names here are ownership contracts, not proof of
|
||||
provisioning. Do not let the bot write directly into oCIS's opaque bucket. It must
|
||||
upload via authorized WebDAV/CS3 so metadata and ACLs remain coherent. Each
|
||||
product uses its own scoped credentials; bucket prefixes are not isolation.
|
||||
|
||||
The target is **R2-only durable file/media payloads, minimal local metadata/config
|
||||
allowed, no local payload cache by design**. POSIX oCIS metadata, Matrix
|
||||
RocksDB/state, Hub metadata rows, identity/config and keys remain local and need
|
||||
backups. Source files, document bodies, thumbnails, extracted full text, pad
|
||||
revisions, attachments and staging/cache copies are payloads. Calling pad text
|
||||
or full-text extraction a database column does not turn it into metadata.
|
||||
Bounded transient RAM processing is allowed; disk/swap/core-dump payload copies
|
||||
are not. This phase does not make existing local Matrix bytes disappear.
|
||||
|
||||
## Telegram-like group file contract
|
||||
|
||||
The shared TypeScript boundary is `packages/core/src/shared-files.ts`; future
|
||||
HTTP/DB adapters must validate these shapes at runtime. Use stable
|
||||
`workspaceId + groupId` (existing Hub groups) and immutable `roomId` to map to an
|
||||
oCIS `spaceId + folderId`. Start with one restricted Space per group; allow
|
||||
multiple rooms to map explicitly to that group. Never infer identity/ACL from
|
||||
room names, aliases or bridge membership. Membership changes must reconcile
|
||||
both Matrix and file grants; mismatches suspend sharing/mirroring until corrected.
|
||||
|
||||
A `GroupFile` records room/group/workspace, sender MXID (not a display name),
|
||||
Matrix event ID, origin (`matrix`, provider-described `bridge`, direct `upload`,
|
||||
or `pad-export`), original name, MIME, verified byte size, SHA-256, UTC ISO
|
||||
`createdAt` and `indexedAt`, stable library resource/version, share link,
|
||||
retention and availability. For bridged sources preserve both Matrix sender
|
||||
and provider/remote event attribution when supplied by the trusted bridge;
|
||||
do not invent attribution. Original `createdAt` comes from event time or accepted
|
||||
upload time, not the replay/index time. Direct uploads and exports use null event
|
||||
IDs until a room announcement supplies one. Hash actual bytes while streaming;
|
||||
S3 ETags, claimed Matrix MIME/size and filenames are not proof. Normalize a safe
|
||||
display filename and sniff MIME; reject over-limit transfers, including when
|
||||
Content-Length is missing or lies.
|
||||
|
||||
Default links require current group membership; downloads/edit launches must
|
||||
recheck ACLs and retention. Public links require an explicit group grant and
|
||||
expiry; revoke them on removal. Persist library links/share IDs and expiry, not
|
||||
presigned object URLs, WOPI tokens or passwords. Keep retention policy separate
|
||||
from link expiry: `keep` is the phase-one default with no automated purge;
|
||||
`until` requires an explicit timestamp, `legalHold` prevents deletion.
|
||||
Future policy must account for source Matrix media and library copies separately;
|
||||
one message redaction does not authorize deleting a blob used by another record.
|
||||
|
||||
## Metadata index and hub-archive seam
|
||||
|
||||
Future `chat_media_archive` persistence is metadata-only. Required constraints:
|
||||
unique `(workspace_id, room_id, event_id)` for attachment events, stable file
|
||||
identity/version, indexes on `(workspace_id, group_id, created_at, id)`,
|
||||
sender, MIME class, name and checksum. List/search return only available,
|
||||
unexpired rows visible through current group/file ACLs. Use a `(createdAt,id)`
|
||||
cursor scoped to workspace/group. Filename/MIME/sender/date/origin queries and
|
||||
oCIS metadata search are the initial searchable/retrievable media experience.
|
||||
Full-text/OCR/Tika is deferred: neither a local text index nor an extra JVM is
|
||||
silently included. Archive rows never contain file bodies or encrypted-file keys.
|
||||
|
||||
`HubArchivePort` defines future archive/revoke operations. A plain sync-loop bot
|
||||
(not an appservice) uses a narrowly scoped Matrix account and a scoped oCIS
|
||||
service identity. Required processing contract:
|
||||
|
||||
1. Load an explicitly enabled group mapping and current room encryption state.
|
||||
Exclude own bot/reply events to prevent loops. Abort on unknown state, lost
|
||||
membership/ACL, redaction, encrypted events or `content.file` attachments.
|
||||
2. Accept only `m.room.message` attachments (`m.file`, `m.image`, `m.audio`,
|
||||
`m.video`) with a valid `mxc://` URI. Use authenticated Matrix media download
|
||||
endpoints (with version negotiation), never arbitrary user-supplied HTTP URLs.
|
||||
Validate any redirect to the configured R2 endpoint and drop Matrix credentials
|
||||
before following; never forward bearer auth to the object store.
|
||||
3. Claim `attachmentArchiveKey(workspace,room,event)` durably. Use a deterministic
|
||||
upload destination/resource lookup and reconcile partial retries to avoid
|
||||
duplicate files across crashes. Stream bounded bytes/hash to WebDAV/CS3;
|
||||
**no temp files on local disk**. Record the committed library reference,
|
||||
verified metadata and a durable reply/outbox entry before advancing the
|
||||
stored Matrix sync cursor. Advancing past a failed event requires a durable
|
||||
retry record; R2/ACL failures must not lose attachments.
|
||||
4. Reply once with an authenticated file link using a stable Matrix transaction
|
||||
ID derived from the replay key. Search is an index query; retrieve by library
|
||||
resource through current ACLs. An encrypted Tactical Media Broadcast Hub library copy, if
|
||||
needed for publishing, goes through the existing encryption path into its
|
||||
own bucket; no cross-product object-key reuse.
|
||||
5. On redaction/retention or membership removal, revoke links and mark the index
|
||||
unavailable; stop serving immediately. A later audited deletion workflow
|
||||
handles library versions/trash, references, source media and legal holds.
|
||||
Backfill, retries and live sync obey the same policy and idempotency rules.
|
||||
|
||||
No implementation of this loop or SQL table is present in phase one; config's
|
||||
`HUB_ARCHIVE_*` variables reserve its interface. `canArchiveAttachment` is a
|
||||
tested pure eligibility check, not complete authorization or byte validation.
|
||||
|
||||
## Encryption policy
|
||||
|
||||
**Encrypted Matrix rooms cannot be server-side mirrored by this architecture.**
|
||||
No bot encryption keys, decryption engine or silent downgrade is added. Group
|
||||
admins designate a separate **unencrypted archive room**, tell members its
|
||||
attachments enter the shared library, and enable its immutable room mapping
|
||||
explicitly. Users may deliberately upload/share a file there; encrypted rooms
|
||||
remain untouched. Treat bridged rooms the same: verify encryption state instead
|
||||
of assuming the bridge made them plaintext. If an archive room becomes encrypted,
|
||||
or its state cannot be established, mirroring stops. R2 can retain encrypted
|
||||
Matrix media as ciphertext in a future Tuwunel deployment, but that does not
|
||||
make it server-searchable or give the bot access to plaintext.
|
||||
|
||||
## Collaborative ODT and pad boundaries
|
||||
|
||||
ODT editing is **oCIS file -> collaboration WOPI host -> Collabora -> save/version
|
||||
back through oCIS -> R2**. Matrix posts links and notifications; it does not edit
|
||||
ODT. The editor gets short-lived, permission-scoped launch tokens and file locks.
|
||||
Test simultaneous edits, reconnects, saves, versions and grant revocation; keep
|
||||
Collabora's working files in transient RAM and block launch if that cannot be
|
||||
guaranteed for the selected image.
|
||||
|
||||
`SharedPad` / `PadExportPort` allow Etherpad- or HedgeDoc-compatible group pad
|
||||
links and revision exports. Pad content, revisions and attachments must be
|
||||
durable R2 payloads; local storage may retain only identifiers, ACLs and session
|
||||
metadata. Standard pad deployments typically persist document text/revisions
|
||||
in their database: they need a separately validated R2 persistence adapter or
|
||||
another compliant service. R2 attachment support or periodic exports alone is
|
||||
insufficient. Consequently there is **no running pad service** in this scaffold,
|
||||
and `PADS_ENABLED=false`. Approved exports become ordinary versioned oCIS
|
||||
resources with checksums and group retention; live ODT collaboration remains
|
||||
Collabora's responsibility. This preserves the lightweight pad option without
|
||||
introducing an incompatible local payload database.
|
||||
41
docs/peertube-setup.md
Normal file
41
docs/peertube-setup.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# PeerTube Setup — Tactical Media Broadcast Hub's video/live backbone
|
||||
|
||||
PeerTube runs as a **subproject** (`infra/peertube/`) on a **different server** than the Tactical Media Broadcast Hub API/web. It stores all media in **S3** and carries the **Tactical Media Broadcast Hub brand**, propagated from this repo.
|
||||
|
||||
## Deploy (on the media server)
|
||||
1. `cp infra/peertube/.env.example infra/peertube/.env` — fill in hostname, S3 keys (dedicated buckets: `tactical-media-broadcast-hub-peertube-{videos,playlists,redundancy}`), DB password
|
||||
2. `docker compose -f infra/peertube/docker-compose.yml up -d`
|
||||
3. Put the instance behind HTTPS (nginx/caddy) — federation + OAuth require it
|
||||
4. First boot creates the admin account (PEERTUBE_ADMIN_EMAIL)
|
||||
|
||||
## S3 storage
|
||||
PeerTube's object-storage env handles ALL media (videos, playlists, redundancy). Point it at the same S3-compatible infra the Tactical Media Broadcast Hub media plane uses (MinIO or cloud), with dedicated buckets + credentials. The Tactical Media Broadcast Hub API never proxies PeerTube bytes — both speak S3 natively.
|
||||
|
||||
## Branding — propagation rule
|
||||
**Source of truth: this repo** (`apps/web/public/brand/` + design tokens in `apps/web/src/app/globals.css`). After ANY brand change:
|
||||
```
|
||||
infra/peertube/brand-sync/sync-brand.sh
|
||||
```
|
||||
It pushes instance name, custom CSS (accent #FF1610 on black), and the solid disk mark to the instance via PeerTube's config API (needs `PEERTUBE_URL` + `PEERTUBE_TOKEN` with settings scope). Idempotent — run it whenever the brand moves. (Later: a "Sync branding" button in Tactical Media Broadcast Hub settings that calls the same API.)
|
||||
|
||||
## Connecting Tactical Media Broadcast Hub → PeerTube
|
||||
- **OAuth bridge**: stock PeerTube exposes a password grant only. Our connect flow (docs/oauth-connect.md) supports a same-instance OAuth bridge when configured; otherwise use an instance with a bridge plugin or a token-entry connection. The connector (`packages/connector-peertube`) handles publish, livestream keys, per-instance federation.
|
||||
- **Live ingest**: the media worker's compositor pushes RTMP to `<host>:1935` (opened in the compose file). Livestream keys flow through the connector.
|
||||
- **Live chat**: install the **livechat plugin** (peertube-plugin-livechat) on the instance — S5's chat hub consumes chat via the XMPP/plugin surface (phase 2 of S5, per the streaming architecture doc).
|
||||
|
||||
## Different-server topology
|
||||
```
|
||||
[app server] [media server]
|
||||
apps/api apps/web scheduler peertube (+redis +postfix)
|
||||
media-worker compose (pg, S3 buckets (dedicated peertube-*)
|
||||
redis, temporal...) │
|
||||
└────── RTMP :1935 ───────┘
|
||||
└────── OAuth/API over HTTPS ──┘
|
||||
```
|
||||
Firewall: app server → media server :443 (API) and :1935 (RTMP). Nothing else required between them; S3 is the shared storage plane.
|
||||
|
||||
## Settings checklist (instance admin UI, after first boot)
|
||||
- Enable live + allow replay + transcode lives
|
||||
- Registration: closed (users join via Tactical Media Broadcast Hub)
|
||||
- Federation: follow only trusted instances (moderated homepage)
|
||||
- Plugins: peertube-plugin-livechat (chat), optionally peertube-plugin-rtc
|
||||
BIN
docs/s27-brief.pdf
Normal file
BIN
docs/s27-brief.pdf
Normal file
Binary file not shown.
40
docs/s27-broadcast-station-brief.md
Normal file
40
docs/s27-broadcast-station-brief.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# Tactical Media Broadcast Hub
|
||||
|
||||
*A horizontal coordination platform for media analysis, production, and diffusion — self-hosted, multi-platform, no algorithm in between.*
|
||||
|
||||
The Hub is built for one thing: **letting many people run many media outlets together, in real time.** Dozens of contributors, dozens of social accounts across Mastodon, Telegram, PeerTube, Bluesky, Instagram, Twitch and more — one shared newsroom where the collective edits, decides, and publishes as a single coordinated force. It is not a scheduling tool with a broadcast feature attached; coordination IS the product. The rest — live mixing, the 24/7 channel, the archives — grows out of that spine.
|
||||
|
||||
## The newsroom loop: analysis → production → diffusion
|
||||
|
||||
**ANALYZE — see everything, together.** Live feeds from every connected account flow into shared columns. Telegram channels the collective follows are mirrored automatically into a review archive. Every draft, clip, and statement lives in one encrypted library. The room sees what's happening across the whole footprint at once — and what the movement is saying everywhere else.
|
||||
|
||||
**PRODUCE — collective editing in real time.** Many hands, one voice. Any editor can draft into any account; every change is versioned. Nothing publishes in the collective's name alone: the **review workflow** puts each post before the group — a configurable quorum (2-of-3, 3-of-5) must sign off, with roles that can be delegated per account (this crew approves the press account, that crew the stream channel). Votes are transparent, denials require a reason, and the whole trail is tamper-evident. Decisions are made by whoever is present, in the open, in minutes — not routed through one admin's phone.
|
||||
|
||||
**DIFFUSE — smart quoting across platforms.** One approval fans a post out everywhere, native to each platform. Cross-posting uses **smart quotes**: where a platform has native quoting (Mastodon quote posts, Bluesky, Telegram forwards), it's used; where it doesn't, the system generates a proper attributed quote — author, instance, link back — never a silent copy-paste. Amplifying sister collectives' work keeps their names on it by default. Scheduling, rate-awareness, and per-account timing come along for free.
|
||||
|
||||
## Real-time story production
|
||||
|
||||
Breaking events don't wait for consensus rounds. The Hub is built for **speed with accountability**: an editor stages a developing story, reviewers approve from their phones, updates go out as quote-chained revisions. Meanwhile the live layer runs: field streamers push cameras from the **Opticast** Android app straight to our server; a director in the browser **Control Room** watches every feed in a multiview grid, mixes audio, cuts between scenes, burns lower-thirds and scrolling chyrons; the mixed program goes to our **PeerTube channel** — and when no one is live, the channel keeps broadcasting a scheduled playlist. An **Auto-Director** can switch sources by priority rules so the station never goes dark mid-action. Unified chat from every platform lands in one place — viewers heard, moderators acting, the streamer focused on the shot.
|
||||
|
||||
## The archive: memory that can't be purged
|
||||
|
||||
Everything the collective produces — text, photos, video, streams — lands in an **encrypted archive** on our own storage: client-side encrypted, content-addressed, deduplicated. Platforms purge; the Hub remembers. The archive feeds back into production: yesterday's footage is tomorrow's playlist, a deleted thread is still citable, and Telegram-channel history becomes reviewable material with attribution intact.
|
||||
|
||||
## Security & trust model
|
||||
|
||||
Self-hosted, end to end — no corporate cloud, no accounts needed to run the system itself. Multi-tenant with row-level security; platform credentials in an encrypted vault the database never sees in plaintext; every editorial action in a hash-chained audit log; anonymous-infrastructure friendly (the host carries boxes, not content — storage is ciphertext).
|
||||
|
||||
## What we need from you
|
||||
|
||||
| Piece | Spec | Notes |
|
||||
|---|---|---|
|
||||
| App server | 4 vCPU / 8 GB / 100 GB | API, newsroom web, scheduler, live compositor (ffmpeg), Postgres, Redis |
|
||||
| Media server (optional, same box OK) | 4 vCPU / 8 GB + S3 storage | PeerTube channel + VODs; RTMP ingest :1935 |
|
||||
| Object storage | S3-compatible, 500 GB+ | MinIO works; content is client-side encrypted |
|
||||
| Domain + DNS | you have it | newsroom / channel / chat subdomains; anonymous registration |
|
||||
| Uplink | 100 Mbps+ recommended | streaming out |
|
||||
| Optional | tiny Matrix box (1 vCPU/1 GB) | encrypted team chat + Telegram bridge |
|
||||
|
||||
**Status, honestly:** publishing + review + archive + connectors + OAuth connecting (Mastodon full, Telegram bots) run today; multi-signoff quorum, live compositor, control-room UI, unified chat and auto-director are landing now. Bring the servers and the domain — the collective brings the newsroom.
|
||||
|
||||
*— the Tactical Media Broadcast Hub collective (Tactical Media Broadcast Hub × Opticast × PeerTube)*
|
||||
169
docs/tactical-media-broadcast-hub-brief.md
Normal file
169
docs/tactical-media-broadcast-hub-brief.md
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
# Tactical Media Broadcast Hub
|
||||
|
||||
## Many collectives. Many minds. Many signals.
|
||||
|
||||
Media activists often face the same problem: a group wants to work together, but the tools are organized around individual accounts.
|
||||
|
||||
One person ends up holding the passwords. A few people become bottlenecks. New contributors cannot help without being given dangerous levels of access. Work gets scattered across chats, documents, drives, and platform dashboards.
|
||||
|
||||
The Tactical Media Broadcast Hub is designed to change that.
|
||||
|
||||
It gives collectives a shared place to organize, create, review, and publish media without turning one person into the gatekeeper of every account.
|
||||
|
||||
**Speak with us, not for us.**
|
||||
|
||||
## What Groups Can Do
|
||||
|
||||
### Work together without sharing passwords
|
||||
|
||||
People get the access they need for the work they are doing, not the keys to everything.
|
||||
|
||||
A contributor can be allowed to:
|
||||
|
||||
- Prepare a post.
|
||||
- Edit a video description.
|
||||
- Review another person’s work.
|
||||
- Publish to a specific channel.
|
||||
- Manage a group or project.
|
||||
- Remove their own access when they move on.
|
||||
|
||||
The group can see who has access and change it when responsibilities change.
|
||||
|
||||
### Organize around projects and campaigns
|
||||
|
||||
Create groups for a campaign, a local action, a publication, a show, or a temporary media team.
|
||||
|
||||
Each group can have its own members, conversations, files, drafts, approval process, and publishing permissions. Groups can work independently while still sharing material and supporting one another.
|
||||
|
||||
### Create channels together
|
||||
|
||||
Groups can build media channels that span several platforms instead of treating every platform as a separate job.
|
||||
|
||||
The same team can coordinate:
|
||||
|
||||
- Video and livestreams.
|
||||
- Social posts and threads.
|
||||
- Telegram and Matrix conversations.
|
||||
- Press statements and newsletters.
|
||||
- Archives, show notes, and background material.
|
||||
|
||||
The Hub helps people adapt work for each place without losing track of the original source, the people who made it, or the decisions around publication.
|
||||
|
||||
### Produce livestreams and TV-style channels
|
||||
|
||||
Groups can produce more than individual posts. They can operate live and scheduled channels with:
|
||||
|
||||
- Multiple live sources and remote contributors.
|
||||
- Playlists, scheduled programs, and looping broadcasts.
|
||||
- Scene layouts, picture-in-picture, shared screens, and source changes.
|
||||
- Lower thirds, logos, chyrons, clocks, and other on-air graphics.
|
||||
- Preview and program feeds so a team can prepare the next moment before taking it live.
|
||||
- Live switching, source replacement, and graceful recovery when a source drops.
|
||||
- Simultaneous distribution to PeerTube and other supported platforms.
|
||||
- Recordings, replays, clips, and show archives after the broadcast.
|
||||
|
||||
This makes a group channel feel like a small community television station: live when something is happening, scheduled when it is not, and built from the work of many contributors.
|
||||
|
||||
### Edit documents together
|
||||
|
||||
Groups can work together on statements, articles, scripts, captions, briefings, and press materials.
|
||||
|
||||
Office documents such as ODT files can be edited collectively. People can review changes, prepare a final version, and keep the publication history with the project instead of passing files around in private messages.
|
||||
|
||||
### Use shared pads for fast work
|
||||
|
||||
Every project also needs a lightweight space for unfinished thoughts.
|
||||
|
||||
Shared pads are useful for:
|
||||
|
||||
- Live event notes.
|
||||
- Breaking-news coordination.
|
||||
- Interview preparation.
|
||||
- Production schedules.
|
||||
- Run sheets and checklists.
|
||||
- Meeting notes.
|
||||
- Quick drafts that do not need a full document workflow.
|
||||
|
||||
Pads are for moving quickly. Documents are for work that needs editing, review, and a lasting home.
|
||||
|
||||
### Find group files and media
|
||||
|
||||
Files should not disappear into one person’s private drive.
|
||||
|
||||
Groups can keep shared images, audio, video, documents, exports, source material, and finished work together with useful context:
|
||||
|
||||
- Which group owns or manages it.
|
||||
- Where it came from.
|
||||
- Who contributed it.
|
||||
- When it was created.
|
||||
- What project or campaign it belongs to.
|
||||
- Whether it is a draft, approved, published, or archived.
|
||||
|
||||
Members should be able to find the file connected to a conversation, a campaign, or a published piece without searching through somebody else’s account.
|
||||
|
||||
### Curate and amplify voices
|
||||
|
||||
The Hub is for finding, organizing, contextualizing, and amplifying work made by people and communities.
|
||||
|
||||
It is not a machine for speaking on behalf of them. Attribution, consent, authorship, and editorial decisions remain visible to the people doing the work.
|
||||
|
||||
## How A Group Uses It
|
||||
|
||||
1. A collective creates a group for a project, campaign, channel, or event.
|
||||
2. Members receive roles that match their responsibilities.
|
||||
3. The group gathers conversations, files, source material, and drafts.
|
||||
4. Members edit and review work together.
|
||||
5. The group decides what is ready to publish.
|
||||
6. Approved work is adapted for the platforms where people will see it.
|
||||
7. The group keeps a record of what was made, who contributed, and where it went.
|
||||
|
||||
## What This Is Not
|
||||
|
||||
- Not a replacement for the communities using it.
|
||||
- Not a single centralized newsroom.
|
||||
- Not a reason to hand one administrator every password.
|
||||
- Not an advertising or engagement farm.
|
||||
- Not a system that hides editorial decisions behind one organization.
|
||||
- Not a request for someone else to provide or operate servers.
|
||||
|
||||
## A Network Of Independent Collectives
|
||||
|
||||
Each collective should be able to decide how it works, who it trusts, where it stores its material, and what it shares.
|
||||
|
||||
The Hub helps groups cooperate without requiring them to become one organization. A local media team, a campaign group, a publication, and a network of independent channels can all coordinate while keeping their own identities and responsibilities.
|
||||
|
||||
The long-term aim is a horizontally coordinated network of media-making communities:
|
||||
|
||||
**Many collectives. Many minds. Many signals.**
|
||||
|
||||
## For Groups Who Want To Run It Themselves
|
||||
|
||||
The Hub can be run independently by a collective or network. They do not need to depend on the original maintainers for access to their own work.
|
||||
|
||||
An independent operator needs:
|
||||
|
||||
- A reliable computer or hosted machine.
|
||||
- A stable internet connection and a domain name.
|
||||
- A secure place for shared files and media.
|
||||
- Someone responsible for updates, backups, and member access.
|
||||
- A clear agreement about who can publish, administer, and recover the system.
|
||||
|
||||
The system is designed so large files and media live in durable object storage rather than filling the application machine. Small records about groups, permissions, conversations, and file locations still need to be protected and backed up.
|
||||
|
||||
For a larger deployment, it is best to separate the everyday coordination tools from heavy video processing. A group can begin small and add dedicated capacity for livestreaming, video production, collaborative documents, and search as its work grows.
|
||||
|
||||
Self-hosting should include:
|
||||
|
||||
- Separate storage areas for chat media, shared files, and application media.
|
||||
- Regular backups of both the files and the records that describe them.
|
||||
- Secure delegated connections to publishing platforms.
|
||||
- Clear rules for access, departures, moderation, and recovery.
|
||||
- A policy for which group conversations can have files copied into the shared library.
|
||||
|
||||
Encrypted conversations need special care: a server cannot automatically read or archive files from a conversation it cannot decrypt. Groups can choose designated archive spaces for material that needs to be searchable and shared across the wider media system.
|
||||
|
||||
## Closing
|
||||
|
||||
**Tools to curate and amplify voices.**
|
||||
|
||||
**Speak with us, not for us.**
|
||||
BIN
docs/tactical-media-broadcast-hub-brief.pdf
Normal file
BIN
docs/tactical-media-broadcast-hub-brief.pdf
Normal file
Binary file not shown.
268
docs/tactical-media-broadcast-hub-technical-brief.md
Normal file
268
docs/tactical-media-broadcast-hub-technical-brief.md
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# Tactical Media Broadcast Hub
|
||||
|
||||
## Independent Self-Hosting: Technical Brief
|
||||
|
||||
This document describes what an independent collective needs to operate its own Tactical Media Broadcast Hub.
|
||||
|
||||
It is intentionally separate from the public presentation brief. The public brief explains what people can do with the Hub; this document explains the systems, storage, security, and operational responsibilities behind it.
|
||||
|
||||
## 1. Operating Principle
|
||||
|
||||
The Hub should keep user file and media payloads in durable object storage, not on the application host.
|
||||
|
||||
The practical boundary is:
|
||||
|
||||
- **R2/S3:** user files, media, document payloads, pad exports, and large objects.
|
||||
- **Local service state:** small metadata databases, indexes, room state, configuration, queues, and temporary processing data.
|
||||
|
||||
A literally zero-local-state deployment is not supported by every component. The goal is zero local storage for user payloads, with the remaining metadata protected by backups and volume snapshots.
|
||||
|
||||
## 2. Recommended System Layout
|
||||
|
||||
### Control plane
|
||||
|
||||
- Tactical Media Broadcast Hub web application.
|
||||
- API and authentication.
|
||||
- PostgreSQL.
|
||||
- Redis.
|
||||
- Temporal or an equivalent workflow service.
|
||||
- Scheduler and media-worker control services.
|
||||
|
||||
### Conversation plane
|
||||
|
||||
- Tuwunel Matrix homeserver.
|
||||
- mautrix-telegram bridge.
|
||||
- Element or another Matrix client.
|
||||
- A small archive bot for approved rooms and attachments.
|
||||
|
||||
The current deployment uses Conduwuit, but Conduwuit is archived and does not provide the native S3 media storage needed for the long-term design. Plan a controlled Conduwuit-to-Tuwunel migration before moving Matrix media to R2.
|
||||
|
||||
### Media plane
|
||||
|
||||
- PeerTube.
|
||||
- RTMP/RTMPS ingest.
|
||||
- Transcoding and broadcast workers.
|
||||
- Preview and archive processing.
|
||||
|
||||
The media plane should be separate from the files/docs plane when the collective runs sustained livestreams or video processing.
|
||||
|
||||
The broadcast system supports persistent virtual channels rather than only one-off uploads. A channel can combine live sources, playlists, and prepared programs into a continuous output.
|
||||
|
||||
Core production capabilities include:
|
||||
|
||||
- Source ingest from live feeds, PeerTube, files, and playlists.
|
||||
- Scene composition with grids, picture-in-picture, shared sources, and custom layouts.
|
||||
- Preview and program buses for preparing and taking a source live.
|
||||
- A/B switching and frame-continuous source replacement.
|
||||
- Lower thirds, chyrons, logos, clocks, and other typed overlays.
|
||||
- Per-source audio selection, gain, mixing, and master output control.
|
||||
- Scheduled playlists with slate fallback when an asset is unavailable.
|
||||
- RTMP/RTMPS output to PeerTube and other supported destinations.
|
||||
- JPEG and low-latency HLS preview taps for operators and control-room clients.
|
||||
- Recording, replay, archive, and later clip/export workflows.
|
||||
|
||||
The media worker is the production engine. The control plane owns channel state, permissions, schedules, and signed commands; it should not carry raw video bytes.
|
||||
|
||||
### Files and documents plane
|
||||
|
||||
- ownCloud Infinite Scale (oCIS) for folders, group files, search, versions, and share links.
|
||||
- Collabora CODE or OnlyOffice for collaborative ODT and office-document editing.
|
||||
- Tika or equivalent content extraction if full-text search is required.
|
||||
|
||||
oCIS is the recommended primary files layer because its S3NG storage driver keeps file blobs in R2 while retaining only necessary metadata on a local POSIX volume.
|
||||
|
||||
### Pad plane
|
||||
|
||||
- Etherpad or HedgeDoc for fast shared pads.
|
||||
- Pads are for live notes, run sheets, planning, and event coordination.
|
||||
- Pad text and revisions should remain in the pad service, with exports stored in R2 when retained as formal project records.
|
||||
|
||||
Matrix is the conversation layer, not the office editor or files library.
|
||||
|
||||
## 3. R2 Bucket Layout
|
||||
|
||||
Use separate buckets. Do not share an exclusive or opaque application bucket between services.
|
||||
|
||||
The initial dedicated buckets are:
|
||||
|
||||
| Bucket | Purpose |
|
||||
|---|---|
|
||||
| `hub-matrix-media` | Matrix/Tuwunel media payloads |
|
||||
| `hub-files-blobs` | oCIS file and document blobs |
|
||||
| `hub-media-library` | Tactical Media Broadcast Hub working media and library assets |
|
||||
| `hub-pads` | Retained pad exports and attachments |
|
||||
|
||||
Each service needs its own scoped R2 S3 credential. Cloudflare account API credentials are not the same thing as R2 S3 access keys.
|
||||
|
||||
Do not place credentials in Git. Store them in an encrypted secret manager or protected deployment environment.
|
||||
|
||||
## 4. Group File Contract
|
||||
|
||||
Every archived group file should retain enough context to answer: what is this, who contributed it, where did it come from, and who may use it?
|
||||
|
||||
Minimum metadata:
|
||||
|
||||
- `workspaceId`
|
||||
- `groupId`
|
||||
- `roomId`
|
||||
- `eventId`
|
||||
- `senderId`
|
||||
- `origin` such as Matrix, Telegram bridge, upload, or application
|
||||
- `filename`
|
||||
- `mimeType`
|
||||
- `sizeBytes`
|
||||
- `sha256`
|
||||
- `createdAt`
|
||||
- `storageKey`
|
||||
- `shareUrl` when one exists
|
||||
- `retention`
|
||||
- `status` such as pending, archived, published, revoked, or deleted
|
||||
|
||||
The Hub database stores metadata and indexing information. The file bytes live in the appropriate R2 bucket.
|
||||
|
||||
Archive writes must be idempotent on the combination of workspace, room, and event ID. A retried sync event must not create duplicate files.
|
||||
|
||||
## 5. Matrix and Telegram Media Flow
|
||||
|
||||
1. Telegram content enters through mautrix-telegram.
|
||||
2. The Matrix room remains the conversation and attribution source.
|
||||
3. An archive bot watches designated rooms.
|
||||
4. For eligible file/image/video/audio events, the bot streams the payload into the files platform or R2-backed archive.
|
||||
5. The bot records room, sender, event, origin, checksum, and retention metadata.
|
||||
6. The bot can post a share link back to the room instead of duplicating the file.
|
||||
|
||||
### Archive policy
|
||||
|
||||
Encrypted rooms cannot be read by a server-side archive bot without a deliberate decryption path. The default safe policy is:
|
||||
|
||||
- Archive only designated unencrypted archive rooms.
|
||||
- Do not silently decrypt or mirror private E2E rooms.
|
||||
- Make the archive policy visible to room members.
|
||||
- Preserve attribution and the original event ID.
|
||||
|
||||
The first implementation should be a plain sync-loop bot using matrix-js-sdk, not a full appservice. An appservice can be considered later if room volume requires it.
|
||||
|
||||
## 6. Collaborative ODT Editing
|
||||
|
||||
ODT and other office documents should be stored in oCIS and opened through a WOPI-compatible editor such as Collabora or OnlyOffice.
|
||||
|
||||
The Hub should store:
|
||||
|
||||
- The document identity.
|
||||
- The owning group and workspace.
|
||||
- The current share or launch reference.
|
||||
- Review and publication status.
|
||||
- A rendered export or snapshot when a document is approved.
|
||||
|
||||
The Hub should not copy the full document revision history into its own database. The files platform and office editor remain responsible for collaborative editing and document versions.
|
||||
|
||||
## 7. Shared Pads
|
||||
|
||||
Pads should have a small integration boundary:
|
||||
|
||||
- Group and room ownership.
|
||||
- Pad URL and service identity.
|
||||
- Membership and access policy.
|
||||
- Export-to-file action.
|
||||
- Optional retention policy.
|
||||
|
||||
Pad content stays with the pad service during editing. When a pad becomes an official project artifact, export it to ODT, Markdown, HTML, or PDF and store the export in R2-backed group files.
|
||||
|
||||
## 8. Access and Security
|
||||
|
||||
- Use group roles rather than shared account passwords.
|
||||
- Prefer OAuth/OIDC/delegated authorization for external platforms.
|
||||
- Scope each platform connection to the smallest useful permission set.
|
||||
- Keep publishing, reviewing, editing, and administration as separate capabilities.
|
||||
- Record access changes and publication actions in the audit trail.
|
||||
- Rotate credentials when a member leaves or a platform connection changes.
|
||||
- Use separate R2 credentials per bucket and service.
|
||||
- Do not put R2 secrets, Matrix registration tokens, or platform refresh tokens in the repository.
|
||||
|
||||
## 9. Compute and Capacity
|
||||
|
||||
### Small pilot
|
||||
|
||||
- 4 vCPUs and 8 GB RAM is a comfortable starting point for control, chat, and files without heavy video processing.
|
||||
- 2 vCPUs and 4 GB RAM can support a reduced pilot, but not the full stack with Collabora, search, PeerTube, and sustained livestreaming.
|
||||
- Keep large media processing on a separate host whenever possible.
|
||||
|
||||
### Full deployment
|
||||
|
||||
The honest target for the complete Hub is approximately 4 vCPUs and 8 GB RAM or more. The main elastic components are:
|
||||
|
||||
- Collabora/OnlyOffice document sessions.
|
||||
- Tika content extraction.
|
||||
- PeerTube transcoding.
|
||||
- Live broadcast workers.
|
||||
- Initial search/indexing jobs.
|
||||
|
||||
Disk capacity is less important than keeping payloads in R2. A self-hosting operator should still maintain enough local space for metadata volumes, temporary processing, logs, and recovery snapshots.
|
||||
|
||||
## 10. Domains and Network
|
||||
|
||||
At minimum, an independent deployment needs:
|
||||
|
||||
- One main web domain.
|
||||
- Separate hostnames for API, Matrix, files, collaboration, pads, and video.
|
||||
- HTTPS for every public service.
|
||||
- A tunnel or reverse proxy.
|
||||
- Long upload and WebDAV/WOPI timeouts.
|
||||
|
||||
Large-upload hostnames should not use a proxy path with a low request-size limit. Use a tunnel or grey-cloud DNS path, or use chunked/TUS uploads.
|
||||
|
||||
## 11. Backups and Recovery
|
||||
|
||||
Back up both payloads and the metadata that makes them meaningful:
|
||||
|
||||
| Artifact | Backup |
|
||||
|---|---|
|
||||
| R2 buckets | Replication, lifecycle protection, or a second-account backup |
|
||||
| Tuwunel room state | Volume snapshot and config backup |
|
||||
| oCIS metadata volume | Volume snapshot; use a filesystem suited to the service |
|
||||
| Hub PostgreSQL | Regular `pg_dump` plus tested restore |
|
||||
| Redis/Temporal | Rebuildable state documented; preserve durable workflow data where required |
|
||||
| Deployment config | Versioned config with secrets stored separately |
|
||||
|
||||
Test restoring a group, its files, document links, and publication history onto a clean host. R2 durability alone does not protect against a bad lifecycle rule, deleted metadata, or leaked credentials.
|
||||
|
||||
## 12. Deployment Phases
|
||||
|
||||
### Phase 1: Foundation
|
||||
|
||||
- Create separate R2 buckets and scoped S3 credentials.
|
||||
- Deploy the files platform without migrating existing Matrix or PeerTube data.
|
||||
- Establish group/file metadata contracts.
|
||||
- Add backup and restore procedures.
|
||||
- Add a basic pad service.
|
||||
|
||||
### Phase 2: Collaboration
|
||||
|
||||
- Add Collabora or OnlyOffice.
|
||||
- Connect ODT launch and export flows.
|
||||
- Add group library views and share links.
|
||||
- Add Matrix room-to-group mappings.
|
||||
|
||||
### Phase 3: Archive integration
|
||||
|
||||
- Upgrade Conduwuit to Tuwunel with the permanent server name preserved.
|
||||
- Enable Tuwunel media storage providers for R2.
|
||||
- Run the archive bot for designated unencrypted rooms.
|
||||
- Add event-to-file indexing and share-link replies.
|
||||
|
||||
### Phase 4: Production hardening
|
||||
|
||||
- Add OIDC or another shared identity layer.
|
||||
- Add retention and deletion workflows.
|
||||
- Add search extraction and indexing.
|
||||
- Run restore drills and permission audits.
|
||||
- Separate heavy media processing onto its own host.
|
||||
|
||||
## 13. Explicit Non-Goals
|
||||
|
||||
- Do not run Nextcloud AIO as the default; it is local-datadir-oriented and wants to own the Docker host.
|
||||
- Do not use Seafile CE if R2 storage and content search are requirements.
|
||||
- Do not make Matrix the files library.
|
||||
- Do not run a second chat platform beside Matrix.
|
||||
- Do not migrate live Matrix data to R2 until the Tuwunel migration is planned and backed up.
|
||||
- Do not claim that user payloads are R2-backed until the service has been configured with real scoped R2 credentials and verified with an upload/restore test.
|
||||
BIN
docs/tactical-media-broadcast-hub-technical-brief.pdf
Normal file
BIN
docs/tactical-media-broadcast-hub-technical-brief.pdf
Normal file
Binary file not shown.
BIN
docs/tmbh-promo.pdf
Normal file
BIN
docs/tmbh-promo.pdf
Normal file
Binary file not shown.
34
docs/tweetdeck-on-steroids-ES.md
Normal file
34
docs/tweetdeck-on-steroids-ES.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# TweetDeck con Esteroides
|
||||
|
||||
**Un newsroom multiplataforma autoalojado para colectivos — un muro de columnas que vigila todas las plataformas a la vez, conectado a un pipeline completo de transmisión en vivo.**
|
||||
|
||||
TweetDeck murió cuando X cerró sus puertas. Construimos lo que debió reemplazarlo: columnas que vigilan Mastodon, Telegram, Bluesky, PeerTube, Twitch y más — conectadas a publicación, aprobaciones, mezcla en vivo y un archivo que no se puede purgar. Autoalojado, sin algoritmo, sin plataformas guardando tus llaves.
|
||||
|
||||
## Cómo trabajamos juntos
|
||||
|
||||
- **El muro es la sala.** Columnas en vivo vigilan todas tus cuentas conectadas a la vez — cada post renderizado seguro, media proxy, duplicados deduplicados al archivo. Todo el colectivo ve la misma imagen.
|
||||
- **Redacta una vez, aprueben juntos, publiquen en todas partes.** Cualquiera redacta en cualquier cuenta. Cada borrador pasa por aprobación configurable del grupo antes de moverse. Al aprobarse, se distribuye nativo por plataforma — citas correctas, atribución intacta, auditoría a prueba de manipulación de quién decidió qué.
|
||||
- **Roles, no contraseñas.** Editores redactan, revisores aprueban, publicadores publican — por cuenta, por canal. Enlaces de invitación integran a la gente; irse quita el acceso. Sin logins compartidos, sin cuellos de botella.
|
||||
- **Programa como una estación.** Posts programados con zona horaria correcta y motor resistente al horario de verano. Cancela, revisa, reprograma.
|
||||
- **Sala de control en el navegador.** Cuadrícula multivista, TAKE/CUT, preview/programa, lower-thirds y chyrons, mezcla de audio — el programa sale a tu canal de PeerTube, playlist 24/7 cuando estás apagado. Funciona desde una laptop.
|
||||
- **Auto-Director.** Reglas de prioridad cambian fuentes automáticamente para que el stream nunca muera en medio de la acción.
|
||||
- **Chats de todas las plataformas** puenteados a un solo overlay — espectadores escuchados, moderadores actuando, el streamer enfocado en el shot.
|
||||
- **Workers de transcodificación propios.** Laptops y GPUs como workers de render por Tailscale — una RTX 3090 ya transcodifica trabajos reales de PeerTube con NVENC. Más workers, más canales.
|
||||
- **Un archivo que no se puede purgar.** Cifrado, deduplicado, direccionado por contenido. Las plataformas eliminan; el archivo recuerda.
|
||||
|
||||
|
||||
## Lo que viene — y dónde necesitamos manos
|
||||
|
||||
- **Columnas de búsqueda entre plataformas.** Una consulta ("acampada26") como columna en vivo que extrae de cada red que lo permita — diseño listo, esquema en construcción.
|
||||
- **Chats dirigidos por la asamblea.** Encuestas dentro del chat puenteado: "¿publicamos esto?" — pasa con x/y administradores o x/y participantes, configurable por encuesta — y un bot publica automáticamente al aprobarse.
|
||||
- **Bots de publicación configurables.** Cada cuenta tiene sus propias reglas: inmediato, por quórum, solo administradores. Por tema, por tipo de contenido.
|
||||
- **Subchats con equipos.** Temas de foro como grupos de trabajo; cuentas asignadas a temas; cada tema con sus propias reglas de publicación.
|
||||
- **El chat es el control remoto.** Bots de Telegram manejan la transmisión: /take, /cut, cambiar fuentes — mapeados a tu rol, con confirmación, todo auditado.
|
||||
- **Archivos del chat al almacenamiento.** Lo que se publica en el chat llega al almacenamiento del grupo; admins lo organizan en carpetas por comando del bot o pantalla móvil; busca en toda la biblioteca desde el chat.
|
||||
- **Más puertas:** conectores de YouTube y TikTok, crossposting con citas inteligentes, y un bot de ingesta comunitaria — reenvía un URL, lo fetchamos y archivamos.
|
||||
|
||||
**Stack:** Next.js + NestJS + Postgres (RLS) + Temporal + Redis + S3 + FFmpeg/PeerTube. Autoalojado de punta a punta, multi-tenant, todo auditado.
|
||||
|
||||
## Buscamos: testers y hackers de ingesta
|
||||
|
||||
Prueba todo el flujo con tus feeds y cuentas reales. Y ayúdanos con el **problema difícil: ingesta a escala** — formas confiables de extraer contenido de las plataformas cerradas sin quemar cuentas. Si has construido scrapers o corrido pipelines de ingesta, queremos tu diseño.
|
||||
26
docs/tweetdeck-on-steroids.md
Normal file
26
docs/tweetdeck-on-steroids.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# TweetDeck on Steroids
|
||||
|
||||
**A self-hosted, multi-platform broadcast newsroom for collectives — a column wall that watches every platform at once, wired into a full live broadcast pipeline.**
|
||||
|
||||
TweetDeck died when X locked its doors. We built what should have replaced it: columns that watch Mastodon, Telegram, Bluesky, PeerTube, Twitch and more in one wall — wired into publishing, approvals, live mixing, and an archive that can't be purged. Self-hosted. No algorithm in between. No platform holding your keys.
|
||||
|
||||
## How people work together in it
|
||||
|
||||
- **The wall is the room.** Live columns watch all your connected accounts at once — every post rendered safe, media proxied, duplicates deduplicated into the archive. The whole collective sees the same picture.
|
||||
- **Draft once, approve together, publish everywhere.** Anyone can draft into any account. Every draft goes through configurable group approval before it moves. On sign-off it fans out natively per platform — proper quotes, attribution intact, tamper-evident audit of who decided what.
|
||||
- **Roles, not passwords.** Editors draft, reviewers approve, publishers publish — per account, per channel. Invite links bring people in; leaving removes access. No shared logins, no bottlenecks.
|
||||
- **Schedule like a station.** Timezone-correct scheduled posts with a DST-proof occurrence engine. Cancel, review, re-schedule — the calendar survives clock changes.
|
||||
- **Live control room in the browser.** Multiview grid, TAKE/CUT, preview/program, lower-thirds and chyrons, audio mixing — program out to your PeerTube channel, 24/7 playlist when you're dark. Works from a laptop.
|
||||
- **Auto-Director.** Priority rules switch sources automatically so the stream never dies mid-action.
|
||||
- **Chat from every platform** bridged into one overlay — viewers heard, moderators acting, the streamer focused on the shot.
|
||||
- **Your own transcode workers.** Laptops and GPUs join as render workers over Tailscale — an RTX 3090 laptop is already transcoding real PeerTube jobs. More workers, more channels, all self-hosted.
|
||||
- **An archive that can't be purged.** Content-addressed, encrypted, deduplicated. Platforms purge; the archive remembers. Deleted threads stay citable.
|
||||
- **Analytics.** Month-by-month snapshots: what published where, what failed, what got archived.
|
||||
|
||||
## What we need: testers and scrapers
|
||||
|
||||
We're looking for **testers** across the flow — post through approvals, run a column wall against your real feeds, mix a stream, break the scheduler across timezones. And we're looking for help on the **hard problem: intake at scale.** Watching RSS, the fediverse and Telegram is solved. What we want next: reliable ways to pull from the locked platforms — hashtags, accounts, search columns — without burning accounts. If you've built scrapers, run intake pipelines, or know the platform-integrity arms race from the inside, help us design the intake layer: per-URL fetchers, community-submitted links, archive harvesters, bridge services.
|
||||
|
||||
## Stack
|
||||
|
||||
Next.js + NestJS + Postgres (RLS) + Temporal + Redis + S3 + FFmpeg/PeerTube. Self-hosted end to end, multi-tenant, everything audited.
|
||||
79
docs/user-guide.md
Normal file
79
docs/user-guide.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# Tactical Media Broadcast Hub — User Guide
|
||||
|
||||
Post, schedule, monitor, and amplify across your social platforms from one dark, keyboard-friendly console. Admin setup help: [admin-guide.md](./admin-guide.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Getting started (onboarding)
|
||||
|
||||
The 3-step wizard (Onboarding screen):
|
||||
|
||||
1. **Name your workspace** — the tenant boundary for accounts, drafts, and the audit trail.
|
||||
2. **Invite your team** — comma-separated emails; roles are assigned after they accept.
|
||||
3. **Connect your first accounts** — pick platforms and Connect.
|
||||
|
||||
Only OAuth tokens (or bot tokens, for Telegram) are ever stored — **the system never asks for or keeps platform passwords**. Tokens live encrypted in the workspace vault; you can rotate or revoke them any time.
|
||||
|
||||
## 2. Streams — your monitoring console
|
||||
|
||||
The Streams screen is TweetDeck-style columns:
|
||||
|
||||
- **Columns** track a feed per platform (Home, Mentions, saved searches). *(Configurable per-column subscriptions — accounts, lists, search terms — arrive with the archive/grabber update.)*
|
||||
- **Cards** show author, age, text, and engagement. Hover or navigate to a card and its **amplify bar** appears.
|
||||
- **Amplify buttons are honest:** each platform's API is different, so each column shows only what that platform truly supports. Mastodon: **Boost** and **Quote**. Telegram: **Forward**. Some platforms show *"no amplify — this platform's API exposes no reshare action"*. That's not a bug; it's us not pretending.
|
||||
|
||||
### Keyboard first
|
||||
| Key | Action |
|
||||
|---|---|
|
||||
| `j` / `↓` | next item |
|
||||
| `k` / `↑` | previous item |
|
||||
| `r` | repost/boost the selected item |
|
||||
| `q` | quote the selected item (opens composer popover with char limit) |
|
||||
| `Esc` | close popover |
|
||||
| `⌘K` / `Ctrl+K` | command palette — jump anywhere, quick actions |
|
||||
|
||||
## 3. Compose — write once, adapt everywhere
|
||||
|
||||
- Rich editor with markdown shortcuts; **per-platform character counters** — add targets (Mastodon, Telegram, Facebook, Instagram) and the limit shown is the *tightest* one; each preview pane shows its own count and turns red when over.
|
||||
- **Visibility** (public / followers / private), **approval state** (draft / submit for review / approved), and **scheduling** (date-time) per post.
|
||||
- Right pane previews each target platform.
|
||||
|
||||
## 4. The approval flow
|
||||
|
||||
If your workspace uses approvals: write → **Submit for review** → an Approver gets it in Approvals → **Grant** (or reject with comments) → schedule or publish. Every step is permission-checked and lands in the audit trail. (Permissions are set by your workspace admin — see the roles table in the admin guide.)
|
||||
|
||||
## 5. Inbox — DMs and conversations
|
||||
|
||||
All direct-message conversations across connected accounts in one list (unread badges, assignees). Open a thread, reply inline. Note the honesty footer: messaging APIs allow replies within each platform's message window (e.g., 24h for Meta platforms); outside it, options narrow — we show you exactly what's possible rather than failing silently.
|
||||
|
||||
## 6. Channels — your connected accounts
|
||||
|
||||
Each account card shows: platform, handle, **capability chips** (what this connection can do), and health. A **token expiring** badge means reconnect soon. The "Connect" flow for new accounts is OAuth — you authenticate on the platform's own page; we only ever receive a token.
|
||||
|
||||
## 7. Team — members, roles, assets
|
||||
|
||||
- **Members:** change roles from the 10 seeded roles (Owner, Admin, Publisher, Editor, Approver, Analyst, Moderator, VideoPublisher, StreamOperator, BroadcastOperator). Removing a member takes effect immediately — their live sessions fail closed.
|
||||
- **Asset library:** shared media grouped by collection (Brand, Launch, PR…). Items marked **group-locked** are scoped to a group; members outside it see the lock, not the file.
|
||||
|
||||
## 8. Audit trail — who did what
|
||||
|
||||
Every meaningful action (post published, role changed, connection added, credential rotated, approval granted…) is recorded in a **tamper-evident hash chain**. Filter by actor or action; click a row to open the **diff viewer** — what changed, field by field, with removed/added values. Secret values always show as `[REDACTED]`: keys and tokens are never written into the audit log, by construction. Export to CSV for compliance.
|
||||
|
||||
## 9. What's real vs. coming
|
||||
|
||||
**Now (Level 1):** monitoring streams, compose with previews, approvals, scheduling, publishing pipeline, encrypted credential vault, audit trail. *(The screens currently run on representative sample data while the backend wiring lands.)*
|
||||
|
||||
**Coming:**
|
||||
- **Archive & Grabber** — your search subscriptions run server-side; every matching post is archived with media downloaded (encrypted, deduplicated) and searchable; videos included.
|
||||
- **Crosspost** — one action moves content across platforms with your commentary (an Instagram image quoted into Mastodon, a Mastodon video into Facebook), flowing through approvals.
|
||||
- **Level 2** — video uploads, livestreams, the Virtual Channel / Auto Director master-control system, and the operator Control Room.
|
||||
|
||||
## 10. FAQ
|
||||
|
||||
**Why can't I retweet on some columns?** That platform's API doesn't expose resharing. We render only real capabilities.
|
||||
|
||||
**Why did everyone get logged out after a role change?** Security: authorization epochs bump and stale sessions fail closed. One re-login.
|
||||
|
||||
**Where are my passwords?** Nowhere. We store tokens only, envelope-encrypted. If you want to cut access, revoke the connection in-app or on the platform.
|
||||
|
||||
**Who can see my stream keys / tokens?** Nobody through the UI — not even workspace Owners; they're redacted from every screen, log, and audit entry by construction.
|
||||
374
docs/user-manual-v2.md
Normal file
374
docs/user-manual-v2.md
Normal file
|
|
@ -0,0 +1,374 @@
|
|||
# Tactical Media Broadcast Hub — User Manual v2
|
||||
|
||||
*Your platform. Their feeds. No algorithm in between.*
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Build documented | commit `6d5dc7e` (main), web production build |
|
||||
| Verified on | 2026-10-07, against a running API, Postgres, Redis and Temporal |
|
||||
| Screenshots | `docs/manual/screenshots/` (1440×900 desktop, 390×844 mobile) |
|
||||
| Method | Every claim marked **live** was exercised end-to-end. See [§15 Capability table](#15-capability-table--what-actually-works) |
|
||||
|
||||
> **How to read this manual.** Each feature has one of four labels:
|
||||
>
|
||||
> - **live**: we ran a real round-trip through the UI and/or the API and checked the result.
|
||||
> - **sample data**: the screen shows built-in fixtures. Sometimes that happens only when the API is unreachable, and sometimes it always happens (this is noted).
|
||||
> - **partial**: the request works, but part of the pipeline behind it isn't wired yet.
|
||||
> - **stub**: a placeholder screen or an inert control.
|
||||
>
|
||||
> Nothing in this manual is described as working unless we exercised it.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
1. [Quickstart](#1-quickstart)
|
||||
2. [The shell: sidebar, session bar, command palette](#2-the-shell)
|
||||
3. [Onboarding: create a workspace, sign in](#3-onboarding)
|
||||
4. [Streams](#4-streams)
|
||||
5. [Compose](#5-compose)
|
||||
6. [Channels (Connections)](#6-channels-connections)
|
||||
7. [Video](#7-video)
|
||||
8. [Team](#8-team)
|
||||
9. [Audit](#9-audit)
|
||||
10. [Inbox](#10-inbox)
|
||||
11. [Calendar, Approvals, Analytics, Settings](#11-calendar-approvals-analytics-settings)
|
||||
12. [Telegram](#12-telegram)
|
||||
13. [Offline / sample-data mode](#13-offline--sample-data-mode)
|
||||
14. [Mobile](#14-mobile)
|
||||
15. [Capability table](#15-capability-table--what-actually-works)
|
||||
|
||||
---
|
||||
|
||||
## 1. Quickstart
|
||||
|
||||
1. **Open the app** and you land on **Onboarding**. Any protected screen redirects there when you have no session (verified: an anonymous visit to `/streams` is sent to `/onboarding`).
|
||||
2. **Create a workspace.** Enter a workspace name, your name, email, and a password of at least 10 characters, then press **Continue**. This creates the workspace, your owner account and your session in one step (**live**).
|
||||
3. **Invite your team (optional).** Enter comma-separated emails. Each invitee is added as **Editor** with status *invited* (**live**). Email delivery and invite acceptance are not wired yet, so invitees can't sign in.
|
||||
4. **Connect accounts.** This step is **simulated**: the buttons only flip a "connected" badge locally. Nothing is stored. Press **Finish setup**.
|
||||
5. **Write a post** in **Compose** and press **Save draft**. The draft is stored and appears under *Recent drafts* (**live**). Publishing needs a connected channel, and channels can't be connected from the UI yet (see §6).
|
||||
6. **Register a video** in **Video** by dropping an MP4/WebM/MOV file. The asset and a queued processing job are created (**partial**: processing doesn't start, see §7).
|
||||
7. **Check Audit.** Every action above appears in the hash-chained audit trail (**live**).
|
||||
|
||||
> **What you can't do yet:** connect a real social account from the UI, publish to a platform from the UI, read real feeds/DMs, or use Calendar, Approvals, Analytics or Settings.
|
||||
|
||||
---
|
||||
|
||||
## 2. The shell
|
||||
|
||||
|
||||
|
||||
- **Sidebar.** Lists Streams (⌘1), Compose (⌘2), Calendar, Video, Inbox, Approvals, Analytics, Channels, Team, Audit, Onboarding and Settings, with the workspace name under the logo. The **3** next to Inbox and the **2** next to Approvals are fixture counts, not live counts. The footer card reads *"Level 1 · posting — Videos & Live arrive in Level 2"*.
|
||||
- **Session bar (top right).** Shows a green dot, your email and your role, plus **Sign out** (**live**). When the API is unreachable it reads *"offline — sample data"* instead.
|
||||
- **Command palette.** Press **⌘K / Ctrl+K** to jump to Streams, Compose, Video, Inbox, Channels, Team, Audit or Onboarding (**live**: navigation only, no actions).
|
||||
|
||||
|
||||
|
||||
**Session safety (live).** Changing a role, inviting or removing a member bumps the workspace's *authorization epoch*. Every older session token is then rejected with *"authorization epoch stale — re-authenticate"*. The web client automatically exchanges a stale token for a fresh one (`/auth/reissue`), so you stay signed in. We verified this: an old token returned 401 after a role change, and the client recovered transparently on the next screen load.
|
||||
|
||||
---
|
||||
|
||||
## 3. Onboarding
|
||||
|
||||
| Step | Screenshot | Status |
|
||||
|---|---|---|
|
||||
| Register (blank) |  | **live** |
|
||||
| Register (filled) |  | **live** |
|
||||
| Invite team |  | **live** (creates *invited* members; no email is sent) |
|
||||
| Connect accounts |  | **stub**: simulated toggle |
|
||||
| After clicking Connect on Telegram |  | **stub**: badge is local state only |
|
||||
| Done |  | **live** (shows the real workspace name) |
|
||||
| Sign in |  | **live** |
|
||||
| Sign in, wrong password |  | **live**: *"invalid email or password"* |
|
||||
| Visiting Onboarding while signed in |  | **live**: shows your email, role and workspace, with *Sign out* and *Continue to Streams* |
|
||||
|
||||
**How to use it**
|
||||
|
||||
- **Create a workspace.** All four fields are required (the password needs at least 10 characters). **Continue** stays disabled until they are valid.
|
||||
- **Sign in.** Click **I already have an account**, then enter your email and password. On success you go to Streams.
|
||||
- **Session expiry.** If your session expires or is revoked, the app sends you back here with a notice: *"Your session ended (expired, or your role changed). Sign in again."*
|
||||
- **Platform list.** The *Connect* step lists five fixture platforms: Mastodon, Telegram, "Limited API" (a demo placeholder), Facebook and Instagram. The connector packages cover more providers (Bluesky, Discord, Slack, PeerTube, Reddit, LinkedIn, Threads, Twitch…), but none of them are connectable from this screen.
|
||||
|
||||
---
|
||||
|
||||
## 4. Streams
|
||||
|
||||
**Status: sample data (always).** Streams columns are generated fixtures, even when you are signed in. The column headers say "live", but the content is not real feed data, and the screen shows no offline badge.
|
||||
|
||||
|
||||
|
||||
**What it is.** A TweetDeck-style multi-column console. The columns are *Home* (Mastodon), *Mentions* (Telegram) and *Saved: product* (Limited API). Each column states its amplification honesty line, for example *"no native reshare on this API — Quote… with attribution still works"*.
|
||||
|
||||
**Keyboard (works on the fixtures):**
|
||||
|
||||
| Key | Action |
|
||||
|---|---|
|
||||
| `j` / `↓` | select next card (highlighted, shows **Boost** / **Quote** buttons) |
|
||||
| `k` / `↑` | select previous card |
|
||||
| `r` | repost/boost the selected card (shows a toast; nothing is sent) |
|
||||
| `q` | open the quote modal |
|
||||
| `Esc` | close the modal |
|
||||
|
||||
| Selected card | Repost toast | Quote modal |
|
||||
|---|---|---|
|
||||
|  |  |  |
|
||||
|
||||
**+ Column** and **Layout** are inert (**stub**).
|
||||
|
||||
---
|
||||
|
||||
## 5. Compose
|
||||
|
||||
**Status: live (drafts) · partial (publish).**
|
||||
|
||||
| Empty | Typed | Draft saved |
|
||||
|---|---|---|
|
||||
|  |  |  |
|
||||
|
||||
**What it does**
|
||||
|
||||
- **Editor.** A rich-text (Tiptap) editor with a live character counter (`133 / 500`). The limit is the tightest one among the selected targets, or 500 when no target is selected.
|
||||
- **Targets.** When signed in, the target chips are your workspace's real connections (`GET /connections`). In a fresh workspace there are none, so the screen shows *"No channels connected — add one under Channels to publish"* and **Publish** stays disabled.
|
||||
- **Visibility.** public / followers / private.
|
||||
- **Approval.** draft / submit for review / approved.
|
||||
- **Schedule.** A date-time picker. Setting it turns **Publish** into **Schedule**.
|
||||
- **Save draft (live).** Stores the post and shows *"Draft saved."*. The draft appears under **Recent drafts** with its state badge (`editing`), and clicking a recent draft loads its text back into the editor. All verified in the UI; saving was also verified via the API.
|
||||
- **Publish (partial).** For each selected target, Publish runs create → submit → approve → schedule, stopping early if you lack a permission or choose *submit for review*.
|
||||
- Verified via the API: create (`editing`) → submit (`in_review`) → approve (`approved`). Scheduling against a connection that doesn't exist is correctly refused (404 *connection … not found*).
|
||||
- **Not verified:** publishing to a real platform. No workspace can obtain a real connection through the UI, and the dev seeding route `POST /demo/connections` currently returns 500. We couldn't create a target, so the publish button was never enabled during testing.
|
||||
- **Previews pane.** Shows one preview per selected target with its own counter. It is empty when there are no targets.
|
||||
|
||||
---
|
||||
|
||||
## 6. Channels (Connections)
|
||||
|
||||
**Status: live list (read-only) · stub for Connect/Reconnect/Remove · capacity chips need sample data or a real connection.**
|
||||
|
||||
| Live workspace (no channels yet) | Sample data (API unreachable) |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|
||||
**What it does**
|
||||
|
||||
- **Live.** Reads `GET /connections` for the workspace. A new workspace shows *"No channels connected yet"*, followed by one dashed *not connected* card per known platform.
|
||||
- **Connect buttons.** **Connect account**, **Connect**, **Reconnect** and **Remove** are **disabled when live**. Their tooltip reads: *"OAuth connect flows arrive with the connector wiring phase"*. **There is no way to add a channel from the UI today.**
|
||||
- **Each connected card** (seen in sample mode) shows:
|
||||
- a health badge (*healthy*, *token expiring*, *needs credentials*)
|
||||
- capability chips (`publishText`, `amplifyRepost`, `feed`, `directMessages`, `comments`…)
|
||||
- an *Amplify* line naming the platform's real reshare verbs (Mastodon: Boost · Quote; Telegram: Forward; Facebook: Share link; "not exposed by this API" where there is none)
|
||||
|
||||
### Capacity chips and popover
|
||||
|
||||
|
||||
|
||||
The chip on each connection reads either **`258/300 posts left`** or **`15 today · no cap`**. Click it to open the **Rolling 24h capacity** popover, which shows:
|
||||
|
||||
- a usage bar with *used / cap / remaining*
|
||||
- either *"slots available now — each post holds its slot for 24h, then frees it"* or *"next slot frees in …"*
|
||||
- a tier note, and a warning when a configured cap exceeds the platform's documented standard
|
||||
|
||||
The math is real: it uses the connector SDK's `capacitySnapshot` rate policy. In a live workspace the chip uses the connection's real `recentPublishes`. **We could only exercise the popover on sample data**, because no live connection could be created (see §6 above).
|
||||
|
||||
---
|
||||
|
||||
## 7. Video
|
||||
|
||||
**Status: live (register, list, detail, delete) · partial (processing).**
|
||||
|
||||
| Empty library | After upload: detail drawer |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|
||||
| Library grid | Delete confirmation | After delete |
|
||||
|---|---|---|
|
||||
|  |  |  |
|
||||
|
||||
**How to use it**
|
||||
|
||||
1. **Add a video.** Drop a file on *Drop a video here*, paste one, or click **Browse files**. MP4, WebM and MOV are accepted. Other types are refused (verified: a `.txt` file gave *"not-a-video.txt: not a video file (text/plain)"*). Drag-drop and paste use the same code path as Browse; we exercised only Browse.
|
||||
2. **What happens on upload.** The browser computes the file's SHA-256 and registers it with the API as `sha256/<hash>.<ext>` (`POST /video/upload`). The response creates a **video asset** and a **processing job** in state **queued** (**live**, verified twice in the UI and once via the API).
|
||||
3. **Detail drawer.** The drawer opens automatically after upload; close it with **Esc**. It shows:
|
||||
- the pipeline: **Queued → Probing (ffprobe) → Transcoding → Ready**
|
||||
- probe metadata (MIME type and size are filled in; duration, resolution, codecs and bitrate stay "–" until the probe runs)
|
||||
- publications
|
||||
- the asset ID
|
||||
4. **Auto-refresh.** The library refreshes every 6 seconds while any asset is in flight, and **Refresh** reloads it manually.
|
||||
5. **Delete.** Click **Delete** on a card to open the confirmation dialog: *"This removes the asset, its processing jobs, and its probe metadata… Published posts that reference it are not deleted."* Click **Delete video** to remove it and show *"Video deleted."* (**live**). The deletion appears in Audit as `video.deleted`.
|
||||
|
||||
**Why it's partial.** The API registers the asset but never receives the file bytes: there is no multipart upload route. The bytes are meant to reach object storage through the media-worker ingest path, which isn't wired. As a result:
|
||||
|
||||
- every job stays **queued at 0%** (verified 8 minutes after upload, and across auto-refresh cycles, as shown in `desktop-video-after-refresh.png`)
|
||||
- no probe metadata is ever filled in
|
||||
- no thumbnail is generated (cards show the logo placeholder)
|
||||
- cards show the hash, not the original filename
|
||||
|
||||
---
|
||||
|
||||
## 8. Team
|
||||
|
||||
**Status: live (members, invite, role change, remove) · sample data (asset library, always).**
|
||||
|
||||
|
||||
|
||||
**Members table (live):**
|
||||
|
||||
| Feature | What happened in testing |
|
||||
|---|---|
|
||||
| List | Shows member, role and status (*active* / *invited*) |
|
||||
| Invite | Entering an email + role and clicking **Send invite** gives *"Invited … as Editor."* and adds an *invited* row. Verified in the UI and via the API |
|
||||
| Change role | Uses the per-row dropdown. Verified via the API (Editor → Approver); the old session token was then rejected and the reissued token worked |
|
||||
| Remove | **Remove** → **Confirm remove**. Verified via the API (the UI button itself was not clicked) |
|
||||
| Roles | Admin, Analyst, Approver, BroadcastOperator, Editor, Moderator, Owner, Publisher, StreamOperator, VideoPublisher. Only an Owner can assign or modify Owner |
|
||||
|
||||
Invite emails and invite acceptance are **not wired**. The page says so itself.
|
||||
|
||||
**Asset library (lower half): sample data, always.** These six fixture files are not your workspace's assets. Real video assets live on the **Video** screen.
|
||||
|
||||
---
|
||||
|
||||
## 9. Audit
|
||||
|
||||
**Status: live · Export CSV is a stub.**
|
||||
|
||||
| Trail | Expanded entry (diff) |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|
||||
- **What it shows.** Every workspace action, newest first: actor (email, or `system` for API-side events such as `video.uploaded`), action, target and age. We saw these actions recorded by our own test run:
|
||||
- `workspace.created`
|
||||
- `membership.invited`
|
||||
- `draft.created` / `draft.submitted` / `draft.approved`
|
||||
- `video.uploaded` / `video.deleted`
|
||||
- `membership.role_changed` / `membership.removed` / `session.created` (from the API-driven run)
|
||||
- **Diff.** Click a row to expand its diff (for example `role "Editor"`, `email "…"`).
|
||||
- **Filters.** The **All actors** / **All actions** dropdowns narrow the list. Verified: filtering to `draft.approved` cut 11 rows to 1. *Load more* pages through older entries (not exercised: our test workspaces never exceeded one page).
|
||||
- **Integrity.** The trail is labelled *hash-chained · tamper-evident*, and secret-shaped keys render as `[REDACTED]`.
|
||||
- **Export CSV** has no handler. It is a **stub**.
|
||||
|
||||
---
|
||||
|
||||
## 10. Inbox
|
||||
|
||||
**Status: sample data (always).** Fixture conversations are shown even when signed in. DMs are Level 2: the connector `getMessages()` path isn't wired to the API. You can type in the reply box, but nothing is sent.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 11. Calendar, Approvals, Analytics, Settings
|
||||
|
||||
**Status: stub.** Each of these four screens is a placeholder reading *"Arrives with WP3 (publishing pipeline)."*
|
||||
|
||||
| Calendar | Approvals |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|
||||
| Analytics | Settings |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|
||||
The approval *state machine* does exist in the API: drafts move editing → in_review → approved, verified in §5. There is just no review-queue screen yet.
|
||||
|
||||
---
|
||||
|
||||
## 12. Telegram
|
||||
|
||||
**Status: connector-level only. There is no UI connect flow.**
|
||||
|
||||
- **What exists.** The `@tactical-media-broadcast-hub/connector-telegram` package publishes to channels and amplifies by **forwarding**:
|
||||
- `forwardMessage` for a plain forward
|
||||
- `copyMessage` when you add commentary, because a forward cannot carry a caption
|
||||
- removing an amplification is supported
|
||||
- **Auth.** Telegram uses a **bot token** created with @BotFather, not OAuth.
|
||||
- **Tests.** Its contract test suite passes (34/34, run on 2026-10-07). The tests run against a mocked Bot API, not a real bot.
|
||||
- **In the UI.** Telegram appears as a fixture column in Streams, a fixture card in Channels sample mode (capability `amplifyShare`, *Amplify: Forward*), and a simulated toggle in Onboarding. **None of these talk to Telegram.** There is no screen to enter a bot token, so a Telegram channel can't be connected from the app today.
|
||||
|
||||
---
|
||||
|
||||
## 13. Offline / sample-data mode
|
||||
|
||||
If the browser can't reach the API at all, the session bar shows *offline — sample data*, and the screens built on the API (Channels, Compose, Video, Team, Audit) render fixtures with an **"offline — showing sample data"** badge. Sample data is read-only: actions answer *"Offline — sample data is read-only."* or *"Offline — sample mode, nothing was sent."*. This mode is a development convenience, not a feature.
|
||||
|
||||
| Channels | Video | Team | Audit |
|
||||
|---|---|---|---|
|
||||
|  |  |  |  |
|
||||
|
||||
Compose and Onboarding in this mode: `desktop-compose-sample.png`, `desktop-onboarding-sample.png`.
|
||||
|
||||
---
|
||||
|
||||
## 14. Mobile
|
||||
|
||||
**Status: a mobile PWA is in progress. Today, the mobile experience is the desktop site on a phone.**
|
||||
|
||||
At 390×844 the full desktop sidebar stays fixed and takes roughly half the width. The content column is squeezed:
|
||||
|
||||
- Streams columns are a few characters wide.
|
||||
- **The Compose editor is collapsed to zero usable width.** Our automated test couldn't click into it at all ("element is not visible").
|
||||
- The onboarding inputs are only **84 px wide**, so registering or signing in on a phone is cramped at best.
|
||||
|
||||
Treat phones as **read-only, at best** until the PWA ships.
|
||||
|
||||
| Streams | Compose | Video | Onboarding (register) |
|
||||
|---|---|---|---|
|
||||
|  |  |  |  |
|
||||
|
||||
Other mobile captures:
|
||||
|
||||
- `mobile-connections.png`
|
||||
- `mobile-team.png`
|
||||
- `mobile-audit.png`
|
||||
- `mobile-inbox.png`
|
||||
- `mobile-calendar.png`
|
||||
- `mobile-approvals.png`
|
||||
- `mobile-settings.png`
|
||||
- `mobile-analytics.png`
|
||||
- `mobile-onboarding.png` (signed in)
|
||||
- `mobile-onboarding-signin.png`
|
||||
|
||||
---
|
||||
|
||||
## 15. Capability table — what actually works
|
||||
|
||||
| Feature | Status | Evidence |
|
||||
|---|---|---|
|
||||
| Register workspace + owner | **live** | UI registration and API `POST /auth/register` → 201 |
|
||||
| Sign in / wrong-password error | **live** | UI sign-in → /streams; wrong password → 401 "invalid email or password" |
|
||||
| Sign out | **live** | Session bar button: token cleared from storage; next protected route → /onboarding |
|
||||
| Auth gate (anonymous → onboarding) | **live** | Anonymous `/streams` redirected to `/onboarding` |
|
||||
| Epoch fail-closed + auto reissue | **live** | Old token → 401 "authorization epoch stale" after a role change; client reissued automatically |
|
||||
| Onboarding invite step | **live** | 2 invites → 201 each; members appear as *invited* |
|
||||
| Invite email delivery / acceptance | **stub** | Not wired (stated in the UI) |
|
||||
| Onboarding "Connect accounts" | **stub** | Simulated local toggle |
|
||||
| Streams columns, j/k/r/q | **sample data** (always) | Fixtures; repost/quote don't call any API |
|
||||
| Streams + Column / Layout | **stub** | Inert buttons |
|
||||
| Compose editor, counters, previews | **live** (client-side) | Exercised in the UI |
|
||||
| Save draft / Recent drafts | **live** | UI "Draft saved." + `GET /publishing/drafts` |
|
||||
| Draft submit → approve (state machine) | **live** (API only) | `in_review` → `approved` via the API |
|
||||
| Schedule / publish to a platform | **partial**, not exercised end-to-end | No connection obtainable; schedule against an unknown connection → 404 as designed |
|
||||
| Channels list | **live** (read-only) | `GET /connections` → `[]` for a new workspace |
|
||||
| Connect / Reconnect / Remove channel | **stub** | Disabled when live ("OAuth connect flows arrive with the connector wiring phase") |
|
||||
| Capacity chip + 24h popover | **sample data** in testing | Real rate-policy math; no live connection to show it on |
|
||||
| `POST /demo/connections` (dev seed) | **broken** | Returns 500 |
|
||||
| Video register (upload) | **live** | 2 UI uploads + 1 API upload → asset + job `queued` |
|
||||
| Video library, detail drawer, refresh | **live** | `GET /video`, `GET /video/:id/status` |
|
||||
| Video delete (confirm modal) | **live** | `DELETE /video/:id` → 200, "Video deleted.", audit `video.deleted` |
|
||||
| Video probe / transcode / thumbnails | **partial** | Job stays `queued 0%`; bytes never reach storage (no ingest path) |
|
||||
| Team list / invite / role change / remove | **live** | UI invite; API role change + remove |
|
||||
| Team asset library | **sample data** (always) | Fixtures |
|
||||
| Audit trail, diffs, filters | **live** | Our own actions recorded with actor/diff; action filter `draft.approved` narrowed 11 rows → 1 |
|
||||
| Audit Export CSV | **stub** | Button has no handler |
|
||||
| Command palette (⌘K) | **live** (navigation only) | Opened and listed routes |
|
||||
| Inbox | **sample data** (always) | Fixtures; reply doesn't send |
|
||||
| Calendar | **stub** | Placeholder "Arrives with WP3" |
|
||||
| Approvals screen | **stub** | Placeholder (approval API exists, see above) |
|
||||
| Analytics | **stub** | Placeholder |
|
||||
| Settings | **stub** | Placeholder |
|
||||
| Telegram publish + forward-amplify | **connector-level only** | Contract tests 34/34 against a mocked Bot API; no UI connect (bot-token entry) flow |
|
||||
| Other connectors (Mastodon, Bluesky, Discord, Slack, PeerTube…) | **connector-level only** | Not exercised here; none connectable from the UI |
|
||||
| Offline sample-data mode | **sample data** | API blocked → fixtures + "offline" badge |
|
||||
| Mobile | **in progress** | Desktop site on a phone; Compose unusable at 390px |
|
||||
|
||||
---
|
||||
|
||||
*Generated from a verification run; details, logs and the screenshot inventory are in `runs/manual/screenshots-manual-report.md`.*
|
||||
29
docs/wiki/EN/getting-started.md
Normal file
29
docs/wiki/EN/getting-started.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
---
|
||||
title: Getting Started
|
||||
lang: en
|
||||
---
|
||||
|
||||
# Getting Started
|
||||
|
||||
## 1. Create your account
|
||||
|
||||
Go to `app.occupymedia.cc/onboarding` and fill in your display name, username, and password. Registering creates a personal workspace for you automatically — you are the Owner.
|
||||
|
||||
## 2. Understand your roles
|
||||
|
||||
The Hub has roles that control what each person can do:
|
||||
|
||||
| Role | Can do | Cannot do |
|
||||
|------|--------|-----------|
|
||||
| **Owner** | Everything, including deleting the workspace | — |
|
||||
| **Admin** | Manage members, connections, and content | Delete the workspace |
|
||||
| **Editor** | Draft, edit, and review content | Publish without approval |
|
||||
| **Viewer** | Read-only access | Change anything |
|
||||
|
||||
## 3. Invite your crew
|
||||
|
||||
Go to **More → Team** and use the invite link. Share the link with your crew — they join the workspace with the role you assigned. No passwords shared, no admin phone in the loop.
|
||||
|
||||
## 4. Connect your first accounts
|
||||
|
||||
Go to **Connections** and add your platform accounts (Mastodon, Telegram, Bluesky, etc.). Each account's credentials are stored in an encrypted vault — the database never sees a plaintext key.
|
||||
29
docs/wiki/ES/getting-started.md
Normal file
29
docs/wiki/ES/getting-started.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
---
|
||||
title: Primeros pasos
|
||||
lang: es
|
||||
---
|
||||
|
||||
# Primeros pasos
|
||||
|
||||
## 1. Crea tu cuenta
|
||||
|
||||
Ve a `app.occupymedia.cc/onboarding` y completa tu nombre visible, nombre de usuario y contraseña. Al registrarte se crea automáticamente un espacio de trabajo personal — eres el propietario.
|
||||
|
||||
## 2. Entiende los roles
|
||||
|
||||
El Hub tiene roles que controlan lo que cada persona puede hacer:
|
||||
|
||||
| Rol | Puede hacer | No puede hacer |
|
||||
|-----|------------|----------------|
|
||||
| **Propietario** | Todo, incluyendo borrar el espacio | — |
|
||||
| **Administrador** | Gestionar miembros, conexiones y contenido | Borrar el espacio |
|
||||
| **Editor** | Redactar, editar y revisar contenido | Publicar sin aprobación |
|
||||
| **Observador** | Solo lectura | Cambiar nada |
|
||||
|
||||
## 3. Invita a tu equipo
|
||||
|
||||
Ve a **Más → Equipo** y usa el enlace de invitación. Comparte el enlace con tu equipo — se unen al espacio con el rol que asignaste. Sin compartir contraseñas, sin teléfonos de administrador en el medio.
|
||||
|
||||
## 4. Conecta tus primeras cuentas
|
||||
|
||||
Ve a **Conexiones** y agrega tus cuentas (Mastodon, Telegram, Bluesky, etc.). Las credenciales de cada cuenta se guardan en una bóveda cifrada — la base de datos nunca ve una clave en texto plano.
|
||||
Loading…
Add table
Add a link
Reference in a new issue