From b3c208543690f92d62039818b1526503085ed58f Mon Sep 17 00:00:00 2001 From: glass beadian Date: Fri, 16 Oct 2026 02:33:46 +0000 Subject: [PATCH] docs: producer & media ingest, ops changes, library updates (October 2026) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - engineering-guide §10: media attachments, alt-text delivery, producer, audio-less silence bed, chunked ingest, upload caps - admin-guide §10: oCIS retired from the payload path, storage_key (041), publishing media resolver + dedupe origins, produce-worker, api restart rate-limit note; §6.3 marked legacy - user-guide §9: 2 GiB Library uploads, compose attachments + alt text, producer, chunked large-video uploads --- docs/admin-guide.md | 25 ++++++++++++++++++++++++- docs/engineering-guide.md | 30 +++++++++++++++++++++++++++++- docs/user-guide.md | 11 +++++++++-- 3 files changed, 62 insertions(+), 4 deletions(-) diff --git a/docs/admin-guide.md b/docs/admin-guide.md index e984ed1..127f475 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -128,6 +128,7 @@ Operational notes: - Every switch is idempotent (`Idempotency-Key`) and lands in the audit chain as `director.` — incident review replays commands from the audit log. ## 6.3 Files SSO (Hub as the OIDC issuer for oCIS) +> **Retired (October 2026):** oCIS is out of the payload path and the in-app Drive view is gone — see §10.1. This section remains only as reference for the legacy `files.occupymedia.cc` host. `files.occupymedia.cc` signs in through the Hub: the issuer is `https://api.occupymedia.cc/oidc`, client id `web`, PKCE only, auto-consent for that first-party client. Phase 2 (migration 034) makes sessions revocable and adds rotating refresh tokens bound to the Hub session — Hub logout reaches Drive within one access-token TTL (≤1 h). Operational notes: - `OIDC_KEY_DIR` must be a persistent volume (the API generates and seals its ES256 keyset there on first boot; losing it logs every Drive session out). @@ -172,7 +173,29 @@ The governance sim is re-runnable (all fixtures prefixed `e2e-gov-`) and writes | MinIO image pull denied | registry degradation | retry later or point `S3_ENDPOINT` at any S3 | | Streams columns empty in UI | mock-registry data (API not wired yet) | expected pre-WP3-integration | -## 10. Upgrading +## 10. Operations changes (October 2026) + +### 10.1 oCIS retired from the payload path +- **All media** — Files tab, video, submissions, media posts, produced files — now stores in the **R2 bucket** (`S3_*` env, `S3_KEK` envelope encryption). oCIS holds no payloads. +- `files.occupymedia.cc` / the **Drive tab** is gone from the web UI. +- `192.168.13.210` is no longer in the serving path and may be decommissioned. + +### 10.2 Payload pointer (migration 041) +- `chat_media_archive.storage_key` is the **authoritative** payload pointer. +- `ocis_resource_id` remains only as a legacy column — do not build against it. + +### 10.3 Publishing media +- The scheduler resolves media at dispatch (**PublishingMediaResolver**: RLS workspace lookup + R2 download + SPENC1 open). +- `MASTODON_PUBLISH_DEDUPE_ORIGINS=https://social.occupymedia.cc` **must stay set**. Media posts are two provider mutations (media upload + status create) and the dispatch gate refuses the second without a verified dedupe origin. + +### 10.4 Produce worker +- Runs as a **separate container** (`produce-worker`) and needs **ffmpeg + a font** — `DejaVuSans-Bold` at `/usr/share/fonts/truetype/dejavu/`. +- `MEDIA_DEBUG_STDERR=1` routes ffmpeg stderr to `/tmp/ffmpeg-debug.log` for debugging. + +### 10.5 Known ops note +- Restarting the api container repeatedly can trip the homeserver login rate limit (`M_LIMIT_EXCEEDED`) and crash bootstrap — wait **~2–3 min** between api restarts. + +## 11. Upgrading 1. Back up DB + confirm KEK backups (§4.3). 2. `git pull && pnpm install`. diff --git a/docs/engineering-guide.md b/docs/engineering-guide.md index 3dc0d1d..c91070d 100644 --- a/docs/engineering-guide.md +++ b/docs/engineering-guide.md @@ -162,7 +162,35 @@ Shipped and wired end-to-end: Still planned: media-engine graph rewiring beyond the current FFmpeg-normalize path, LLM video AI (design only), content search rollout. -## 10. Key file map +## 10. Producer & media ingest (October 2026) + +### 10.1 Media attachments for publishing +- `POST /media` (multipart `file`); content type is **magic-byte sniffed** (jpeg/png/webp/mp4/webm) — the extension is never trusted. +- Caps: image **10 MiB**, video **64 MiB**, and at most **4 attachments per post** (`MAX_MEDIA_PER_POST`). +- Read back via `GET /media/:id` (auth-scoped). Bytes live in R2 object storage — SPENC1 envelope-encrypted, content-addressed `sha256` keys (§3.8 scheme). + +### 10.2 Alt texts +- Drafts carry `altTexts` positionally aligned with `mediaAssetIds` (0–4 entries; `null` = no alt text for that slot). +- At publish, the scheduler maps them into `providerMetadata.altText` (asset-id keyed); Mastodon media uploads receive them as descriptions. + +### 10.3 Video producer +- `POST /video/:id/produce` accepts `{ overlays?, voiceover?, outputLabel?, lane? }` — closed enums with a strict key allowlist; unknown keys are rejected. Overlay kinds: `lower_third | logo | clock`; overlay text runs through the filtergraph-metacharacter blocklist before it reaches an FFmpeg graph. +- Jobs execute in the **produce-worker** (its own container, `MEDIA_WORKER_LANE=local`): claim/lease via polling with `SKIP LOCKED`; stale leases are re-queued. +- The output is a **derived library asset** (`derived_from`), publishable like any other asset. + +### 10.4 Audio-less sources +- The command-builder synthesizes an `anullsrc` silence bed — configured at the output preset's rate/layout — instead of emitting `[0:a]` maps that ffmpeg rejects. + +### 10.5 Chunked video ingest +- `POST /video/upload/init` → `PUT /video/upload/:uploadId/chunk/:index` → `POST /video/upload/:uploadId/complete`. +- 10 MiB chunks (`CHUNK_SIZE_BYTES`) sidestep Cloudflare's ~100 MB proxy body cap. The server reassembles, seals, stores to R2, then runs the normal probe pipeline. +- Max chunked upload: **100 GiB** (`MAX_CHUNKED_UPLOAD_BYTES`). + +### 10.6 Upload caps +- Whole-body video upload cap: **2 GiB** (`MAX_VIDEO_UPLOAD_BYTES`). +- Files group upload cap: **2 GiB** (was 20 MiB). + +## 11. Key file map | Concern | Path | |---|---| diff --git a/docs/user-guide.md b/docs/user-guide.md index a642cfc..eba4f80 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -59,7 +59,14 @@ Each account card shows: platform, handle, **capability chips** (what this conne Every meaningful action (post published, role changed, connection added, credential rotated, approval granted…) is recorded in a **tamper-evident hash chain**. Filter by actor or action; click a row to open the **diff viewer** — what changed, field by field, with removed/added values. Secret values always show as `[REDACTED]`: keys and tokens are never written into the audit log, by construction. Export to CSV for compliance. -## 9. What's real vs. coming +## 9. New in the Library & Publishing + +- **Library (Files) tab:** upload up to **2 GiB per file**. Files are encrypted and stored in the hub's own object storage. The old 20 MiB limit and the separate Drive tab are gone — the Library replaces both. +- **Compose:** attach up to **4 images or videos** per post (10 MiB per image, 64 MiB per video) and add **alt text** descriptions — they appear on Mastodon. +- **Video producer:** apply **lower-third, logo, or clock overlays** and **voiceover** to any library video. Clips with no audio get a synthesized silence bed, so they always work. Results land back in the library and can be published like any other asset. +- **Large videos:** uploads are automatically split into 10 MB chunks — big files work even through Cloudflare. + +## 10. What's real vs. coming **Now (Level 1):** monitoring streams, compose with previews, approvals, scheduling, publishing pipeline, encrypted credential vault, audit trail. *(The screens currently run on representative sample data while the backend wiring lands.)* @@ -68,7 +75,7 @@ Every meaningful action (post published, role changed, connection added, credent - **Crosspost** — one action moves content across platforms with your commentary (an Instagram image quoted into Mastodon, a Mastodon video into Facebook), flowing through approvals. - **Level 2** — video uploads, livestreams, the Virtual Channel / Auto Director master-control system, and the operator Control Room. -## 10. FAQ +## 11. FAQ **Why can't I retweet on some columns?** That platform's API doesn't expose resharing. We render only real capabilities.