# WP3 — Editable Multi-Platform Streams Columns: Design **Status:** decision-complete design (no implementation) · **Author:** glm-5.3 design agent · **Inputs:** `internal-survey.md`, `platform-research.md`, `docs/work-breakdown.md` WP3, current `streams/page.tsx` + `lib/streams-data.ts`. --- ## 0. Ground rules (non-negotiable constraints this design obeys) 1. **No logged-in scraping, no ToS-gray bridges** except where explicitly signed off (Telegram via RSS-Bridge is the only candidate; deferred, not in scope of the base plan). 2. **All provider content renders only through `packages/safe-content`** (typed rich-text AST; raw HTML never reaches the client). See §5. 3. **Every outbound request goes through the `egress` client** and draws from the **per-host budget** (`app.take_host_budget`) machinery shipped by WP7/WP8 — columns add *callers* of that machinery, not a parallel limiter. 4. **Back-compat:** `GET /feeds` (merged all-feeds view, `meta.connections`, cursor shape) keeps working unchanged. Columns are additive. 5. **Workspace-scoped everything** (multi-tenant; `workspace_id` in every unique constraint, RLS-checked reads, `draft.view`-class RBAC extended with a `streams.columns.manage` permission — see §5.6). --- ## 1. Column types, schema, migration, API surface ### 1.1 The four column types | Type | What it shows | Sources resolved at runtime | Example | |---|---|---|---| | `platform_feed` | One connected account's own home/mentions (today's implicit column, made explicit & editable) | 1 connection × kinds | "Mastodon · @acampada" | | `keyword` | One query across N platforms (each platform adapter shapes the query: full-text, hashtag, video-search) | per-(connection or keyless-source) search job | "acampada26" on Bluesky+Mastodon+YouTube | | `account_list` | A curated list of accounts across platforms (author timelines / RSS) | per-account author-feed job | "Movement voices" = 5 handles × 3 platforms | | `saved_search` | A keyword column created via the existing saved-search UX (localStorage migrated to server) | same as `keyword` | "product" | `saved_search` is **not a separate engine** — it is `keyword` with `origin: "saved_search"`. It exists as a persisted type so the UI can label, migrate, and later diverge (e.g. notifications) without another migration. Decision: single `type` enum column with the four values; the keyword engine is shared. ### 1.2 Persisted schema (workspace-scoped) New tables (forward-only numbered SQL migration, additive, no changes to existing tables): ```sql -- 1. Column definitions CREATE TABLE stream_columns ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), workspace_id UUID NOT NULL, type TEXT NOT NULL CHECK (type IN ('platform_feed','keyword','account_list','saved_search')), title TEXT NOT NULL, -- user-editable, max 80 chars position INTEGER NOT NULL DEFAULT 0, -- dense rank; reorder rewrites the tail spec JSONB NOT NULL, -- typed per-type, validated in app layer (below) refresh_policy JSONB NOT NULL DEFAULT '{}', -- { "intervalMs"?: number, "paused"?: boolean } created_by UUID NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), deleted_at TIMESTAMPTZ -- soft delete keeps feed_items FK-stable ); CREATE UNIQUE INDEX stream_columns_ws_order_uq ON stream_columns (workspace_id, position) WHERE deleted_at IS NULL; CREATE INDEX stream_columns_ws_idx ON stream_columns (workspace_id) WHERE deleted_at IS NULL; -- 2. Resolved sources: what the scheduler actually polls for a column CREATE TABLE column_sources ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), workspace_id UUID NOT NULL, column_id UUID NOT NULL REFERENCES stream_columns(id), -- identity of the thing polled: provider TEXT NOT NULL, -- 'bluesky' | 'mastodon' | 'youtube' | ... connection_id UUID NULL, -- NULL for keyless sources (YouTube RSS, public Bluesky/Mastodon) source_kind TEXT NOT NULL CHECK (source_kind IN ('home','mentions','search','author')), query TEXT NULL, -- platform-adapted query (post-adaptation, what gets sent) actor_ref TEXT NULL, -- canonical account ref for source_kind='author': '{provider}:{handle@host}' state TEXT NOT NULL DEFAULT 'pending' CHECK (state IN ('pending','ready','stale','error','unsupported','rate_limited','budget_paused','disabled')), last_error TEXT NULL, last_polled_at TIMESTAMPTZ NULL, next_poll_at TIMESTAMPTZ NULL, consecutive_failures INTEGER NOT NULL DEFAULT 0, UNIQUE (workspace_id, column_id, provider, source_kind, COALESCE(connection_id::text,''), COALESCE(query,''), COALESCE(actor_ref,'')) ); CREATE INDEX column_sources_due_idx ON column_sources (next_poll_at) WHERE state NOT IN ('disabled'); CREATE INDEX column_sources_column_idx ON column_sources (column_id); -- 3. Ingested items (replaces "cache-only" depth; enables deep pagination + per-column serving) CREATE TABLE feed_items ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), workspace_id UUID NOT NULL, dedup_key TEXT NOT NULL, -- see §3.3 provider TEXT NOT NULL, remote_id TEXT NOT NULL, author_handle TEXT NOT NULL, author_ref TEXT NULL, -- canonical '{provider}:{handle@host}' when resolvable url TEXT NULL, -- provider permalink (http/https only) created_at_remote TIMESTAMPTZ NOT NULL, -- provider timestamp (sort key) fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(), content JSONB NOT NULL, -- safe-content typed AST (NEVER raw HTML) — §5 plain_text TEXT NOT NULL, -- derived flat text (for saved-search client filter & search preview) media JSONB NOT NULL DEFAULT '[]', -- [{kind:'image'|'video', proxyUrl, width?, height?, alt?}] — proxy-minted only metrics JSONB NOT NULL DEFAULT '{}', -- finite numbers only source_id UUID NOT NULL REFERENCES column_sources(id), expires_at TIMESTAMPTZ NULL, -- set for YouTube Data-API-sourced items: fetched_at + 30d (ToS) UNIQUE (workspace_id, dedup_key) ); CREATE INDEX feed_items_ws_time_idx ON feed_items (workspace_id, created_at_remote DESC, dedup_key DESC); -- keyset CREATE INDEX feed_items_source_idx ON feed_items (source_id, created_at_remote DESC); CREATE INDEX feed_items_expiry_idx ON feed_items (expires_at) WHERE expires_at IS NOT NULL; -- 4. Join: which columns show an item (an item ingested for a source is visible to every column sharing that source) CREATE TABLE feed_item_columns ( workspace_id UUID NOT NULL, item_id UUID NOT NULL REFERENCES feed_items(id) ON DELETE CASCADE, column_id UUID NOT NULL REFERENCES stream_columns(id), PRIMARY KEY (item_id, column_id) ); -- 5. Per-provider budget/spend ledger (X pay-per-use, IG 30-tags/7d, YT search bucket) CREATE TABLE provider_budget_ledger ( workspace_id TEXT NOT NULL, -- '' = global (API keys are workspace-owned; see §2.5) provider TEXT NOT NULL, bucket TEXT NOT NULL, -- 'x_post_reads' | 'ig_hashtag_window' | 'yt_search_calls' window_start TIMESTAMPTZ NOT NULL, units_used NUMERIC NOT NULL DEFAULT 0, units_cap NUMERIC NOT NULL, UNIQUE (workspace_id, provider, bucket, window_start) ); ``` **Why items are persisted, not just cached:** today's 40-item Redis cache + TTL means cursors die and "Load older" is shallow. Columns are user-curated surfaces; they need stable keyset pagination and cross-column dedup. Redis stays as the hot read-through cache in front of `feed_items`. **`spec` JSONB shapes** (validated by a zod-style schema in `packages/core`; invalid spec → `422`, never stored): ```ts // platform_feed { connectionId: string, kinds: ("home"|"mentions")[] } // default ["home"] // keyword / saved_search { query: string, // 1..200 chars, trimmed; no NUL/control chars platforms: string[], // provider ids, e.g. ["bluesky","mastodon","youtube"] matchMode?: "any" } // reserved; v1 = single term set, no boolean ops // account_list { accounts: Array<{ provider: string, handle: string }> } // 1..50 entries; handle free-text "@user@instance.social" | "@user" | "UC..." for YT ``` `refresh_policy` (clamped server-side): `{ intervalMs?: 60_000..3_600_000, paused?: boolean }`. Defaults per type: `platform_feed` 60 000 (inherits today's effective cadence), `keyword` 120 000, `account_list` 180 000, `saved_search` 300 000. The clamp floor is **never below the per-connection budget-derived interval** — a user setting 60 s on a source whose budget says 300 s gets 300 s and a `meta` hint `effectiveIntervalMs`. **Keyless sources:** `column_sources.connection_id` is NULL for YouTube-RSS channels, public Bluesky (`getAuthorFeed`/`searchPosts` unauthenticated), and Mastodon public/RSS reads. These draw from the **per-host budget only** (plus a global keyless concurrency cap of 2 in-flight per host), not from any connection's token bucket. This is what lets an account-list column follow YouTube/Mastodon accounts with **zero connections**. ### 1.3 Migration approach 1. **DDL migration** (additive, numbered): the five tables above. No existing table touched. 2. **Lazy seeding (no backfill job):** on first `GET /feeds/columns` for a workspace with zero rows, the API inserts one `platform_feed` column per feed-capable visible connection (kinds `["home"]`, title `` `${displayName} · ${handle}` ``) — exactly what `liveColumns()` derives client-side today. Seeding is idempotent under the unique `(workspace_id, position)` index (transaction + `ON CONFLICT DO NOTHING` re-check). 3. **Saved-search migration (opt-in, client-driven):** `POST /feeds/columns/import-saved-searches { terms: string[] }` creates `saved_search` columns (platforms = all search-capable connected/keyless providers). The web app calls it once when localStorage `tmbh.saved-searches` is non-empty and the server list lacks them; on success it clears localStorage and dispatches `saved-searches-change`. LocalStorage remains the fallback when the API is unreachable (sample mode) — no behavior regression offline. 4. **Rollback story:** columns are additive; disabling the feature flag (`STREAMS_COLUMNS_ENABLED`, default on) falls the UI back to today's derivation path. `/feeds` legacy shape never changes. ### 1.4 API surface All under the existing feeds module, RBAC `draft.view` to read; **new permission `streams.columns.manage`** (granted to Owner/Admin/Editor by default) to write. Group-visibility: a `platform_feed` column referencing a non-visible connection is rejected `403`; keyword/account columns only see what their sources legitimately fetch. ``` GET /feeds/columns → { columns: ColumnView[] } (ordered, with per-source status) POST /feeds/columns → 201 ColumnView (validates spec, resolves column_sources) PATCH /feeds/columns/:id → ColumnView (title, spec, refresh_policy, position-delta) DELETE /feeds/columns/:id → 204 (soft delete; source items expire naturally) POST /feeds/columns/reorder → { columns: ColumnView[] } { orderedIds: string[] } — rewrites positions POST /feeds/columns/import-saved-searches → { columns: ColumnView[] } GET /feeds/columns/:id/items?cursor&limit(1..200,def 40) → { items: FeedItemDTO[], nextCursor, meta: ColumnMeta } POST /feeds/columns/:id/refresh → 202 (shares lease/cooldown/budget machinery) ``` `ColumnView`: ```ts { id, type, title, position, spec, refreshPolicy, effectiveIntervalMs, sources: Array<{ id, provider, connectionId, sourceKind, query, actorRef, state, lastPolledAt, lastError?, handle?, displayName? }>, aggregateState: "ready"|"pending"|"stale"|"error"|"partial"|"empty"|"paused" } // worst-of, §3.5 ``` `FeedItemDTO` (column items endpoint; superset of today's `/feeds` item — the UI's `StreamItem` maps 1:1): ```ts { id, remoteId, provider, connectionId: string|null, sourceId, authorHandle, authorRef?, url?, createdAt, content: RichTextAst, // §5 — the ONLY render source plainText, media: MediaRef[], metrics: Record } ``` `ColumnMeta`: `{ sources: [...same as ColumnView.sources], fetchedAt, nextCursor semantics }`. **Mapping into `/feeds` serving (back-compat + one merged view):** - `GET /feeds` **unchanged**: merged home-timeline view over capable connections, `meta.connections`, `excludedConnections`, cursor format, `POST /feeds/refresh`. It now reads through the same `feed_items` store (cache-first) — external shape identical. - `GET /feeds` gains **one additive** field: `meta.columns?: { id, title, type, position, aggregateState }[]` (omitted when the feature flag is off). The "All feeds" chip and offline/sample mode keep working off the old shape. - Per-column serving is **only** `GET /feeds/columns/:id/items`. The merged `/feeds` never mixes column-filtered results, so the current switcher ("All feeds" / "Columns" / per-column / per-saved-term) keeps exact semantics. **Cursor rule:** every cursor is the existing base64url `{at, key}` position tuple over a **deterministic sort** (`created_at_remote DESC, dedup_key DESC`). Per-column cursors are stable because they page persisted rows, not a Redis snapshot. --- ## 2. Feed acquisition strategy per platform Order follows the research matrix ("Low effort, $0" → up). **Build order: Bluesky → Mastodon → YouTube → X(paid, gated) → Instagram(hashtag, gated) → TikTok(embed-only, out of scope).** ### 2.1 Platform decision table | Platform | Keyword column | Account-list column | Mechanism | Auth / credential | Budget | Phase | |---|---|---|---|---|---|---| | **Bluesky** | `app.bsky.feed.searchPosts` (`sort=latest`, `q`) | `app.bsky.feed.getAuthorFeed(actor)` | native REST via connector; **poll first**, jetstream later (§2.4) | **optional app password** — created via `createAppPassword`, vaulted per workspace as a *column credential*; unauthenticated mode is best-effort fallback (api.bsky.app works; design never *requires* keyless) | AppView "generous"; per-host bucket default 30 req/10 min/host; keyless concurrency cap 2 | **P2 / P3** | | **Mastodon** | hashtag timeline `timelines/tag/:tag` only (no global search — hard platform fact); non-tag query → provider marked `unsupported` | `accounts/:id/statuses` (public) **or** `@user.rss` (verified keyless) — prefer RSS for keyless follows, `/statuses` when a connection exists | native via existing connector | none for public/RSS; user token (existing connection) for streaming later | 300 req/5 min per account/IP → per-host bucket 10 req/10 s default (egress), per-connection 0.5 RPS cap as today | **P2 / P3** | | **YouTube** | Data API `search.list` (`type=video`, `publishedAfter` = last poll − slack) | **channel RSS** `feeds/videos.xml?channel_id=` (15 latest, keyless, verified) as primary; `playlistItems.list` (1 unit/50) for backfill/deep history | RSS fetcher (new shared `rss-source` reader in ingest, not a full connector) + Data API client | **API key** in vault, workspace-scoped `youtube_data` credential; RSS needs nothing | `search.list` own bucket **100 calls/day** → ledger `yt_search_calls` cap 90/day (10% headroom); `playlistItems` from shared 10k units; RSS = per-host bucket only. **30-day refresh-or-delete** on Data-API items → `expires_at`, attribution required in UI | **P2 (search) / P3 (RSS)** | | **X / Twitter** | `tweets/search/recent` (last 7–10 d) | `users/:id/tweets` | native connector, **behind a pay-per-use gate** | OAuth2 app Bearer; billing = operator's console credits (pay-per-use $0.005/post read; free tier has no reads) | **$ ledger**: `x_post_reads` cap per workspace (default **0 = off**); every read debited; hard stop at cap → source `budget_paused`; platform 3M reads/mo cap noted but irrelevant vs our caps | **P5 (opt-in)** | | **Instagram** | Graph API `ig_hasid_search` → `top_media`+`recent_media` (hashtag columns only) | **not possible** (others' accounts) — UI shows embed-card fallback for known URLs only, never auto-follow | native connector extension | Meta app + App Review; Business/Creator account connection | **30 unique hashtags / rolling 7 d** → ledger `ig_hashtag_window`; source `rate_limited` + "next tag slot frees {time}" when exhausted; `recent_media` ~50-item window → treat as shallow | **P5 (gated on App Review)** | | **TikTok** | none compliant (Research API = academic/nonprofit only; Commercial = ads) | embed-only for known URLs | none — **out of scope**; render link-cards | n/a | n/a | not built | | **Telegram / Discord / Slack** | n/a | n/a (channel-picker columns are WP5 inbox work) | existing `getFeed` home paths unchanged | existing | existing | n/a | | **Threads** | n/a (API is own-account only) | n/a | unchanged | n/a | n/a | n/a | **Bridge policy:** no RSS-Bridge/RSSHub/Nitter in any phase of this plan. Telegram public-channel via RSS-Bridge is recorded as a possible later opt-in with legal sign-off; not designed in. ### 2.2 SDK interface changes (decided) Extend `PlatformConnector` (additive, optional methods — existing connectors untouched): ```ts // keyword search — lifts the existing getFeed kind:"search" branches verbatim search?(connection | null, { query: string, cursor?: string, limit: number }): Promise>; // author timeline — new getUserFeed?(connection | null, { actorRef: string, cursor?: string, limit: number }): Promise>; ``` `connection | null` formalizes **keyless sources** (Bluesky public, Mastodon RSS/public, YouTube RSS). Capability flags added to `ConnectorCapabilities` + manifests: `search: boolean`, `authorFeed: boolean`. These mirror into `/feeds` `meta.connections[].capabilities` (already a string array) so the UI can gate affordances with zero new endpoints. Reddit's degenerate "search = subreddit hot" is **removed** from the search path (no `search()` implementation → `unsupported`), fixing a current semantic lie. A small **`rss-source` reader** (XML→FeedItem) lives in the ingest package for YouTube/Mastodon RSS — it is not a `PlatformConnector` because it has no publish/identity surface; it registers per-host budget usage like any egress caller. ### 2.3 Credential & key residence (vault) - **Connection credentials** (Mastodon/Bluesky/IG/X user tokens): unchanged — `vault.open/seal` per `(workspaceId, connectionId)` exactly as `feeds.dependencies.ts` does today. - **Column credentials (new class):** workspace-scoped secrets not tied to a connection — Bluesky app password (optional), YouTube Data API key, X app Bearer. Stored vaulted under `(workspaceId, "columns", provider)`; the poller resolves provider→credential handle at poll time. **Never** in `stream_columns.spec`, never logged, never sent to the client (the API returns only `{ provider, configured: boolean, label }`). - **Keyless sources carry no secret at all** — egress policy + per-host budget is the only control (which is why they're preferred where equal). ### 2.4 Polling vs streaming - **v1 = polling everywhere.** The generalized scheduler (§3.4) with per-source `next_poll_at`, exponential backoff on failure (×2 up to 30 min), jitter ±20%. - **Bluesky Jetstream (P6, optional):** a single workspace-shared subscriber (`collections=app.bsky.feed.post`) maintains a **local keyword index** (FTS table) that backs keyword columns for *Bluesky only*, giving near-real-time + historical depth beyond `searchPosts` page depth. Rules: self-hostable binary or public endpoint, no auth, cursor replay on reconnect, **index rows follow the same safe-content AST + 30-day retention**, and keyword columns **fall back to `searchPosts` polling** whenever the subscriber is down (state `stale`, never `error`). Jetstream is an accelerator, never a dependency. - **Mastodon streaming** (user token + `read:statuses`) is the same pattern later; hashtag polling is good enough for v1. ### 2.5 Per-host rate budgets (reusing WP7/WP8 machinery) No new limiter. The column poller: 1. computes the target host from the source's provider+instance (Mastodon instances vary! host = the connection's instance, not `mastodon.social` globally); 2. acquires via `egress` → `app.take_host_budget(host_id, 'monitor', n, lease_seconds)` exactly like the live-monitor pollers — so columns, monitor polls, and any other caller of the same instance share one bucket; 3. honors `429`/`Retry-After` → cooldown key + source state `rate_limited`; 4. keeps the existing per-connection Redis lease/cooldown (`feeds:{ws}:{conn}:…` extended with `:{columnSourceId}`) so **manual refresh cannot bypass budgets** and replicas never double-poll (identical pattern to today's `FeedsService`, keyed finer). Per-provider defaults (conservative, env-overridable, `CONNECTOR_RATE__RPS` style retained): Bluesky host 0.2 RPS sustained (keyless 0.05 RPS), Mastodon per-instance 0.1 RPS, YouTube RSS host 0.05 RPS, YouTube API search 90 calls/day ledger, X gated by spend ledger, Instagram by hashtag-window ledger. The **provider-budget ledger** (table 5) is checked *before* the host-budget debit; exhausted → `budget_paused` (distinct from `rate_limited`: no retry storm, requires user action or window rollover). --- ## 3. Ingest / search pipeline ### 3.1 Source resolution (column spec → column_sources) On create/update of a column, a **resolution step** (no network for keyword columns; bounded network for account_list) writes `column_sources`: - `platform_feed`: 1–2 sources (per kind) on the named connection. - `keyword`: for each `platforms[]` entry → one source per (provider, connection-or-keyless) that **declares the capability** (`search: true` or RSS-search path). Providers in `platforms[]` without capability → **no row**; the API response lists them under `spec.unsupportedPlatforms` so the editor can show "TikTok: not available" without inventing a failing source. Query adaptation happens here, once, per provider: - Bluesky: query as-is (full-text). - Mastodon: extract first `#hashtag` token; if none → this provider is `unsupported` for the column (UI states "Mastodon supports #hashtag queries only"). - YouTube: query as-is (`search.list`, `type=video`). - X: query as-is (`search/recent`), only if spend ledger enabled. - Instagram: only if the entire query is a single hashtag; else `unsupported`. - `account_list`: each entry `{provider, handle}` → resolve to `actor_ref` via provider-specific resolution (`resolveHandle` for Bluesky DID, Mastodon `accounts/lookup`, YouTube channel-id via RSS/`channels.list` if key present). Resolution uses the bounded resolver pattern (≤4 in-flight/host, 10-min positive cache / 1-min negative). Unresolvable handle → source row with `state:'error', last_error:'handle_not_found'` — **it does not fail the column**. ### 3.2 Normalization (into the existing shape, extended) Every source type funnels into one `normalize()` (today's `feeds.service.ts` validation retained: finite dates, string remoteId/handle, finite metrics; invalid items dropped + counted in source meta): | Provider source | author_handle | content (AST) | media | metrics | url | |---|---|---|---|---|---| | Bluesky post | handle | facets→AST (links, mentions) | embed images/video → **proxy** | likes/reposts/replies | `bsky.app/profile/…/post/…` | | Mastodon status/RSS | `@user@instance` | HTML→AST via allowlist parser (strip scripts/styles/iframes; keep p/br/a/strong/em; links http/https only) | media attachments → proxy | reblogs/favourites/replies | status permalink | | YouTube video/RSS item | channel title | title+description→AST (plain paragraphs; links auto-linked http/https only) | thumbnail → proxy | viewCount/likeCount (API only) | watch URL | | X tweet | handle | entities→AST | media → proxy | likes/reposts/replies | tweet permalink | | IG hashtag media | username | caption→AST | media → proxy | like/comment count | permalink | `plain_text` is derived at ingest (AST→text) so saved-search-style client filtering and the column preview keep working. **`media[].proxyUrl` is always our media proxy URL** (WP3 media path) — raw provider CDN URLs never reach the browser. ### 3.3 Dedup & identity - `dedup_key = sha256(workspace_id | provider | canonical_remote_id)` where `canonical_remote_id` = `remoteId` lower-cased and, for handle-bearing providers, `{host}/{remoteId}` with the IDNA-canonical host (same canonicalization rule as WP7 `remote_ref`: lowercase host, punycode, default-port stripped). One object reached via two connections (or a connection + keyless) collapses to one row; the second ingest updates metrics/fetched_at in place (`ON CONFLICT … DO UPDATE` on metrics + fetched_at). - **Cross-provider dedup is explicitly NOT attempted** (no fuzzy matching) — an item is provider-scoped. The UI dedups by `streamItemKey` client-side as today. - `feed_item_columns` is filled by fan-out at ingest: an item lands in every column that shares that `column_source` identity — i.e., columns referencing the same (provider, actor_ref) or (provider, query) share ingested rows, so following the same account in two columns costs **one** poll (mirrors the WP8 "one remote live, three channels, one poller" rule). ### 3.4 Scheduler (generalizing FeedsService) The existing 5 s discovery loop stays; `pollAll()` becomes "poll due `column_sources`" (the legacy per-connection home poll is literally a `platform_feed` source, so old behavior is the degenerate case): - select due sources (`next_poll_at <= now`, `state NOT IN ('disabled')`), `FEED_CONCURRENCY=4` waiter queue retained; - per-source flow: lease → resolve credential (vault or keyless) → budget/ledger check → egress call via connector `getFeed/search/getUserFeed` or RSS reader → normalize → upsert `feed_items` + fan-out join → write hot Redis slice (`feeds:{ws}:{column}:items` head, TTL = interval×1.2+30 s) → update source state/`next_poll_at` (success: interval; failure: backoff ×2 cap 30 min, `consecutive_failures`≥3 → `error`); - `RateLimitError.retryAfterMs` → cooldown, state `rate_limited`; ledger empty → `budget_paused`; - manual `POST /feeds/columns/:id/refresh` → same path (202; cooldown may make it a no-op — response carries `effectiveNextPollAt`). ### 3.5 Stale / failed / empty behavior in a column Per-source `state` (§1.2) aggregates to `aggregateState` shown in the column header: - `ready` — all sources fresh (< 2× effective interval). - `stale` — any source's `last_polled_at` older than 2× interval but data present → header "Updated HH:MM" + amber dot; **old items remain visible** (persisted store — the big UX win over today's TTL'd cache). - `error` — all sources failed ≥3× → red dot + per-source error line (from `last_error`, provider message scrubbed); last-known items stay visible with "showing last data from …". - `partial` — some sources fine, some error/stale → amber; per-source chips in column menu. - `rate_limited` / `budget_paused` — distinct icon + actionable copy ("X reads paused: budget reached — raise cap in settings" / "Instagram hashtag slots full until {time}"). - `empty` — source healthy, zero items ever → per-platform empty-state copy ("No public posts match #acampada26 on Mastodon yet"). - `unsupported` — never a source row; surfaced in editor only. Deleting the underlying connection → its sources flip `disabled`; column survives with remaining sources; a `platform_feed` column whose connection dies becomes `error` with a reconnect CTA (not auto-deleted). **Retention / expiry sweeper:** daily job deletes `feed_items WHERE expires_at < now()` (YouTube Data-API sourced) and everything older than 30 days regardless of source (matches the strictest ToS we're under; RSS items keep 30 d too for uniformity). Deletion is the default; re-polling a live column continuously refreshes the visible window, satisfying "refresh or delete". --- ## 4. UI / UX spec (Streams page integration) ### 4.1 Model change in the web app `lib/streams-data.ts` gains `columnViewsToColumns(ColumnView + items)` producing today's `ColumnDef` (key = column id, title, platformId from dominant provider, staleness from `aggregateState`/`lastPolledAt`). The sample/offline path (`MOCK_COLUMN_DEFS` + fixtures) is untouched. Data flow: - mount → `GET /feeds/columns` (once, plus after any mutation) → for the visible columns (and lazily the next offscreen ones) `GET /feeds/columns/:id/items` per column, 30 s polling retained per visible column (staggered), merged into `ColumnDef[]`. - "All feeds" chip → unchanged `GET /feeds` merged view. "Columns" chip → column array as above. Per-column chips → single column. Saved-term chips now resolve to their migrated server columns (id `saved:{term}` → real id after migration; localStorage terms still render offline). ### 4.2 Column editor - **"+ Column" button** opens a right-side drawer (desktop) / bottom sheet (mobile): type selector (Platform feed · Keyword · Accounts · Saved search), then type-specific fields: - *Platform feed*: connection picker (visible connections only, feed-capable, capability badge), kinds toggles (Home / Mentions). - *Keyword*: query input with live per-platform feasibility hints as you type ("Bluesky ✓ full-text · Mastodon #hashtag only · YouTube ✓ · TikTok ✗"), platform multi-select chips (search-capable enabled, others visible-but-disabled with reason tooltip), interval select (1/2/5/10/30 min — clamped label shows effective value). - *Accounts*: multi-entry list (paste handles one per line or chip input; provider inferred from handle shape `@x@bsky.social`/`@x@mastodon.social`/`UC…`; `@x` on TikTok/IG → blocked with reason), live resolution status per row (✓ resolved / ✗ not found / ⏳ resolving), 50-entry cap enforced client+server. - *Saved search*: term input; identical to keyword with `origin` label. - **Edit**: column header "⋯" menu → same drawer prefilled; changing spec re-resolves sources (existing items for retained sources persist; removed sources' items age out via retention, no mass delete). - **Reorder**: drag handle on column headers (desktop) / up-down arrows in the columns sheet (mobile) → `POST /reorder`. - **Remove**: "⋯" → Remove (soft delete; confirm if the column has fresh data) and Pause (refresh_policy.paused). - Optimistic UI on create/reorder/remove with server reconciliation; failure → toast + revert. ### 4.3 States, keyboard, mobile - **Empty states:** no columns → existing `EmptyState` + "Add column" CTA. Column with no items → per-source empty copy (§3.5). No connections at all → today's sample-mode fixtures with a "Sample data — connect an account" banner (unchanged). - **Error states:** header dot + per-source detail in the column menu; a "source status" strip lists each source (provider, handle/query, state, last polled, error). - **Sample state:** identical to today (API unreachable / 0 configured feeds → fixtures); the editor still opens but Create is disabled with "offline — changes need the server". - **Keyboard:** j/k flat navigation retained; `c` opens the column editor; `1..9` jumps to column n. - **Mobile:** single-column reading with the existing `SourceChips` bar (All feeds · Columns · each column · saved terms); Columns mode = swipeable pager (one column per screen, `PullToRefresh` triggers the column refresh endpoint); editor as bottom sheet; drag-reorder replaced by arrows. Desktop keeps the multi-column wall. ### 4.4 Back-compat guarantee The source switcher's four modes (all / columns / per-column / saved-term) all survive with identical semantics; when the flag is off or seeding hasn't run, the page behaves exactly as the current build (derived columns from `/feeds` meta). Saved-search localStorage remains the offline fallback. --- ## 5. Security design 1. **safe-content as the only render path.** `FeedItemDTO.content` is the safe-content **typed rich-text AST**, produced **at ingestion** (server-side): Mastodon HTML → AST via the allowlist parser inside the connector/ingest boundary; Bluesky facets, YouTube/RSS text, X entities → AST builders. Raw provider HTML **never** crosses the API boundary; there is no field a client could accidentally render. The web `StreamColumn`/`PostCard` render exclusively ``; the repo lint rule fails on `dangerouslySetInnerHTML` (existing WP3 acceptance) and additionally on any new string prop named `html`. 2. **Links:** AST link nodes are http/https-only (other schemes dropped at ingest, not CSS-hidden); unsafe-scheme links render as inert text, not clickable. 3. **Media:** `media[].proxyUrl` only — provider CDN URLs are proxied through the media path (CSP `img-src`/`media-src` never include provider origins). No `