glass beadian
aa5e262f3a
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
385 lines
40 KiB
Markdown
385 lines
40 KiB
Markdown
# WP3 — Editable Multi-Platform Streams Columns: Design
|
||
**Status:** decision-complete design (no implementation) · **Author:** glm-5.3 design agent · **Inputs:** `internal-survey.md`, `platform-research.md`, `docs/work-breakdown.md` WP3, current `streams/page.tsx` + `lib/streams-data.ts`.
|
||
|
||
---
|
||
|
||
## 0. Ground rules (non-negotiable constraints this design obeys)
|
||
|
||
1. **No logged-in scraping, no ToS-gray bridges** except where explicitly signed off (Telegram via RSS-Bridge is the only candidate; deferred, not in scope of the base plan).
|
||
2. **All provider content renders only through `packages/safe-content`** (typed rich-text AST; raw HTML never reaches the client). See §5.
|
||
3. **Every outbound request goes through the `egress` client** and draws from the **per-host budget** (`app.take_host_budget`) machinery shipped by WP7/WP8 — columns add *callers* of that machinery, not a parallel limiter.
|
||
4. **Back-compat:** `GET /feeds` (merged all-feeds view, `meta.connections`, cursor shape) keeps working unchanged. Columns are additive.
|
||
5. **Workspace-scoped everything** (multi-tenant; `workspace_id` in every unique constraint, RLS-checked reads, `draft.view`-class RBAC extended with a `streams.columns.manage` permission — see §5.6).
|
||
|
||
---
|
||
|
||
## 1. Column types, schema, migration, API surface
|
||
|
||
### 1.1 The four column types
|
||
|
||
| Type | What it shows | Sources resolved at runtime | Example |
|
||
|---|---|---|---|
|
||
| `platform_feed` | One connected account's own home/mentions (today's implicit column, made explicit & editable) | 1 connection × kinds | "Mastodon · @acampada" |
|
||
| `keyword` | One query across N platforms (each platform adapter shapes the query: full-text, hashtag, video-search) | per-(connection or keyless-source) search job | "acampada26" on Bluesky+Mastodon+YouTube |
|
||
| `account_list` | A curated list of accounts across platforms (author timelines / RSS) | per-account author-feed job | "Movement voices" = 5 handles × 3 platforms |
|
||
| `saved_search` | A keyword column created via the existing saved-search UX (localStorage migrated to server) | same as `keyword` | "product" |
|
||
|
||
`saved_search` is **not a separate engine** — it is `keyword` with `origin: "saved_search"`. It exists as a persisted type so the UI can label, migrate, and later diverge (e.g. notifications) without another migration. Decision: single `type` enum column with the four values; the keyword engine is shared.
|
||
|
||
### 1.2 Persisted schema (workspace-scoped)
|
||
|
||
New tables (forward-only numbered SQL migration, additive, no changes to existing tables):
|
||
|
||
```sql
|
||
-- 1. Column definitions
|
||
CREATE TABLE stream_columns (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
workspace_id UUID NOT NULL,
|
||
type TEXT NOT NULL CHECK (type IN ('platform_feed','keyword','account_list','saved_search')),
|
||
title TEXT NOT NULL, -- user-editable, max 80 chars
|
||
position INTEGER NOT NULL DEFAULT 0, -- dense rank; reorder rewrites the tail
|
||
spec JSONB NOT NULL, -- typed per-type, validated in app layer (below)
|
||
refresh_policy JSONB NOT NULL DEFAULT '{}', -- { "intervalMs"?: number, "paused"?: boolean }
|
||
created_by UUID NOT NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||
deleted_at TIMESTAMPTZ -- soft delete keeps feed_items FK-stable
|
||
);
|
||
CREATE UNIQUE INDEX stream_columns_ws_order_uq ON stream_columns (workspace_id, position) WHERE deleted_at IS NULL;
|
||
CREATE INDEX stream_columns_ws_idx ON stream_columns (workspace_id) WHERE deleted_at IS NULL;
|
||
|
||
-- 2. Resolved sources: what the scheduler actually polls for a column
|
||
CREATE TABLE column_sources (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
workspace_id UUID NOT NULL,
|
||
column_id UUID NOT NULL REFERENCES stream_columns(id),
|
||
-- identity of the thing polled:
|
||
provider TEXT NOT NULL, -- 'bluesky' | 'mastodon' | 'youtube' | ...
|
||
connection_id UUID NULL, -- NULL for keyless sources (YouTube RSS, public Bluesky/Mastodon)
|
||
source_kind TEXT NOT NULL CHECK (source_kind IN ('home','mentions','search','author')),
|
||
query TEXT NULL, -- platform-adapted query (post-adaptation, what gets sent)
|
||
actor_ref TEXT NULL, -- canonical account ref for source_kind='author': '{provider}:{handle@host}'
|
||
state TEXT NOT NULL DEFAULT 'pending'
|
||
CHECK (state IN ('pending','ready','stale','error','unsupported','rate_limited','budget_paused','disabled')),
|
||
last_error TEXT NULL,
|
||
last_polled_at TIMESTAMPTZ NULL,
|
||
next_poll_at TIMESTAMPTZ NULL,
|
||
consecutive_failures INTEGER NOT NULL DEFAULT 0,
|
||
UNIQUE (workspace_id, column_id, provider, source_kind, COALESCE(connection_id::text,''), COALESCE(query,''), COALESCE(actor_ref,''))
|
||
);
|
||
CREATE INDEX column_sources_due_idx ON column_sources (next_poll_at) WHERE state NOT IN ('disabled');
|
||
CREATE INDEX column_sources_column_idx ON column_sources (column_id);
|
||
|
||
-- 3. Ingested items (replaces "cache-only" depth; enables deep pagination + per-column serving)
|
||
CREATE TABLE feed_items (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
workspace_id UUID NOT NULL,
|
||
dedup_key TEXT NOT NULL, -- see §3.3
|
||
provider TEXT NOT NULL,
|
||
remote_id TEXT NOT NULL,
|
||
author_handle TEXT NOT NULL,
|
||
author_ref TEXT NULL, -- canonical '{provider}:{handle@host}' when resolvable
|
||
url TEXT NULL, -- provider permalink (http/https only)
|
||
created_at_remote TIMESTAMPTZ NOT NULL, -- provider timestamp (sort key)
|
||
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||
content JSONB NOT NULL, -- safe-content typed AST (NEVER raw HTML) — §5
|
||
plain_text TEXT NOT NULL, -- derived flat text (for saved-search client filter & search preview)
|
||
media JSONB NOT NULL DEFAULT '[]', -- [{kind:'image'|'video', proxyUrl, width?, height?, alt?}] — proxy-minted only
|
||
metrics JSONB NOT NULL DEFAULT '{}', -- finite numbers only
|
||
source_id UUID NOT NULL REFERENCES column_sources(id),
|
||
expires_at TIMESTAMPTZ NULL, -- set for YouTube Data-API-sourced items: fetched_at + 30d (ToS)
|
||
UNIQUE (workspace_id, dedup_key)
|
||
);
|
||
CREATE INDEX feed_items_ws_time_idx ON feed_items (workspace_id, created_at_remote DESC, dedup_key DESC); -- keyset
|
||
CREATE INDEX feed_items_source_idx ON feed_items (source_id, created_at_remote DESC);
|
||
CREATE INDEX feed_items_expiry_idx ON feed_items (expires_at) WHERE expires_at IS NOT NULL;
|
||
|
||
-- 4. Join: which columns show an item (an item ingested for a source is visible to every column sharing that source)
|
||
CREATE TABLE feed_item_columns (
|
||
workspace_id UUID NOT NULL,
|
||
item_id UUID NOT NULL REFERENCES feed_items(id) ON DELETE CASCADE,
|
||
column_id UUID NOT NULL REFERENCES stream_columns(id),
|
||
PRIMARY KEY (item_id, column_id)
|
||
);
|
||
|
||
-- 5. Per-provider budget/spend ledger (X pay-per-use, IG 30-tags/7d, YT search bucket)
|
||
CREATE TABLE provider_budget_ledger (
|
||
workspace_id TEXT NOT NULL, -- '' = global (API keys are workspace-owned; see §2.5)
|
||
provider TEXT NOT NULL,
|
||
bucket TEXT NOT NULL, -- 'x_post_reads' | 'ig_hashtag_window' | 'yt_search_calls'
|
||
window_start TIMESTAMPTZ NOT NULL,
|
||
units_used NUMERIC NOT NULL DEFAULT 0,
|
||
units_cap NUMERIC NOT NULL,
|
||
UNIQUE (workspace_id, provider, bucket, window_start)
|
||
);
|
||
```
|
||
|
||
**Why items are persisted, not just cached:** today's 40-item Redis cache + TTL means cursors die and "Load older" is shallow. Columns are user-curated surfaces; they need stable keyset pagination and cross-column dedup. Redis stays as the hot read-through cache in front of `feed_items`.
|
||
|
||
**`spec` JSONB shapes** (validated by a zod-style schema in `packages/core`; invalid spec → `422`, never stored):
|
||
|
||
```ts
|
||
// platform_feed
|
||
{ connectionId: string, kinds: ("home"|"mentions")[] } // default ["home"]
|
||
// keyword / saved_search
|
||
{ query: string, // 1..200 chars, trimmed; no NUL/control chars
|
||
platforms: string[], // provider ids, e.g. ["bluesky","mastodon","youtube"]
|
||
matchMode?: "any" } // reserved; v1 = single term set, no boolean ops
|
||
// account_list
|
||
{ accounts: Array<{ provider: string, handle: string }> } // 1..50 entries; handle free-text "@user@instance.social" | "@user" | "UC..." for YT
|
||
```
|
||
|
||
`refresh_policy` (clamped server-side): `{ intervalMs?: 60_000..3_600_000, paused?: boolean }`. Defaults per type: `platform_feed` 60 000 (inherits today's effective cadence), `keyword` 120 000, `account_list` 180 000, `saved_search` 300 000. The clamp floor is **never below the per-connection budget-derived interval** — a user setting 60 s on a source whose budget says 300 s gets 300 s and a `meta` hint `effectiveIntervalMs`.
|
||
|
||
**Keyless sources:** `column_sources.connection_id` is NULL for YouTube-RSS channels, public Bluesky (`getAuthorFeed`/`searchPosts` unauthenticated), and Mastodon public/RSS reads. These draw from the **per-host budget only** (plus a global keyless concurrency cap of 2 in-flight per host), not from any connection's token bucket. This is what lets an account-list column follow YouTube/Mastodon accounts with **zero connections**.
|
||
|
||
### 1.3 Migration approach
|
||
|
||
1. **DDL migration** (additive, numbered): the five tables above. No existing table touched.
|
||
2. **Lazy seeding (no backfill job):** on first `GET /feeds/columns` for a workspace with zero rows, the API inserts one `platform_feed` column per feed-capable visible connection (kinds `["home"]`, title `` `${displayName} · ${handle}` ``) — exactly what `liveColumns()` derives client-side today. Seeding is idempotent under the unique `(workspace_id, position)` index (transaction + `ON CONFLICT DO NOTHING` re-check).
|
||
3. **Saved-search migration (opt-in, client-driven):** `POST /feeds/columns/import-saved-searches { terms: string[] }` creates `saved_search` columns (platforms = all search-capable connected/keyless providers). The web app calls it once when localStorage `tmbh.saved-searches` is non-empty and the server list lacks them; on success it clears localStorage and dispatches `saved-searches-change`. LocalStorage remains the fallback when the API is unreachable (sample mode) — no behavior regression offline.
|
||
4. **Rollback story:** columns are additive; disabling the feature flag (`STREAMS_COLUMNS_ENABLED`, default on) falls the UI back to today's derivation path. `/feeds` legacy shape never changes.
|
||
|
||
### 1.4 API surface
|
||
|
||
All under the existing feeds module, RBAC `draft.view` to read; **new permission `streams.columns.manage`** (granted to Owner/Admin/Editor by default) to write. Group-visibility: a `platform_feed` column referencing a non-visible connection is rejected `403`; keyword/account columns only see what their sources legitimately fetch.
|
||
|
||
```
|
||
GET /feeds/columns → { columns: ColumnView[] } (ordered, with per-source status)
|
||
POST /feeds/columns → 201 ColumnView (validates spec, resolves column_sources)
|
||
PATCH /feeds/columns/:id → ColumnView (title, spec, refresh_policy, position-delta)
|
||
DELETE /feeds/columns/:id → 204 (soft delete; source items expire naturally)
|
||
POST /feeds/columns/reorder → { columns: ColumnView[] } { orderedIds: string[] } — rewrites positions
|
||
POST /feeds/columns/import-saved-searches → { columns: ColumnView[] }
|
||
GET /feeds/columns/:id/items?cursor&limit(1..200,def 40) → { items: FeedItemDTO[], nextCursor, meta: ColumnMeta }
|
||
POST /feeds/columns/:id/refresh → 202 (shares lease/cooldown/budget machinery)
|
||
```
|
||
|
||
`ColumnView`:
|
||
```ts
|
||
{ id, type, title, position, spec, refreshPolicy, effectiveIntervalMs,
|
||
sources: Array<{ id, provider, connectionId, sourceKind, query, actorRef,
|
||
state, lastPolledAt, lastError?, handle?, displayName? }>,
|
||
aggregateState: "ready"|"pending"|"stale"|"error"|"partial"|"empty"|"paused" } // worst-of, §3.5
|
||
```
|
||
|
||
`FeedItemDTO` (column items endpoint; superset of today's `/feeds` item — the UI's `StreamItem` maps 1:1):
|
||
```ts
|
||
{ id, remoteId, provider, connectionId: string|null, sourceId,
|
||
authorHandle, authorRef?, url?, createdAt,
|
||
content: RichTextAst, // §5 — the ONLY render source
|
||
plainText, media: MediaRef[], metrics: Record<string,number> }
|
||
```
|
||
`ColumnMeta`: `{ sources: [...same as ColumnView.sources], fetchedAt, nextCursor semantics }`.
|
||
|
||
**Mapping into `/feeds` serving (back-compat + one merged view):**
|
||
- `GET /feeds` **unchanged**: merged home-timeline view over capable connections, `meta.connections`, `excludedConnections`, cursor format, `POST /feeds/refresh`. It now reads through the same `feed_items` store (cache-first) — external shape identical.
|
||
- `GET /feeds` gains **one additive** field: `meta.columns?: { id, title, type, position, aggregateState }[]` (omitted when the feature flag is off). The "All feeds" chip and offline/sample mode keep working off the old shape.
|
||
- Per-column serving is **only** `GET /feeds/columns/:id/items`. The merged `/feeds` never mixes column-filtered results, so the current switcher ("All feeds" / "Columns" / per-column / per-saved-term) keeps exact semantics.
|
||
|
||
**Cursor rule:** every cursor is the existing base64url `{at, key}` position tuple over a **deterministic sort** (`created_at_remote DESC, dedup_key DESC`). Per-column cursors are stable because they page persisted rows, not a Redis snapshot.
|
||
|
||
---
|
||
|
||
## 2. Feed acquisition strategy per platform
|
||
|
||
Order follows the research matrix ("Low effort, $0" → up). **Build order: Bluesky → Mastodon → YouTube → X(paid, gated) → Instagram(hashtag, gated) → TikTok(embed-only, out of scope).**
|
||
|
||
### 2.1 Platform decision table
|
||
|
||
| Platform | Keyword column | Account-list column | Mechanism | Auth / credential | Budget | Phase |
|
||
|---|---|---|---|---|---|---|
|
||
| **Bluesky** | `app.bsky.feed.searchPosts` (`sort=latest`, `q`) | `app.bsky.feed.getAuthorFeed(actor)` | native REST via connector; **poll first**, jetstream later (§2.4) | **optional app password** — created via `createAppPassword`, vaulted per workspace as a *column credential*; unauthenticated mode is best-effort fallback (api.bsky.app works; design never *requires* keyless) | AppView "generous"; per-host bucket default 30 req/10 min/host; keyless concurrency cap 2 | **P2 / P3** |
|
||
| **Mastodon** | hashtag timeline `timelines/tag/:tag` only (no global search — hard platform fact); non-tag query → provider marked `unsupported` | `accounts/:id/statuses` (public) **or** `@user.rss` (verified keyless) — prefer RSS for keyless follows, `/statuses` when a connection exists | native via existing connector | none for public/RSS; user token (existing connection) for streaming later | 300 req/5 min per account/IP → per-host bucket 10 req/10 s default (egress), per-connection 0.5 RPS cap as today | **P2 / P3** |
|
||
| **YouTube** | Data API `search.list` (`type=video`, `publishedAfter` = last poll − slack) | **channel RSS** `feeds/videos.xml?channel_id=` (15 latest, keyless, verified) as primary; `playlistItems.list` (1 unit/50) for backfill/deep history | RSS fetcher (new shared `rss-source` reader in ingest, not a full connector) + Data API client | **API key** in vault, workspace-scoped `youtube_data` credential; RSS needs nothing | `search.list` own bucket **100 calls/day** → ledger `yt_search_calls` cap 90/day (10% headroom); `playlistItems` from shared 10k units; RSS = per-host bucket only. **30-day refresh-or-delete** on Data-API items → `expires_at`, attribution required in UI | **P2 (search) / P3 (RSS)** |
|
||
| **X / Twitter** | `tweets/search/recent` (last 7–10 d) | `users/:id/tweets` | native connector, **behind a pay-per-use gate** | OAuth2 app Bearer; billing = operator's console credits (pay-per-use $0.005/post read; free tier has no reads) | **$ ledger**: `x_post_reads` cap per workspace (default **0 = off**); every read debited; hard stop at cap → source `budget_paused`; platform 3M reads/mo cap noted but irrelevant vs our caps | **P5 (opt-in)** |
|
||
| **Instagram** | Graph API `ig_hasid_search` → `top_media`+`recent_media` (hashtag columns only) | **not possible** (others' accounts) — UI shows embed-card fallback for known URLs only, never auto-follow | native connector extension | Meta app + App Review; Business/Creator account connection | **30 unique hashtags / rolling 7 d** → ledger `ig_hashtag_window`; source `rate_limited` + "next tag slot frees {time}" when exhausted; `recent_media` ~50-item window → treat as shallow | **P5 (gated on App Review)** |
|
||
| **TikTok** | none compliant (Research API = academic/nonprofit only; Commercial = ads) | embed-only for known URLs | none — **out of scope**; render link-cards | n/a | n/a | not built |
|
||
| **Telegram / Discord / Slack** | n/a | n/a (channel-picker columns are WP5 inbox work) | existing `getFeed` home paths unchanged | existing | existing | n/a |
|
||
| **Threads** | n/a (API is own-account only) | n/a | unchanged | n/a | n/a | n/a |
|
||
|
||
**Bridge policy:** no RSS-Bridge/RSSHub/Nitter in any phase of this plan. Telegram public-channel via RSS-Bridge is recorded as a possible later opt-in with legal sign-off; not designed in.
|
||
|
||
### 2.2 SDK interface changes (decided)
|
||
|
||
Extend `PlatformConnector` (additive, optional methods — existing connectors untouched):
|
||
|
||
```ts
|
||
// keyword search — lifts the existing getFeed kind:"search" branches verbatim
|
||
search?(connection | null, { query: string, cursor?: string, limit: number }): Promise<Page<FeedItem>>;
|
||
// author timeline — new
|
||
getUserFeed?(connection | null, { actorRef: string, cursor?: string, limit: number }): Promise<Page<FeedItem>>;
|
||
```
|
||
`connection | null` formalizes **keyless sources** (Bluesky public, Mastodon RSS/public, YouTube RSS). Capability flags added to `ConnectorCapabilities` + manifests: `search: boolean`, `authorFeed: boolean`. These mirror into `/feeds` `meta.connections[].capabilities` (already a string array) so the UI can gate affordances with zero new endpoints. Reddit's degenerate "search = subreddit hot" is **removed** from the search path (no `search()` implementation → `unsupported`), fixing a current semantic lie.
|
||
|
||
A small **`rss-source` reader** (XML→FeedItem) lives in the ingest package for YouTube/Mastodon RSS — it is not a `PlatformConnector` because it has no publish/identity surface; it registers per-host budget usage like any egress caller.
|
||
|
||
### 2.3 Credential & key residence (vault)
|
||
|
||
- **Connection credentials** (Mastodon/Bluesky/IG/X user tokens): unchanged — `vault.open/seal` per `(workspaceId, connectionId)` exactly as `feeds.dependencies.ts` does today.
|
||
- **Column credentials (new class):** workspace-scoped secrets not tied to a connection — Bluesky app password (optional), YouTube Data API key, X app Bearer. Stored vaulted under `(workspaceId, "columns", provider)`; the poller resolves provider→credential handle at poll time. **Never** in `stream_columns.spec`, never logged, never sent to the client (the API returns only `{ provider, configured: boolean, label }`).
|
||
- **Keyless sources carry no secret at all** — egress policy + per-host budget is the only control (which is why they're preferred where equal).
|
||
|
||
### 2.4 Polling vs streaming
|
||
|
||
- **v1 = polling everywhere.** The generalized scheduler (§3.4) with per-source `next_poll_at`, exponential backoff on failure (×2 up to 30 min), jitter ±20%.
|
||
- **Bluesky Jetstream (P6, optional):** a single workspace-shared subscriber (`collections=app.bsky.feed.post`) maintains a **local keyword index** (FTS table) that backs keyword columns for *Bluesky only*, giving near-real-time + historical depth beyond `searchPosts` page depth. Rules: self-hostable binary or public endpoint, no auth, cursor replay on reconnect, **index rows follow the same safe-content AST + 30-day retention**, and keyword columns **fall back to `searchPosts` polling** whenever the subscriber is down (state `stale`, never `error`). Jetstream is an accelerator, never a dependency.
|
||
- **Mastodon streaming** (user token + `read:statuses`) is the same pattern later; hashtag polling is good enough for v1.
|
||
|
||
### 2.5 Per-host rate budgets (reusing WP7/WP8 machinery)
|
||
|
||
No new limiter. The column poller:
|
||
1. computes the target host from the source's provider+instance (Mastodon instances vary! host = the connection's instance, not `mastodon.social` globally);
|
||
2. acquires via `egress` → `app.take_host_budget(host_id, 'monitor', n, lease_seconds)` exactly like the live-monitor pollers — so columns, monitor polls, and any other caller of the same instance share one bucket;
|
||
3. honors `429`/`Retry-After` → cooldown key + source state `rate_limited`;
|
||
4. keeps the existing per-connection Redis lease/cooldown (`feeds:{ws}:{conn}:…` extended with `:{columnSourceId}`) so **manual refresh cannot bypass budgets** and replicas never double-poll (identical pattern to today's `FeedsService`, keyed finer).
|
||
|
||
Per-provider defaults (conservative, env-overridable, `CONNECTOR_RATE_<PROVIDER>_RPS` style retained): Bluesky host 0.2 RPS sustained (keyless 0.05 RPS), Mastodon per-instance 0.1 RPS, YouTube RSS host 0.05 RPS, YouTube API search 90 calls/day ledger, X gated by spend ledger, Instagram by hashtag-window ledger. The **provider-budget ledger** (table 5) is checked *before* the host-budget debit; exhausted → `budget_paused` (distinct from `rate_limited`: no retry storm, requires user action or window rollover).
|
||
|
||
---
|
||
|
||
## 3. Ingest / search pipeline
|
||
|
||
### 3.1 Source resolution (column spec → column_sources)
|
||
|
||
On create/update of a column, a **resolution step** (no network for keyword columns; bounded network for account_list) writes `column_sources`:
|
||
|
||
- `platform_feed`: 1–2 sources (per kind) on the named connection.
|
||
- `keyword`: for each `platforms[]` entry → one source per (provider, connection-or-keyless) that **declares the capability** (`search: true` or RSS-search path). Providers in `platforms[]` without capability → **no row**; the API response lists them under `spec.unsupportedPlatforms` so the editor can show "TikTok: not available" without inventing a failing source. Query adaptation happens here, once, per provider:
|
||
- Bluesky: query as-is (full-text).
|
||
- Mastodon: extract first `#hashtag` token; if none → this provider is `unsupported` for the column (UI states "Mastodon supports #hashtag queries only").
|
||
- YouTube: query as-is (`search.list`, `type=video`).
|
||
- X: query as-is (`search/recent`), only if spend ledger enabled.
|
||
- Instagram: only if the entire query is a single hashtag; else `unsupported`.
|
||
- `account_list`: each entry `{provider, handle}` → resolve to `actor_ref` via provider-specific resolution (`resolveHandle` for Bluesky DID, Mastodon `accounts/lookup`, YouTube channel-id via RSS/`channels.list` if key present). Resolution uses the bounded resolver pattern (≤4 in-flight/host, 10-min positive cache / 1-min negative). Unresolvable handle → source row with `state:'error', last_error:'handle_not_found'` — **it does not fail the column**.
|
||
|
||
### 3.2 Normalization (into the existing shape, extended)
|
||
|
||
Every source type funnels into one `normalize()` (today's `feeds.service.ts` validation retained: finite dates, string remoteId/handle, finite metrics; invalid items dropped + counted in source meta):
|
||
|
||
| Provider source | author_handle | content (AST) | media | metrics | url |
|
||
|---|---|---|---|---|---|
|
||
| Bluesky post | handle | facets→AST (links, mentions) | embed images/video → **proxy** | likes/reposts/replies | `bsky.app/profile/…/post/…` |
|
||
| Mastodon status/RSS | `@user@instance` | HTML→AST via allowlist parser (strip scripts/styles/iframes; keep p/br/a/strong/em; links http/https only) | media attachments → proxy | reblogs/favourites/replies | status permalink |
|
||
| YouTube video/RSS item | channel title | title+description→AST (plain paragraphs; links auto-linked http/https only) | thumbnail → proxy | viewCount/likeCount (API only) | watch URL |
|
||
| X tweet | handle | entities→AST | media → proxy | likes/reposts/replies | tweet permalink |
|
||
| IG hashtag media | username | caption→AST | media → proxy | like/comment count | permalink |
|
||
|
||
`plain_text` is derived at ingest (AST→text) so saved-search-style client filtering and the column preview keep working. **`media[].proxyUrl` is always our media proxy URL** (WP3 media path) — raw provider CDN URLs never reach the browser.
|
||
|
||
### 3.3 Dedup & identity
|
||
|
||
- `dedup_key = sha256(workspace_id | provider | canonical_remote_id)` where `canonical_remote_id` = `remoteId` lower-cased and, for handle-bearing providers, `{host}/{remoteId}` with the IDNA-canonical host (same canonicalization rule as WP7 `remote_ref`: lowercase host, punycode, default-port stripped). One object reached via two connections (or a connection + keyless) collapses to one row; the second ingest updates metrics/fetched_at in place (`ON CONFLICT … DO UPDATE` on metrics + fetched_at).
|
||
- **Cross-provider dedup is explicitly NOT attempted** (no fuzzy matching) — an item is provider-scoped. The UI dedups by `streamItemKey` client-side as today.
|
||
- `feed_item_columns` is filled by fan-out at ingest: an item lands in every column that shares that `column_source` identity — i.e., columns referencing the same (provider, actor_ref) or (provider, query) share ingested rows, so following the same account in two columns costs **one** poll (mirrors the WP8 "one remote live, three channels, one poller" rule).
|
||
|
||
### 3.4 Scheduler (generalizing FeedsService)
|
||
|
||
The existing 5 s discovery loop stays; `pollAll()` becomes "poll due `column_sources`" (the legacy per-connection home poll is literally a `platform_feed` source, so old behavior is the degenerate case):
|
||
- select due sources (`next_poll_at <= now`, `state NOT IN ('disabled')`), `FEED_CONCURRENCY=4` waiter queue retained;
|
||
- per-source flow: lease → resolve credential (vault or keyless) → budget/ledger check → egress call via connector `getFeed/search/getUserFeed` or RSS reader → normalize → upsert `feed_items` + fan-out join → write hot Redis slice (`feeds:{ws}:{column}:items` head, TTL = interval×1.2+30 s) → update source state/`next_poll_at` (success: interval; failure: backoff ×2 cap 30 min, `consecutive_failures`≥3 → `error`);
|
||
- `RateLimitError.retryAfterMs` → cooldown, state `rate_limited`; ledger empty → `budget_paused`;
|
||
- manual `POST /feeds/columns/:id/refresh` → same path (202; cooldown may make it a no-op — response carries `effectiveNextPollAt`).
|
||
|
||
### 3.5 Stale / failed / empty behavior in a column
|
||
|
||
Per-source `state` (§1.2) aggregates to `aggregateState` shown in the column header:
|
||
- `ready` — all sources fresh (< 2× effective interval).
|
||
- `stale` — any source's `last_polled_at` older than 2× interval but data present → header "Updated HH:MM" + amber dot; **old items remain visible** (persisted store — the big UX win over today's TTL'd cache).
|
||
- `error` — all sources failed ≥3× → red dot + per-source error line (from `last_error`, provider message scrubbed); last-known items stay visible with "showing last data from …".
|
||
- `partial` — some sources fine, some error/stale → amber; per-source chips in column menu.
|
||
- `rate_limited` / `budget_paused` — distinct icon + actionable copy ("X reads paused: budget reached — raise cap in settings" / "Instagram hashtag slots full until {time}").
|
||
- `empty` — source healthy, zero items ever → per-platform empty-state copy ("No public posts match #acampada26 on Mastodon yet").
|
||
- `unsupported` — never a source row; surfaced in editor only.
|
||
Deleting the underlying connection → its sources flip `disabled`; column survives with remaining sources; a `platform_feed` column whose connection dies becomes `error` with a reconnect CTA (not auto-deleted).
|
||
|
||
**Retention / expiry sweeper:** daily job deletes `feed_items WHERE expires_at < now()` (YouTube Data-API sourced) and everything older than 30 days regardless of source (matches the strictest ToS we're under; RSS items keep 30 d too for uniformity). Deletion is the default; re-polling a live column continuously refreshes the visible window, satisfying "refresh or delete".
|
||
|
||
---
|
||
|
||
## 4. UI / UX spec (Streams page integration)
|
||
|
||
### 4.1 Model change in the web app
|
||
|
||
`lib/streams-data.ts` gains `columnViewsToColumns(ColumnView + items)` producing today's `ColumnDef` (key = column id, title, platformId from dominant provider, staleness from `aggregateState`/`lastPolledAt`). The sample/offline path (`MOCK_COLUMN_DEFS` + fixtures) is untouched. Data flow:
|
||
|
||
- mount → `GET /feeds/columns` (once, plus after any mutation) → for the visible columns (and lazily the next offscreen ones) `GET /feeds/columns/:id/items` per column, 30 s polling retained per visible column (staggered), merged into `ColumnDef[]`.
|
||
- "All feeds" chip → unchanged `GET /feeds` merged view. "Columns" chip → column array as above. Per-column chips → single column. Saved-term chips now resolve to their migrated server columns (id `saved:{term}` → real id after migration; localStorage terms still render offline).
|
||
|
||
### 4.2 Column editor
|
||
|
||
- **"+ Column" button** opens a right-side drawer (desktop) / bottom sheet (mobile): type selector (Platform feed · Keyword · Accounts · Saved search), then type-specific fields:
|
||
- *Platform feed*: connection picker (visible connections only, feed-capable, capability badge), kinds toggles (Home / Mentions).
|
||
- *Keyword*: query input with live per-platform feasibility hints as you type ("Bluesky ✓ full-text · Mastodon #hashtag only · YouTube ✓ · TikTok ✗"), platform multi-select chips (search-capable enabled, others visible-but-disabled with reason tooltip), interval select (1/2/5/10/30 min — clamped label shows effective value).
|
||
- *Accounts*: multi-entry list (paste handles one per line or chip input; provider inferred from handle shape `@x@bsky.social`/`@x@mastodon.social`/`UC…`; `@x` on TikTok/IG → blocked with reason), live resolution status per row (✓ resolved / ✗ not found / ⏳ resolving), 50-entry cap enforced client+server.
|
||
- *Saved search*: term input; identical to keyword with `origin` label.
|
||
- **Edit**: column header "⋯" menu → same drawer prefilled; changing spec re-resolves sources (existing items for retained sources persist; removed sources' items age out via retention, no mass delete).
|
||
- **Reorder**: drag handle on column headers (desktop) / up-down arrows in the columns sheet (mobile) → `POST /reorder`.
|
||
- **Remove**: "⋯" → Remove (soft delete; confirm if the column has fresh data) and Pause (refresh_policy.paused).
|
||
- Optimistic UI on create/reorder/remove with server reconciliation; failure → toast + revert.
|
||
|
||
### 4.3 States, keyboard, mobile
|
||
|
||
- **Empty states:** no columns → existing `EmptyState` + "Add column" CTA. Column with no items → per-source empty copy (§3.5). No connections at all → today's sample-mode fixtures with a "Sample data — connect an account" banner (unchanged).
|
||
- **Error states:** header dot + per-source detail in the column menu; a "source status" strip lists each source (provider, handle/query, state, last polled, error).
|
||
- **Sample state:** identical to today (API unreachable / 0 configured feeds → fixtures); the editor still opens but Create is disabled with "offline — changes need the server".
|
||
- **Keyboard:** j/k flat navigation retained; `c` opens the column editor; `1..9` jumps to column n.
|
||
- **Mobile:** single-column reading with the existing `SourceChips` bar (All feeds · Columns · each column · saved terms); Columns mode = swipeable pager (one column per screen, `PullToRefresh` triggers the column refresh endpoint); editor as bottom sheet; drag-reorder replaced by arrows. Desktop keeps the multi-column wall.
|
||
|
||
### 4.4 Back-compat guarantee
|
||
|
||
The source switcher's four modes (all / columns / per-column / saved-term) all survive with identical semantics; when the flag is off or seeding hasn't run, the page behaves exactly as the current build (derived columns from `/feeds` meta). Saved-search localStorage remains the offline fallback.
|
||
|
||
---
|
||
|
||
## 5. Security design
|
||
|
||
1. **safe-content as the only render path.** `FeedItemDTO.content` is the safe-content **typed rich-text AST**, produced **at ingestion** (server-side): Mastodon HTML → AST via the allowlist parser inside the connector/ingest boundary; Bluesky facets, YouTube/RSS text, X entities → AST builders. Raw provider HTML **never** crosses the API boundary; there is no field a client could accidentally render. The web `StreamColumn`/`PostCard` render exclusively `<SafeContent ast={item.content}/>`; the repo lint rule fails on `dangerouslySetInnerHTML` (existing WP3 acceptance) and additionally on any new string prop named `html`.
|
||
2. **Links:** AST link nodes are http/https-only (other schemes dropped at ingest, not CSS-hidden); unsafe-scheme links render as inert text, not clickable.
|
||
3. **Media:** `media[].proxyUrl` only — provider CDN URLs are proxied through the media path (CSP `img-src`/`media-src` never include provider origins). No `<iframe>` embeds for IG/TikTok in v1 (an embed-only fallback, if ever added, gets its own sandboxed frame + CSP review — explicitly out of scope here).
|
||
4. **SSRF:** no user-supplied URL is ever fetched. Account handles and queries go through provider APIs / the bounded resolver, all via `egress` (scheme allowlist, private-range denial, resolve-once pinning). YouTube/Mastodon RSS host URLs are constructed from validated channel-id/handle forms, never pasted URLs.
|
||
5. **Secrets:** vault-only (§2.3); API responses expose `configured: boolean` at most. Ledger/credential management endpoints are admin-gated.
|
||
6. **Authz:** read = `draft.view` (+ group visibility for connection-backed sources); write = `streams.columns.manage`; every route workspace-scoped and RLS-checked; import-saved-searches is rate-limited per user (it creates rows).
|
||
7. **Cost safety:** X/IG gated ledgers with per-workspace caps default-off/pending-review; a keyword column targeting X shows a **pre-flight cost estimate** (posts × $0.005) before creation.
|
||
|
||
---
|
||
|
||
## 6. Phased build plan (agent-executable slices)
|
||
|
||
Tiers: **flash** = small fully-specified slice; **glm** = medium feature slice; **sol** = large/complex or cross-cutting. Phases strictly dependency-ordered; marked slices parallelize.
|
||
|
||
### P0 — Foundations (flash ×3, parallel)
|
||
- **P0a Schema & validation:** migrations for the 5 tables; `spec`/`refresh_policy` zod schemas + clamps in `packages/core`; unit tests for clamps and invalid-spec rejection. **AC:** migration applies on a populated DB; invalid specs rejected `422`; `(workspace_id, position)` unique holds under concurrent inserts.
|
||
- **P0b SDK contract:** optional `search()` / `getUserFeed()` (`connection | null`), capability flags `search`/`authorFeed`, manifest validation, contract-test fixtures. **AC:** extended contract tests pass; existing connectors compile untouched; capability strings surface in `/feeds` meta.
|
||
- **P0c FeedItem extension + normalize v2:** extended normalization (author_ref, url, AST content, proxy media refs, expires_at) with per-connector fixtures incl. hostile HTML (script tag, `javascript:` link, iframe → AST contains none). **AC:** malicious fixture yields an AST with zero raw remnants; plain_text derivation stable.
|
||
|
||
### P1 — Column CRUD + platform_feed persistence (glm)
|
||
Tables wired into the API; `GET/POST/PATCH/DELETE /feeds/columns`, `reorder`, `import-saved-searches`; lazy seeding; RBAC `streams.columns.manage`; `GET /feeds/columns/:id/items` keyset-cursored over `feed_items`; scheduler generalization (due-sources loop, per-source lease/cooldown, backoff — legacy home poll becomes the degenerate `platform_feed` source); `/feeds` served cache-first from the store, external shape unchanged + additive `meta.columns`.
|
||
**AC:** seeded workspace reproduces today's columns exactly (title/key/staleness); CRUD + reorder RBAC-enforced (403 for ungranted roles); manual refresh cannot bypass cooldown (existing test extended to column endpoints); cursor survives a Redis flush; re-running seeding creates no duplicates.
|
||
|
||
### P2 — Keyword columns: Bluesky + Mastodon hashtag + YouTube search (glm ×2 + sol + parallel UI glm)
|
||
- **P2a (glm):** Bluesky `search()` (optional app-password credential, keyless fallback), Mastodon hashtag `search()` (+ explicit `unsupported` for non-tag queries), query-adaptation resolver, keyword column creation flow, per-source states.
|
||
- **P2b (glm):** YouTube Data-API `search()` client, `yt_search_calls` ledger (cap 90/day, pre-check, `budget_paused`), `publishedAfter` sliding window, `expires_at` = +30 d, attribution line in UI.
|
||
- **P2c (sol):** ingest fan-out/dedup correctness (two columns sharing a query → one poll, shared rows; one object via two connections → one row), retention sweeper.
|
||
- **UI slice (glm, parallel after P1):** column editor drawer (keyword type first), per-platform feasibility hints, source-status strip, mobile sheet.
|
||
**AC:** "acampada26" column on Bluesky+Mastodon+YouTube returns merged, deduped, AST-rendered items; exhausting the API key quota → YouTube source `budget_paused`, others unaffected; a `#tag`-less query shows Mastodon as unsupported (never an error); Playwright malicious-content corpus renders inert through the column UI; YouTube items auto-expire at 30 d.
|
||
|
||
### P3 — Account-list columns (glm ×2)
|
||
- **P3a:** `getUserFeed()` for Bluesky (`getAuthorFeed`) + Mastodon (`/accounts/:id/statuses` with connection, RSS keyless) + YouTube channel RSS via the `rss-source` reader; bounded handle resolver with positive/negative caching.
|
||
- **P3b:** account-list editor UX (paste list, live resolution, 50-cap), mixed connection/keyless column, connection-loss behavior (`disabled`/reconnect CTA).
|
||
**AC:** a 50-account column across 3 platforms polls each host within budget (mock server counts requests); unresolvable handle → row error, column healthy; revoking the Mastodon connection flips its sources to `disabled` while Bluesky keyless sources are unaffected; the same account in two columns = one poll.
|
||
|
||
### P4 — Saved-search migration + polish (flash + glm)
|
||
localStorage import endpoint wired into the page; offline fallback preserved; "Layout" stub becomes real (column widths persisted client-side); keyboard `c`/`1..9`; empty/error copy pass.
|
||
**AC:** a fresh browser with legacy localStorage sees identical saved columns server-side, localStorage cleared, no duplicates on double-run; feature flag off → behavior identical to the pre-columns build.
|
||
|
||
### P5 — Gated paid/limited platforms (sol ×2, parallel after P2's ledger exists)
|
||
- **P5a X:** connector `search()`/`getUserFeed()` vs `search/recent` + `users/:id/tweets`, spend ledger ($/post-read, per-workspace cap default 0=off), pre-flight cost estimate UI, `budget_paused` UX.
|
||
- **P5b Instagram:** Graph-API hashtag sources (30-tags/7d ledger window math, `top_media`+`recent_media`, shallow-window handling), App-Review notes; account-follow explicitly surfaced as impossible with embed-card fallback design.
|
||
**AC:** an X column with a $2 cap stops exactly at the cap (mock billing counts reads) and resumes only on explicit cap raise; the 31st unique IG hashtag in a window → `rate_limited` with next-slot time; both default-disabled via env.
|
||
|
||
### P6 — Jetstream accelerator + realtime (sol, optional, after P2a)
|
||
Workspace-shared Jetstream subscriber → local keyword index (same AST/retention); keyword columns prefer the index with `searchPosts` polling fallback; WS push of column item deltas via the existing `RealtimeModule` (versioned, last-value-wins per column).
|
||
**AC:** killing the subscriber degrades columns to polling (`stale`, not `error`) within one interval; a new matching post appears < 5 s locally; index rows expire at 30 d; no duplicate rows vs the polling path.
|
||
|
||
**Dependency chain:** P0 → P1 → P2 → P3 → P4; P5 ∥ after P2; P6 after P2a. Malicious-content Playwright acceptance runs in every phase touching rendering or ingest (P2, P3, P4 at minimum).
|