T-HUB/docs/design/2026-10-09-columns-design.md

WP3 — Editable Multi-Platform Streams Columns: Design

Status: decision-complete design (no implementation) · Author: glm-5.3 design agent · Inputs: internal-survey.md, platform-research.md, docs/work-breakdown.md WP3, current streams/page.tsx + lib/streams-data.ts.

---

0. Ground rules (non-negotiable constraints this design obeys)

1. No logged-in scraping, no ToS-gray bridges except where explicitly signed off (Telegram via RSS-Bridge is the only candidate; deferred, not in scope of the base plan).
2. All provider content renders only through packages/safe-content (typed rich-text AST; raw HTML never reaches the client). See §5.
3. Every outbound request goes through the egress client and draws from the per-host budget (app.take_host_budget) machinery shipped by WP7/WP8 — columns add callers of that machinery, not a parallel limiter.
4. Back-compat: GET /feeds (merged all-feeds view, meta.connections, cursor shape) keeps working unchanged. Columns are additive.
5. Workspace-scoped everything (multi-tenant; workspace_id in every unique constraint, RLS-checked reads, draft.view-class RBAC extended with a streams.columns.manage permission — see §5.6).

---

1. Column types, schema, migration, API surface

1.1 The four column types

Type	What it shows	Sources resolved at runtime	Example
platform_feed	One connected account's own home/mentions (today's implicit column, made explicit & editable)	1 connection × kinds	"Mastodon · @acampada"
keyword	One query across N platforms (each platform adapter shapes the query: full-text, hashtag, video-search)	per-(connection or keyless-source) search job	"acampada26" on Bluesky+Mastodon+YouTube
account_list	A curated list of accounts across platforms (author timelines / RSS)	per-account author-feed job	"Movement voices" = 5 handles × 3 platforms
saved_search	A keyword column created via the existing saved-search UX (localStorage migrated to server)	same as keyword	"product"

saved_search is not a separate engine — it is keyword with origin: "saved_search". It exists as a persisted type so the UI can label, migrate, and later diverge (e.g. notifications) without another migration. Decision: single type enum column with the four values; the keyword engine is shared.

1.2 Persisted schema (workspace-scoped)

New tables (forward-only numbered SQL migration, additive, no changes to existing tables):

-- 1. Column definitions
CREATE TABLE stream_columns (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id    UUID NOT NULL,
  type            TEXT NOT NULL CHECK (type IN ('platform_feed','keyword','account_list','saved_search')),
  title           TEXT NOT NULL,                    -- user-editable, max 80 chars
  position        INTEGER NOT NULL DEFAULT 0,       -- dense rank; reorder rewrites the tail
  spec            JSONB NOT NULL,                   -- typed per-type, validated in app layer (below)
  refresh_policy  JSONB NOT NULL DEFAULT '{}',      -- { "intervalMs"?: number, "paused"?: boolean }
  created_by      UUID NOT NULL,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  deleted_at      TIMESTAMPTZ                       -- soft delete keeps feed_items FK-stable
);
CREATE UNIQUE INDEX stream_columns_ws_order_uq ON stream_columns (workspace_id, position) WHERE deleted_at IS NULL;
CREATE INDEX stream_columns_ws_idx ON stream_columns (workspace_id) WHERE deleted_at IS NULL;

-- 2. Resolved sources: what the scheduler actually polls for a column
CREATE TABLE column_sources (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id    UUID NOT NULL,
  column_id       UUID NOT NULL REFERENCES stream_columns(id),
  -- identity of the thing polled:
  provider        TEXT NOT NULL,                    -- 'bluesky' | 'mastodon' | 'youtube' | ...
  connection_id   UUID NULL,                        -- NULL for keyless sources (YouTube RSS, public Bluesky/Mastodon)
  source_kind     TEXT NOT NULL CHECK (source_kind IN ('home','mentions','search','author')),
  query           TEXT NULL,                        -- platform-adapted query (post-adaptation, what gets sent)
  actor_ref       TEXT NULL,                        -- canonical account ref for source_kind='author': '{provider}:{handle@host}'
  state           TEXT NOT NULL DEFAULT 'pending'
                  CHECK (state IN ('pending','ready','stale','error','unsupported','rate_limited','budget_paused','disabled')),
  last_error      TEXT NULL,
  last_polled_at  TIMESTAMPTZ NULL,
  next_poll_at    TIMESTAMPTZ NULL,
  consecutive_failures INTEGER NOT NULL DEFAULT 0,
  UNIQUE (workspace_id, column_id, provider, source_kind, COALESCE(connection_id::text,''), COALESCE(query,''), COALESCE(actor_ref,''))
);
CREATE INDEX column_sources_due_idx ON column_sources (next_poll_at) WHERE state NOT IN ('disabled');
CREATE INDEX column_sources_column_idx ON column_sources (column_id);

-- 3. Ingested items (replaces "cache-only" depth; enables deep pagination + per-column serving)
CREATE TABLE feed_items (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id    UUID NOT NULL,
  dedup_key       TEXT NOT NULL,                    -- see §3.3
  provider        TEXT NOT NULL,
  remote_id       TEXT NOT NULL,
  author_handle   TEXT NOT NULL,
  author_ref      TEXT NULL,                        -- canonical '{provider}:{handle@host}' when resolvable
  url             TEXT NULL,                        -- provider permalink (http/https only)
  created_at_remote TIMESTAMPTZ NOT NULL,           -- provider timestamp (sort key)
  fetched_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  content         JSONB NOT NULL,                   -- safe-content typed AST (NEVER raw HTML) — §5
  plain_text      TEXT NOT NULL,                    -- derived flat text (for saved-search client filter & search preview)
  media           JSONB NOT NULL DEFAULT '[]',      -- [{kind:'image'|'video', proxyUrl, width?, height?, alt?}] — proxy-minted only
  metrics         JSONB NOT NULL DEFAULT '{}',      -- finite numbers only
  source_id       UUID NOT NULL REFERENCES column_sources(id),
  expires_at      TIMESTAMPTZ NULL,                 -- set for YouTube Data-API-sourced items: fetched_at + 30d (ToS)
  UNIQUE (workspace_id, dedup_key)
);
CREATE INDEX feed_items_ws_time_idx ON feed_items (workspace_id, created_at_remote DESC, dedup_key DESC); -- keyset
CREATE INDEX feed_items_source_idx ON feed_items (source_id, created_at_remote DESC);
CREATE INDEX feed_items_expiry_idx ON feed_items (expires_at) WHERE expires_at IS NOT NULL;

-- 4. Join: which columns show an item (an item ingested for a source is visible to every column sharing that source)
CREATE TABLE feed_item_columns (
  workspace_id  UUID NOT NULL,
  item_id       UUID NOT NULL REFERENCES feed_items(id) ON DELETE CASCADE,
  column_id     UUID NOT NULL REFERENCES stream_columns(id),
  PRIMARY KEY (item_id, column_id)
);

-- 5. Per-provider budget/spend ledger (X pay-per-use, IG 30-tags/7d, YT search bucket)
CREATE TABLE provider_budget_ledger (
  workspace_id TEXT NOT NULL,                       -- '' = global (API keys are workspace-owned; see §2.5)
  provider     TEXT NOT NULL,
  bucket       TEXT NOT NULL,                       -- 'x_post_reads' | 'ig_hashtag_window' | 'yt_search_calls'
  window_start TIMESTAMPTZ NOT NULL,
  units_used   NUMERIC NOT NULL DEFAULT 0,
  units_cap    NUMERIC NOT NULL,
  UNIQUE (workspace_id, provider, bucket, window_start)
);

Why items are persisted, not just cached: today's 40-item Redis cache + TTL means cursors die and "Load older" is shallow. Columns are user-curated surfaces; they need stable keyset pagination and cross-column dedup. Redis stays as the hot read-through cache in front of feed_items.

spec JSONB shapes (validated by a zod-style schema in packages/core; invalid spec → 422, never stored):

// platform_feed
{ connectionId: string, kinds: ("home"|"mentions")[] }   // default ["home"]
// keyword / saved_search
{ query: string,                       // 1..200 chars, trimmed; no NUL/control chars
  platforms: string[],                 // provider ids, e.g. ["bluesky","mastodon","youtube"]
  matchMode?: "any" }                  // reserved; v1 = single term set, no boolean ops
// account_list
{ accounts: Array<{ provider: string, handle: string }> } // 1..50 entries; handle free-text "@user@instance.social" | "@user" | "UC..." for YT

refresh_policy (clamped server-side): { intervalMs?: 60_000..3_600_000, paused?: boolean }. Defaults per type: platform_feed 60 000 (inherits today's effective cadence), keyword 120 000, account_list 180 000, saved_search 300 000. The clamp floor is never below the per-connection budget-derived interval — a user setting 60 s on a source whose budget says 300 s gets 300 s and a meta hint effectiveIntervalMs.

Keyless sources: column_sources.connection_id is NULL for YouTube-RSS channels, public Bluesky (getAuthorFeed/searchPosts unauthenticated), and Mastodon public/RSS reads. These draw from the per-host budget only (plus a global keyless concurrency cap of 2 in-flight per host), not from any connection's token bucket. This is what lets an account-list column follow YouTube/Mastodon accounts with zero connections.

1.3 Migration approach

1. DDL migration (additive, numbered): the five tables above. No existing table touched.
2. Lazy seeding (no backfill job): on first GET /feeds/columns for a workspace with zero rows, the API inserts one platform_feed column per feed-capable visible connection (kinds ["home"], title `${displayName} · ${handle}`) — exactly what liveColumns() derives client-side today. Seeding is idempotent under the unique (workspace_id, position) index (transaction + ON CONFLICT DO NOTHING re-check).
3. Saved-search migration (opt-in, client-driven): POST /feeds/columns/import-saved-searches { terms: string[] } creates saved_search columns (platforms = all search-capable connected/keyless providers). The web app calls it once when localStorage tmbh.saved-searches is non-empty and the server list lacks them; on success it clears localStorage and dispatches saved-searches-change. LocalStorage remains the fallback when the API is unreachable (sample mode) — no behavior regression offline.
4. Rollback story: columns are additive; disabling the feature flag (STREAMS_COLUMNS_ENABLED, default on) falls the UI back to today's derivation path. /feeds legacy shape never changes.

1.4 API surface

All under the existing feeds module, RBAC draft.view to read; new permission streams.columns.manage (granted to Owner/Admin/Editor by default) to write. Group-visibility: a platform_feed column referencing a non-visible connection is rejected 403; keyword/account columns only see what their sources legitimately fetch.

GET    /feeds/columns                       → { columns: ColumnView[] }   (ordered, with per-source status)
POST   /feeds/columns                       → 201 ColumnView              (validates spec, resolves column_sources)
PATCH  /feeds/columns/:id                   → ColumnView                  (title, spec, refresh_policy, position-delta)
DELETE /feeds/columns/:id                   → 204                         (soft delete; source items expire naturally)
POST   /feeds/columns/reorder               → { columns: ColumnView[] }   { orderedIds: string[] } — rewrites positions
POST   /feeds/columns/import-saved-searches → { columns: ColumnView[] }
GET    /feeds/columns/:id/items?cursor&limit(1..200,def 40) → { items: FeedItemDTO[], nextCursor, meta: ColumnMeta }
POST   /feeds/columns/:id/refresh           → 202                         (shares lease/cooldown/budget machinery)

ColumnView:

{ id, type, title, position, spec, refreshPolicy, effectiveIntervalMs,
  sources: Array<{ id, provider, connectionId, sourceKind, query, actorRef,
                   state, lastPolledAt, lastError?, handle?, displayName? }>,
  aggregateState: "ready"|"pending"|"stale"|"error"|"partial"|"empty"|"paused" }  // worst-of, §3.5

FeedItemDTO (column items endpoint; superset of today's /feeds item — the UI's StreamItem maps 1:1):

{ id, remoteId, provider, connectionId: string|null, sourceId,
  authorHandle, authorRef?, url?, createdAt,
  content: RichTextAst,          // §5 — the ONLY render source
  plainText, media: MediaRef[], metrics: Record<string,number> }

ColumnMeta: { sources: [...same as ColumnView.sources], fetchedAt, nextCursor semantics }.

Mapping into /feeds serving (back-compat + one merged view):

• GET /feeds unchanged: merged home-timeline view over capable connections, meta.connections, excludedConnections, cursor format, POST /feeds/refresh. It now reads through the same feed_items store (cache-first) — external shape identical.
• GET /feeds gains one additive field: meta.columns?: { id, title, type, position, aggregateState }[] (omitted when the feature flag is off). The "All feeds" chip and offline/sample mode keep working off the old shape.
• Per-column serving is only GET /feeds/columns/:id/items. The merged /feeds never mixes column-filtered results, so the current switcher ("All feeds" / "Columns" / per-column / per-saved-term) keeps exact semantics.

Cursor rule: every cursor is the existing base64url {at, key} position tuple over a deterministic sort (created_at_remote DESC, dedup_key DESC). Per-column cursors are stable because they page persisted rows, not a Redis snapshot.

---

2. Feed acquisition strategy per platform

Order follows the research matrix ("Low effort, $0" → up). Build order: Bluesky → Mastodon → YouTube → X(paid, gated) → Instagram(hashtag, gated) → TikTok(embed-only, out of scope).

2.1 Platform decision table

Platform	Keyword column	Account-list column	Mechanism	Auth / credential	Budget	Phase
Bluesky	app.bsky.feed.searchPosts (sort=latest, q)	app.bsky.feed.getAuthorFeed(actor)	native REST via connector; poll first, jetstream later (§2.4)	optional app password — created via createAppPassword, vaulted per workspace as a column credential; unauthenticated mode is best-effort fallback (api.bsky.app works; design never requires keyless)	AppView "generous"; per-host bucket default 30 req/10 min/host; keyless concurrency cap 2	P2 / P3
Mastodon	hashtag timeline timelines/tag/:tag only (no global search — hard platform fact); non-tag query → provider marked unsupported	accounts/:id/statuses (public) or @user.rss (verified keyless) — prefer RSS for keyless follows, /statuses when a connection exists	native via existing connector	none for public/RSS; user token (existing connection) for streaming later	300 req/5 min per account/IP → per-host bucket 10 req/10 s default (egress), per-connection 0.5 RPS cap as today	P2 / P3
YouTube	Data API search.list (type=video, publishedAfter = last poll − slack)	channel RSS feeds/videos.xml?channel_id= (15 latest, keyless, verified) as primary; playlistItems.list (1 unit/50) for backfill/deep history	RSS fetcher (new shared rss-source reader in ingest, not a full connector) + Data API client	API key in vault, workspace-scoped youtube_data credential; RSS needs nothing	search.list own bucket 100 calls/day → ledger yt_search_calls cap 90/day (10% headroom); playlistItems from shared 10k units; RSS = per-host bucket only. 30-day refresh-or-delete on Data-API items → expires_at, attribution required in UI	P2 (search) / P3 (RSS)
X / Twitter	tweets/search/recent (last 7–10 d)	users/:id/tweets	native connector, behind a pay-per-use gate	OAuth2 app Bearer; billing = operator's console credits (pay-per-use $0.005/post read; free tier has no reads)	$ ledger: x_post_reads cap per workspace (default 0 = off); every read debited; hard stop at cap → source budget_paused; platform 3M reads/mo cap noted but irrelevant vs our caps	P5 (opt-in)
Instagram	Graph API ig_hasid_search → top_media+recent_media (hashtag columns only)	not possible (others' accounts) — UI shows embed-card fallback for known URLs only, never auto-follow	native connector extension	Meta app + App Review; Business/Creator account connection	30 unique hashtags / rolling 7 d → ledger ig_hashtag_window; source rate_limited + "next tag slot frees {time}" when exhausted; recent_media ~50-item window → treat as shallow	P5 (gated on App Review)
TikTok	none compliant (Research API = academic/nonprofit only; Commercial = ads)	embed-only for known URLs	none — out of scope; render link-cards	n/a	n/a	not built
Telegram / Discord / Slack	n/a	n/a (channel-picker columns are WP5 inbox work)	existing getFeed home paths unchanged	existing	existing	n/a
Threads	n/a (API is own-account only)	n/a	unchanged	n/a	n/a	n/a

Bridge policy: no RSS-Bridge/RSSHub/Nitter in any phase of this plan. Telegram public-channel via RSS-Bridge is recorded as a possible later opt-in with legal sign-off; not designed in.

2.2 SDK interface changes (decided)

Extend PlatformConnector (additive, optional methods — existing connectors untouched):

// keyword search — lifts the existing getFeed kind:"search" branches verbatim
search?(connection | null, { query: string, cursor?: string, limit: number }): Promise<Page<FeedItem>>;
// author timeline — new
getUserFeed?(connection | null, { actorRef: string, cursor?: string, limit: number }): Promise<Page<FeedItem>>;

connection | null formalizes keyless sources (Bluesky public, Mastodon RSS/public, YouTube RSS). Capability flags added to ConnectorCapabilities + manifests: search: boolean, authorFeed: boolean. These mirror into /feeds meta.connections[].capabilities (already a string array) so the UI can gate affordances with zero new endpoints. Reddit's degenerate "search = subreddit hot" is removed from the search path (no search() implementation → unsupported), fixing a current semantic lie.

A small rss-source reader (XML→FeedItem) lives in the ingest package for YouTube/Mastodon RSS — it is not a PlatformConnector because it has no publish/identity surface; it registers per-host budget usage like any egress caller.

2.3 Credential & key residence (vault)

• Connection credentials (Mastodon/Bluesky/IG/X user tokens): unchanged — vault.open/seal per (workspaceId, connectionId) exactly as feeds.dependencies.ts does today.
• Column credentials (new class): workspace-scoped secrets not tied to a connection — Bluesky app password (optional), YouTube Data API key, X app Bearer. Stored vaulted under (workspaceId, "columns", provider); the poller resolves provider→credential handle at poll time. Never in stream_columns.spec, never logged, never sent to the client (the API returns only { provider, configured: boolean, label }).
• Keyless sources carry no secret at all — egress policy + per-host budget is the only control (which is why they're preferred where equal).

2.4 Polling vs streaming

• v1 = polling everywhere. The generalized scheduler (§3.4) with per-source next_poll_at, exponential backoff on failure (×2 up to 30 min), jitter ±20%.
• Bluesky Jetstream (P6, optional): a single workspace-shared subscriber (collections=app.bsky.feed.post) maintains a local keyword index (FTS table) that backs keyword columns for Bluesky only, giving near-real-time + historical depth beyond searchPosts page depth. Rules: self-hostable binary or public endpoint, no auth, cursor replay on reconnect, index rows follow the same safe-content AST + 30-day retention, and keyword columns fall back to searchPosts polling whenever the subscriber is down (state stale, never error). Jetstream is an accelerator, never a dependency.
• Mastodon streaming (user token + read:statuses) is the same pattern later; hashtag polling is good enough for v1.

2.5 Per-host rate budgets (reusing WP7/WP8 machinery)

No new limiter. The column poller:

1. computes the target host from the source's provider+instance (Mastodon instances vary! host = the connection's instance, not mastodon.social globally);
2. acquires via egress → app.take_host_budget(host_id, 'monitor', n, lease_seconds) exactly like the live-monitor pollers — so columns, monitor polls, and any other caller of the same instance share one bucket;
3. honors 429/Retry-After → cooldown key + source state rate_limited;
4. keeps the existing per-connection Redis lease/cooldown (feeds:{ws}:{conn}:… extended with :{columnSourceId}) so manual refresh cannot bypass budgets and replicas never double-poll (identical pattern to today's FeedsService, keyed finer).

Per-provider defaults (conservative, env-overridable, CONNECTOR_RATE_<PROVIDER>_RPS style retained): Bluesky host 0.2 RPS sustained (keyless 0.05 RPS), Mastodon per-instance 0.1 RPS, YouTube RSS host 0.05 RPS, YouTube API search 90 calls/day ledger, X gated by spend ledger, Instagram by hashtag-window ledger. The provider-budget ledger (table 5) is checked before the host-budget debit; exhausted → budget_paused (distinct from rate_limited: no retry storm, requires user action or window rollover).

---

3. Ingest / search pipeline

3.1 Source resolution (column spec → column_sources)

On create/update of a column, a resolution step (no network for keyword columns; bounded network for account_list) writes column_sources:

• platform_feed: 1–2 sources (per kind) on the named connection.
• keyword: for each platforms[] entry → one source per (provider, connection-or-keyless) that declares the capability (search: true or RSS-search path). Providers in platforms[] without capability → no row; the API response lists them under spec.unsupportedPlatforms so the editor can show "TikTok: not available" without inventing a failing source. Query adaptation happens here, once, per provider:
  • Bluesky: query as-is (full-text).
  • Mastodon: extract first #hashtag token; if none → this provider is unsupported for the column (UI states "Mastodon supports #hashtag queries only").
  • YouTube: query as-is (search.list, type=video).
  • X: query as-is (search/recent), only if spend ledger enabled.
  • Instagram: only if the entire query is a single hashtag; else unsupported.
• account_list: each entry {provider, handle} → resolve to actor_ref via provider-specific resolution (resolveHandle for Bluesky DID, Mastodon accounts/lookup, YouTube channel-id via RSS/channels.list if key present). Resolution uses the bounded resolver pattern (≤4 in-flight/host, 10-min positive cache / 1-min negative). Unresolvable handle → source row with state:'error', last_error:'handle_not_found' — it does not fail the column.

3.2 Normalization (into the existing shape, extended)

Every source type funnels into one normalize() (today's feeds.service.ts validation retained: finite dates, string remoteId/handle, finite metrics; invalid items dropped + counted in source meta):

Provider source	author_handle	content (AST)	media	metrics	url
Bluesky post	handle	facets→AST (links, mentions)	embed images/video → proxy	likes/reposts/replies	bsky.app/profile/…/post/…
Mastodon status/RSS	@user@instance	HTML→AST via allowlist parser (strip scripts/styles/iframes; keep p/br/a/strong/em; links http/https only)	media attachments → proxy	reblogs/favourites/replies	status permalink
YouTube video/RSS item	channel title	title+description→AST (plain paragraphs; links auto-linked http/https only)	thumbnail → proxy	viewCount/likeCount (API only)	watch URL
X tweet	handle	entities→AST	media → proxy	likes/reposts/replies	tweet permalink
IG hashtag media	username	caption→AST	media → proxy	like/comment count	permalink

plain_text is derived at ingest (AST→text) so saved-search-style client filtering and the column preview keep working. media[].proxyUrl is always our media proxy URL (WP3 media path) — raw provider CDN URLs never reach the browser.

3.3 Dedup & identity

• dedup_key = sha256(workspace_id | provider | canonical_remote_id) where canonical_remote_id = remoteId lower-cased and, for handle-bearing providers, {host}/{remoteId} with the IDNA-canonical host (same canonicalization rule as WP7 remote_ref: lowercase host, punycode, default-port stripped). One object reached via two connections (or a connection + keyless) collapses to one row; the second ingest updates metrics/fetched_at in place (ON CONFLICT … DO UPDATE on metrics + fetched_at).
• Cross-provider dedup is explicitly NOT attempted (no fuzzy matching) — an item is provider-scoped. The UI dedups by streamItemKey client-side as today.
• feed_item_columns is filled by fan-out at ingest: an item lands in every column that shares that column_source identity — i.e., columns referencing the same (provider, actor_ref) or (provider, query) share ingested rows, so following the same account in two columns costs one poll (mirrors the WP8 "one remote live, three channels, one poller" rule).

3.4 Scheduler (generalizing FeedsService)

The existing 5 s discovery loop stays; pollAll() becomes "poll due column_sources" (the legacy per-connection home poll is literally a platform_feed source, so old behavior is the degenerate case):

• select due sources (next_poll_at <= now, state NOT IN ('disabled')), FEED_CONCURRENCY=4 waiter queue retained;
• per-source flow: lease → resolve credential (vault or keyless) → budget/ledger check → egress call via connector getFeed/search/getUserFeed or RSS reader → normalize → upsert feed_items + fan-out join → write hot Redis slice (feeds:{ws}:{column}:items head, TTL = interval×1.2+30 s) → update source state/next_poll_at (success: interval; failure: backoff ×2 cap 30 min, consecutive_failures≥3 → error);
• RateLimitError.retryAfterMs → cooldown, state rate_limited; ledger empty → budget_paused;
• manual POST /feeds/columns/:id/refresh → same path (202; cooldown may make it a no-op — response carries effectiveNextPollAt).

3.5 Stale / failed / empty behavior in a column

Per-source state (§1.2) aggregates to aggregateState shown in the column header:

• ready — all sources fresh (< 2× effective interval).
• stale — any source's last_polled_at older than 2× interval but data present → header "Updated HH:MM" + amber dot; old items remain visible (persisted store — the big UX win over today's TTL'd cache).
• error — all sources failed ≥3× → red dot + per-source error line (from last_error, provider message scrubbed); last-known items stay visible with "showing last data from …".
• partial — some sources fine, some error/stale → amber; per-source chips in column menu.
• rate_limited / budget_paused — distinct icon + actionable copy ("X reads paused: budget reached — raise cap in settings" / "Instagram hashtag slots full until {time}").
• empty — source healthy, zero items ever → per-platform empty-state copy ("No public posts match #acampada26 on Mastodon yet").
• unsupported — never a source row; surfaced in editor only. Deleting the underlying connection → its sources flip disabled; column survives with remaining sources; a platform_feed column whose connection dies becomes error with a reconnect CTA (not auto-deleted).

Retention / expiry sweeper: daily job deletes feed_items WHERE expires_at < now() (YouTube Data-API sourced) and everything older than 30 days regardless of source (matches the strictest ToS we're under; RSS items keep 30 d too for uniformity). Deletion is the default; re-polling a live column continuously refreshes the visible window, satisfying "refresh or delete".

---

4. UI / UX spec (Streams page integration)

4.1 Model change in the web app

lib/streams-data.ts gains columnViewsToColumns(ColumnView + items) producing today's ColumnDef (key = column id, title, platformId from dominant provider, staleness from aggregateState/lastPolledAt). The sample/offline path (MOCK_COLUMN_DEFS + fixtures) is untouched. Data flow:

• mount → GET /feeds/columns (once, plus after any mutation) → for the visible columns (and lazily the next offscreen ones) GET /feeds/columns/:id/items per column, 30 s polling retained per visible column (staggered), merged into ColumnDef[].
• "All feeds" chip → unchanged GET /feeds merged view. "Columns" chip → column array as above. Per-column chips → single column. Saved-term chips now resolve to their migrated server columns (id saved:{term} → real id after migration; localStorage terms still render offline).

4.2 Column editor

• "+ Column" button opens a right-side drawer (desktop) / bottom sheet (mobile): type selector (Platform feed · Keyword · Accounts · Saved search), then type-specific fields:
  • Platform feed: connection picker (visible connections only, feed-capable, capability badge), kinds toggles (Home / Mentions).
  • Keyword: query input with live per-platform feasibility hints as you type ("Bluesky ✓ full-text · Mastodon #hashtag only · YouTube ✓ · TikTok ✗"), platform multi-select chips (search-capable enabled, others visible-but-disabled with reason tooltip), interval select (1/2/5/10/30 min — clamped label shows effective value).
  • Accounts: multi-entry list (paste handles one per line or chip input; provider inferred from handle shape @x@bsky.social/@x@mastodon.social/UC…; @x on TikTok/IG → blocked with reason), live resolution status per row (✓ resolved / ✗ not found / ⏳ resolving), 50-entry cap enforced client+server.
  • Saved search: term input; identical to keyword with origin label.
• Edit: column header "⋯" menu → same drawer prefilled; changing spec re-resolves sources (existing items for retained sources persist; removed sources' items age out via retention, no mass delete).
• Reorder: drag handle on column headers (desktop) / up-down arrows in the columns sheet (mobile) → POST /reorder.
• Remove: "⋯" → Remove (soft delete; confirm if the column has fresh data) and Pause (refresh_policy.paused).
• Optimistic UI on create/reorder/remove with server reconciliation; failure → toast + revert.

4.3 States, keyboard, mobile

• Empty states: no columns → existing EmptyState + "Add column" CTA. Column with no items → per-source empty copy (§3.5). No connections at all → today's sample-mode fixtures with a "Sample data — connect an account" banner (unchanged).
• Error states: header dot + per-source detail in the column menu; a "source status" strip lists each source (provider, handle/query, state, last polled, error).
• Sample state: identical to today (API unreachable / 0 configured feeds → fixtures); the editor still opens but Create is disabled with "offline — changes need the server".
• Keyboard: j/k flat navigation retained; c opens the column editor; 1..9 jumps to column n.
• Mobile: single-column reading with the existing SourceChips bar (All feeds · Columns · each column · saved terms); Columns mode = swipeable pager (one column per screen, PullToRefresh triggers the column refresh endpoint); editor as bottom sheet; drag-reorder replaced by arrows. Desktop keeps the multi-column wall.

4.4 Back-compat guarantee

The source switcher's four modes (all / columns / per-column / saved-term) all survive with identical semantics; when the flag is off or seeding hasn't run, the page behaves exactly as the current build (derived columns from /feeds meta). Saved-search localStorage remains the offline fallback.

---

5. Security design

1. safe-content as the only render path. FeedItemDTO.content is the safe-content typed rich-text AST, produced at ingestion (server-side): Mastodon HTML → AST via the allowlist parser inside the connector/ingest boundary; Bluesky facets, YouTube/RSS text, X entities → AST builders. Raw provider HTML never crosses the API boundary; there is no field a client could accidentally render. The web StreamColumn/PostCard render exclusively <SafeContent ast={item.content}/>; the repo lint rule fails on dangerouslySetInnerHTML (existing WP3 acceptance) and additionally on any new string prop named html.
2. Links: AST link nodes are http/https-only (other schemes dropped at ingest, not CSS-hidden); unsafe-scheme links render as inert text, not clickable.
3. Media: media[].proxyUrl only — provider CDN URLs are proxied through the media path (CSP img-src/media-src never include provider origins). No <iframe> embeds for IG/TikTok in v1 (an embed-only fallback, if ever added, gets its own sandboxed frame + CSP review — explicitly out of scope here).
4. SSRF: no user-supplied URL is ever fetched. Account handles and queries go through provider APIs / the bounded resolver, all via egress (scheme allowlist, private-range denial, resolve-once pinning). YouTube/Mastodon RSS host URLs are constructed from validated channel-id/handle forms, never pasted URLs.
5. Secrets: vault-only (§2.3); API responses expose configured: boolean at most. Ledger/credential management endpoints are admin-gated.
6. Authz: read = draft.view (+ group visibility for connection-backed sources); write = streams.columns.manage; every route workspace-scoped and RLS-checked; import-saved-searches is rate-limited per user (it creates rows).
7. Cost safety: X/IG gated ledgers with per-workspace caps default-off/pending-review; a keyword column targeting X shows a pre-flight cost estimate (posts × $0.005) before creation.

---

6. Phased build plan (agent-executable slices)

Tiers: flash = small fully-specified slice; glm = medium feature slice; sol = large/complex or cross-cutting. Phases strictly dependency-ordered; marked slices parallelize.

P0 — Foundations (flash ×3, parallel)

• P0a Schema & validation: migrations for the 5 tables; spec/refresh_policy zod schemas + clamps in packages/core; unit tests for clamps and invalid-spec rejection. AC: migration applies on a populated DB; invalid specs rejected 422; (workspace_id, position) unique holds under concurrent inserts.
• P0b SDK contract: optional search() / getUserFeed() (connection | null), capability flags search/authorFeed, manifest validation, contract-test fixtures. AC: extended contract tests pass; existing connectors compile untouched; capability strings surface in /feeds meta.
• P0c FeedItem extension + normalize v2: extended normalization (author_ref, url, AST content, proxy media refs, expires_at) with per-connector fixtures incl. hostile HTML (script tag, javascript: link, iframe → AST contains none). AC: malicious fixture yields an AST with zero raw remnants; plain_text derivation stable.

P1 — Column CRUD + platform_feed persistence (glm)

Tables wired into the API; GET/POST/PATCH/DELETE /feeds/columns, reorder, import-saved-searches; lazy seeding; RBAC streams.columns.manage; GET /feeds/columns/:id/items keyset-cursored over feed_items; scheduler generalization (due-sources loop, per-source lease/cooldown, backoff — legacy home poll becomes the degenerate platform_feed source); /feeds served cache-first from the store, external shape unchanged + additive meta.columns. AC: seeded workspace reproduces today's columns exactly (title/key/staleness); CRUD + reorder RBAC-enforced (403 for ungranted roles); manual refresh cannot bypass cooldown (existing test extended to column endpoints); cursor survives a Redis flush; re-running seeding creates no duplicates.

P2 — Keyword columns: Bluesky + Mastodon hashtag + YouTube search (glm ×2 + sol + parallel UI glm)

• P2a (glm): Bluesky search() (optional app-password credential, keyless fallback), Mastodon hashtag search() (+ explicit unsupported for non-tag queries), query-adaptation resolver, keyword column creation flow, per-source states.
• P2b (glm): YouTube Data-API search() client, yt_search_calls ledger (cap 90/day, pre-check, budget_paused), publishedAfter sliding window, expires_at = +30 d, attribution line in UI.
• P2c (sol): ingest fan-out/dedup correctness (two columns sharing a query → one poll, shared rows; one object via two connections → one row), retention sweeper.
• UI slice (glm, parallel after P1): column editor drawer (keyword type first), per-platform feasibility hints, source-status strip, mobile sheet. AC: "acampada26" column on Bluesky+Mastodon+YouTube returns merged, deduped, AST-rendered items; exhausting the API key quota → YouTube source budget_paused, others unaffected; a #tag-less query shows Mastodon as unsupported (never an error); Playwright malicious-content corpus renders inert through the column UI; YouTube items auto-expire at 30 d.

P3 — Account-list columns (glm ×2)

• P3a: getUserFeed() for Bluesky (getAuthorFeed) + Mastodon (/accounts/:id/statuses with connection, RSS keyless) + YouTube channel RSS via the rss-source reader; bounded handle resolver with positive/negative caching.
• P3b: account-list editor UX (paste list, live resolution, 50-cap), mixed connection/keyless column, connection-loss behavior (disabled/reconnect CTA). AC: a 50-account column across 3 platforms polls each host within budget (mock server counts requests); unresolvable handle → row error, column healthy; revoking the Mastodon connection flips its sources to disabled while Bluesky keyless sources are unaffected; the same account in two columns = one poll.

P4 — Saved-search migration + polish (flash + glm)

localStorage import endpoint wired into the page; offline fallback preserved; "Layout" stub becomes real (column widths persisted client-side); keyboard c/1..9; empty/error copy pass. AC: a fresh browser with legacy localStorage sees identical saved columns server-side, localStorage cleared, no duplicates on double-run; feature flag off → behavior identical to the pre-columns build.

P5 — Gated paid/limited platforms (sol ×2, parallel after P2's ledger exists)

• P5a X: connector search()/getUserFeed() vs search/recent + users/:id/tweets, spend ledger ($/post-read, per-workspace cap default 0=off), pre-flight cost estimate UI, budget_paused UX.
• P5b Instagram: Graph-API hashtag sources (30-tags/7d ledger window math, top_media+recent_media, shallow-window handling), App-Review notes; account-follow explicitly surfaced as impossible with embed-card fallback design. AC: an X column with a $2 cap stops exactly at the cap (mock billing counts reads) and resumes only on explicit cap raise; the 31st unique IG hashtag in a window → rate_limited with next-slot time; both default-disabled via env.

P6 — Jetstream accelerator + realtime (sol, optional, after P2a)

Workspace-shared Jetstream subscriber → local keyword index (same AST/retention); keyword columns prefer the index with searchPosts polling fallback; WS push of column item deltas via the existing RealtimeModule (versioned, last-value-wins per column). AC: killing the subscriber degrades columns to polling (stale, not error) within one interval; a new matching post appears < 5 s locally; index rows expire at 30 d; no duplicate rows vs the polling path.

Dependency chain: P0 → P1 → P2 → P3 → P4; P5 ∥ after P2; P6 after P2a. Malicious-content Playwright acceptance runs in every phase touching rendering or ingest (P2, P3, P4 at minimum).