fix: produce worker claim — SECURITY DEFINER fn fixes 08P01 poll crash + stale-lease reclaim (040)
The store's hand-built claim interpolated the lane predicate into the SQL text while still passing a bind parameter — node-postgres sent 1 param for a 0-param statement (08P01) on every poll, so the live produce worker store-errored every ~2s and never claimed. Migration 040 replaces the inline claim with produce_job_claim(p_lane, p_stale_after): SECURITY DEFINER (the 035/oidc_refresh_lookup pattern — the poller is lane-global, FORCE RLS hides every row without a tenant context), atomic FOR UPDATE SKIP LOCKED with the queued→transcoding bump as the lease, and stale- transcoding reclaim so a killed worker's rows return to the queue. produce_job_workspace(p_job_id) discovers the owning tenant for the follow-up writes, which stay RLS-bound as sp_app. pg-store rewired; store errors now surface err.message instead of err.name; pg-store tests added. 38/38 producer tests green.
This commit is contained in:
parent
d9e8cdf789
commit
ee8d5c6e76
5 changed files with 512 additions and 91 deletions
88
apps/api/migrations/040_produce_claim.sql
Normal file
88
apps/api/migrations/040_produce_claim.sql
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
-- 040: produce-worker job claim — the queue half of 039's producer feature.
|
||||
--
|
||||
-- WHY A SECURITY DEFINER CLAIM: video_processing_jobs is FORCE ROW LEVEL SECURITY
|
||||
-- with a tenant_isolation policy (009), but the produce worker is a lane-global
|
||||
-- poller with NO tenant context: it must see queued rows in EVERY workspace, then
|
||||
-- work each claimed row inside that workspace's scope. 006/012/017/035 established
|
||||
-- the pattern for exactly this shape — a narrow definer-rights claim that
|
||||
-- discovers the tenant and nothing else; the caller enters the tenant context
|
||||
-- afterwards and every subsequent statement stays RLS-bound as sp_app.
|
||||
--
|
||||
-- The same migration also fixes the production bug this claim surfaced
|
||||
-- (2026-10 produce worker logging a store error every ~2 s): the store's inline
|
||||
-- claim interpolated its lane predicate into the SQL text while still passing a
|
||||
-- bind parameter for it, so node-postgres sent `bind message supplies 1
|
||||
-- parameters, but prepared statement "" requires 0` (08P01) on EVERY poll. A
|
||||
-- stored function with a real $1 takes the driver out of the hand-built-SQL
|
||||
-- business entirely; lane becomes a typed parameter that cannot desynchronise.
|
||||
--
|
||||
-- Leasing: the `queued -> transcoding` bump inside FOR UPDATE SKIP LOCKED is the
|
||||
-- lease (no separate lease column). A worker that dies mid-encode leaves its row
|
||||
-- in `transcoding` forever, so the claim also requeues rows whose lease has gone
|
||||
-- stale (transcoding for longer than p_stale_after with no progress): the row
|
||||
-- returns to `queued` and any worker may retake it. Requeue happens BEFORE the
|
||||
-- claim sub-select in the same statement snapshot, so a row released by the sweep
|
||||
-- is claimable by the very same call.
|
||||
|
||||
CREATE FUNCTION produce_job_claim(p_lane text, p_stale_after interval)
|
||||
RETURNS TABLE (
|
||||
id uuid,
|
||||
workspace_id uuid,
|
||||
kind text,
|
||||
input_json jsonb,
|
||||
video_asset_id uuid
|
||||
)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = public AS $$
|
||||
BEGIN
|
||||
-- Lease reclaim: a crashed/killed worker's rows go back to the queue. The
|
||||
-- progress write (the runner's per-stage checkpoint) is the liveness signal;
|
||||
-- `updated_at` is bumped by every progress/claim write, so `updated_at` older
|
||||
-- than the stale window means the owner is gone, not merely slow.
|
||||
UPDATE video_processing_jobs stale
|
||||
SET state = 'queued', progress_percent = 0, error = NULL, updated_at = now()
|
||||
WHERE stale.state = 'transcoding'
|
||||
AND stale.updated_at < now() - p_stale_after;
|
||||
|
||||
-- The claim: one lane-eligible queued row, locked so a concurrent worker cannot
|
||||
-- take the same one. The kind check + lane check live in the sub-select, so a
|
||||
-- worker never locks a row it would only have to release.
|
||||
RETURN QUERY
|
||||
UPDATE video_processing_jobs j
|
||||
SET state = 'transcoding', progress_percent = 5, error = NULL, updated_at = now()
|
||||
WHERE j.id IN (
|
||||
SELECT c.id
|
||||
FROM video_processing_jobs c
|
||||
WHERE c.kind IN ('compose', 'mix')
|
||||
AND c.state = 'queued'
|
||||
AND (
|
||||
(p_lane = 'remote' AND c.input_json->>'lane' = 'remote')
|
||||
OR (p_lane <> 'remote' AND c.input_json->>'lane' IS DISTINCT FROM 'remote')
|
||||
)
|
||||
ORDER BY c.created_at, c.id
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING j.id, j.workspace_id, j.kind, j.input_json, j.video_asset_id;
|
||||
END
|
||||
$$;
|
||||
|
||||
REVOKE ALL ON FUNCTION produce_job_claim(text, interval) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION produce_job_claim(text, interval) TO sp_app;
|
||||
|
||||
-- The worker's follow-up writes are workspace-scoped already (SET LOCAL
|
||||
-- app.workspace_id inside a transaction), so no definer rights are needed there:
|
||||
-- FORCE RLS binds those statements as sp_app, which is the point.
|
||||
|
||||
-- The claim's follow-up writes (progress / complete / fail / the derived asset
|
||||
-- insert) run inside the job's tenant context, so they stay RLS-bound as sp_app —
|
||||
-- but the worker only holds a job id at that point, and an ordinary SELECT cannot
|
||||
-- discover which workspace owns it (FORCE RLS hides every row without a tenant
|
||||
-- context). This is the 012 live_monitor_source_workspace shape verbatim: a narrow
|
||||
-- definer lookup that discovers the owning tenant and nothing else.
|
||||
CREATE FUNCTION produce_job_workspace(p_job_id uuid) RETURNS uuid
|
||||
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$
|
||||
SELECT workspace_id FROM video_processing_jobs WHERE id = p_job_id
|
||||
$$;
|
||||
|
||||
REVOKE ALL ON FUNCTION produce_job_workspace(uuid) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION produce_job_workspace(uuid) TO sp_app;
|
||||
Loading…
Add table
Add a link
Reference in a new issue