T-HUB/docs/design/2026-10-09-transcode-pool-design.md

NVIDIA GPU Extension + Dynamic Pool Scaling — Design & Research

PeerTube transcoding worker cluster · design wave · 2026-10-09

Companion to the Intel QSV worker implementation landing in infra/peertube-workers/ (the enrolled worker laptop). This document depends only on the Intel worker's CONTRACT (official @peertube/peertube-runner, per-worker admission controller, pluggable capability probe, status JSON, drain control, fail-closed hardware rule), never on its internals.

Empirical anchors in this doc are marked [E] and were measured on this machine on 2026-10-09: WSL2 host, 2× NVIDIA GeForce RTX 3090 (Ampere GA102), Windows-side driver 595.58.03 / CUDA 13.2, Debian 12 ffmpeg 5.1.9 (0+deb12u1, libmfx build), 56-core Xeon E5-2690 v4, 128 GiB RAM. Numbers not marked [E] are from NVIDIA primary documentation (Video Codec SDK 13.x support matrix, NVENC help page, driver release notes, CUDA-on-WSL guide) as recalled; no web access was available in this session, so treat vendor-doc-derived figures as "verify against the live matrix when the first NVIDIA box is onboarded" — every one of them is also re-derived at install time by the probe (§5.3), so a stale number here degrades to a warning, never to a wrong production decision.

---

0. Executive summary

1. NVENC on GeForce is no longer meaningfully session-capped for our use case: the driver limit is 8 concurrent encode sessions per GPU since driver branch 536 (mid-2023); professional/datacenter boards have no driver cap. Our per-worker concurrency will be bounded by VRAM, NVDEC+filter throughput and bandwidth long before 8 sessions matter — but the cap must still be tracked, because PeerTube fans one job out to many resolution renditions and a 5-rendition job on one GPU is 5 sessions.
2. The hardware rule generalizes cleanly: the same fail-closed contract that keeps QSV from falling back to x264 keeps NVENC from falling back to x264. The only NVIDIA-specific additions are device selection (-gpu N), driver/kernel-module installation, session accounting, and VRAM floors.
3. Do not put NVIDIA workers on WSL2. [E] The API surface exists (h264_nvenc/hevc_nvenc registered, libnvidia-encode.so.595.58.03 present) but the GPU is shared with the Windows host, NVML encoder queries are reduced, and there is no /dev/dri, no persistence-mode control and no isolation from Windows-side VRAM consumers. On this very box NVENC is currently unusable because two co-tenant Windows processes hold 23.8/24 GiB on each GPU. WSL2 is a dev/probe platform; production NVIDIA workers are native Linux.
4. Dynamic pool scaling does not require touching PeerTube's scheduler. Runners register once and persist server-side; "joining the pool" is systemctl start + admission AVAILABLE, "leaving" is graceful drain + optional systemctl stop. The pool manager nudges which runners are eligible (and at what concurrency), PeerTube still decides job assignment. Mixed-vendor preference is expressed the same way: selective availability + concurrency, not scheduler surgery.
5. The one interface the Intel implementation must get right now: the capability probe must be a mandatory, fail-closed, versioned plugin interface, and the status JSON must be vendor-generic with a gpu.vendor discriminator from day one.

---

1. NVIDIA capability research (2026)

1.1 NVENC concurrent-session limits

GPU class	Driver-imposed concurrent NVENC sessions	Notes
GeForce (consumer)	8 per GPU	Raised 3→5 in 530.41.03/531.29 (2023-03), 5→8 in branch 536 (2023-07). Applies per GPU, across all processes; an ffmpeg process using -threads still counts as 1 session per encode context.
Professional / workstation (RTX A-series, RTX 5000/6000 Ada, RTX Pro Blackwell; ex-Quadro)	No driver cap	Bounded only by hardware/VRAM.
Datacenter media-class (T4, A2, L4, L40, L40S)	No driver cap	These are the right datacenter parts for transcoding.
Datacenter compute-class (A100, H100, H200, B100/B200)	0 — no NVENC at all	Compute dies ship no encode engine (and Hopper/A100 have reduced/absent NVDEC). Never buy these for a transcode pool.

Design consequences:

• The cap counts encode sessions, not processes and not ffmpeg "engines". A single PeerTube VOD job that produces 4 renditions from one decode can hold 4 encode sessions (PeerTube today typically runs one ffmpeg per rendition, so it holds 4 decode sessions too — see §3.4).
• Session pressure must be measured, not assumed from the table: NVML exposes nvmlDeviceGetEncoderCapacity / encoder-session counts on bare-metal Linux, but [E] on WSL2 the capacity query returned a non-success code in our ctypes probe, so the probe design (§5.3) treats NVML as best-effort with a driver-class lookup table and our own accounting as ground truth.

1.2 NVENC encode support matrix by generation

Generational numbering below is the commonly used NVENC-generation scheme (1=Kepler … 7=Blackwell); NVIDIA's own docs speak of "Turing NVENC", "Ampere NVENC", etc. Quality-relevant jumps: Turing (large HEVC/H.264 quality+lookahead gains) and Ada (AV1 + HEVC 4:4:4).

NVENC generation (arch)	Example parts	H.264	HEVC 8-bit	HEVC 10-bit	HEVC B-frames	HEVC 4:4:4	AV1
Gen 1 (Kepler)	GTX 680	✓	—	—	—	—	—
Gen 2 (Maxwell)	GTX 750Ti/970; GM206 (950/960)	✓	GM206 only	—	—	—	—
Gen 3 (Pascal)	GTX 10xx, Quadro P	✓	✓	✓	—	—	—
Gen 4 (Turing)	GTX 16xx, RTX 20xx, T4	✓	✓	✓	✓	—	—
Gen 5 (Ampere)	RTX 30xx, A-series	✓	✓	✓	✓	✓ (Pro/A-series)	—
Gen 6 (Ada)	RTX 40xx, L4, L40/L40S	✓	✓	✓	✓	✓	✓
Gen 7 (Blackwell)	RTX 50xx, RTX Pro	✓	✓	✓	✓	✓ + 4:2:2	✓ (+UHQ)

[E] Consistency check on this box (RTX 3090 = Ampere = gen 5): ffmpeg 5.1.9 lists h264_nvenc and hevc_nvenc and does not list av1_nvenc; av1_cuvid decode is listed. Matches the matrix exactly (Ampere: AV1 decode yes, AV1 encode no).

1.3 NVDEC decode support matrix

NVDEC generation (arch)	H.264	HEVC 8/10-bit	VP9	HEVC 4:4:4	AV1
Maxwell	✓	GM206: 8-bit only	—	—	—
Pascal	✓	✓	✓	—	—
Turing	✓	✓	✓ 10/12-bit	✓	—
Ampere	✓	✓	✓	✓	✓ 8/10-bit
Ada	✓	✓	✓	✓	✓ (incl. higher bit depths)
Blackwell	✓	✓ (+4:2:2)	✓	✓	✓ (+4:4:4/12-bit)

Practical rule for the pool: any card we would actually buy (Turing or newer) decodes H.264/HEVC/VP9; AV1 decode needs Ampere+; AV1 encode needs Ada+. An Ampere pool (like this box) can ingest AV1 sources and emit H.264/HEVC, which fits the stated "H.264 primary, HEVC secondary, no AV1 output" policy with headroom.

1.4 Linux driver stack

• Open vs proprietary kernel modules: NVIDIA's open kernel modules (nvidia-open / open-gpu-kernel-modules) are the default recommended path for Turing+ since the 560-series; proprietary remains for pre-Turing and a few feature gaps. All realistic pool hardware (Turing+) → use open modules. Blackwell effectively requires them.
• 2026 driver branches: current feature branches run in the 5xx range; [E] this host's Windows-side driver is 595.58.03 (CUDA 13.2). Debian packages: nvidia-open-kernel-dkms (or prebuilt linux-image-amd64 + signed modules), nvidia-driver-libs, nvidia-smi, nvidia-utils; firmware from non-free-firmware. Keep driver version pinned per pool fleet and record driver + kernel + module flavor in the status JSON.
• Userspace pieces that matter to us: libnvidia-encode.so.1 (NVENC), CUDA driver (libcuda.so.1) used by ffmpeg's nvenc/cuvid code paths, libnvidia-ml.so.1 (NVML for nvidia-smi). ffmpeg needs no CUDA toolkit — only the driver libs. (A statically-built ffmpeg with --enable-ffnvcodec / --enable-cuda-nvcc is only needed for scale_npp; scale_cuda is covered by the header-only ffnvcodec path.)
• Access control: user in video group (or a dedicated worker user with a udev/tagged device ACL); nvidia-persistenced (or NVreg_PreserveVideoMemoryAllocations=1) to avoid multi-second GPU re-init between jobs on an idle pool box — relevant to dynamic scale-up latency (§3.7).
• MIG/compute-mode: leave compute mode DEFAULT [E]; nothing to configure, but note it in the probe report.

1.5 WSL2: CUDA/NVENC status

What works [E, this host]:

• CUDA user-mode stack present via /usr/lib/x86_64-linux-gnu/libnvidia-*.so (WSL-provided, version-matched to the Windows driver).
• ffmpeg registers h264_nvenc, hevc_nvenc, *_cuvid decoders and the cuda hwaccel; libnvidia-encode.so.595.58.03 is installed.
• nvidia-smi works (2 GPUs visible, temps, clocks, throttle counters).

What breaks it (all [E] on this host):

• No isolation from Windows GPU consumers. Two Windows processes hold 23 856 MiB of each 24 GB GPU → any CUDA context creation fails (CUDA_ERROR_OUT_OF_MEMORY), so h264_nvenc init fails with "No capable devices found" and h264_cuvid device setup fails. This is the textbook GPU_UNAVAILABLE state, and it is caused by another tenant, not by the driver.
• Reduced NVML: encoder-capacity/session queries returned non-success in a ctypes probe; nvidia-smi -q shows Encoder Stats fields that cannot be validated.
• No /dev/dri [E: directory absent/empty] → no VAAPI/QSV fallback on the same box, no intel_gpu_top-style counters, no display-path controls.
• No control of persistence mode, ECC, or power limits from inside WSL.

Verdict: transcoding in WSL2 is feasible in the "API present, demo works" sense but is not a production worker platform — the GPU is a shared resource with the Windows host, NVML is limited, and admission signals are unreliable. Use WSL2 to develop the nvidia-nvenc probe; run production workers on native Linux (or pass a whole GPU to a VM with VFIO if the box must stay Windows-adjacent).

1.6 FFmpeg flags: the NVIDIA pipeline

The NVIDIA analogue of hw decode → hw frames → hw scale → hw encode:

# full hardware pipeline, frames never leave the GPU
ffmpeg -y -init_hw_device cuda=cu:0 -filter_hw_device cu \
  -hwaccel cuda -hwaccel_output_format cuda -c:v h264_cuvid -i INPUT \
  -vf scale_cuda=1280:720:format=yuv420p \
  -c:v h264_nvenc -preset p5 -tune hq -rc vbr -cq 23 -b:v 0 \
  -rc-lookahead 20 -spatial_aq 1 -temporal_aq 1 \
  -bf 3 -b_ref_mode middle \
  -c:a copy OUTPUT

Key flags (PeerTube-runner passes its own argv; these are what the capability probe and any wrapper/profile must emit):

Purpose	Flag	Notes
HW decode	-hwaccel cuda -hwaccel_output_format cuda or explicit -c:v h264_cuvid/hevc_cuvid/av1_cuvid	explicit *_cuvid gives named-decoder certainty; -hwaccel_output_format cuda keeps frames on-GPU
HW upload (when input is CPU-decoded)	-vf hwupload_cuda	needed for e.g. audio-merge jobs with CPU-side frames
HW scale	scale_cuda=w:h (part of ffmpeg) or scale_npp (needs nvcc-built ffmpeg)	scale_cuda is sufficient and dependency-light
Multi-GPU	-init_hw_device cuda=cu:1, encoder -gpu N	probe must test the specific device index it will advertise
Encode	-c:v h264_nvenc / hevc_nvenc / av1_nvenc	av1_nvenc exists only on Ada+ builds of ffmpeg ≥ 5.0
Quality	-preset p1..p7 (p7 slowest/best), -tune hq|ll|ull, -multipass qres|fullres	p4–p5 is the sweet spot for VOD
Rate control	-rc vbr -cq N -b:v 0 (quality-targeted VBR) or -rc constqp	prefer -cq + -b:v 0 for PeerTube's CRF-style outputs
Lookahead/AQ	-rc-lookahead, -spatial_aq, -temporal_aq, -bf, -b_ref_mode middle (Turing+)	the Turing+ quality levers
10-bit HEVC	-profile:v main10 -pix_fmt p010le	verify per generation (§1.2)
Session hygiene	-delay 0, and one encode context per session	sessions are the scarce resource on GeForce (§1.1)
Verification	run under nvidia-smi dmon -s u (enc/dec columns), check encoder-session count delta	this is the no-silent-CPU evidence, see §5.4

Minimal VRAM budget per 1080p hw pipeline ≈ 200–400 MiB (decode surfaces + scale + encode lookahead); 4K ≈ 0.8–1.5 GiB. The probe measures real usage per resolution class instead of trusting these numbers.

---

2. What an NVIDIA worker is (profile)

Same shape as the Intel worker — official runner, admission controller, status JSON, drain control — differing only in the probe module, deps, and concurrency inputs.

2.1 Differences for install-peertube-worker.sh

Aspect	Intel QSV worker	NVIDIA worker
Kernel/driver deps	intel-media-va-driver-non-free, oneVPL runtime, /dev/dri/renderD128 + render/video groups	nvidia-open kernel module + pinned driver branch, libnvidia-encode, nvidia-smi, nvidia-persistenced, user in video group
Device nodes	/dev/dri/* (always present)	/dev/nvidia* (created by module; nvidia-modprobe/udev)
Probe module	intel-qsv	nvidia-nvenc (new module, same interface)
Config additions	—	PT_WORKER_GPU_DEVICES=0 (or 0,1), PT_WORKER_SESSIONS_RESERVE=1 (headroom sessions), PT_WORKER_VRAM_FLOOR_MB=1024, PT_WORKER_DRIVER_MIN=550
Self-test	vainfo + QSV transcode	nvidia-smi gates + per-device NVENC transcode + session-slot test + VRAM floor (§5.3)
Concurrency inputs	engine util, CPU/RAM	min(sessions_free, cpu_bound, ram_bound, vram_bound, disk_bound, bw_bound)
Runner invocation	as Intel contract	plus -gpu N device pinning in the ffmpeg argv and one runner instance per GPU when a box has >1 GPU (see §2.2)

Everything else — systemd unit shape, service account, Tailscale posture, registration, VOD-only job types, drain semantics — is inherited unchanged.

2.2 Multi-GPU boxes

A box with K GPUs runs K runner instances (peertube-worker@gpu0…gpuK-1 template units), each pinned to one device, rather than one runner juggling devices. Reasons: session accounting and VRAM floors are per-device; admission states (GPU_UNAVAILABLE for one card after a co-tenant/ECC event) should not drain healthy siblings; PeerTube sees N independent runners and load-balances them natively. [E: this host has 2× RTX 3090 and would onboard as …:gpu0 and …:gpu1.]

---

3. Dynamic pool design

3.1 Architecture

                ┌────────────────────────────────────────────┐
                │ pool-manager (new, small, stateless-ish)   │
                │  signals → policy engine → per-worker      │
                │  desired state (AVAILABLE/DRAINED +        │
                │  concurrency hint + job-class eligibility) │
                └───────┬───────────────────┬────────────────┘
        read-only admin │                   │ control channel (SSH over
        API polls       │                   │ Tailscale → pt-worker CLI)
   PeerTube ◄───────────┘                   ▼
   keeps THE job queue          worker-1 (Intel, QSV)   worker-2 (NV gpu0) …
   and does assignment          admission controller     admission controller
                                + peertube-runner        + peertube-runner

The pool manager never talks to PeerTube as a scheduler. It only (a) observes queue pressure read-only, (b) starts/stops/drains worker services, (c) sets admission state and concurrency. Assignment remains PeerTube's, via the official runner request/accept cycle. This honors the "do not build a new job scheduler" constraint.

3.2 (a) Load signals

Signal	Source	Cadence	Notes
pending_depth	PeerTube admin API: runner-jobs endpoint filtered by state=pending, per job type (vod-web-video / vod-hls / vod-audio-merge)	15 s	the primary scale-up input
oldest_pending_age_s	same endpoint, createdAt of oldest pending job	15 s	catches stall at low depth (e.g. one 4K job nobody can serve)
completed_rate_ewma	per-worker status JSON jobs_completed counters	30 s	throughput of the currently-active set
enc_util, dec_util	NVIDIA: nvidia-smi --query-gpu=encoder.utilization.../NVML; Intel: media-engine counters per Intel contract	10 s	saturation of the active set
sessions_active/cap	our own accounting + NVML where supported	per-job event	GeForce cap awareness (§1.1); NVML unsupported on WSL2 [E]
vram_free_mb	NVML	10 s	hard eligibility floor
recent_download_mbps / upload_mbps (EWMA)	per-job transfer observations (Intel contract already defines these)	per-job	job-size-aware eligibility input
realtime_factor_ewma	completed jobs (encode wall-clock vs duration)	per-job	learned worker weight (§3.3)
throttle_reasons, temp_c	NVML throttle-cause bits / thermal zones	10 s	THERMAL_THROTTLED mapping [E: counters visible on this host]

Pressure index (normalized 0..1), computed over eligible workers only:

P = clamp01( max(
    pending_depth / (2 * Σ active_capacity_slots),
    oldest_pending_age_s / 900s
))

Σ active_capacity_slots = Σ over AVAILABLE/BUSY workers of their current concurrency hint. The 2× makes P=1 mean "queue is two full waves deep".

3.3 (b) Policy engine

Pool-worker states (per worker; these are manager-side views layered on the worker's own admission states from the Intel contract):

Manager state	Meaning	Transition in
COLD	powered off / suspended (WoL-able, §3.8)	manual or WoL policy
SPARE	service up, admission DRAINED (registered, idle)	default resting state
WARM	service up, DRAINED, persistence-mode on, probe recently green	pre-warm tier
ACTIVE	admission AVAILABLE at concurrency C	scale-up decision
DRAINING	admission DRAINING, in-flight jobs finishing	scale-down or fault
COOLDOWN	admission DRAINING after repeated job failures	failure policy

Hysteresis (per-worker and pool-wide):

• Scale up when P ≥ 0.6 sustained ≥ 90 s (two consecutive 45 s windows) → promote the highest-weight SPARE/WARM worker to ACTIVE.
• Scale down when P ≤ 0.15 sustained ≥ 20 min and worker has been idle ≥ 15 min → demote ACTIVE → SPARE (drain, then optional service stop).
• Never demote below min_active = 1 (or 2 once multi-tenant), regardless of idle, to keep probe freshness and WoL-free response.
• Promotions are staggered ≥ 30 s apart and probe-gated: a worker may only become ACTIVE if its last probe is < 5 min old and green. This prevents thundering-herd registration wake-ups and GPU re-init spikes.
• Failure fast-path bypasses all hysteresis: GPU_UNAVAILABLE, thermal, or a hardware-evidence-failed job → immediate DRAINING.

Per-worker learned weights (used for promotion order and drain order):

w_i = EWMA_λ=0.3 over jobs of ( output_pixel_rate_i / class_baseline )
      shrunk toward class prior:  ŵ = (n·w + k·w_prior) / (n + k),  k=3

output_pixel_rate = output pixels/sec during encode phase only (excludes download/upload), bucketed by job class so a worker isn't punished for getting small jobs. Cold-start workers inherit the class prior, so a new box's first promotion comes from queue pressure, not fabricated history. Drain order prefers low ŵ and low recent_download_mbps (cheapest to re-warm).

Job-size-aware eligibility. Job classes from PeerTube job metadata (source resolution × duration, known at pending time):

Class	Definition (approx)	Requires
S	≤1080p, ≤10 min	any ACTIVE worker
M	≤1080p long, or 1440p	enc_util headroom ≥ 30%
L	4K, or 4K-source→1080p	realtime_factor_ewma ≥ 3, recent_download_mbps ≥ 50, vram_free ≥ 4 GiB (NVIDIA) or equivalent QSV gate
XL	4K60 long-form	L gates + ŵ ≥ 1.0

Enforcement is, again, selective availability: when an L/XL job is pending and no one satisfies the gates, the manager keeps the small workers at their normal concurrency but does not count them as capacity for that job (PeerTube will still let a small worker take it if it asks — acceptable fallback, not silent CPU fallback, and it still satisfies the hardware rule).

3.4 (c) Scale-up / scale-down mechanics

The constraint: PeerTube runner registration is static. A runner row exists server-side from first register (with a registration token) until an admin deletes it; the runner process merely polls. Therefore:

Joining the pool =

1. box provisioned + runner registered once at install time (install-peertube-worker.sh --name w2:gpu0 …),
2. service exists but may be stopped or admission-DRAINED,
3. manager: systemctl start peertube-worker@… (if stopped) then pt-worker enable → admission AVAILABLE → runner starts requesting jobs on its normal poll cycle.

Leaving the pool (grace) =

1. pt-worker drain → admission DRAINING; the controller stops offering the runner for new jobs (per Intel contract: runner paused/disabled, not killed),
2. wait for in-flight jobs: active_jobs → 0, with a max-drain timeout (e.g. 2× p99 job wall-clock, floor 30 min),
3. then either leave at SPARE (service up, DRAINED — cheapest re-join) or systemctl stop for overnight-class quiescence, or suspend (§3.8).

Notes:

• Repeated start/stop never touches registration tokens; tokens live only in the install-time secret path on the worker.
• Because a drained runner is still registered, PeerTube's per-runner job-request rate is unaffected; there is no server-side cleanup needed.
• One job → many sessions: PeerTube runs separate ffmpeg instances per output rendition, so a "concurrency 1" job can still consume 4+ NVENC sessions. The admission controller must therefore count actual encode sessions (its own children's NVENC contexts), not just jobs, when enforcing sessions_active < cap (§1.1, §5.3).

3.5 (d) Mixed-vendor scheduling

We cannot intercept PeerTube's runner→job assignment, so vendor preference is expressed as eligibility, decided at the moment a job class is pending:

Job / condition	Prefer	Mechanism
AV1 source	NVIDIA Ampere+ (NVDEC AV1) or CPU-decode on QSV box	gate QSV workers out of the AV1-decode class via probe capability (av1_dec)
AV1 output	NVIDIA Ada+ only (av1_enc)	same gate; policy says: not enabled in production anyway
4K HEVC/H.264 outputs	NVIDIA (throughput, VRAM for lookahead)	weight-based promotion pulls NVENC boxes in first at L/XL classes
≤1080p H.264 bulk	whichever has session/enc headroom; QSV is typically more power-efficient per 1080p job	neutral; capacity-driven
Session pressure (NVENC near cap, QSV idle)	QSV	temporarily demote NVENC availability for class S/M
One vendor fleet-wide drained (driver CVE, bad driver branch)	the other	vendor-level kill switch in manager config: vendor_disabled: [nvidia]

Probe capability keys that drive this (identical names across vendors): h264_enc, hevc_enc, av1_enc, h264_dec, hevc_dec, av1_dec, max10bit, max_sessions, max_width/height. The Intel probe fills the same keys (i5-1240P: h264_enc=1, hevc_enc=1, av1_enc=0, av1_dec=1), which is the whole point of the pluggable module.

3.6 (e) Unified status/metrics schema

Extends the Intel status JSON with a versioned, vendor-generic gpu block. Bump schema_version; additive changes only.

{
  "schema_version": 2,
  "hostname": "w2", "runner_name": "w2:gpu0",
  "tailscale_ip": "100.x.y.z", "net_path": "DIRECT|DERP",
  "state": "AVAILABLE",            // Intel-contract states, unchanged
  "registration_state": "registered",
  "active_jobs": 1, "concurrency": 3, "concurrency_limiters": {
      "cpu": 4, "ram": 4, "disk": 8, "bw": 3,
      "gpu_sessions": 3,           // NVIDIA only; null on Intel
      "vram": 3
  },
  "gpu": {                          // NEW: vendor-generic
    "vendor": "nvidia",             // "intel" | "nvidia" | "none"
    "probe_module": "nvidia-nvenc", // "intel-qsv" | "nvidia-nvenc"
    "model": "RTX 3090", "generation": "ampere",
    "nvenc_generation": 5,
    "driver": "595.58.03", "kernel_module": "open",
    "capabilities": { "h264_enc": 1, "hevc_enc": 1, "av1_enc": 0,
                      "h264_dec": 1, "hevc_dec": 1, "av1_dec": 1,
                      "max10bit": 1 },
    "sessions": { "active": 2, "cap": 8,
                  "cap_source": "driver_class_table|nvml|probe" },
    "devices": [ { "index": 0, "uuid": "GPU-2095…",
                   "vram_total_mb": 24576, "vram_free_mb": 22000,
                   "enc_util_pct": 71, "dec_util_pct": 40,
                   "temp_c": 66, "throttle_reasons": [] } ],
    "self_test": { "ts": "2026-10-09T14:00:00Z", "ok": true,
                   "evidence": "enc_util_delta=+68%", "detail": "…" }
  },
  "hw_evidence_last_job": { "ok": true,
      "method": "encoder_session_delta>=1", "ts": "…" },
  "cpu": {"model": "…", "util_pct": 40}, "temps_c": {"cpu": 55},
  "disk_free_gb": 180,
  "recent_download_mbps": 92.4, "recent_upload_mbps": 88.1,
  "jobs": {"completed": 214, "failed": 3, "hw_fallback_prevented": 0},
  "last_job": {"duration_s": 412, "realtime_factor": 7.1,
               "class": "M", "output_pixel_rate_mpx_s": 22.6},
  "manager": {"desired": "ACTIVE", "weight": 1.14, "class_gates": "ok"}
}

Prometheus mapping is mechanical: one pt_worker_up, one pt_worker_gpu_sessions{vendor,cap_source}, one pt_worker_hw_evidence_ok (the no-silent-CPU invariant as an alertable series), plus re-exports of the Intel contract's series.

3.7 (f) Future: bare-metal autoscaling hooks (WoL)

Brief by request. If worker boxes are wakeable:

• Manager keeps wol_mac, wake_budget (max 2 concurrent wakes, ≥ 3 min apart), and a boot-readiness check = Tailscale SSH + agent status JSON with fresh probe (typical budget 3–6 min including GPU persistence).
• Wake policy reuses the scale-up rule with an extra tier: only wake when P ≥ 0.8 sustained 5 min and all SPARE/WARM workers are ACTIVE (i.e. waking is the last resort, because boot latency is minutes not seconds).
• Sleep path = graceful drain (§3.4) then systemctl suspend; AC/battery rules from the Intel laptop policy apply first (never sleep a box on battery, never wake one).
• Accounting: track wakes/day and woken-minutes-saved per box; cap wakes (e.g. ≤ 6/day/box) to respect hardware and power-on cycles.

---

4. Rollout plan

4.1 Change the Intel admission controller NOW (small interface list)

Five things, all cheap while the code is fresh:

1. Versioned status JSON: add schema_version: 1 today, and the vendor-generic gpu block shape (§3.6) with "vendor": "intel" — i.e. gpu.probe_module, gpu.capabilities{h264_enc,hevc_enc,av1_enc, h264_dec,hevc_dec,av1_dec}, gpu.model/generation, per-device array (Intel: one entry, /dev/dri/renderD128, enc_util from media-engine counters). Avoid leaking QSV-specific keys to the top level.
2. Probe interface contract (document, even if only one impl exists): probe() → {ok, capabilities, maxSessions?, devices[], evidence} and verifyHardwareDidWork(jobSamples) → {ok, evidence} — both fail closed: any exception, timeout, or absent evidence ⇒ GPU_UNAVAILABLE + drain, never "assume CPU is fine". Selectable via PT_WORKER_GPU_PROBE=intel-qsv|nvidia-nvenc.
3. Concurrency resolver as a pure function of (probe, cpu, ram, disk, bw, [sessions], [vram]) so NVIDIA is a new input set, not a new control path. Expose which limiter bound the choice (concurrency_limiters, §3.6) — this is what the pool manager consumes.
4. Hardware-evidence hook per job: after each completed job, sample the GPU's encoder utilization delta (Intel: media-engine busy% delta during the job; NVIDIA: NVML encoder-session/util delta) and record hw_evidence_last_job. A job that completed without evidence must count as a failure for pool purposes even if PeerTube marks it OK — this is the enforcement teeth of the no-silent-CPU rule, and it is vendor-neutral.
5. Job-class metadata in per-job telemetry (source res/codec, output renditions, wall-clock, realtime factor) so the manager's weights and gates (§3.3) have data from day one on the Intel cluster.

4.2 Onboarding the first NVIDIA box (when one exists)

1. Pick the platform: native Linux (Debian 12/13), Turing or newer; Ada+ if AV1 encode is ever wanted. Reject A100/H100-class compute cards (no NVENC, §1.1).
2. Driver install, pinned branch (≥ 550; open kernel modules for Turing+), nvidia-persistenced, user in video, verify nvidia-smi + libnvidia-encode.so.1 present. Reboot-verify.
3. ffmpeg: distro build is fine if it lists h264_nvenc/hevc_nvenc and *_cuvid [E: Debian 12's ffmpeg 5.1.9 does]; else pin a static BtbN/johnvansickle-style build ≥ 5.0 with ffnvcodec. Record ffmpeg --version in status JSON.
4. install-peertube-worker.sh --name <host>:gpu0 --gpu-probe nvidia-nvenc --gpu-devices 0 … — same script, new profile (§2.1). Per additional GPU on the box, repeat with :gpu1, … (§2.2).
5. Self-test gate (must pass before registration): see §5.3 — nvidia-smi gates → per-device NVENC encode probe under dmon → session-slot test → VRAM floor. Fail ⇒ abort install with GPU_UNAVAILABLE, no runner registration.
6. Register runner (VOD job types only, per Intel contract), start service, run one real E2E job, and assert hardware evidence: encoder-session delta ≥ 1 per rendition and enc-util delta > 0 and CPU-user-time of the ffmpeg process ≪ CPU-encode baseline (belt and braces).
7. Soak: 1 h at target concurrency; watch sessions, VRAM, temps, throttle reasons, throughput; set the box's initial concurrency from measured end-to-end throughput, not single-job fps (Intel Phase 9 methodology, unchanged).
8. Add to pool manager inventory as SPARE with class-prior weight; first promotion happens on the next pressure event.
9. Only then consider multi-GPU boxes and WoL tiers (§3.7).

---

5. Appendices

5.1 Failure-mode → state mapping (NVIDIA additions)

Failure	Detection	State	Action
Driver/module missing or unloaded	nvidia-smi nonzero, no /dev/nvidia*	GPU_UNAVAILABLE	refuse work; re-probe 60 s
VRAM exhausted (co-tenant, leak) [E]	CUDA_ERROR_OUT_OF_MEMORY / "No capable devices found" in ffmpeg init; NVML vram_free below floor	GPU_UNAVAILABLE (device-scoped)	drain that device's runner; sibling GPUs unaffected (§2.2)
Session cap hit	our accounting ≥ cap − reserve	BUSY (capacity)	no new jobs until sessions free; never queue extra ffmpeg per job
Thermal/power throttle	NVML throttle-cause bits, temp ≥ limit	THERMAL_THROTTLED	drain after in-flight; cooldown
Encoder produced no HW evidence	hw_evidence_last_job.ok=false	GPU_UNAVAILABLE + job counted failed	page; never re-run on CPU
Driver/GPU fallen off bus (Xid errors in dmesg)	dmesg NVRM: Xid watch	OFFLINE	drain + host-level alert
Network DERP/throughput collapse	Intel contract's EWMA + path change	NETWORK_DEGRADED	unchanged from Intel

5.2 Config surface (NVIDIA profile)

PT_WORKER_GPU_PROBE=nvidia-nvenc
PT_WORKER_GPU_DEVICES=0            # one runner instance per device
PT_WORKER_SESSIONS_CAP=auto        # nvml → driver_class_table → 8 (GeForce)
PT_WORKER_SESSIONS_RESERVE=1       # keep 1 session headroom
PT_WORKER_VRAM_FLOOR_MB=1024
PT_WORKER_DRIVER_MIN=550
PT_WORKER_FFMPEG_NVENC_PRESET=p5   # map PeerTube quality tier → preset
PT_WORKER_PROBE_INTERVAL=300       # full probe; 10 s counters in between

5.3 nvidia-nvenc probe / self-test design (ordered, fail-closed)

1. Gate: nvidia-smi exits 0; driver ≥ min; module flavor recorded; libnvidia-encode.so.1 resolvable; ≥1 device with vram_free ≥ floor.
2. Capability enumeration: per advertised device, from NVML name → generation → capability table (§1.2/§1.3), then verified empirically: attempt a 1-second h264_nvenc encode on that exact device (-gpu N); attempt hevc_nvenc; attempt av1_nvenc only if generation ≥ Ada. Absence of an encoder in ffmpeg -encoders downgrades capability without failing the probe (e.g. distro ffmpeg without av1_nvenc).
3. Pipeline probe: full hw chain (§1.6) at 1080p on the target device under nvidia-smi dmon -s u; require enc>0 and dec>0 columns observed during the run, and record VRAM delta as the per-class budget.
4. Session-slot test: launch K parallel tiny h264_nvenc encodes (K = advertised cap) and count how many initialize successfully (No capable devices found / init failure = slot unavailable). The largest K′ that succeeds becomes sessions.cap with cap_source=probe. This is the ground truth the controller trusts — it automatically reflects driver-class caps and whatever the platform (e.g. WSL2 [E]) actually allows.
5. Decode matrix: one decode-only pass per codec family (h264/hevc/av1 *_cuvid) on a synthetic stream, asserting dec>0.
6. Report: fill the §3.6 gpu block, include dmon excerpts and VRAM deltas as evidence. Any step erroring or timing out (30 s/step) ⇒ ok=false ⇒ GPU_UNAVAILABLE. No step may conclude "use CPU instead".

5.4 Enforcing the no-silent-CPU rule on NVIDIA

Three independent tripwires, any one sufficient to fail a job:

1. Argv gate: the wrapper/runner must be invoking *_nvenc for video (grep the spawned argv); PeerTube's own fallback logic must be disabled/refused in config.
2. Evidence gate: NVML encoder-session count and encoder utilization must increase while the job's ffmpeg runs (sampled ≥ 1 Hz); a completed encode with flat counters = CPU fallback (or a no-op) ⇒ job marked failed, worker GPU_UNAVAILABLE, manager COOLDOWN.
3. Plausibility gate: ffmpeg CPU user-time for the job must be far below the CPU-encode baseline for that resolution (measured once per box during install); a CPU-encode-shaped CPU profile ⇒ same treatment.

This is deliberately vendor-neutral (Intel uses media-engine busy% in place of NVML counters), which is exactly why it belongs in the shared admission controller rather than in the probe module.

5.5 Open questions for the Intel implementation (blocking-free)

• Exact admin endpoint + auth shape for pending runner-jobs depth (manager needs it read-only; a proxy via the worker's own credentials is acceptable if admin scope is unavailable).
• Whether the Intel drain implementation pauses the runner process or stops the unit — the manager wants to treat both as DRAINED and needs to know which to expect for start/stop idempotency.
• Whether per-rendition ffmpeg invocation is confirmed (drives session-counting assumptions, §3.4).