# NVIDIA GPU Extension + Dynamic Pool Scaling — Design & Research
PeerTube transcoding worker cluster · design wave · 2026-10-09

Companion to the Intel QSV worker implementation landing in
`infra/peertube-workers/` (the enrolled worker laptop). This document depends **only on the
Intel worker's CONTRACT** (official `@peertube/peertube-runner`, per-worker
admission controller, pluggable capability probe, status JSON, drain control,
fail-closed hardware rule), never on its internals.

Empirical anchors in this doc are marked **[E]** and were measured on this
machine on 2026-10-09: WSL2 host, 2× NVIDIA GeForce RTX 3090 (Ampere GA102),
Windows-side driver 595.58.03 / CUDA 13.2, Debian 12 ffmpeg 5.1.9
(`0+deb12u1`, libmfx build), 56-core Xeon E5-2690 v4, 128 GiB RAM. Numbers not
marked [E] are from NVIDIA primary documentation (Video Codec SDK 13.x
support matrix, NVENC help page, driver release notes, CUDA-on-WSL guide) as
recalled; **no web access was available in this session, so treat
vendor-doc-derived figures as "verify against the live matrix when the first
NVIDIA box is onboarded"** — every one of them is also re-derived at install
time by the probe (§5.3), so a stale number here degrades to a warning, never
to a wrong production decision.

---

## 0. Executive summary

1. NVENC on GeForce is no longer meaningfully session-capped for our use
   case: the driver limit is **8 concurrent encode sessions per GPU** since
   driver branch 536 (mid-2023); professional/datacenter boards have **no
   driver cap**. Our per-worker concurrency will be bounded by VRAM,
   NVDEC+filter throughput and bandwidth long before 8 sessions matter —
   but the cap must still be tracked, because PeerTube fans one job out to
   many resolution renditions and a 5-rendition job on one GPU *is* 5
   sessions.
2. The hardware rule generalizes cleanly: the same fail-closed contract that
   keeps QSV from falling back to x264 keeps NVENC from falling back to x264.
   The only NVIDIA-specific additions are device selection (`-gpu N`),
   driver/kernel-module installation, session accounting, and VRAM floors.
3. **Do not put NVIDIA workers on WSL2.** [E] The API surface exists
   (`h264_nvenc`/`hevc_nvenc` registered, `libnvidia-encode.so.595.58.03`
   present) but the GPU is shared with the Windows host, NVML encoder
   queries are reduced, and there is no `/dev/dri`, no persistence-mode
   control and no isolation from Windows-side VRAM consumers. On this very
   box NVENC is currently unusable because two co-tenant Windows processes
   hold 23.8/24 GiB on each GPU. WSL2 is a dev/probe platform; production
   NVIDIA workers are native Linux.
4. Dynamic pool scaling does **not** require touching PeerTube's scheduler.
   Runners register once and persist server-side; "joining the pool" is
   `systemctl start` + admission AVAILABLE, "leaving" is graceful drain +
   optional `systemctl stop`. The pool manager nudges *which* runners are
   eligible (and at what concurrency), PeerTube still decides job
   assignment. Mixed-vendor preference is expressed the same way: selective
   availability + concurrency, not scheduler surgery.
5. The one interface the Intel implementation must get right **now**: the
   capability probe must be a *mandatory, fail-closed, versioned* plugin
   interface, and the status JSON must be vendor-generic with a `gpu.vendor`
   discriminator from day one.

---

## 1. NVIDIA capability research (2026)

### 1.1 NVENC concurrent-session limits

| GPU class | Driver-imposed concurrent NVENC sessions | Notes |
|---|---|---|
| GeForce (consumer) | **8** per GPU | Raised 3→5 in 530.41.03/531.29 (2023-03), 5→8 in branch 536 (2023-07). Applies per GPU, across all processes; an ffmpeg process using `-threads` still counts as 1 session per encode context. |
| Professional / workstation (RTX A-series, RTX 5000/6000 Ada, RTX Pro Blackwell; ex-Quadro) | **No driver cap** | Bounded only by hardware/VRAM. |
| Datacenter media-class (T4, A2, L4, L40, L40S) | **No driver cap** | These are the right datacenter parts for transcoding. |
| Datacenter compute-class (A100, H100, H200, B100/B200) | **0 — no NVENC at all** | Compute dies ship no encode engine (and Hopper/A100 have reduced/absent NVDEC). Never buy these for a transcode pool. |

Design consequences:

- The cap counts **encode sessions**, not processes and not ffmpeg
  "engines". A single PeerTube VOD job that produces 4 renditions from one
  decode can hold 4 encode sessions (PeerTube today typically runs one
  ffmpeg per rendition, so it holds 4 decode sessions too — see §3.4).
- Session pressure must be *measured*, not assumed from the table: NVML
  exposes `nvmlDeviceGetEncoderCapacity` / encoder-session counts on
  bare-metal Linux, but **[E]** on WSL2 the capacity query returned a
  non-success code in our ctypes probe, so the probe design (§5.3) treats
  NVML as best-effort with a driver-class lookup table and our own
  accounting as ground truth.

### 1.2 NVENC encode support matrix by generation

Generational numbering below is the commonly used NVENC-generation scheme
(1=Kepler … 7=Blackwell); NVIDIA's own docs speak of "Turing NVENC",
"Ampere NVENC", etc. Quality-relevant jumps: Turing (large HEVC/H.264
quality+lookahead gains) and Ada (AV1 + HEVC 4:4:4).

| NVENC generation (arch) | Example parts | H.264 | HEVC 8-bit | HEVC 10-bit | HEVC B-frames | HEVC 4:4:4 | AV1 |
|---|---|---|---|---|---|---|---|
| Gen 1 (Kepler) | GTX 680 | ✓ | — | — | — | — | — |
| Gen 2 (Maxwell) | GTX 750Ti/970; GM206 (950/960) | ✓ | GM206 only | — | — | — | — |
| Gen 3 (Pascal) | GTX 10xx, Quadro P | ✓ | ✓ | ✓ | — | — | — |
| Gen 4 (Turing) | GTX 16xx, RTX 20xx, T4 | ✓ | ✓ | ✓ | ✓ | — | — |
| Gen 5 (Ampere) | RTX 30xx, A-series | ✓ | ✓ | ✓ | ✓ | ✓ (Pro/A-series) | — |
| Gen 6 (Ada) | RTX 40xx, L4, L40/L40S | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Gen 7 (Blackwell) | RTX 50xx, RTX Pro | ✓ | ✓ | ✓ | ✓ | ✓ + 4:2:2 | ✓ (+UHQ) |

**[E]** Consistency check on this box (RTX 3090 = Ampere = gen 5):
ffmpeg 5.1.9 lists `h264_nvenc` and `hevc_nvenc` and does **not** list
`av1_nvenc`; `av1_cuvid` decode **is** listed. Matches the matrix exactly
(Ampere: AV1 decode yes, AV1 encode no).

### 1.3 NVDEC decode support matrix

| NVDEC generation (arch) | H.264 | HEVC 8/10-bit | VP9 | HEVC 4:4:4 | AV1 |
|---|---|---|---|---|---|
| Maxwell | ✓ | GM206: 8-bit only | — | — | — |
| Pascal | ✓ | ✓ | ✓ | — | — |
| Turing | ✓ | ✓ | ✓ 10/12-bit | ✓ | — |
| Ampere | ✓ | ✓ | ✓ | ✓ | ✓ 8/10-bit |
| Ada | ✓ | ✓ | ✓ | ✓ | ✓ (incl. higher bit depths) |
| Blackwell | ✓ | ✓ (+4:2:2) | ✓ | ✓ | ✓ (+4:4:4/12-bit) |

Practical rule for the pool: **any card we would actually buy (Turing or
newer) decodes H.264/HEVC/VP9; AV1 decode needs Ampere+; AV1 *encode* needs
Ada+**. An Ampere pool (like this box) can ingest AV1 sources and emit
H.264/HEVC, which fits the stated "H.264 primary, HEVC secondary, no AV1
output" policy with headroom.

### 1.4 Linux driver stack

- **Open vs proprietary kernel modules**: NVIDIA's open kernel modules
  (`nvidia-open` / open-gpu-kernel-modules) are the **default recommended
  path for Turing+ since the 560-series**; proprietary remains for
  pre-Turing and a few feature gaps. All realistic pool hardware (Turing+)
  → use open modules. Blackwell effectively requires them.
- 2026 driver branches: current feature branches run in the 5xx range;
  **[E]** this host's Windows-side driver is 595.58.03 (CUDA 13.2). Debian
  packages: `nvidia-open-kernel-dkms` (or prebuilt `linux-image-amd64` +
  signed modules), `nvidia-driver-libs`, `nvidia-smi`, `nvidia-utils`;
  firmware from `non-free-firmware`. Keep driver version **pinned per pool
  fleet** and record `driver + kernel + module flavor` in the status JSON.
- Userspace pieces that matter to us: `libnvidia-encode.so.1` (NVENC),
  CUDA driver (`libcuda.so.1`) used by ffmpeg's nvenc/cuvid code paths,
  `libnvidia-ml.so.1` (NVML for nvidia-smi). ffmpeg needs no CUDA toolkit
  — only the driver libs. (A statically-built ffmpeg with `--enable-ffnvcodec`
  / `--enable-cuda-nvcc` is only needed for `scale_npp`; `scale_cuda` is
  covered by the header-only ffnvcodec path.)
- Access control: user in `video` group (or a dedicated worker user with a
  udev/tagged device ACL); `nvidia-persistenced` (or
  `NVreg_PreserveVideoMemoryAllocations=1`) to avoid multi-second GPU
  re-init between jobs on an idle pool box — relevant to dynamic scale-up
  latency (§3.7).
- MIG/compute-mode: leave compute mode `DEFAULT` **[E]**; nothing to
  configure, but note it in the probe report.

### 1.5 WSL2: CUDA/NVENC status

What works [E, this host]:
- CUDA user-mode stack present via `/usr/lib/x86_64-linux-gnu/libnvidia-*.so`
  (WSL-provided, version-matched to the Windows driver).
- ffmpeg registers `h264_nvenc`, `hevc_nvenc`, `*_cuvid` decoders and the
  `cuda` hwaccel; `libnvidia-encode.so.595.58.03` is installed.
- `nvidia-smi` works (2 GPUs visible, temps, clocks, throttle counters).

What breaks it (all [E] on this host):
- **No isolation from Windows GPU consumers.** Two Windows processes hold
  23 856 MiB of each 24 GB GPU → any CUDA context creation fails
  (`CUDA_ERROR_OUT_OF_MEMORY`), so `h264_nvenc` init fails with *"No capable
  devices found"* and `h264_cuvid` device setup fails. This is the textbook
  `GPU_UNAVAILABLE` state, and it is caused by another tenant, not by the
  driver.
- **Reduced NVML**: encoder-capacity/session queries returned non-success
  in a ctypes probe; `nvidia-smi -q` shows Encoder Stats fields that cannot
  be validated.
- **No `/dev/dri`** [E: directory absent/empty] → no VAAPI/QSV fallback on
  the same box, no `intel_gpu_top`-style counters, no display-path controls.
- No control of persistence mode, ECC, or power limits from inside WSL.

Verdict: **transcoding in WSL2 is feasible in the "API present, demo works"
sense but is not a production worker platform** — the GPU is a shared
resource with the Windows host, NVML is limited, and admission signals are
unreliable. Use WSL2 to develop the `nvidia-nvenc` probe; run production
workers on native Linux (or pass a whole GPU to a VM with VFIO if the box
must stay Windows-adjacent).

### 1.6 FFmpeg flags: the NVIDIA pipeline

The NVIDIA analogue of `hw decode → hw frames → hw scale → hw encode`:

```bash
# full hardware pipeline, frames never leave the GPU
ffmpeg -y -init_hw_device cuda=cu:0 -filter_hw_device cu \
  -hwaccel cuda -hwaccel_output_format cuda -c:v h264_cuvid -i INPUT \
  -vf scale_cuda=1280:720:format=yuv420p \
  -c:v h264_nvenc -preset p5 -tune hq -rc vbr -cq 23 -b:v 0 \
  -rc-lookahead 20 -spatial_aq 1 -temporal_aq 1 \
  -bf 3 -b_ref_mode middle \
  -c:a copy OUTPUT
```

Key flags (PeerTube-runner passes its own argv; these are what the
capability probe and any wrapper/profile must emit):

| Purpose | Flag | Notes |
|---|---|---|
| HW decode | `-hwaccel cuda -hwaccel_output_format cuda` or explicit `-c:v h264_cuvid`/`hevc_cuvid`/`av1_cuvid` | explicit `*_cuvid` gives named-decoder certainty; `-hwaccel_output_format cuda` keeps frames on-GPU |
| HW upload (when input is CPU-decoded) | `-vf hwupload_cuda` | needed for e.g. audio-merge jobs with CPU-side frames |
| HW scale | `scale_cuda=w:h` (part of ffmpeg) or `scale_npp` (needs nvcc-built ffmpeg) | `scale_cuda` is sufficient and dependency-light |
| Multi-GPU | `-init_hw_device cuda=cu:1`, encoder `-gpu N` | probe must test the *specific* device index it will advertise |
| Encode | `-c:v h264_nvenc / hevc_nvenc / av1_nvenc` | `av1_nvenc` exists only on Ada+ builds of ffmpeg ≥ 5.0 |
| Quality | `-preset p1..p7` (p7 slowest/best), `-tune hq\|ll\|ull`, `-multipass qres\|fullres` | p4–p5 is the sweet spot for VOD |
| Rate control | `-rc vbr -cq N -b:v 0` (quality-targeted VBR) or `-rc constqp` | prefer `-cq` + `-b:v 0` for PeerTube's CRF-style outputs |
| Lookahead/AQ | `-rc-lookahead`, `-spatial_aq`, `-temporal_aq`, `-bf`, `-b_ref_mode middle` (Turing+) | the Turing+ quality levers |
| 10-bit HEVC | `-profile:v main10 -pix_fmt p010le` | verify per generation (§1.2) |
| Session hygiene | `-delay 0`, and **one encode context per session** | sessions are the scarce resource on GeForce (§1.1) |
| Verification | run under `nvidia-smi dmon -s u` (enc/dec columns), check encoder-session count delta | this is the no-silent-CPU *evidence*, see §5.4 |

Minimal VRAM budget per 1080p hw pipeline ≈ 200–400 MiB (decode surfaces +
scale + encode lookahead); 4K ≈ 0.8–1.5 GiB. The probe measures real usage
per resolution class instead of trusting these numbers.

---

## 2. What an NVIDIA worker is (profile)

Same shape as the Intel worker — official runner, admission controller,
status JSON, drain control — differing only in the probe module, deps, and
concurrency inputs.

### 2.1 Differences for `install-peertube-worker.sh`

| Aspect | Intel QSV worker | NVIDIA worker |
|---|---|---|
| Kernel/driver deps | `intel-media-va-driver-non-free`, oneVPL runtime, `/dev/dri/renderD128` + `render`/`video` groups | `nvidia-open` kernel module + pinned driver branch, `libnvidia-encode`, `nvidia-smi`, `nvidia-persistenced`, user in `video` group |
| Device nodes | `/dev/dri/*` (always present) | `/dev/nvidia*` (created by module; `nvidia-modprobe`/udev) |
| Probe module | `intel-qsv` | `nvidia-nvenc` (new module, same interface) |
| Config additions | — | `PT_WORKER_GPU_DEVICES=0` (or `0,1`), `PT_WORKER_SESSIONS_RESERVE=1` (headroom sessions), `PT_WORKER_VRAM_FLOOR_MB=1024`, `PT_WORKER_DRIVER_MIN=550` |
| Self-test | vainfo + QSV transcode | nvidia-smi gates + per-device NVENC transcode + session-slot test + VRAM floor (§5.3) |
| Concurrency inputs | engine util, CPU/RAM | `min(sessions_free, cpu_bound, ram_bound, vram_bound, disk_bound, bw_bound)` |
| Runner invocation | as Intel contract | plus `-gpu N` device pinning in the ffmpeg argv and one runner instance per GPU when a box has >1 GPU (see §2.2) |

Everything else — systemd unit shape, service account, Tailscale posture,
registration, VOD-only job types, drain semantics — is inherited unchanged.

### 2.2 Multi-GPU boxes

A box with K GPUs runs **K runner instances** (`peertube-worker@gpu0…gpuK-1`
template units), each pinned to one device, rather than one runner juggling
devices. Reasons: session accounting and VRAM floors are per-device;
admission states (GPU_UNAVAILABLE for one card after a co-tenant/ECC event)
should not drain healthy siblings; PeerTube sees N independent runners and
load-balances them natively. [E: this host has 2× RTX 3090 and would onboard
as `…:gpu0` and `…:gpu1`.]

---

## 3. Dynamic pool design

### 3.1 Architecture

```
                ┌────────────────────────────────────────────┐
                │ pool-manager (new, small, stateless-ish)   │
                │  signals → policy engine → per-worker      │
                │  desired state (AVAILABLE/DRAINED +        │
                │  concurrency hint + job-class eligibility) │
                └───────┬───────────────────┬────────────────┘
        read-only admin │                   │ control channel (SSH over
        API polls       │                   │ Tailscale → pt-worker CLI)
   PeerTube ◄───────────┘                   ▼
   keeps THE job queue          worker-1 (Intel, QSV)   worker-2 (NV gpu0) …
   and does assignment          admission controller     admission controller
                                + peertube-runner        + peertube-runner
```

The pool manager **never talks to PeerTube as a scheduler**. It only (a)
observes queue pressure read-only, (b) starts/stops/drains worker services,
(c) sets admission state and concurrency. Assignment remains PeerTube's, via
the official runner request/accept cycle. This honors the "do not build a
new job scheduler" constraint.

### 3.2 (a) Load signals

| Signal | Source | Cadence | Notes |
|---|---|---|---|
| `pending_depth` | PeerTube admin API: runner-jobs endpoint filtered by state=pending, per job type (vod-web-video / vod-hls / vod-audio-merge) | 15 s | the primary scale-up input |
| `oldest_pending_age_s` | same endpoint, `createdAt` of oldest pending job | 15 s | catches stall at low depth (e.g. one 4K job nobody can serve) |
| `completed_rate_ewma` | per-worker status JSON `jobs_completed` counters | 30 s | throughput of the currently-active set |
| `enc_util`, `dec_util` | NVIDIA: `nvidia-smi --query-gpu=encoder.utilization...`/NVML; Intel: media-engine counters per Intel contract | 10 s | saturation of the active set |
| `sessions_active/cap` | our own accounting + NVML where supported | per-job event | GeForce cap awareness (§1.1); NVML unsupported on WSL2 [E] |
| `vram_free_mb` | NVML | 10 s | hard eligibility floor |
| `recent_download_mbps` / `upload_mbps` (EWMA) | per-job transfer observations (Intel contract already defines these) | per-job | job-size-aware eligibility input |
| `realtime_factor_ewma` | completed jobs (encode wall-clock vs duration) | per-job | learned worker weight (§3.3) |
| `throttle_reasons`, `temp_c` | NVML throttle-cause bits / thermal zones | 10 s | THERMAL_THROTTLED mapping [E: counters visible on this host] |

Pressure index (normalized 0..1), computed over *eligible* workers only:

```
P = clamp01( max(
    pending_depth / (2 * Σ active_capacity_slots),
    oldest_pending_age_s / 900s
))
```

`Σ active_capacity_slots` = Σ over AVAILABLE/BUSY workers of their current
concurrency hint. The `2×` makes P=1 mean "queue is two full waves deep".

### 3.3 (b) Policy engine

**Pool-worker states** (per worker; these are *manager-side* views layered
on the worker's own admission states from the Intel contract):

| Manager state | Meaning | Transition in |
|---|---|---|
| `COLD` | powered off / suspended (WoL-able, §3.8) | manual or WoL policy |
| `SPARE` | service up, admission DRAINED (registered, idle) | default resting state |
| `WARM` | service up, DRAINED, persistence-mode on, probe recently green | pre-warm tier |
| `ACTIVE` | admission AVAILABLE at concurrency C | scale-up decision |
| `DRAINING` | admission DRAINING, in-flight jobs finishing | scale-down or fault |
| `COOLDOWN` | admission DRAINING after repeated job failures | failure policy |

**Hysteresis** (per-worker and pool-wide):

- Scale **up** when `P ≥ 0.6` sustained ≥ 90 s (two consecutive 45 s windows)
  → promote the highest-weight SPARE/WARM worker to ACTIVE.
- Scale **down** when `P ≤ 0.15` sustained ≥ 20 min **and** worker has been
  idle ≥ 15 min → demote ACTIVE → SPARE (drain, then optional service stop).
- Never demote below `min_active = 1` (or 2 once multi-tenant), regardless
  of idle, to keep probe freshness and WoL-free response.
- Promotions are staggered ≥ 30 s apart and probe-gated: a worker may only
  become ACTIVE if its last probe is < 5 min old and green. This prevents
  thundering-herd registration wake-ups and GPU re-init spikes.
- Failure fast-path bypasses all hysteresis: GPU_UNAVAILABLE, thermal, or a
  hardware-evidence-failed job → immediate DRAINING.

**Per-worker learned weights** (used for promotion order and drain order):

```
w_i = EWMA_λ=0.3 over jobs of ( output_pixel_rate_i / class_baseline )
      shrunk toward class prior:  ŵ = (n·w + k·w_prior) / (n + k),  k=3
```

`output_pixel_rate` = output pixels/sec during encode phase only (excludes
download/upload), bucketed by job class so a worker isn't punished for
getting small jobs. Cold-start workers inherit the class prior, so a new
box's first promotion comes from queue pressure, not fabricated history.
Drain order prefers low `ŵ` *and* low `recent_download_mbps` (cheapest to
re-warm).

**Job-size-aware eligibility.** Job classes from PeerTube job metadata
(source resolution × duration, known at pending time):

| Class | Definition (approx) | Requires |
|---|---|---|
| S | ≤1080p, ≤10 min | any ACTIVE worker |
| M | ≤1080p long, or 1440p | enc_util headroom ≥ 30% |
| L | 4K, or 4K-source→1080p | `realtime_factor_ewma ≥ 3`, `recent_download_mbps ≥ 50`, `vram_free ≥ 4 GiB` (NVIDIA) or equivalent QSV gate |
| XL | 4K60 long-form | L gates + `ŵ ≥ 1.0` |

Enforcement is, again, selective availability: when an L/XL job is pending
and no one satisfies the gates, the manager *keeps* the small workers at
their normal concurrency but does not count them as capacity for that job
(PeerTube will still let a small worker take it if it asks — acceptable
fallback, not silent CPU fallback, and it still satisfies the hardware
rule).

### 3.4 (c) Scale-up / scale-down mechanics

The constraint: **PeerTube runner registration is static.** A runner row
exists server-side from first `register` (with a registration token) until
an admin deletes it; the runner process merely polls. Therefore:

*Joining the pool* =
1. box provisioned + runner registered **once** at install time
   (`install-peertube-worker.sh --name w2:gpu0 …`),
2. service exists but may be stopped or admission-DRAINED,
3. manager: `systemctl start peertube-worker@…` (if stopped) then
   `pt-worker enable` → admission AVAILABLE → runner starts
   requesting jobs on its normal poll cycle.

*Leaving the pool* (grace) =
1. `pt-worker drain` → admission DRAINING; the controller stops offering
   the runner for new jobs (per Intel contract: runner paused/disabled, not
   killed),
2. wait for in-flight jobs: `active_jobs → 0`, with a max-drain timeout
   (e.g. 2× p99 job wall-clock, floor 30 min),
3. then either leave at SPARE (service up, DRAINED — cheapest re-join) or
   `systemctl stop` for overnight-class quiescence, or suspend (§3.8).

Notes:
- Repeated start/stop never touches registration tokens; tokens live only
  in the install-time secret path on the worker.
- Because a drained runner is still *registered*, PeerTube's per-runner
  job-request rate is unaffected; there is no server-side cleanup needed.
- One job → many sessions: PeerTube runs separate ffmpeg instances per
  output rendition, so a "concurrency 1" job can still consume 4+ NVENC
  sessions. The admission controller must therefore count **actual encode
  sessions** (its own children's NVENC contexts), not just jobs, when
  enforcing `sessions_active < cap` (§1.1, §5.3).

### 3.5 (d) Mixed-vendor scheduling

We cannot intercept PeerTube's runner→job assignment, so vendor preference
is expressed as *eligibility*, decided at the moment a job class is pending:

| Job / condition | Prefer | Mechanism |
|---|---|---|
| AV1 **source** | NVIDIA Ampere+ (NVDEC AV1) or CPU-decode on QSV box | gate QSV workers out of the AV1-decode class via probe capability (`av1_dec`) |
| AV1 **output** | NVIDIA **Ada+ only** (`av1_enc`) | same gate; policy says: not enabled in production anyway |
| 4K HEVC/H.264 outputs | NVIDIA (throughput, VRAM for lookahead) | weight-based promotion pulls NVENC boxes in first at L/XL classes |
| ≤1080p H.264 bulk | whichever has session/enc headroom; QSV is typically *more* power-efficient per 1080p job | neutral; capacity-driven |
| Session pressure (NVENC near cap, QSV idle) | QSV | temporarily demote NVENC availability for class S/M |
| One vendor fleet-wide drained (driver CVE, bad driver branch) | the other | vendor-level kill switch in manager config: `vendor_disabled: [nvidia]` |

Probe capability keys that drive this (identical names across vendors):
`h264_enc, hevc_enc, av1_enc, h264_dec, hevc_dec, av1_dec, max10bit`,
`max_sessions`, `max_width/height`. The Intel probe fills the same keys
(i5-1240P: `h264_enc=1, hevc_enc=1, av1_enc=0, av1_dec=1`), which is the
whole point of the pluggable module.

### 3.6 (e) Unified status/metrics schema

Extends the Intel status JSON with a versioned, vendor-generic `gpu` block.
Bump `schema_version`; additive changes only.

```jsonc
{
  "schema_version": 2,
  "hostname": "w2", "runner_name": "w2:gpu0",
  "tailscale_ip": "100.x.y.z", "net_path": "DIRECT|DERP",
  "state": "AVAILABLE",            // Intel-contract states, unchanged
  "registration_state": "registered",
  "active_jobs": 1, "concurrency": 3, "concurrency_limiters": {
      "cpu": 4, "ram": 4, "disk": 8, "bw": 3,
      "gpu_sessions": 3,           // NVIDIA only; null on Intel
      "vram": 3
  },
  "gpu": {                          // NEW: vendor-generic
    "vendor": "nvidia",             // "intel" | "nvidia" | "none"
    "probe_module": "nvidia-nvenc", // "intel-qsv" | "nvidia-nvenc"
    "model": "RTX 3090", "generation": "ampere",
    "nvenc_generation": 5,
    "driver": "595.58.03", "kernel_module": "open",
    "capabilities": { "h264_enc": 1, "hevc_enc": 1, "av1_enc": 0,
                      "h264_dec": 1, "hevc_dec": 1, "av1_dec": 1,
                      "max10bit": 1 },
    "sessions": { "active": 2, "cap": 8,
                  "cap_source": "driver_class_table|nvml|probe" },
    "devices": [ { "index": 0, "uuid": "GPU-2095…",
                   "vram_total_mb": 24576, "vram_free_mb": 22000,
                   "enc_util_pct": 71, "dec_util_pct": 40,
                   "temp_c": 66, "throttle_reasons": [] } ],
    "self_test": { "ts": "2026-10-09T14:00:00Z", "ok": true,
                   "evidence": "enc_util_delta=+68%", "detail": "…" }
  },
  "hw_evidence_last_job": { "ok": true,
      "method": "encoder_session_delta>=1", "ts": "…" },
  "cpu": {"model": "…", "util_pct": 40}, "temps_c": {"cpu": 55},
  "disk_free_gb": 180,
  "recent_download_mbps": 92.4, "recent_upload_mbps": 88.1,
  "jobs": {"completed": 214, "failed": 3, "hw_fallback_prevented": 0},
  "last_job": {"duration_s": 412, "realtime_factor": 7.1,
               "class": "M", "output_pixel_rate_mpx_s": 22.6},
  "manager": {"desired": "ACTIVE", "weight": 1.14, "class_gates": "ok"}
}
```

Prometheus mapping is mechanical: one `pt_worker_up`, one
`pt_worker_gpu_sessions{vendor,cap_source}`, one
`pt_worker_hw_evidence_ok` (the no-silent-CPU invariant as an alertable
series), plus re-exports of the Intel contract's series.

### 3.7 (f) Future: bare-metal autoscaling hooks (WoL)

Brief by request. If worker boxes are wakeable:

- Manager keeps `wol_mac`, `wake_budget` (max 2 concurrent wakes, ≥ 3 min
  apart), and a boot-readiness check = Tailscale SSH + agent status JSON
  with fresh probe (typical budget 3–6 min including GPU persistence).
- Wake policy reuses the scale-up rule with an extra tier: only wake when
  `P ≥ 0.8` sustained 5 min and all SPARE/WARM workers are ACTIVE
  (i.e. waking is the last resort, because boot latency is minutes not
  seconds).
- Sleep path = graceful drain (§3.4) then `systemctl suspend`; AC/battery
  rules from the Intel laptop policy apply first (never sleep a box on
  battery, never wake one).
- Accounting: track `wakes/day` and `woken-minutes-saved` per box; cap wakes
  (e.g. ≤ 6/day/box) to respect hardware and power-on cycles.

---

## 4. Rollout plan

### 4.1 Change the Intel admission controller NOW (small interface list)

Five things, all cheap while the code is fresh:

1. **Versioned status JSON**: add `schema_version: 1` today, and the
   vendor-generic `gpu` block shape (§3.6) with `"vendor": "intel"` —
   i.e. `gpu.probe_module`, `gpu.capabilities{h264_enc,hevc_enc,av1_enc,
   h264_dec,hevc_dec,av1_dec}`, `gpu.model/generation`, per-device array
   (Intel: one entry, `/dev/dri/renderD128`, `enc_util` from media-engine
   counters). Avoid leaking QSV-specific keys to the top level.
2. **Probe interface contract** (document, even if only one impl exists):
   `probe() → {ok, capabilities, maxSessions?, devices[], evidence}` and
   `verifyHardwareDidWork(jobSamples) → {ok, evidence}` — both **fail
   closed**: any exception, timeout, or absent evidence ⇒ GPU_UNAVAILABLE +
   drain, never "assume CPU is fine". Selectable via
   `PT_WORKER_GPU_PROBE=intel-qsv|nvidia-nvenc`.
3. **Concurrency resolver** as a pure function of
   `(probe, cpu, ram, disk, bw, [sessions], [vram])` so NVIDIA is a new
   input set, not a new control path. Expose which limiter bound the choice
   (`concurrency_limiters`, §3.6) — this is what the pool manager consumes.
4. **Hardware-evidence hook per job**: after each completed job, sample the
   GPU's encoder utilization delta (Intel: media-engine busy% delta during
   the job; NVIDIA: NVML encoder-session/util delta) and record
   `hw_evidence_last_job`. A job that completed without evidence must count
   as a *failure* for pool purposes even if PeerTube marks it OK — this is
   the enforcement teeth of the no-silent-CPU rule, and it is vendor-neutral.
5. **Job-class metadata in per-job telemetry** (source res/codec, output
   renditions, wall-clock, realtime factor) so the manager's weights and
   gates (§3.3) have data from day one on the Intel cluster.

### 4.2 Onboarding the first NVIDIA box (when one exists)

1. Pick the platform: **native Linux** (Debian 12/13), Turing or newer;
   Ada+ if AV1 encode is ever wanted. Reject A100/H100-class compute cards
   (no NVENC, §1.1).
2. Driver install, pinned branch (≥ 550; open kernel modules for Turing+),
   `nvidia-persistenced`, user in `video`, verify
   `nvidia-smi` + `libnvidia-encode.so.1` present. Reboot-verify.
3. ffmpeg: distro build is fine if it lists `h264_nvenc`/`hevc_nvenc` and
   `*_cuvid` **[E: Debian 12's ffmpeg 5.1.9 does]**; else pin a static
   BtbN/johnvansickle-style build ≥ 5.0 with ffnvcodec. Record
   `ffmpeg --version` in status JSON.
4. `install-peertube-worker.sh --name <host>:gpu0 --gpu-probe nvidia-nvenc
   --gpu-devices 0 …` — same script, new profile (§2.1). Per additional GPU
   on the box, repeat with `:gpu1`, … (§2.2).
5. Self-test gate (must pass before registration): see §5.3 — nvidia-smi
   gates → per-device NVENC encode probe under `dmon` → session-slot test →
   VRAM floor. Fail ⇒ abort install with `GPU_UNAVAILABLE`, no runner
   registration.
6. Register runner (VOD job types only, per Intel contract), start service,
   run one real E2E job, and assert hardware evidence: encoder-session
   delta ≥ 1 per rendition and enc-util delta > 0 **and** CPU-user-time of
   the ffmpeg process ≪ CPU-encode baseline (belt and braces).
7. Soak: 1 h at target concurrency; watch sessions, VRAM, temps, throttle
   reasons, throughput; set the box's initial `concurrency` from measured
   end-to-end throughput, not single-job fps (Intel Phase 9 methodology,
   unchanged).
8. Add to pool manager inventory as `SPARE` with class-prior weight; first
   promotion happens on the next pressure event.
9. Only then consider multi-GPU boxes and WoL tiers (§3.7).

---

## 5. Appendices

### 5.1 Failure-mode → state mapping (NVIDIA additions)

| Failure | Detection | State | Action |
|---|---|---|---|
| Driver/module missing or unloaded | nvidia-smi nonzero, no `/dev/nvidia*` | GPU_UNAVAILABLE | refuse work; re-probe 60 s |
| VRAM exhausted (co-tenant, leak) [E] | CUDA_ERROR_OUT_OF_MEMORY / "No capable devices found" in ffmpeg init; NVML `vram_free` below floor | GPU_UNAVAILABLE (device-scoped) | drain that device's runner; sibling GPUs unaffected (§2.2) |
| Session cap hit | our accounting ≥ cap − reserve | BUSY (capacity) | no new jobs until sessions free; never queue extra ffmpeg per job |
| Thermal/power throttle | NVML throttle-cause bits, temp ≥ limit | THERMAL_THROTTLED | drain after in-flight; cooldown |
| Encoder produced no HW evidence | `hw_evidence_last_job.ok=false` | GPU_UNAVAILABLE + job counted failed | page; **never** re-run on CPU |
| Driver/GPU fallen off bus (Xid errors in dmesg) | dmesg `NVRM: Xid` watch | OFFLINE | drain + host-level alert |
| Network DERP/throughput collapse | Intel contract's EWMA + path change | NETWORK_DEGRADED | unchanged from Intel |

### 5.2 Config surface (NVIDIA profile)

```
PT_WORKER_GPU_PROBE=nvidia-nvenc
PT_WORKER_GPU_DEVICES=0            # one runner instance per device
PT_WORKER_SESSIONS_CAP=auto        # nvml → driver_class_table → 8 (GeForce)
PT_WORKER_SESSIONS_RESERVE=1       # keep 1 session headroom
PT_WORKER_VRAM_FLOOR_MB=1024
PT_WORKER_DRIVER_MIN=550
PT_WORKER_FFMPEG_NVENC_PRESET=p5   # map PeerTube quality tier → preset
PT_WORKER_PROBE_INTERVAL=300       # full probe; 10 s counters in between
```

### 5.3 `nvidia-nvenc` probe / self-test design (ordered, fail-closed)

1. **Gate**: `nvidia-smi` exits 0; driver ≥ min; module flavor recorded;
   `libnvidia-encode.so.1` resolvable; ≥1 device with `vram_free ≥ floor`.
2. **Capability enumeration**: per advertised device, from NVML name →
   generation → capability table (§1.2/§1.3), *then verified empirically*:
   attempt a 1-second `h264_nvenc` encode on that exact device (`-gpu N`);
   attempt `hevc_nvenc`; attempt `av1_nvenc` only if generation ≥ Ada.
   Absence of an encoder in `ffmpeg -encoders` downgrades capability
   without failing the probe (e.g. distro ffmpeg without av1_nvenc).
3. **Pipeline probe**: full hw chain (§1.6) at 1080p on the target device
   under `nvidia-smi dmon -s u`; require `enc>0` and `dec>0` columns
   observed during the run, and record VRAM delta as the per-class budget.
4. **Session-slot test**: launch K parallel tiny `h264_nvenc` encodes
   (K = advertised cap) and count how many initialize successfully
   (`No capable devices found` / init failure = slot unavailable). The
   largest K′ that succeeds becomes `sessions.cap` with
   `cap_source=probe`. This is the ground truth the controller trusts —
   it automatically reflects driver-class caps **and** whatever the
   platform (e.g. WSL2 [E]) actually allows.
5. **Decode matrix**: one decode-only pass per codec family
   (h264/hevc/av1 `*_cuvid`) on a synthetic stream, asserting `dec>0`.
6. **Report**: fill the §3.6 `gpu` block, include `dmon` excerpts and VRAM
   deltas as `evidence`. Any step erroring or timing out (30 s/step) ⇒
   `ok=false` ⇒ GPU_UNAVAILABLE. No step may conclude "use CPU instead".

### 5.4 Enforcing the no-silent-CPU rule on NVIDIA

Three independent tripwires, any one sufficient to fail a job:

1. **Argv gate**: the wrapper/runner must be invoking `*_nvenc` for video
   (grep the spawned argv); PeerTube's own fallback logic must be
   disabled/refused in config.
2. **Evidence gate**: NVML encoder-session count and encoder utilization
   must increase while the job's ffmpeg runs (sampled ≥ 1 Hz); a completed
   encode with flat counters = CPU fallback (or a no-op) ⇒ job marked
   failed, worker GPU_UNAVAILABLE, manager COOLDOWN.
3. **Plausibility gate**: ffmpeg CPU user-time for the job must be far
   below the CPU-encode baseline for that resolution (measured once per
   box during install); a CPU-encode-shaped CPU profile ⇒ same treatment.

This is deliberately vendor-neutral (Intel uses media-engine busy% in place
of NVML counters), which is exactly why it belongs in the shared admission
controller rather than in the probe module.

### 5.5 Open questions for the Intel implementation (blocking-free)

- Exact admin endpoint + auth shape for pending runner-jobs depth (manager
  needs it read-only; a proxy via the worker's own credentials is
  acceptable if admin scope is unavailable).
- Whether the Intel drain implementation pauses the runner process or
  stops the unit — the manager wants to treat both as DRAINED and needs to
  know which to expect for start/stop idempotency.
- Whether per-rendition ffmpeg invocation is confirmed (drives
  session-counting assumptions, §3.4).