glass beadian
aa5e262f3a
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
632 lines
35 KiB
Markdown
632 lines
35 KiB
Markdown
# NVIDIA GPU Extension + Dynamic Pool Scaling — Design & Research
|
||
PeerTube transcoding worker cluster · design wave · 2026-10-09
|
||
|
||
Companion to the Intel QSV worker implementation landing in
|
||
`infra/peertube-workers/` (the enrolled worker laptop). This document depends **only on the
|
||
Intel worker's CONTRACT** (official `@peertube/peertube-runner`, per-worker
|
||
admission controller, pluggable capability probe, status JSON, drain control,
|
||
fail-closed hardware rule), never on its internals.
|
||
|
||
Empirical anchors in this doc are marked **[E]** and were measured on this
|
||
machine on 2026-10-09: WSL2 host, 2× NVIDIA GeForce RTX 3090 (Ampere GA102),
|
||
Windows-side driver 595.58.03 / CUDA 13.2, Debian 12 ffmpeg 5.1.9
|
||
(`0+deb12u1`, libmfx build), 56-core Xeon E5-2690 v4, 128 GiB RAM. Numbers not
|
||
marked [E] are from NVIDIA primary documentation (Video Codec SDK 13.x
|
||
support matrix, NVENC help page, driver release notes, CUDA-on-WSL guide) as
|
||
recalled; **no web access was available in this session, so treat
|
||
vendor-doc-derived figures as "verify against the live matrix when the first
|
||
NVIDIA box is onboarded"** — every one of them is also re-derived at install
|
||
time by the probe (§5.3), so a stale number here degrades to a warning, never
|
||
to a wrong production decision.
|
||
|
||
---
|
||
|
||
## 0. Executive summary
|
||
|
||
1. NVENC on GeForce is no longer meaningfully session-capped for our use
|
||
case: the driver limit is **8 concurrent encode sessions per GPU** since
|
||
driver branch 536 (mid-2023); professional/datacenter boards have **no
|
||
driver cap**. Our per-worker concurrency will be bounded by VRAM,
|
||
NVDEC+filter throughput and bandwidth long before 8 sessions matter —
|
||
but the cap must still be tracked, because PeerTube fans one job out to
|
||
many resolution renditions and a 5-rendition job on one GPU *is* 5
|
||
sessions.
|
||
2. The hardware rule generalizes cleanly: the same fail-closed contract that
|
||
keeps QSV from falling back to x264 keeps NVENC from falling back to x264.
|
||
The only NVIDIA-specific additions are device selection (`-gpu N`),
|
||
driver/kernel-module installation, session accounting, and VRAM floors.
|
||
3. **Do not put NVIDIA workers on WSL2.** [E] The API surface exists
|
||
(`h264_nvenc`/`hevc_nvenc` registered, `libnvidia-encode.so.595.58.03`
|
||
present) but the GPU is shared with the Windows host, NVML encoder
|
||
queries are reduced, and there is no `/dev/dri`, no persistence-mode
|
||
control and no isolation from Windows-side VRAM consumers. On this very
|
||
box NVENC is currently unusable because two co-tenant Windows processes
|
||
hold 23.8/24 GiB on each GPU. WSL2 is a dev/probe platform; production
|
||
NVIDIA workers are native Linux.
|
||
4. Dynamic pool scaling does **not** require touching PeerTube's scheduler.
|
||
Runners register once and persist server-side; "joining the pool" is
|
||
`systemctl start` + admission AVAILABLE, "leaving" is graceful drain +
|
||
optional `systemctl stop`. The pool manager nudges *which* runners are
|
||
eligible (and at what concurrency), PeerTube still decides job
|
||
assignment. Mixed-vendor preference is expressed the same way: selective
|
||
availability + concurrency, not scheduler surgery.
|
||
5. The one interface the Intel implementation must get right **now**: the
|
||
capability probe must be a *mandatory, fail-closed, versioned* plugin
|
||
interface, and the status JSON must be vendor-generic with a `gpu.vendor`
|
||
discriminator from day one.
|
||
|
||
---
|
||
|
||
## 1. NVIDIA capability research (2026)
|
||
|
||
### 1.1 NVENC concurrent-session limits
|
||
|
||
| GPU class | Driver-imposed concurrent NVENC sessions | Notes |
|
||
|---|---|---|
|
||
| GeForce (consumer) | **8** per GPU | Raised 3→5 in 530.41.03/531.29 (2023-03), 5→8 in branch 536 (2023-07). Applies per GPU, across all processes; an ffmpeg process using `-threads` still counts as 1 session per encode context. |
|
||
| Professional / workstation (RTX A-series, RTX 5000/6000 Ada, RTX Pro Blackwell; ex-Quadro) | **No driver cap** | Bounded only by hardware/VRAM. |
|
||
| Datacenter media-class (T4, A2, L4, L40, L40S) | **No driver cap** | These are the right datacenter parts for transcoding. |
|
||
| Datacenter compute-class (A100, H100, H200, B100/B200) | **0 — no NVENC at all** | Compute dies ship no encode engine (and Hopper/A100 have reduced/absent NVDEC). Never buy these for a transcode pool. |
|
||
|
||
Design consequences:
|
||
|
||
- The cap counts **encode sessions**, not processes and not ffmpeg
|
||
"engines". A single PeerTube VOD job that produces 4 renditions from one
|
||
decode can hold 4 encode sessions (PeerTube today typically runs one
|
||
ffmpeg per rendition, so it holds 4 decode sessions too — see §3.4).
|
||
- Session pressure must be *measured*, not assumed from the table: NVML
|
||
exposes `nvmlDeviceGetEncoderCapacity` / encoder-session counts on
|
||
bare-metal Linux, but **[E]** on WSL2 the capacity query returned a
|
||
non-success code in our ctypes probe, so the probe design (§5.3) treats
|
||
NVML as best-effort with a driver-class lookup table and our own
|
||
accounting as ground truth.
|
||
|
||
### 1.2 NVENC encode support matrix by generation
|
||
|
||
Generational numbering below is the commonly used NVENC-generation scheme
|
||
(1=Kepler … 7=Blackwell); NVIDIA's own docs speak of "Turing NVENC",
|
||
"Ampere NVENC", etc. Quality-relevant jumps: Turing (large HEVC/H.264
|
||
quality+lookahead gains) and Ada (AV1 + HEVC 4:4:4).
|
||
|
||
| NVENC generation (arch) | Example parts | H.264 | HEVC 8-bit | HEVC 10-bit | HEVC B-frames | HEVC 4:4:4 | AV1 |
|
||
|---|---|---|---|---|---|---|---|
|
||
| Gen 1 (Kepler) | GTX 680 | ✓ | — | — | — | — | — |
|
||
| Gen 2 (Maxwell) | GTX 750Ti/970; GM206 (950/960) | ✓ | GM206 only | — | — | — | — |
|
||
| Gen 3 (Pascal) | GTX 10xx, Quadro P | ✓ | ✓ | ✓ | — | — | — |
|
||
| Gen 4 (Turing) | GTX 16xx, RTX 20xx, T4 | ✓ | ✓ | ✓ | ✓ | — | — |
|
||
| Gen 5 (Ampere) | RTX 30xx, A-series | ✓ | ✓ | ✓ | ✓ | ✓ (Pro/A-series) | — |
|
||
| Gen 6 (Ada) | RTX 40xx, L4, L40/L40S | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||
| Gen 7 (Blackwell) | RTX 50xx, RTX Pro | ✓ | ✓ | ✓ | ✓ | ✓ + 4:2:2 | ✓ (+UHQ) |
|
||
|
||
**[E]** Consistency check on this box (RTX 3090 = Ampere = gen 5):
|
||
ffmpeg 5.1.9 lists `h264_nvenc` and `hevc_nvenc` and does **not** list
|
||
`av1_nvenc`; `av1_cuvid` decode **is** listed. Matches the matrix exactly
|
||
(Ampere: AV1 decode yes, AV1 encode no).
|
||
|
||
### 1.3 NVDEC decode support matrix
|
||
|
||
| NVDEC generation (arch) | H.264 | HEVC 8/10-bit | VP9 | HEVC 4:4:4 | AV1 |
|
||
|---|---|---|---|---|---|
|
||
| Maxwell | ✓ | GM206: 8-bit only | — | — | — |
|
||
| Pascal | ✓ | ✓ | ✓ | — | — |
|
||
| Turing | ✓ | ✓ | ✓ 10/12-bit | ✓ | — |
|
||
| Ampere | ✓ | ✓ | ✓ | ✓ | ✓ 8/10-bit |
|
||
| Ada | ✓ | ✓ | ✓ | ✓ | ✓ (incl. higher bit depths) |
|
||
| Blackwell | ✓ | ✓ (+4:2:2) | ✓ | ✓ | ✓ (+4:4:4/12-bit) |
|
||
|
||
Practical rule for the pool: **any card we would actually buy (Turing or
|
||
newer) decodes H.264/HEVC/VP9; AV1 decode needs Ampere+; AV1 *encode* needs
|
||
Ada+**. An Ampere pool (like this box) can ingest AV1 sources and emit
|
||
H.264/HEVC, which fits the stated "H.264 primary, HEVC secondary, no AV1
|
||
output" policy with headroom.
|
||
|
||
### 1.4 Linux driver stack
|
||
|
||
- **Open vs proprietary kernel modules**: NVIDIA's open kernel modules
|
||
(`nvidia-open` / open-gpu-kernel-modules) are the **default recommended
|
||
path for Turing+ since the 560-series**; proprietary remains for
|
||
pre-Turing and a few feature gaps. All realistic pool hardware (Turing+)
|
||
→ use open modules. Blackwell effectively requires them.
|
||
- 2026 driver branches: current feature branches run in the 5xx range;
|
||
**[E]** this host's Windows-side driver is 595.58.03 (CUDA 13.2). Debian
|
||
packages: `nvidia-open-kernel-dkms` (or prebuilt `linux-image-amd64` +
|
||
signed modules), `nvidia-driver-libs`, `nvidia-smi`, `nvidia-utils`;
|
||
firmware from `non-free-firmware`. Keep driver version **pinned per pool
|
||
fleet** and record `driver + kernel + module flavor` in the status JSON.
|
||
- Userspace pieces that matter to us: `libnvidia-encode.so.1` (NVENC),
|
||
CUDA driver (`libcuda.so.1`) used by ffmpeg's nvenc/cuvid code paths,
|
||
`libnvidia-ml.so.1` (NVML for nvidia-smi). ffmpeg needs no CUDA toolkit
|
||
— only the driver libs. (A statically-built ffmpeg with `--enable-ffnvcodec`
|
||
/ `--enable-cuda-nvcc` is only needed for `scale_npp`; `scale_cuda` is
|
||
covered by the header-only ffnvcodec path.)
|
||
- Access control: user in `video` group (or a dedicated worker user with a
|
||
udev/tagged device ACL); `nvidia-persistenced` (or
|
||
`NVreg_PreserveVideoMemoryAllocations=1`) to avoid multi-second GPU
|
||
re-init between jobs on an idle pool box — relevant to dynamic scale-up
|
||
latency (§3.7).
|
||
- MIG/compute-mode: leave compute mode `DEFAULT` **[E]**; nothing to
|
||
configure, but note it in the probe report.
|
||
|
||
### 1.5 WSL2: CUDA/NVENC status
|
||
|
||
What works [E, this host]:
|
||
- CUDA user-mode stack present via `/usr/lib/x86_64-linux-gnu/libnvidia-*.so`
|
||
(WSL-provided, version-matched to the Windows driver).
|
||
- ffmpeg registers `h264_nvenc`, `hevc_nvenc`, `*_cuvid` decoders and the
|
||
`cuda` hwaccel; `libnvidia-encode.so.595.58.03` is installed.
|
||
- `nvidia-smi` works (2 GPUs visible, temps, clocks, throttle counters).
|
||
|
||
What breaks it (all [E] on this host):
|
||
- **No isolation from Windows GPU consumers.** Two Windows processes hold
|
||
23 856 MiB of each 24 GB GPU → any CUDA context creation fails
|
||
(`CUDA_ERROR_OUT_OF_MEMORY`), so `h264_nvenc` init fails with *"No capable
|
||
devices found"* and `h264_cuvid` device setup fails. This is the textbook
|
||
`GPU_UNAVAILABLE` state, and it is caused by another tenant, not by the
|
||
driver.
|
||
- **Reduced NVML**: encoder-capacity/session queries returned non-success
|
||
in a ctypes probe; `nvidia-smi -q` shows Encoder Stats fields that cannot
|
||
be validated.
|
||
- **No `/dev/dri`** [E: directory absent/empty] → no VAAPI/QSV fallback on
|
||
the same box, no `intel_gpu_top`-style counters, no display-path controls.
|
||
- No control of persistence mode, ECC, or power limits from inside WSL.
|
||
|
||
Verdict: **transcoding in WSL2 is feasible in the "API present, demo works"
|
||
sense but is not a production worker platform** — the GPU is a shared
|
||
resource with the Windows host, NVML is limited, and admission signals are
|
||
unreliable. Use WSL2 to develop the `nvidia-nvenc` probe; run production
|
||
workers on native Linux (or pass a whole GPU to a VM with VFIO if the box
|
||
must stay Windows-adjacent).
|
||
|
||
### 1.6 FFmpeg flags: the NVIDIA pipeline
|
||
|
||
The NVIDIA analogue of `hw decode → hw frames → hw scale → hw encode`:
|
||
|
||
```bash
|
||
# full hardware pipeline, frames never leave the GPU
|
||
ffmpeg -y -init_hw_device cuda=cu:0 -filter_hw_device cu \
|
||
-hwaccel cuda -hwaccel_output_format cuda -c:v h264_cuvid -i INPUT \
|
||
-vf scale_cuda=1280:720:format=yuv420p \
|
||
-c:v h264_nvenc -preset p5 -tune hq -rc vbr -cq 23 -b:v 0 \
|
||
-rc-lookahead 20 -spatial_aq 1 -temporal_aq 1 \
|
||
-bf 3 -b_ref_mode middle \
|
||
-c:a copy OUTPUT
|
||
```
|
||
|
||
Key flags (PeerTube-runner passes its own argv; these are what the
|
||
capability probe and any wrapper/profile must emit):
|
||
|
||
| Purpose | Flag | Notes |
|
||
|---|---|---|
|
||
| HW decode | `-hwaccel cuda -hwaccel_output_format cuda` or explicit `-c:v h264_cuvid`/`hevc_cuvid`/`av1_cuvid` | explicit `*_cuvid` gives named-decoder certainty; `-hwaccel_output_format cuda` keeps frames on-GPU |
|
||
| HW upload (when input is CPU-decoded) | `-vf hwupload_cuda` | needed for e.g. audio-merge jobs with CPU-side frames |
|
||
| HW scale | `scale_cuda=w:h` (part of ffmpeg) or `scale_npp` (needs nvcc-built ffmpeg) | `scale_cuda` is sufficient and dependency-light |
|
||
| Multi-GPU | `-init_hw_device cuda=cu:1`, encoder `-gpu N` | probe must test the *specific* device index it will advertise |
|
||
| Encode | `-c:v h264_nvenc / hevc_nvenc / av1_nvenc` | `av1_nvenc` exists only on Ada+ builds of ffmpeg ≥ 5.0 |
|
||
| Quality | `-preset p1..p7` (p7 slowest/best), `-tune hq\|ll\|ull`, `-multipass qres\|fullres` | p4–p5 is the sweet spot for VOD |
|
||
| Rate control | `-rc vbr -cq N -b:v 0` (quality-targeted VBR) or `-rc constqp` | prefer `-cq` + `-b:v 0` for PeerTube's CRF-style outputs |
|
||
| Lookahead/AQ | `-rc-lookahead`, `-spatial_aq`, `-temporal_aq`, `-bf`, `-b_ref_mode middle` (Turing+) | the Turing+ quality levers |
|
||
| 10-bit HEVC | `-profile:v main10 -pix_fmt p010le` | verify per generation (§1.2) |
|
||
| Session hygiene | `-delay 0`, and **one encode context per session** | sessions are the scarce resource on GeForce (§1.1) |
|
||
| Verification | run under `nvidia-smi dmon -s u` (enc/dec columns), check encoder-session count delta | this is the no-silent-CPU *evidence*, see §5.4 |
|
||
|
||
Minimal VRAM budget per 1080p hw pipeline ≈ 200–400 MiB (decode surfaces +
|
||
scale + encode lookahead); 4K ≈ 0.8–1.5 GiB. The probe measures real usage
|
||
per resolution class instead of trusting these numbers.
|
||
|
||
---
|
||
|
||
## 2. What an NVIDIA worker is (profile)
|
||
|
||
Same shape as the Intel worker — official runner, admission controller,
|
||
status JSON, drain control — differing only in the probe module, deps, and
|
||
concurrency inputs.
|
||
|
||
### 2.1 Differences for `install-peertube-worker.sh`
|
||
|
||
| Aspect | Intel QSV worker | NVIDIA worker |
|
||
|---|---|---|
|
||
| Kernel/driver deps | `intel-media-va-driver-non-free`, oneVPL runtime, `/dev/dri/renderD128` + `render`/`video` groups | `nvidia-open` kernel module + pinned driver branch, `libnvidia-encode`, `nvidia-smi`, `nvidia-persistenced`, user in `video` group |
|
||
| Device nodes | `/dev/dri/*` (always present) | `/dev/nvidia*` (created by module; `nvidia-modprobe`/udev) |
|
||
| Probe module | `intel-qsv` | `nvidia-nvenc` (new module, same interface) |
|
||
| Config additions | — | `PT_WORKER_GPU_DEVICES=0` (or `0,1`), `PT_WORKER_SESSIONS_RESERVE=1` (headroom sessions), `PT_WORKER_VRAM_FLOOR_MB=1024`, `PT_WORKER_DRIVER_MIN=550` |
|
||
| Self-test | vainfo + QSV transcode | nvidia-smi gates + per-device NVENC transcode + session-slot test + VRAM floor (§5.3) |
|
||
| Concurrency inputs | engine util, CPU/RAM | `min(sessions_free, cpu_bound, ram_bound, vram_bound, disk_bound, bw_bound)` |
|
||
| Runner invocation | as Intel contract | plus `-gpu N` device pinning in the ffmpeg argv and one runner instance per GPU when a box has >1 GPU (see §2.2) |
|
||
|
||
Everything else — systemd unit shape, service account, Tailscale posture,
|
||
registration, VOD-only job types, drain semantics — is inherited unchanged.
|
||
|
||
### 2.2 Multi-GPU boxes
|
||
|
||
A box with K GPUs runs **K runner instances** (`peertube-worker@gpu0…gpuK-1`
|
||
template units), each pinned to one device, rather than one runner juggling
|
||
devices. Reasons: session accounting and VRAM floors are per-device;
|
||
admission states (GPU_UNAVAILABLE for one card after a co-tenant/ECC event)
|
||
should not drain healthy siblings; PeerTube sees N independent runners and
|
||
load-balances them natively. [E: this host has 2× RTX 3090 and would onboard
|
||
as `…:gpu0` and `…:gpu1`.]
|
||
|
||
---
|
||
|
||
## 3. Dynamic pool design
|
||
|
||
### 3.1 Architecture
|
||
|
||
```
|
||
┌────────────────────────────────────────────┐
|
||
│ pool-manager (new, small, stateless-ish) │
|
||
│ signals → policy engine → per-worker │
|
||
│ desired state (AVAILABLE/DRAINED + │
|
||
│ concurrency hint + job-class eligibility) │
|
||
└───────┬───────────────────┬────────────────┘
|
||
read-only admin │ │ control channel (SSH over
|
||
API polls │ │ Tailscale → pt-worker CLI)
|
||
PeerTube ◄───────────┘ ▼
|
||
keeps THE job queue worker-1 (Intel, QSV) worker-2 (NV gpu0) …
|
||
and does assignment admission controller admission controller
|
||
+ peertube-runner + peertube-runner
|
||
```
|
||
|
||
The pool manager **never talks to PeerTube as a scheduler**. It only (a)
|
||
observes queue pressure read-only, (b) starts/stops/drains worker services,
|
||
(c) sets admission state and concurrency. Assignment remains PeerTube's, via
|
||
the official runner request/accept cycle. This honors the "do not build a
|
||
new job scheduler" constraint.
|
||
|
||
### 3.2 (a) Load signals
|
||
|
||
| Signal | Source | Cadence | Notes |
|
||
|---|---|---|---|
|
||
| `pending_depth` | PeerTube admin API: runner-jobs endpoint filtered by state=pending, per job type (vod-web-video / vod-hls / vod-audio-merge) | 15 s | the primary scale-up input |
|
||
| `oldest_pending_age_s` | same endpoint, `createdAt` of oldest pending job | 15 s | catches stall at low depth (e.g. one 4K job nobody can serve) |
|
||
| `completed_rate_ewma` | per-worker status JSON `jobs_completed` counters | 30 s | throughput of the currently-active set |
|
||
| `enc_util`, `dec_util` | NVIDIA: `nvidia-smi --query-gpu=encoder.utilization...`/NVML; Intel: media-engine counters per Intel contract | 10 s | saturation of the active set |
|
||
| `sessions_active/cap` | our own accounting + NVML where supported | per-job event | GeForce cap awareness (§1.1); NVML unsupported on WSL2 [E] |
|
||
| `vram_free_mb` | NVML | 10 s | hard eligibility floor |
|
||
| `recent_download_mbps` / `upload_mbps` (EWMA) | per-job transfer observations (Intel contract already defines these) | per-job | job-size-aware eligibility input |
|
||
| `realtime_factor_ewma` | completed jobs (encode wall-clock vs duration) | per-job | learned worker weight (§3.3) |
|
||
| `throttle_reasons`, `temp_c` | NVML throttle-cause bits / thermal zones | 10 s | THERMAL_THROTTLED mapping [E: counters visible on this host] |
|
||
|
||
Pressure index (normalized 0..1), computed over *eligible* workers only:
|
||
|
||
```
|
||
P = clamp01( max(
|
||
pending_depth / (2 * Σ active_capacity_slots),
|
||
oldest_pending_age_s / 900s
|
||
))
|
||
```
|
||
|
||
`Σ active_capacity_slots` = Σ over AVAILABLE/BUSY workers of their current
|
||
concurrency hint. The `2×` makes P=1 mean "queue is two full waves deep".
|
||
|
||
### 3.3 (b) Policy engine
|
||
|
||
**Pool-worker states** (per worker; these are *manager-side* views layered
|
||
on the worker's own admission states from the Intel contract):
|
||
|
||
| Manager state | Meaning | Transition in |
|
||
|---|---|---|
|
||
| `COLD` | powered off / suspended (WoL-able, §3.8) | manual or WoL policy |
|
||
| `SPARE` | service up, admission DRAINED (registered, idle) | default resting state |
|
||
| `WARM` | service up, DRAINED, persistence-mode on, probe recently green | pre-warm tier |
|
||
| `ACTIVE` | admission AVAILABLE at concurrency C | scale-up decision |
|
||
| `DRAINING` | admission DRAINING, in-flight jobs finishing | scale-down or fault |
|
||
| `COOLDOWN` | admission DRAINING after repeated job failures | failure policy |
|
||
|
||
**Hysteresis** (per-worker and pool-wide):
|
||
|
||
- Scale **up** when `P ≥ 0.6` sustained ≥ 90 s (two consecutive 45 s windows)
|
||
→ promote the highest-weight SPARE/WARM worker to ACTIVE.
|
||
- Scale **down** when `P ≤ 0.15` sustained ≥ 20 min **and** worker has been
|
||
idle ≥ 15 min → demote ACTIVE → SPARE (drain, then optional service stop).
|
||
- Never demote below `min_active = 1` (or 2 once multi-tenant), regardless
|
||
of idle, to keep probe freshness and WoL-free response.
|
||
- Promotions are staggered ≥ 30 s apart and probe-gated: a worker may only
|
||
become ACTIVE if its last probe is < 5 min old and green. This prevents
|
||
thundering-herd registration wake-ups and GPU re-init spikes.
|
||
- Failure fast-path bypasses all hysteresis: GPU_UNAVAILABLE, thermal, or a
|
||
hardware-evidence-failed job → immediate DRAINING.
|
||
|
||
**Per-worker learned weights** (used for promotion order and drain order):
|
||
|
||
```
|
||
w_i = EWMA_λ=0.3 over jobs of ( output_pixel_rate_i / class_baseline )
|
||
shrunk toward class prior: ŵ = (n·w + k·w_prior) / (n + k), k=3
|
||
```
|
||
|
||
`output_pixel_rate` = output pixels/sec during encode phase only (excludes
|
||
download/upload), bucketed by job class so a worker isn't punished for
|
||
getting small jobs. Cold-start workers inherit the class prior, so a new
|
||
box's first promotion comes from queue pressure, not fabricated history.
|
||
Drain order prefers low `ŵ` *and* low `recent_download_mbps` (cheapest to
|
||
re-warm).
|
||
|
||
**Job-size-aware eligibility.** Job classes from PeerTube job metadata
|
||
(source resolution × duration, known at pending time):
|
||
|
||
| Class | Definition (approx) | Requires |
|
||
|---|---|---|
|
||
| S | ≤1080p, ≤10 min | any ACTIVE worker |
|
||
| M | ≤1080p long, or 1440p | enc_util headroom ≥ 30% |
|
||
| L | 4K, or 4K-source→1080p | `realtime_factor_ewma ≥ 3`, `recent_download_mbps ≥ 50`, `vram_free ≥ 4 GiB` (NVIDIA) or equivalent QSV gate |
|
||
| XL | 4K60 long-form | L gates + `ŵ ≥ 1.0` |
|
||
|
||
Enforcement is, again, selective availability: when an L/XL job is pending
|
||
and no one satisfies the gates, the manager *keeps* the small workers at
|
||
their normal concurrency but does not count them as capacity for that job
|
||
(PeerTube will still let a small worker take it if it asks — acceptable
|
||
fallback, not silent CPU fallback, and it still satisfies the hardware
|
||
rule).
|
||
|
||
### 3.4 (c) Scale-up / scale-down mechanics
|
||
|
||
The constraint: **PeerTube runner registration is static.** A runner row
|
||
exists server-side from first `register` (with a registration token) until
|
||
an admin deletes it; the runner process merely polls. Therefore:
|
||
|
||
*Joining the pool* =
|
||
1. box provisioned + runner registered **once** at install time
|
||
(`install-peertube-worker.sh --name w2:gpu0 …`),
|
||
2. service exists but may be stopped or admission-DRAINED,
|
||
3. manager: `systemctl start peertube-worker@…` (if stopped) then
|
||
`pt-worker enable` → admission AVAILABLE → runner starts
|
||
requesting jobs on its normal poll cycle.
|
||
|
||
*Leaving the pool* (grace) =
|
||
1. `pt-worker drain` → admission DRAINING; the controller stops offering
|
||
the runner for new jobs (per Intel contract: runner paused/disabled, not
|
||
killed),
|
||
2. wait for in-flight jobs: `active_jobs → 0`, with a max-drain timeout
|
||
(e.g. 2× p99 job wall-clock, floor 30 min),
|
||
3. then either leave at SPARE (service up, DRAINED — cheapest re-join) or
|
||
`systemctl stop` for overnight-class quiescence, or suspend (§3.8).
|
||
|
||
Notes:
|
||
- Repeated start/stop never touches registration tokens; tokens live only
|
||
in the install-time secret path on the worker.
|
||
- Because a drained runner is still *registered*, PeerTube's per-runner
|
||
job-request rate is unaffected; there is no server-side cleanup needed.
|
||
- One job → many sessions: PeerTube runs separate ffmpeg instances per
|
||
output rendition, so a "concurrency 1" job can still consume 4+ NVENC
|
||
sessions. The admission controller must therefore count **actual encode
|
||
sessions** (its own children's NVENC contexts), not just jobs, when
|
||
enforcing `sessions_active < cap` (§1.1, §5.3).
|
||
|
||
### 3.5 (d) Mixed-vendor scheduling
|
||
|
||
We cannot intercept PeerTube's runner→job assignment, so vendor preference
|
||
is expressed as *eligibility*, decided at the moment a job class is pending:
|
||
|
||
| Job / condition | Prefer | Mechanism |
|
||
|---|---|---|
|
||
| AV1 **source** | NVIDIA Ampere+ (NVDEC AV1) or CPU-decode on QSV box | gate QSV workers out of the AV1-decode class via probe capability (`av1_dec`) |
|
||
| AV1 **output** | NVIDIA **Ada+ only** (`av1_enc`) | same gate; policy says: not enabled in production anyway |
|
||
| 4K HEVC/H.264 outputs | NVIDIA (throughput, VRAM for lookahead) | weight-based promotion pulls NVENC boxes in first at L/XL classes |
|
||
| ≤1080p H.264 bulk | whichever has session/enc headroom; QSV is typically *more* power-efficient per 1080p job | neutral; capacity-driven |
|
||
| Session pressure (NVENC near cap, QSV idle) | QSV | temporarily demote NVENC availability for class S/M |
|
||
| One vendor fleet-wide drained (driver CVE, bad driver branch) | the other | vendor-level kill switch in manager config: `vendor_disabled: [nvidia]` |
|
||
|
||
Probe capability keys that drive this (identical names across vendors):
|
||
`h264_enc, hevc_enc, av1_enc, h264_dec, hevc_dec, av1_dec, max10bit`,
|
||
`max_sessions`, `max_width/height`. The Intel probe fills the same keys
|
||
(i5-1240P: `h264_enc=1, hevc_enc=1, av1_enc=0, av1_dec=1`), which is the
|
||
whole point of the pluggable module.
|
||
|
||
### 3.6 (e) Unified status/metrics schema
|
||
|
||
Extends the Intel status JSON with a versioned, vendor-generic `gpu` block.
|
||
Bump `schema_version`; additive changes only.
|
||
|
||
```jsonc
|
||
{
|
||
"schema_version": 2,
|
||
"hostname": "w2", "runner_name": "w2:gpu0",
|
||
"tailscale_ip": "100.x.y.z", "net_path": "DIRECT|DERP",
|
||
"state": "AVAILABLE", // Intel-contract states, unchanged
|
||
"registration_state": "registered",
|
||
"active_jobs": 1, "concurrency": 3, "concurrency_limiters": {
|
||
"cpu": 4, "ram": 4, "disk": 8, "bw": 3,
|
||
"gpu_sessions": 3, // NVIDIA only; null on Intel
|
||
"vram": 3
|
||
},
|
||
"gpu": { // NEW: vendor-generic
|
||
"vendor": "nvidia", // "intel" | "nvidia" | "none"
|
||
"probe_module": "nvidia-nvenc", // "intel-qsv" | "nvidia-nvenc"
|
||
"model": "RTX 3090", "generation": "ampere",
|
||
"nvenc_generation": 5,
|
||
"driver": "595.58.03", "kernel_module": "open",
|
||
"capabilities": { "h264_enc": 1, "hevc_enc": 1, "av1_enc": 0,
|
||
"h264_dec": 1, "hevc_dec": 1, "av1_dec": 1,
|
||
"max10bit": 1 },
|
||
"sessions": { "active": 2, "cap": 8,
|
||
"cap_source": "driver_class_table|nvml|probe" },
|
||
"devices": [ { "index": 0, "uuid": "GPU-2095…",
|
||
"vram_total_mb": 24576, "vram_free_mb": 22000,
|
||
"enc_util_pct": 71, "dec_util_pct": 40,
|
||
"temp_c": 66, "throttle_reasons": [] } ],
|
||
"self_test": { "ts": "2026-10-09T14:00:00Z", "ok": true,
|
||
"evidence": "enc_util_delta=+68%", "detail": "…" }
|
||
},
|
||
"hw_evidence_last_job": { "ok": true,
|
||
"method": "encoder_session_delta>=1", "ts": "…" },
|
||
"cpu": {"model": "…", "util_pct": 40}, "temps_c": {"cpu": 55},
|
||
"disk_free_gb": 180,
|
||
"recent_download_mbps": 92.4, "recent_upload_mbps": 88.1,
|
||
"jobs": {"completed": 214, "failed": 3, "hw_fallback_prevented": 0},
|
||
"last_job": {"duration_s": 412, "realtime_factor": 7.1,
|
||
"class": "M", "output_pixel_rate_mpx_s": 22.6},
|
||
"manager": {"desired": "ACTIVE", "weight": 1.14, "class_gates": "ok"}
|
||
}
|
||
```
|
||
|
||
Prometheus mapping is mechanical: one `pt_worker_up`, one
|
||
`pt_worker_gpu_sessions{vendor,cap_source}`, one
|
||
`pt_worker_hw_evidence_ok` (the no-silent-CPU invariant as an alertable
|
||
series), plus re-exports of the Intel contract's series.
|
||
|
||
### 3.7 (f) Future: bare-metal autoscaling hooks (WoL)
|
||
|
||
Brief by request. If worker boxes are wakeable:
|
||
|
||
- Manager keeps `wol_mac`, `wake_budget` (max 2 concurrent wakes, ≥ 3 min
|
||
apart), and a boot-readiness check = Tailscale SSH + agent status JSON
|
||
with fresh probe (typical budget 3–6 min including GPU persistence).
|
||
- Wake policy reuses the scale-up rule with an extra tier: only wake when
|
||
`P ≥ 0.8` sustained 5 min and all SPARE/WARM workers are ACTIVE
|
||
(i.e. waking is the last resort, because boot latency is minutes not
|
||
seconds).
|
||
- Sleep path = graceful drain (§3.4) then `systemctl suspend`; AC/battery
|
||
rules from the Intel laptop policy apply first (never sleep a box on
|
||
battery, never wake one).
|
||
- Accounting: track `wakes/day` and `woken-minutes-saved` per box; cap wakes
|
||
(e.g. ≤ 6/day/box) to respect hardware and power-on cycles.
|
||
|
||
---
|
||
|
||
## 4. Rollout plan
|
||
|
||
### 4.1 Change the Intel admission controller NOW (small interface list)
|
||
|
||
Five things, all cheap while the code is fresh:
|
||
|
||
1. **Versioned status JSON**: add `schema_version: 1` today, and the
|
||
vendor-generic `gpu` block shape (§3.6) with `"vendor": "intel"` —
|
||
i.e. `gpu.probe_module`, `gpu.capabilities{h264_enc,hevc_enc,av1_enc,
|
||
h264_dec,hevc_dec,av1_dec}`, `gpu.model/generation`, per-device array
|
||
(Intel: one entry, `/dev/dri/renderD128`, `enc_util` from media-engine
|
||
counters). Avoid leaking QSV-specific keys to the top level.
|
||
2. **Probe interface contract** (document, even if only one impl exists):
|
||
`probe() → {ok, capabilities, maxSessions?, devices[], evidence}` and
|
||
`verifyHardwareDidWork(jobSamples) → {ok, evidence}` — both **fail
|
||
closed**: any exception, timeout, or absent evidence ⇒ GPU_UNAVAILABLE +
|
||
drain, never "assume CPU is fine". Selectable via
|
||
`PT_WORKER_GPU_PROBE=intel-qsv|nvidia-nvenc`.
|
||
3. **Concurrency resolver** as a pure function of
|
||
`(probe, cpu, ram, disk, bw, [sessions], [vram])` so NVIDIA is a new
|
||
input set, not a new control path. Expose which limiter bound the choice
|
||
(`concurrency_limiters`, §3.6) — this is what the pool manager consumes.
|
||
4. **Hardware-evidence hook per job**: after each completed job, sample the
|
||
GPU's encoder utilization delta (Intel: media-engine busy% delta during
|
||
the job; NVIDIA: NVML encoder-session/util delta) and record
|
||
`hw_evidence_last_job`. A job that completed without evidence must count
|
||
as a *failure* for pool purposes even if PeerTube marks it OK — this is
|
||
the enforcement teeth of the no-silent-CPU rule, and it is vendor-neutral.
|
||
5. **Job-class metadata in per-job telemetry** (source res/codec, output
|
||
renditions, wall-clock, realtime factor) so the manager's weights and
|
||
gates (§3.3) have data from day one on the Intel cluster.
|
||
|
||
### 4.2 Onboarding the first NVIDIA box (when one exists)
|
||
|
||
1. Pick the platform: **native Linux** (Debian 12/13), Turing or newer;
|
||
Ada+ if AV1 encode is ever wanted. Reject A100/H100-class compute cards
|
||
(no NVENC, §1.1).
|
||
2. Driver install, pinned branch (≥ 550; open kernel modules for Turing+),
|
||
`nvidia-persistenced`, user in `video`, verify
|
||
`nvidia-smi` + `libnvidia-encode.so.1` present. Reboot-verify.
|
||
3. ffmpeg: distro build is fine if it lists `h264_nvenc`/`hevc_nvenc` and
|
||
`*_cuvid` **[E: Debian 12's ffmpeg 5.1.9 does]**; else pin a static
|
||
BtbN/johnvansickle-style build ≥ 5.0 with ffnvcodec. Record
|
||
`ffmpeg --version` in status JSON.
|
||
4. `install-peertube-worker.sh --name <host>:gpu0 --gpu-probe nvidia-nvenc
|
||
--gpu-devices 0 …` — same script, new profile (§2.1). Per additional GPU
|
||
on the box, repeat with `:gpu1`, … (§2.2).
|
||
5. Self-test gate (must pass before registration): see §5.3 — nvidia-smi
|
||
gates → per-device NVENC encode probe under `dmon` → session-slot test →
|
||
VRAM floor. Fail ⇒ abort install with `GPU_UNAVAILABLE`, no runner
|
||
registration.
|
||
6. Register runner (VOD job types only, per Intel contract), start service,
|
||
run one real E2E job, and assert hardware evidence: encoder-session
|
||
delta ≥ 1 per rendition and enc-util delta > 0 **and** CPU-user-time of
|
||
the ffmpeg process ≪ CPU-encode baseline (belt and braces).
|
||
7. Soak: 1 h at target concurrency; watch sessions, VRAM, temps, throttle
|
||
reasons, throughput; set the box's initial `concurrency` from measured
|
||
end-to-end throughput, not single-job fps (Intel Phase 9 methodology,
|
||
unchanged).
|
||
8. Add to pool manager inventory as `SPARE` with class-prior weight; first
|
||
promotion happens on the next pressure event.
|
||
9. Only then consider multi-GPU boxes and WoL tiers (§3.7).
|
||
|
||
---
|
||
|
||
## 5. Appendices
|
||
|
||
### 5.1 Failure-mode → state mapping (NVIDIA additions)
|
||
|
||
| Failure | Detection | State | Action |
|
||
|---|---|---|---|
|
||
| Driver/module missing or unloaded | nvidia-smi nonzero, no `/dev/nvidia*` | GPU_UNAVAILABLE | refuse work; re-probe 60 s |
|
||
| VRAM exhausted (co-tenant, leak) [E] | CUDA_ERROR_OUT_OF_MEMORY / "No capable devices found" in ffmpeg init; NVML `vram_free` below floor | GPU_UNAVAILABLE (device-scoped) | drain that device's runner; sibling GPUs unaffected (§2.2) |
|
||
| Session cap hit | our accounting ≥ cap − reserve | BUSY (capacity) | no new jobs until sessions free; never queue extra ffmpeg per job |
|
||
| Thermal/power throttle | NVML throttle-cause bits, temp ≥ limit | THERMAL_THROTTLED | drain after in-flight; cooldown |
|
||
| Encoder produced no HW evidence | `hw_evidence_last_job.ok=false` | GPU_UNAVAILABLE + job counted failed | page; **never** re-run on CPU |
|
||
| Driver/GPU fallen off bus (Xid errors in dmesg) | dmesg `NVRM: Xid` watch | OFFLINE | drain + host-level alert |
|
||
| Network DERP/throughput collapse | Intel contract's EWMA + path change | NETWORK_DEGRADED | unchanged from Intel |
|
||
|
||
### 5.2 Config surface (NVIDIA profile)
|
||
|
||
```
|
||
PT_WORKER_GPU_PROBE=nvidia-nvenc
|
||
PT_WORKER_GPU_DEVICES=0 # one runner instance per device
|
||
PT_WORKER_SESSIONS_CAP=auto # nvml → driver_class_table → 8 (GeForce)
|
||
PT_WORKER_SESSIONS_RESERVE=1 # keep 1 session headroom
|
||
PT_WORKER_VRAM_FLOOR_MB=1024
|
||
PT_WORKER_DRIVER_MIN=550
|
||
PT_WORKER_FFMPEG_NVENC_PRESET=p5 # map PeerTube quality tier → preset
|
||
PT_WORKER_PROBE_INTERVAL=300 # full probe; 10 s counters in between
|
||
```
|
||
|
||
### 5.3 `nvidia-nvenc` probe / self-test design (ordered, fail-closed)
|
||
|
||
1. **Gate**: `nvidia-smi` exits 0; driver ≥ min; module flavor recorded;
|
||
`libnvidia-encode.so.1` resolvable; ≥1 device with `vram_free ≥ floor`.
|
||
2. **Capability enumeration**: per advertised device, from NVML name →
|
||
generation → capability table (§1.2/§1.3), *then verified empirically*:
|
||
attempt a 1-second `h264_nvenc` encode on that exact device (`-gpu N`);
|
||
attempt `hevc_nvenc`; attempt `av1_nvenc` only if generation ≥ Ada.
|
||
Absence of an encoder in `ffmpeg -encoders` downgrades capability
|
||
without failing the probe (e.g. distro ffmpeg without av1_nvenc).
|
||
3. **Pipeline probe**: full hw chain (§1.6) at 1080p on the target device
|
||
under `nvidia-smi dmon -s u`; require `enc>0` and `dec>0` columns
|
||
observed during the run, and record VRAM delta as the per-class budget.
|
||
4. **Session-slot test**: launch K parallel tiny `h264_nvenc` encodes
|
||
(K = advertised cap) and count how many initialize successfully
|
||
(`No capable devices found` / init failure = slot unavailable). The
|
||
largest K′ that succeeds becomes `sessions.cap` with
|
||
`cap_source=probe`. This is the ground truth the controller trusts —
|
||
it automatically reflects driver-class caps **and** whatever the
|
||
platform (e.g. WSL2 [E]) actually allows.
|
||
5. **Decode matrix**: one decode-only pass per codec family
|
||
(h264/hevc/av1 `*_cuvid`) on a synthetic stream, asserting `dec>0`.
|
||
6. **Report**: fill the §3.6 `gpu` block, include `dmon` excerpts and VRAM
|
||
deltas as `evidence`. Any step erroring or timing out (30 s/step) ⇒
|
||
`ok=false` ⇒ GPU_UNAVAILABLE. No step may conclude "use CPU instead".
|
||
|
||
### 5.4 Enforcing the no-silent-CPU rule on NVIDIA
|
||
|
||
Three independent tripwires, any one sufficient to fail a job:
|
||
|
||
1. **Argv gate**: the wrapper/runner must be invoking `*_nvenc` for video
|
||
(grep the spawned argv); PeerTube's own fallback logic must be
|
||
disabled/refused in config.
|
||
2. **Evidence gate**: NVML encoder-session count and encoder utilization
|
||
must increase while the job's ffmpeg runs (sampled ≥ 1 Hz); a completed
|
||
encode with flat counters = CPU fallback (or a no-op) ⇒ job marked
|
||
failed, worker GPU_UNAVAILABLE, manager COOLDOWN.
|
||
3. **Plausibility gate**: ffmpeg CPU user-time for the job must be far
|
||
below the CPU-encode baseline for that resolution (measured once per
|
||
box during install); a CPU-encode-shaped CPU profile ⇒ same treatment.
|
||
|
||
This is deliberately vendor-neutral (Intel uses media-engine busy% in place
|
||
of NVML counters), which is exactly why it belongs in the shared admission
|
||
controller rather than in the probe module.
|
||
|
||
### 5.5 Open questions for the Intel implementation (blocking-free)
|
||
|
||
- Exact admin endpoint + auth shape for pending runner-jobs depth (manager
|
||
needs it read-only; a proxy via the worker's own credentials is
|
||
acceptable if admin scope is unavailable).
|
||
- Whether the Intel drain implementation pauses the runner process or
|
||
stops the unit — the manager wants to treat both as DRAINED and needs to
|
||
know which to expect for start/stop idempotency.
|
||
- Whether per-rendition ffmpeg invocation is confirmed (drives
|
||
session-counting assumptions, §3.4).
|