T-HUB/infra/peertube-workers

This directory prepares Intel QSV workers for the inspected PeerTube 7.3.0 instance. It uses the published @peertube/peertube-runner 0.2.0, pinned with npm integrity hashes. PeerTube owns the queue; the controller admits or drains a whole runner. No PeerTube fork, scheduler, public worker listener, live delegation, or viewer-serving component is introduced.

Recommended next step: use native Linux with a working Intel DRM render device, then rerun the real QSV self-test before enrolling. A Windows-native alternative requires a separate adapter and supervisor: FFmpeg documents h264_mf -hw_encoding 1 to force Media Foundation hardware encoding, with a verified D3D11 decode/scale path and Windows power/GPU telemetry. The existing Linux QSV wrapper and systemd service cannot be reused unchanged. No Windows hardware encode has been tested. WSL's Mesa D3D12 VAAPI path is a different implementation from this libmfx/Intel DRM QSV path and was not deployed. Windows also reports an RTX 3090, but that is not a validated NVIDIA worker.

The actual 7.3.0 VOD key is transcoding.remote_runners.enabled (not vod_enabled). The override is /config/production.yaml in the effective Docker config volume. Only PeerTube was restarted; remote live/studio/transcription remain disabled. The actual deployed database/table are peertube / runnerRegistrationToken. The existing registration token was checked privately using the installed validator and an exact database lookup; no new token or runner was created. A successful registration handshake and real VOD E2E remain untested because the hardware gate failed.

The installed runner source was compared to the v7.3.0 tag: apps/peertube-runner/src/server/process/shared/common.ts has SHA256 d28fe0f4db9ca46edaeac2f12c269f0fad52fd5cf92d9ba90d529c4a1d7c3627 in both. The 7.3.0 CLI docs require Node >=20 and FFmpeg/ffprobe >=4.3. This installer uses isolated Node 22 and refuses other PeerTube releases until their source is checked. The current latest Node 22 tarball is downloaded over HTTPS, verified against Node's SHA256 manifest, and retained on reruns. The runner version stays pinned.

Runner behavior verified in 7.3.0 source	Consequence
register uses a local Unix IPC socket; the server must already be running	Enrollment starts a VOD-only server with concurrency 0, then uses the official IPC library
Socket.IO /runners, websocket-only transport, available-jobs notifications	Workers initiate API and WS connections; no inbound listener is needed
Requests up to ten pending jobs; accepts the first supported job; checks again on connect, notifications, completion, and after 2.5 seconds when there is spare capacity	Queue selection, acceptance, and concurrency remain with the official runner
Request payload includes input URLs and requested resolution/fps, but no input size or duration	Whole-worker admission; no per-job byte prediction
Downloads use POST carrying runner/job tokens, then transcode and POST multipart results	Workers have only their dedicated runner credential after enrollment
VOD handlers remove inputs/results in finally; startup removes leftover transcode files	Output cleanup is upstream behavior; smoke test verifies it
Progress update every 60 seconds; VOD stale threshold 2 minutes; MAX_FAILURES=5	Server requeues eligible failures until its limit; controller does not implement retries
SIGTERM/SIGINT abort active jobs; graceful-shutdown stops acceptance and waits for completion	All ordinary controller drains use the official graceful command
Download timeout is 2 hours; there is no independent periodic idle polling timer	A network drain can remain pending while an upstream transfer times out; reconnection triggers a queue check

Remote profile plugins are not an integration option in this release: getCommonFFmpegOptions() sets profile: 'default' and the default video encoder list contains only libx264. Installing a server profile plugin would not reach this runner. No maintained remote QSV plugin was established during the investigation. The implementation therefore uses preference (4), a minimal worker wrapper, rather than patching the runner or PeerTube.

FFMPEG_PATH=ffmpeg-qsv.py redirects commands generated by the runner. For video it replaces the inspected default x264 options with explicit QSV decode, QSV hardware frames, scale_qsv, and h264_qsv. It keeps PeerTube's muxing, HLS segmenting, fps, GOP, and rate caps. H.264/HEVC/VP9/AV1 source paths use explicit QSV decoders. Unsupported pixel formats, source codecs, rotation, unknown filters, and unknown encoder commands fail closed. Static cover images for audio-merge jobs require image decode and one upload of frames; their scaling and video encoding use QSV. Audio-only renditions can use the upstream AAC encoder. No video CPU fallback exists. Capability queries are passed through to FFmpeg.

The self-test runs real H.264 decode -> GPU scale -> H.264/HEVC encode, HEVC decode when encoding succeeded, and separate AV1 decode/encode attempts. It records cumulative Intel DRM media-engine counters from FFmpeg's /proc/PID/fdinfo, output ffprobe results, temperatures, and memory. A successful exit alone is insufficient: H.264 must also have positive media-engine counters. Missing counters or permissions block enrollment. AV1 encode is expected to fail on an i5-1240P; this expectation does not substitute for inspecting the actual CPU/GPU. See Intel's driver features and FFmpeg QSV filters. HEVC is a self-test/investigation output only; production remains H.264.

observe.cjs measures real response/request byte counts and elapsed transfer times without recording bodies, headers, credentials, or URLs. It correlates downloaded temporary files and encode events with job UUIDs, including concurrent jobs. Input payloads use the public canonical origin; just that origin is mapped to the configured Tailscale API address. Other storage origins retain HTTPS and normal redirects. The route preserves API paths and authentication. Interrupted response streams explicitly destroy the request so upstream download promises reject instead of waiting indefinitely. HTTP on TAILSCALE_IP:9000 travels inside encrypted Tailscale; public viewer URLs remain untouched. Safe runner milestones are retained; arbitrary upstream logs are suppressed because they can contain tokens.

pt-worker.py exposes JSON status and optional Prometheus text. State is updated roughly every five seconds plus probe/IPC/API time. Admission states are AVAILABLE, BUSY, DRAINING, NETWORK_DEGRADED, GPU_UNAVAILABLE, LOW_DISK, THERMAL_THROTTLED, OFFLINE, and PAUSED. Degraded states request a graceful drain. Already accepted work finishes. Detection and graceful IPC have a bounded polling race; a hardware failure nevertheless cannot select a CPU encoder.

Real transfer observations use EWMA alpha 0.3. Defaults: 10 Mbps download/upload floor, recovery at 1.25 times those floors, 3 good samples to reopen, 2 bad samples to degrade, 150 ms maximum latency, 60 seconds after a transfer failure, 15-minute measurement expiry, DERP disallowed, 20 GiB free disk, and an 85 C thermal ceiling with 120-second cooldown. DIRECT/DERP transitions and suspend gaps invalidate throughput history. Floors are initial policy values, not measured laptop capacity. Tune them after real jobs.

The optional server probe prevents stale bandwidth from permanently blocking an idle or rebooted worker. It is already installed on TAILSCALE_IP:9044, binds only that Tailscale IP, discards uploads, caps each transfer at 16 MiB, permits two concurrent transfers, applies per-client/per-direction rate limits, and runs without media, Docker, DB, Redis, or filesystem write access. Ordinary checks transfer 8 MiB each way after an idle runner drains, no more often than every five minutes. Real job observations remain the preferred measurements. The probe measures the worker-to-PeerTube path, not a future external object-storage path; object-storage transfers need separate successful job observations.

On AC loss or unknown laptop power state, admission drains. Native machines with no battery are treated as permanently powered. Battery capacity/status, thermal sensors, CPU throttle counters, memory, load, disk, and kernel type are included when visible; unsupported measurements are null. There is no forced suspend, reboot, perf-policy change, or firmware change. The controller reruns its hardware test after a changed boot ID. Manual pause survives reboot. With the probe and healthy hardware it can recover network admission unattended.

Shortest additional-worker procedure, once a hardware worker passes validation: join Tailscale; provide a native Debian 12 / Ubuntu 24.04+ Linux environment with Intel /dev/dri/renderD128, systemd, and adequate disk; copy this directory to the worker; use an owner-authorized registration token (the existing token is authorized for this deployment); then run:

sudo ./install-peertube-worker.sh --name WORKER2 --peertube-url http://TAILSCALE_IP:9000
# The registration token is read at a hidden prompt, never as a CLI argument.
sudo pt-worker start
sudo pt-worker status

The installer derives this instance's canonical origin from a fallback asset in /api/v1/config. For other branding or instances supply --canonical-origin https://INSTANCE and --tailscale-peer TAILSCALE_IP; deploy the probe at that IP or adjust probe_url. For noninteractive enrollment use a private file/pipe FD, e.g. sudo -i, then exec 3</root/runner-registration-token followed by the installer with --registration-token-fd 3; revoke/delete the temporary enrollment secret after confirming registration. Do not put tokens in argv, shell history, environment files, git, or reports.

The installer inspects hardware, systemd, Tailscale, and server release before installing packages. It chooses Intel's full-feature media driver when available in existing package sources, otherwise the free driver and relies on actual tests. It does not alter repositories or kernel settings. Dependencies include FFmpeg, VAAPI tools, Intel GPU tools, Python >=3.11, and Node >=22. Service user pt-worker receives the actual render-device group and existing render/video groups; no chmod 777. Code and settings are root-owned; registration TOML and runtime state are worker-owned and private. The worker service is enabled at boot and initially paused. Uncertain enrollment outcomes leave a marker and refuse automatic re-registration, preventing duplicate runners; inspect/revoke the named server runner before resolving that marker. Normal reruns preserve credentials, settings, and concurrency and back up existing installation/configuration before replacement.

Command (run with sudo)	Behavior
pt-worker status	JSON; unknown fields are null, stale controller state is OFFLINE
pt-worker status --prometheus	Numeric metrics without inventing absent measurements
pt-worker start	Starts the supervisor and opens admission when checks pass
pt-worker start --calibrate	Explicit ten-minute initial real-job measurement window if the probe is unavailable; still enforces GPU, AC, disk, latency, and direct path
pt-worker stop / drain	Persists PAUSED; gracefully finishes accepted work; supervisor stays alive for status
pt-worker restart	Graceful child restart; preserves the runner registration
pt-worker self-test	Refuses while a runner is running; renews hardware proof after drain
pt-worker network-test	Real bounded probe download/upload, or explicit disposable --download-url / --upload-url endpoints
pt-worker benchmark --source /path/source.mp4	After drain, local QSV 720p diagnostic batches at 1/2/3/4, FPS/RTF/wall time/RAM/temperature/CPU/media utilization/throttle observations; does not choose production concurrency

Native systemd uses Restart=always, 30-second restart delay, start throttling, an unprivileged account, read-only system/home protection, and no capabilities. KillMode=mixed gives the supervisor the initial stop signal so it can drain the official runner. Stop timeout is infinite to preserve accepted work; forced operator kills/crashes rely on upstream abort/watchdog behavior. GPU and memory metrics are observational and never justify a CPU fallback.

WSL must be inspected before using this installer. The visible Windows Tailscale peer does not identify the SSH shell or WSL kernel. Check Windows wsl.exe --list --verbose, then inside the intended distro check uname -a, /proc/1/comm, /etc/wsl.conf, /dev/dri, /dev/dxg, lscpu, lspci, vainfo, FFmpeg, and driver capabilities. Microsoft documents enabling systemd with [boot] systemd=true; applying it requires a distro restart. That alone does not prove Intel Linux DRM/QSV support. /dev/dxg alone cannot satisfy this adapter. If the laptop exposes only the Windows GPU path, use native Linux or separately implement and validate a Windows QSV/oneVPL/D3D adapter and Windows service/power/telemetry. None of these host changes have been made.

Production validation still required, after resolving the hardware blocker: VOD remote delegation is already enabled; preserve existing resolutions and viewer configuration and keep live/studio/transcription remote flags disabled. Install/register the first proven hardware worker with the authorized existing token, and open admission at concurrency 2. Upload a disposable private synthetic A/V clip through an existing admin/user channel. Capture the real runner job UUIDs and Pending -> Processing -> Completing -> Completed, source/output probes, QSV engine counters, encode and accept-to-success wall times, actual download/upload observations, imported HLS/renditions, playback, A/V sync, and temporary cleanup. Repeat equivalent batches at concurrency 1/2/3/4 after draining and changing both worker.json and runner TOML; choose by completed media seconds per end-to-end wall time, with thermal and network headroom. The local benchmark cannot make that decision.

Failure validation on the worker, after the normal VOD test: drain/restart and verify no new registration; interrupt only the worker's Tailscale path under a recoverable admin connection and verify drain/recovery/server retries; configure an inaccessible test render path and ensure no CPU output; constrain a disposable probe path and verify low-bandwidth hysteresis; disconnect AC and observe drain; finally reboot the worker and verify service/Tailscale/hardware proof/network admission/registration recovery. Do not interrupt or reboot the production PeerTube host for these tests.

Suggested Tailscale policy: put worker devices in tag:pt-worker, server in tag:pt-server; allow worker -> server TCP 9000 and 9044 only for this deployment; authorize admin -> worker SSH separately; deny worker DB/Redis access and inbound worker connections except explicitly authorized administration. ACLs are centrally managed and were not changed here. Retain HTTPS egress to approved storage origins if object storage is later enabled. The server's existing TCP 1935/public/tunnel behavior was not changed. Live delegation later requires live-rtmp-hls-transcoding, worker-initiated access to the job's PeerTube RTMP input URL, sustained encode capacity and continuous HLS upload with measured upload headroom, low stable latency, a stricter direct-path policy, and separate failure/load validation. Workers do not serve viewers.

NVIDIA readiness: admission consumes generic gpu_ready, gpu_tested_at, GPU device accessibility, and gpu_failure events; network, queue, AC, disk, thermal policy, and graceful IPC are independent of Intel. A future provider can supply gpu_health_file with hardware_video_ok and tested_at, and gpu_device. NVIDIA still needs a separately tested CUDA/NVDEC -> CUDA scale -> NVENC adapter, driver/container permissions, an NVENC hardware self-test, NVIDIA utilization/temperature collection, codec/session-limit validation, and an installer package set. The existing Intel installer and fdinfo collector do not configure NVIDIA.

Tests are executable without a GPU or production token:

python3 -m unittest discover -s tests -v
node tests/test_observer.cjs
# Install the pinned runtime into a temporary directory and use Node 22:
python3 tests/runner_smoke.py --node /path/node --runtime /path/runtime
systemd-analyze verify systemd/*.service

The smoke test uses a loopback API and the published official runner. It proves enrollment at concurrency 0, secure IPC, persistence/restart, source routing, refusal/reporting without a GPU, graceful shutdown, and cleanup. It does not prove a production job, QSV acceleration, import/playback, laptop service/reboot behavior, or real laptop throughput. Keep private runtime evidence outside git. The worker jobs_completed counter counts acknowledged result submissions; server-side import success must still be verified centrally with the admin job API. Safe event logs currently need an operator retention policy; they are not automatically rotated. @peertube/net-ipc@2.2.0 is deprecated on npm but matches the installed runner's IPC protocol; review this helper together with any runner upgrade.

Windows NVENC

The Windows-native RTX 3090 worker, executable adapter, installer phases, admission policy, service elevation requirements, and onboarding steps are documented in WINDOWS.md. Use that runbook for Windows workers; the Linux QSV installer is unchanged.