glass beadian
aa5e262f3a
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
190 lines
12 KiB
Markdown
190 lines
12 KiB
Markdown
# Windows native NVENC worker
|
||
|
||
Use Windows 10/11 x64 with an NVIDIA GPU and a working driver. WSL is the SSH
|
||
transport only: Node, FFmpeg, the shim, admission, and IPC all run as Windows
|
||
processes. The initial worker name is `worker-laptop-win`; root is `C:\pt-worker`.
|
||
The canonical instance is `https://tube.occupymedia.cc`; worker traffic uses
|
||
`http://TAILSCALE_IP:9000` inside encrypted Tailscale. Viewers use PeerTube.
|
||
No server configuration change is needed: remote VOD is already enabled.
|
||
|
||
## Prepare and enroll
|
||
|
||
Run the installer from this directory. Python 3.12+ and Windows Tailscale are
|
||
build/access prerequisites. Python is embedded in the frozen executables, so
|
||
services do not depend on the developer's Python installation. Pins are in
|
||
[win/dependencies.json](win/dependencies.json); the FFmpeg URL is a dated archive,
|
||
verified by SHA256. The two executables are built from reviewed Python source
|
||
using the versions in `win/build-requirements.txt`. Runner and IPC dependencies
|
||
use the existing runtime lockfile. Do not substitute a `.cmd` shim: Node's
|
||
shell-free spawn rejects it with EINVAL.
|
||
|
||
```powershell
|
||
# Elevated PowerShell: installs Node MSI silently plus isolated Node 22.
|
||
.\install-peertube-worker-windows.ps1 -Phase Prepare -Name WORKERNAME-win
|
||
# A filtered SSH/WSL token can prepare everything except the MSI:
|
||
.\install-peertube-worker-windows.ps1 -Phase Prepare -UserLevel -Name WORKERNAME-win
|
||
.\install-peertube-worker-windows.ps1 -Phase SelfTest
|
||
```
|
||
|
||
Both H.264 and HEVC must produce a real 5-second 1080p encode **and positive
|
||
NVIDIA encoder utilization**. GPU identity, UUID, driver, boot, device index,
|
||
exit codes, samples, and session statistics are recorded in `state\nvenc.json`.
|
||
No proof means no enrollment. The installer keeps admission PAUSED. A failed
|
||
init, absent telemetry signal, reboot, or driver mismatch closes admission.
|
||
Session-0 proof runs again when services are installed; enrollment-session
|
||
success alone does not prove service-context access.
|
||
|
||
Supply the **existing** registration token through stdin. Never pass it as a
|
||
PowerShell/Node argument, environment variable, transcript, or source file.
|
||
For SSH transport, pipe PostgreSQL's selected token output directly into the
|
||
Windows Node `bin\ipc-nvenc.cjs register` process. The version-controlled
|
||
`win/register-from-ssh.py --worker USER@HOST --name NAME` implements this using two subprocesses and pipe handles; it fetches
|
||
`runnerRegistrationToken.registrationToken`, database `peertube`, using the
|
||
PostgreSQL container's existing `POSTGRES_USER`. No token is printed or retained.
|
||
Before doing so, run `bin\enroll-server.cjs`: it requires concurrency 0 and drops
|
||
all upstream logs. After registration use `ipc-nvenc.cjs graceful-shutdown`, wait
|
||
for the listener to exit, and set TOML concurrency to 2. Alternatively use:
|
||
|
||
```powershell
|
||
# The calling process must give this process the secret on inherited stdin.
|
||
.\install-peertube-worker-windows.ps1 -Phase Enroll -RegistrationTokenStdin
|
||
```
|
||
|
||
The **registration secret** is never saved. The official runner necessarily
|
||
saves its separate issued **runner credential** in
|
||
`profile\Roaming\peertube-runner-nodejs\Config\default\config.toml`.
|
||
NTFS ACLs restrict this tree to the owner, SYSTEM, administrators, and LocalService.
|
||
Never print that TOML, arbitrary runner logs, database token columns, or HTTP
|
||
headers. Enrollment pending markers prohibit blind duplicate registrations.
|
||
|
||
## Services and operations
|
||
|
||
```powershell
|
||
# ServiceFiles prepares concrete WinSW XML/exe files without touching SCM.
|
||
.\install-peertube-worker-windows.ps1 -Phase ServiceFiles
|
||
# Elevated PowerShell; services initially PAUSED, session-0 proof first.
|
||
.\install-peertube-worker-windows.ps1 -Phase Services
|
||
C:\pt-worker\pt-worker-win.ps1 status
|
||
C:\pt-worker\pt-worker-win.ps1 network-test
|
||
C:\pt-worker\pt-worker-win.ps1 start
|
||
C:\pt-worker\pt-worker-win.ps1 drain
|
||
C:\pt-worker\pt-worker-win.ps1 restart
|
||
C:\pt-worker\pt-worker-win.ps1 self-test
|
||
```
|
||
|
||
WinSW 2.12.0 wraps `pt-runner` (Node supervisor) and `pt-admission` (frozen Python).
|
||
Both use the [WinSW 2.12 service-account and stop configuration](https://github.com/winsw/winsw/blob/v2.12.0/doc/xmlConfigFile.md), running as LocalService with automatic delayed startup, a 30-second service
|
||
restart delay, and limited rotated logs. The supervisor uses a further 15-second
|
||
runner restart delay. It retains only safe milestones and observer events;
|
||
unfiltered upstream logs are discarded because they can contain tokens.
|
||
A service stop asks the supervisor to drain via the official named-pipe IPC,
|
||
with an explicit one-hour SCM stop ceiling. For jobs that can exceed this ceiling,
|
||
use `pt-worker-win drain`, wait for `runner_running=false`, then stop services.
|
||
No admission loss forcibly kills an active encode.
|
||
|
||
State decisions use the Linux Meter/Admission semantics: AVAILABLE, BUSY,
|
||
DRAINING, NETWORK_DEGRADED, GPU_UNAVAILABLE, LOW_DISK, THERMAL_THROTTLED, OFFLINE,
|
||
PAUSED. Status preserves Linux field names (`status`, `disk_free_bytes`,
|
||
`registration_state`, `last_job_duration_seconds`, `last_job_realtime_factor`,
|
||
`util`, etc.) and adds `state`, NVENC capability flags, and fleet aliases. QSV
|
||
flags remain false; a successful NVENC test is never reported as QSV success.
|
||
The supervisor requires a fresh admission document (<30 seconds), an explicit
|
||
unpaused control document, and no GPU-refusal marker. Missing/stale admission
|
||
always triggers the official runner graceful-shutdown request.
|
||
|
||
AC policy uses `GetSystemPowerStatus`: battery or unknown AC drains. Thermal
|
||
zones are read through `root/wmi:MSAcpi_ThermalZoneTemperature` when supported;
|
||
unavailable CPU sensors remain null. NVIDIA temperature remains a real reading.
|
||
Defaults: 20 GiB disk reserve, 85 C thermal ceiling, 120 s thermal cooldown,
|
||
10 Mbps each direction, 1.25x bandwidth recovery margin, three good/two bad
|
||
samples, 900 s bandwidth expiry, no DERP admission. The existing Tailscale-only
|
||
server probe (`TAILSCALE_IP:9044`) uses bounded 8 MiB GET download and PUT upload
|
||
checks, EWMA, and runs while the runner is drained. GPU failure events remain
|
||
latched until a later real self-test. Boot/driver changes trigger new proof.
|
||
`powercfg /change standby-timeout-ac 0` is an elevated install step. The laptop
|
||
must remain on AC; lid-close/Modern Standby behavior still needs owner review.
|
||
|
||
## Adapter contract and tests
|
||
|
||
The pinned runner requests `libx264`. The executable accepts only reviewed
|
||
command shapes and rewrites them before executing FFmpeg: CUDA/NVDEC hardware
|
||
frames -> `scale_cuda` -> `h264_nvenc`, preset p4, preserving bitrate cap/GOP.
|
||
The original libx264 command is never executed. H.264, SDR HEVC, and VP9 sources
|
||
have reviewed hardware-decode routes; rotated, HDR, unsupported pixel formats,
|
||
unknown options, stream codec overrides, extra outputs, and unknown codecs fail
|
||
closed with exit 78. Audio-only and explicit stream-copy/remux jobs are permitted
|
||
without video encoding. Static audio-merge images use software image processing
|
||
and `hwupload_cuda`; video encoding remains NVENC. AV1 input is refused until a
|
||
reviewed decode route is tested; AV1 capability is not inferred from FFmpeg's
|
||
encoder list. A local diagnostic can set `PT_NVENC_TARGET=hevc_nvenc`; any other target is refused. Never change the PeerTube service target without a reviewed codec contract. HEVC encoding is proven by the gate but the PeerTube default output
|
||
remains H.264, matching the pinned runner/server contract.
|
||
|
||
```bash
|
||
python3 -m unittest discover -s infra/peertube-workers/win/tests -p test_nvenc.py -v
|
||
```
|
||
|
||
A running encode that fails, or lasts >=3 seconds without a readable positive
|
||
encoder signal, returns failure and latches GPU_UNAVAILABLE. Very short encodes
|
||
may finish within NVIDIA's utilization averaging window; their explicit NVENC
|
||
encoder and current boot/driver proof remain required. For a real E2E test record
|
||
positive encode telemetry, job UUID, server Pending/Processing/Completed states,
|
||
input/output codec and resolution, wall time, RTF, and playable result. Local
|
||
synthetic success is not a PeerTube E2E result.
|
||
|
||
Failure checks: drain -> runner gracefully exits; start/restart -> reconnect
|
||
milestone; create `state\gpu-unavailable` -> admission GPU_UNAVAILABLE and shim
|
||
exit 78; remove marker, drain, rerun self-test, then start. Do not unload the
|
||
owner's desktop GPU driver to simulate failure. Test reboot recovery and AC loss
|
||
only after services and the owner's physical AC/lid policy can be exercised.
|
||
|
||
## Deployment-specific elevation blocker / worker 2
|
||
|
||
The verified route is SSH into WSL using `~/.ssh/devserver`, then native
|
||
PowerShell/Node through `/mnt/c`. Interop works. The Windows token is medium
|
||
integrity with administrators marked deny-only. `sc create` fails with
|
||
OpenSCManager error 5; registering a Highest scheduled task returns access denied.
|
||
Enabling alternate-port Windows sshd needs the same unavailable elevation and
|
||
would not remove this blocker. No existing recovery task, UAC policy, driver, or
|
||
server setting was changed to bypass it. The owner must run the prepared Services
|
||
phase in an elevated Windows PowerShell. An interactive native process deployment
|
||
can be validated over the working route but is not a boot-persistent service.
|
||
|
||
Worker 2 changes only name, root (if wanted), GPU device index, and local network
|
||
access. Reuse this server probe and existing authorized token; give each worker a
|
||
unique runner name and isolated profile. Do not copy worker 1's credential TOML.
|
||
Require a fresh real hardware gate on that GPU and service context. Start with
|
||
concurrency 2; benchmark equivalent real workloads at 1–4 before increasing it.
|
||
The RTX 3090's stock session ceiling is not a two-job constraint; no driver patch
|
||
is deployed. Its measured limit is 350 W; an optional 200–250 W cap needs driver
|
||
support and throughput/thermal measurement before adopting it. No power limit or
|
||
driver was changed during deployment. Hardware throughput/session/codec claims
|
||
from research remain estimates until measured on the specific worker.
|
||
|
||
## Authorized E2E fixture
|
||
|
||
`win/e2e-peertube.mjs` is an explicit, privileged server test, never an installer
|
||
autostart step. Run it only for an authorized private upload, with other workers
|
||
drained if a specific worker must own the job. Supply a synthetic H.264 file as
|
||
`/tmp/pt-worker-e2e-source.mp4` inside the PeerTube container, then run the script
|
||
there through Node stdin with `NODE_DB_LOG=false` and `LOGGER_LEVEL=error`. It
|
||
requires the inspected PeerTube 7.3.0 model layout. It registers the installed
|
||
models without the initializer's SQL function/extension mutations, creates one
|
||
10-minute admin OAuth session through the installed token model, performs one
|
||
private upload through the HTTP API, records safe state metadata, checks private
|
||
HLS/media HTTP responses and full decoding, then revokes/removes that session.
|
||
It does not change the root password or runner registration token. Credentials
|
||
remain in process memory and the temporary normal OAuth database row; they never
|
||
appear in command arguments, report JSON, or source. Upstream ORM logging is
|
||
disabled before the token is created. The private test video remains available
|
||
for owner review; delete disposable server/container scratch files afterward.
|
||
|
||
First upload with the target worker drained to capture Pending, then start the
|
||
worker while the helper polls. Report JSON goes to `/tmp/pt-worker-e2e-result.json`
|
||
inside the container; it contains no credentials. This host's successful private
|
||
fixture is [Windows NVENC worker E2E](https://tube.occupymedia.cc/w/7d3EHVeikq3JZ29XqjKJyd),
|
||
video UUID `324566b4-3506-417d-9cca-cc2a3b7a14cc`. Jobs 1/2 completed on runner 1.
|
||
The cold WebVideo encode took 36.359 s with positive 100% NVIDIA encoder samples;
|
||
its warm HLS encode took 2.594 s. These are observed 5-second-fixture timings,
|
||
not a concurrency benchmark. The short HLS run sampled 0% in NVIDIA's averaging
|
||
window; its command was still explicitly NVENC, and the WebVideo job supplies
|
||
mandatory job-correlated GPU proof. Root OAuth sessions were verified back at 0.
|