Tactical Media Broadcast Hub — initial release
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
This commit is contained in:
commit
aa5e262f3a
880 changed files with 111042 additions and 0 deletions
6
infra/peertube-workers/.gitignore
vendored
Normal file
6
infra/peertube-workers/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
__pycache__/
|
||||
*.pyc
|
||||
runtime/
|
||||
state/
|
||||
*.toml
|
||||
*.env
|
||||
89
infra/peertube-workers/README.md
Normal file
89
infra/peertube-workers/README.md
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
This directory prepares Intel QSV workers for the inspected PeerTube **7.3.0** instance. It uses the published **@peertube/peertube-runner 0.2.0**, pinned with npm integrity hashes. PeerTube owns the queue; the controller admits or drains a whole runner. No PeerTube fork, scheduler, public worker listener, live delegation, or viewer-serving component is introduced.
|
||||
|
||||
|
||||
**Recommended next step:** use native Linux with a working Intel DRM render device, then rerun the real QSV self-test before enrolling. A Windows-native alternative requires a separate adapter and supervisor: [FFmpeg documents](https://www.ffmpeg.org/ffmpeg-all.html#MediaFoundation) `h264_mf -hw_encoding 1` to force Media Foundation hardware encoding, with a verified D3D11 decode/scale path and Windows power/GPU telemetry. The existing Linux QSV wrapper and systemd service cannot be reused unchanged. No Windows hardware encode has been tested. [WSL's Mesa D3D12 VAAPI path](https://devblogs.microsoft.com/commandline/d3d12-gpu-video-acceleration-in-the-windows-subsystem-for-linux-now-available/) is a different implementation from this libmfx/Intel DRM QSV path and was not deployed. Windows also reports an RTX 3090, but that is not a validated NVIDIA worker.
|
||||
|
||||
The actual 7.3.0 VOD key is `transcoding.remote_runners.enabled` (not `vod_enabled`). The override is `/config/production.yaml` in the effective Docker config volume. Only PeerTube was restarted; remote live/studio/transcription remain disabled. The actual deployed database/table are `peertube` / `runnerRegistrationToken`. The existing registration token was checked privately using the installed validator and an exact database lookup; no new token or runner was created. A successful registration handshake and real VOD E2E remain untested because the hardware gate failed.
|
||||
|
||||
The installed runner source was compared to the v7.3.0 tag: `apps/peertube-runner/src/server/process/shared/common.ts` has SHA256 `d28fe0f4db9ca46edaeac2f12c269f0fad52fd5cf92d9ba90d529c4a1d7c3627` in both. The [7.3.0 CLI docs](https://github.com/Chocobozzz/PeerTube/blob/v7.3.0/support/doc/tools.md#peertube-runner) require Node >=20 and FFmpeg/ffprobe >=4.3. This installer uses isolated Node 22 and refuses other PeerTube releases until their source is checked. The current latest Node 22 tarball is downloaded over HTTPS, verified against Node's SHA256 manifest, and retained on reruns. The runner version stays pinned.
|
||||
|
||||
| Runner behavior verified in 7.3.0 source | Consequence |
|
||||
| --- | --- |
|
||||
| `register` uses a local Unix IPC socket; the server must already be running | Enrollment starts a VOD-only server with concurrency **0**, then uses the official IPC library |
|
||||
| Socket.IO `/runners`, websocket-only transport, `available-jobs` notifications | Workers initiate API and WS connections; no inbound listener is needed |
|
||||
| Requests up to ten pending jobs; accepts the first supported job; checks again on connect, notifications, completion, and after 2.5 seconds when there is spare capacity | Queue selection, acceptance, and concurrency remain with the official runner |
|
||||
| Request payload includes input URLs and requested resolution/fps, but **no input size or duration** | Whole-worker admission; no per-job byte prediction |
|
||||
| Downloads use POST carrying runner/job tokens, then transcode and POST multipart results | Workers have only their dedicated runner credential after enrollment |
|
||||
| VOD handlers remove inputs/results in `finally`; startup removes leftover transcode files | Output cleanup is upstream behavior; smoke test verifies it |
|
||||
| Progress update every 60 seconds; VOD stale threshold 2 minutes; `MAX_FAILURES=5` | Server requeues eligible failures until its limit; controller does not implement retries |
|
||||
| SIGTERM/SIGINT abort active jobs; `graceful-shutdown` stops acceptance and waits for completion | All ordinary controller drains use the official graceful command |
|
||||
| Download timeout is 2 hours; there is no independent periodic idle polling timer | A network drain can remain pending while an upstream transfer times out; reconnection triggers a queue check |
|
||||
|
||||
Remote profile plugins are not an integration option in this release: `getCommonFFmpegOptions()` sets `profile: 'default'` and the default video encoder list contains only `libx264`. Installing a server profile plugin would not reach this runner. No maintained remote QSV plugin was established during the investigation. The implementation therefore uses preference **(4), a minimal worker wrapper**, rather than patching the runner or PeerTube.
|
||||
|
||||
`FFMPEG_PATH=ffmpeg-qsv.py` redirects commands generated by the runner. For video it replaces the inspected default x264 options with explicit QSV decode, QSV hardware frames, `scale_qsv`, and `h264_qsv`. It keeps PeerTube's muxing, HLS segmenting, fps, GOP, and rate caps. H.264/HEVC/VP9/AV1 source paths use explicit QSV decoders. Unsupported pixel formats, source codecs, rotation, unknown filters, and unknown encoder commands fail closed. Static cover images for audio-merge jobs require image decode and one upload of frames; their scaling and video encoding use QSV. Audio-only renditions can use the upstream AAC encoder. No video CPU fallback exists. Capability queries are passed through to FFmpeg.
|
||||
|
||||
The self-test runs real H.264 decode -> GPU scale -> H.264/HEVC encode, HEVC decode when encoding succeeded, and separate AV1 decode/encode attempts. It records cumulative Intel DRM media-engine counters from FFmpeg's `/proc/PID/fdinfo`, output ffprobe results, temperatures, and memory. A successful exit alone is insufficient: H.264 must also have positive media-engine counters. Missing counters or permissions block enrollment. AV1 encode is expected to fail on an i5-1240P; this expectation does not substitute for inspecting the actual CPU/GPU. See [Intel's driver features](https://github.com/intel/media-driver#decodingencoding-features) and [FFmpeg QSV filters](https://www.ffmpeg.org/ffmpeg-filters.html#QSV-Video-Filters). HEVC is a self-test/investigation output only; production remains H.264.
|
||||
|
||||
`observe.cjs` measures real response/request byte counts and elapsed transfer times without recording bodies, headers, credentials, or URLs. It correlates downloaded temporary files and encode events with job UUIDs, including concurrent jobs. Input payloads use the public canonical origin; just that origin is mapped to the configured Tailscale API address. Other storage origins retain HTTPS and normal redirects. The route preserves API paths and authentication. Interrupted response streams explicitly destroy the request so upstream download promises reject instead of waiting indefinitely. HTTP on `TAILSCALE_IP:9000` travels inside encrypted Tailscale; public viewer URLs remain untouched. Safe runner milestones are retained; arbitrary upstream logs are suppressed because they can contain tokens.
|
||||
|
||||
`pt-worker.py` exposes JSON status and optional Prometheus text. State is updated roughly every five seconds plus probe/IPC/API time. Admission states are `AVAILABLE`, `BUSY`, `DRAINING`, `NETWORK_DEGRADED`, `GPU_UNAVAILABLE`, `LOW_DISK`, `THERMAL_THROTTLED`, `OFFLINE`, and `PAUSED`. Degraded states request a graceful drain. Already accepted work finishes. Detection and graceful IPC have a bounded polling race; a hardware failure nevertheless cannot select a CPU encoder.
|
||||
|
||||
Real transfer observations use EWMA alpha 0.3. Defaults: 10 Mbps download/upload floor, recovery at 1.25 times those floors, 3 good samples to reopen, 2 bad samples to degrade, 150 ms maximum latency, 60 seconds after a transfer failure, 15-minute measurement expiry, DERP disallowed, 20 GiB free disk, and an 85 C thermal ceiling with 120-second cooldown. DIRECT/DERP transitions and suspend gaps invalidate throughput history. Floors are initial policy values, **not measured laptop capacity**. Tune them after real jobs.
|
||||
|
||||
The optional server probe prevents stale bandwidth from permanently blocking an idle or rebooted worker. It is already installed on `TAILSCALE_IP:9044`, binds only that Tailscale IP, discards uploads, caps each transfer at 16 MiB, permits two concurrent transfers, applies per-client/per-direction rate limits, and runs without media, Docker, DB, Redis, or filesystem write access. Ordinary checks transfer 8 MiB each way after an idle runner drains, no more often than every five minutes. Real job observations remain the preferred measurements. The probe measures the worker-to-PeerTube path, not a future external object-storage path; object-storage transfers need separate successful job observations.
|
||||
|
||||
On AC loss or unknown laptop power state, admission drains. Native machines with no battery are treated as permanently powered. Battery capacity/status, thermal sensors, CPU throttle counters, memory, load, disk, and kernel type are included when visible; unsupported measurements are null. There is no forced suspend, reboot, perf-policy change, or firmware change. The controller reruns its hardware test after a changed boot ID. Manual pause survives reboot. With the probe and healthy hardware it can recover network admission unattended.
|
||||
|
||||
**Shortest additional-worker procedure, once a hardware worker passes validation:** join Tailscale; provide a native Debian 12 / Ubuntu 24.04+ Linux environment with Intel `/dev/dri/renderD128`, systemd, and adequate disk; copy this directory to the worker; use an owner-authorized registration token (the existing token is authorized for this deployment); then run:
|
||||
|
||||
```bash
|
||||
sudo ./install-peertube-worker.sh --name WORKER2 --peertube-url http://TAILSCALE_IP:9000
|
||||
# The registration token is read at a hidden prompt, never as a CLI argument.
|
||||
sudo pt-worker start
|
||||
sudo pt-worker status
|
||||
```
|
||||
|
||||
The installer derives this instance's canonical origin from a fallback asset in `/api/v1/config`. For other branding or instances supply `--canonical-origin https://INSTANCE` and `--tailscale-peer TAILSCALE_IP`; deploy the probe at that IP or adjust `probe_url`. For noninteractive enrollment use a private file/pipe FD, e.g. `sudo -i`, then `exec 3</root/runner-registration-token` followed by the installer with `--registration-token-fd 3`; revoke/delete the temporary enrollment secret after confirming registration. Do not put tokens in argv, shell history, environment files, git, or reports.
|
||||
|
||||
The installer inspects hardware, systemd, Tailscale, and server release before installing packages. It chooses Intel's full-feature media driver when available in existing package sources, otherwise the free driver and relies on actual tests. It does not alter repositories or kernel settings. Dependencies include FFmpeg, VAAPI tools, Intel GPU tools, Python >=3.11, and Node >=22. Service user `pt-worker` receives the actual render-device group and existing render/video groups; no `chmod 777`. Code and settings are root-owned; registration TOML and runtime state are worker-owned and private. The worker service is enabled at boot and initially paused. Uncertain enrollment outcomes leave a marker and refuse automatic re-registration, preventing duplicate runners; inspect/revoke the named server runner before resolving that marker. Normal reruns preserve credentials, settings, and concurrency and back up existing installation/configuration before replacement.
|
||||
|
||||
| Command (run with sudo) | Behavior |
|
||||
| --- | --- |
|
||||
| `pt-worker status` | JSON; unknown fields are null, stale controller state is OFFLINE |
|
||||
| `pt-worker status --prometheus` | Numeric metrics without inventing absent measurements |
|
||||
| `pt-worker start` | Starts the supervisor and opens admission when checks pass |
|
||||
| `pt-worker start --calibrate` | Explicit ten-minute initial real-job measurement window if the probe is unavailable; still enforces GPU, AC, disk, latency, and direct path |
|
||||
| `pt-worker stop` / `drain` | Persists PAUSED; gracefully finishes accepted work; supervisor stays alive for status |
|
||||
| `pt-worker restart` | Graceful child restart; preserves the runner registration |
|
||||
| `pt-worker self-test` | Refuses while a runner is running; renews hardware proof after drain |
|
||||
| `pt-worker network-test` | Real bounded probe download/upload, or explicit disposable `--download-url` / `--upload-url` endpoints |
|
||||
| `pt-worker benchmark --source /path/source.mp4` | After drain, local QSV 720p diagnostic batches at 1/2/3/4, FPS/RTF/wall time/RAM/temperature/CPU/media utilization/throttle observations; does not choose production concurrency |
|
||||
|
||||
Native systemd uses `Restart=always`, 30-second restart delay, start throttling, an unprivileged account, read-only system/home protection, and no capabilities. `KillMode=mixed` gives the supervisor the initial stop signal so it can drain the official runner. Stop timeout is infinite to preserve accepted work; forced operator kills/crashes rely on upstream abort/watchdog behavior. GPU and memory metrics are observational and never justify a CPU fallback.
|
||||
|
||||
WSL must be inspected before using this installer. The visible Windows Tailscale peer does not identify the SSH shell or WSL kernel. Check Windows `wsl.exe --list --verbose`, then inside the intended distro check `uname -a`, `/proc/1/comm`, `/etc/wsl.conf`, `/dev/dri`, `/dev/dxg`, `lscpu`, `lspci`, `vainfo`, FFmpeg, and driver capabilities. [Microsoft documents enabling systemd](https://learn.microsoft.com/en-us/windows/wsl/systemd) with `[boot] systemd=true`; applying it requires a distro restart. That alone does not prove Intel Linux DRM/QSV support. `/dev/dxg` alone cannot satisfy this adapter. If the laptop exposes only the Windows GPU path, use native Linux or separately implement and validate a Windows QSV/oneVPL/D3D adapter and Windows service/power/telemetry. None of these host changes have been made.
|
||||
|
||||
**Production validation still required, after resolving the hardware blocker:** VOD remote delegation is already enabled; preserve existing resolutions and viewer configuration and keep live/studio/transcription remote flags disabled. Install/register the first proven hardware worker with the authorized existing token, and open admission at concurrency 2. Upload a disposable private synthetic A/V clip through an existing admin/user channel. Capture the real runner job UUIDs and Pending -> Processing -> Completing -> Completed, source/output probes, QSV engine counters, encode and accept-to-success wall times, actual download/upload observations, imported HLS/renditions, playback, A/V sync, and temporary cleanup. Repeat equivalent batches at concurrency 1/2/3/4 after draining and changing **both** worker.json and runner TOML; choose by completed media seconds per end-to-end wall time, with thermal and network headroom. The local benchmark cannot make that decision.
|
||||
|
||||
Failure validation on the worker, after the normal VOD test: drain/restart and verify no new registration; interrupt only the worker's Tailscale path under a recoverable admin connection and verify drain/recovery/server retries; configure an inaccessible test render path and ensure no CPU output; constrain a disposable probe path and verify low-bandwidth hysteresis; disconnect AC and observe drain; finally reboot the worker and verify service/Tailscale/hardware proof/network admission/registration recovery. Do not interrupt or reboot the production PeerTube host for these tests.
|
||||
|
||||
Suggested Tailscale policy: put worker devices in `tag:pt-worker`, server in `tag:pt-server`; allow worker -> server TCP 9000 and 9044 only for this deployment; authorize admin -> worker SSH separately; deny worker DB/Redis access and inbound worker connections except explicitly authorized administration. ACLs are centrally managed and were not changed here. Retain HTTPS egress to approved storage origins if object storage is later enabled. The server's existing TCP 1935/public/tunnel behavior was not changed. Live delegation later requires `live-rtmp-hls-transcoding`, worker-initiated access to the job's PeerTube RTMP input URL, sustained encode capacity and continuous HLS upload with measured upload headroom, low stable latency, a stricter direct-path policy, and separate failure/load validation. Workers do not serve viewers.
|
||||
|
||||
NVIDIA readiness: admission consumes generic `gpu_ready`, `gpu_tested_at`, GPU device accessibility, and `gpu_failure` events; network, queue, AC, disk, thermal policy, and graceful IPC are independent of Intel. A future provider can supply `gpu_health_file` with `hardware_video_ok` and `tested_at`, and `gpu_device`. NVIDIA still needs a separately tested CUDA/NVDEC -> CUDA scale -> NVENC adapter, driver/container permissions, an NVENC hardware self-test, NVIDIA utilization/temperature collection, codec/session-limit validation, and an installer package set. The existing Intel installer and fdinfo collector do not configure NVIDIA.
|
||||
|
||||
Tests are executable without a GPU or production token:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s tests -v
|
||||
node tests/test_observer.cjs
|
||||
# Install the pinned runtime into a temporary directory and use Node 22:
|
||||
python3 tests/runner_smoke.py --node /path/node --runtime /path/runtime
|
||||
systemd-analyze verify systemd/*.service
|
||||
```
|
||||
|
||||
The smoke test uses a loopback API and the published official runner. It proves enrollment at concurrency 0, secure IPC, persistence/restart, source routing, refusal/reporting without a GPU, graceful shutdown, and cleanup. It does **not** prove a production job, QSV acceleration, import/playback, laptop service/reboot behavior, or real laptop throughput. Keep private runtime evidence outside git. The worker jobs_completed counter counts acknowledged result submissions; server-side import success must still be verified centrally with the admin job API. Safe event logs currently need an operator retention policy; they are not automatically rotated. `@peertube/net-ipc@2.2.0` is deprecated on npm but matches the installed runner's IPC protocol; review this helper together with any runner upgrade.
|
||||
|
||||
## Windows NVENC
|
||||
|
||||
The Windows-native RTX 3090 worker, executable adapter, installer phases, admission policy, service elevation requirements, and onboarding steps are documented in [WINDOWS.md](WINDOWS.md). Use that runbook for Windows workers; the Linux QSV installer is unchanged.
|
||||
190
infra/peertube-workers/WINDOWS.md
Normal file
190
infra/peertube-workers/WINDOWS.md
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
# Windows native NVENC worker
|
||||
|
||||
Use Windows 10/11 x64 with an NVIDIA GPU and a working driver. WSL is the SSH
|
||||
transport only: Node, FFmpeg, the shim, admission, and IPC all run as Windows
|
||||
processes. The initial worker name is `worker-laptop-win`; root is `C:\pt-worker`.
|
||||
The canonical instance is `https://tube.occupymedia.cc`; worker traffic uses
|
||||
`http://TAILSCALE_IP:9000` inside encrypted Tailscale. Viewers use PeerTube.
|
||||
No server configuration change is needed: remote VOD is already enabled.
|
||||
|
||||
## Prepare and enroll
|
||||
|
||||
Run the installer from this directory. Python 3.12+ and Windows Tailscale are
|
||||
build/access prerequisites. Python is embedded in the frozen executables, so
|
||||
services do not depend on the developer's Python installation. Pins are in
|
||||
[win/dependencies.json](win/dependencies.json); the FFmpeg URL is a dated archive,
|
||||
verified by SHA256. The two executables are built from reviewed Python source
|
||||
using the versions in `win/build-requirements.txt`. Runner and IPC dependencies
|
||||
use the existing runtime lockfile. Do not substitute a `.cmd` shim: Node's
|
||||
shell-free spawn rejects it with EINVAL.
|
||||
|
||||
```powershell
|
||||
# Elevated PowerShell: installs Node MSI silently plus isolated Node 22.
|
||||
.\install-peertube-worker-windows.ps1 -Phase Prepare -Name WORKERNAME-win
|
||||
# A filtered SSH/WSL token can prepare everything except the MSI:
|
||||
.\install-peertube-worker-windows.ps1 -Phase Prepare -UserLevel -Name WORKERNAME-win
|
||||
.\install-peertube-worker-windows.ps1 -Phase SelfTest
|
||||
```
|
||||
|
||||
Both H.264 and HEVC must produce a real 5-second 1080p encode **and positive
|
||||
NVIDIA encoder utilization**. GPU identity, UUID, driver, boot, device index,
|
||||
exit codes, samples, and session statistics are recorded in `state\nvenc.json`.
|
||||
No proof means no enrollment. The installer keeps admission PAUSED. A failed
|
||||
init, absent telemetry signal, reboot, or driver mismatch closes admission.
|
||||
Session-0 proof runs again when services are installed; enrollment-session
|
||||
success alone does not prove service-context access.
|
||||
|
||||
Supply the **existing** registration token through stdin. Never pass it as a
|
||||
PowerShell/Node argument, environment variable, transcript, or source file.
|
||||
For SSH transport, pipe PostgreSQL's selected token output directly into the
|
||||
Windows Node `bin\ipc-nvenc.cjs register` process. The version-controlled
|
||||
`win/register-from-ssh.py --worker USER@HOST --name NAME` implements this using two subprocesses and pipe handles; it fetches
|
||||
`runnerRegistrationToken.registrationToken`, database `peertube`, using the
|
||||
PostgreSQL container's existing `POSTGRES_USER`. No token is printed or retained.
|
||||
Before doing so, run `bin\enroll-server.cjs`: it requires concurrency 0 and drops
|
||||
all upstream logs. After registration use `ipc-nvenc.cjs graceful-shutdown`, wait
|
||||
for the listener to exit, and set TOML concurrency to 2. Alternatively use:
|
||||
|
||||
```powershell
|
||||
# The calling process must give this process the secret on inherited stdin.
|
||||
.\install-peertube-worker-windows.ps1 -Phase Enroll -RegistrationTokenStdin
|
||||
```
|
||||
|
||||
The **registration secret** is never saved. The official runner necessarily
|
||||
saves its separate issued **runner credential** in
|
||||
`profile\Roaming\peertube-runner-nodejs\Config\default\config.toml`.
|
||||
NTFS ACLs restrict this tree to the owner, SYSTEM, administrators, and LocalService.
|
||||
Never print that TOML, arbitrary runner logs, database token columns, or HTTP
|
||||
headers. Enrollment pending markers prohibit blind duplicate registrations.
|
||||
|
||||
## Services and operations
|
||||
|
||||
```powershell
|
||||
# ServiceFiles prepares concrete WinSW XML/exe files without touching SCM.
|
||||
.\install-peertube-worker-windows.ps1 -Phase ServiceFiles
|
||||
# Elevated PowerShell; services initially PAUSED, session-0 proof first.
|
||||
.\install-peertube-worker-windows.ps1 -Phase Services
|
||||
C:\pt-worker\pt-worker-win.ps1 status
|
||||
C:\pt-worker\pt-worker-win.ps1 network-test
|
||||
C:\pt-worker\pt-worker-win.ps1 start
|
||||
C:\pt-worker\pt-worker-win.ps1 drain
|
||||
C:\pt-worker\pt-worker-win.ps1 restart
|
||||
C:\pt-worker\pt-worker-win.ps1 self-test
|
||||
```
|
||||
|
||||
WinSW 2.12.0 wraps `pt-runner` (Node supervisor) and `pt-admission` (frozen Python).
|
||||
Both use the [WinSW 2.12 service-account and stop configuration](https://github.com/winsw/winsw/blob/v2.12.0/doc/xmlConfigFile.md), running as LocalService with automatic delayed startup, a 30-second service
|
||||
restart delay, and limited rotated logs. The supervisor uses a further 15-second
|
||||
runner restart delay. It retains only safe milestones and observer events;
|
||||
unfiltered upstream logs are discarded because they can contain tokens.
|
||||
A service stop asks the supervisor to drain via the official named-pipe IPC,
|
||||
with an explicit one-hour SCM stop ceiling. For jobs that can exceed this ceiling,
|
||||
use `pt-worker-win drain`, wait for `runner_running=false`, then stop services.
|
||||
No admission loss forcibly kills an active encode.
|
||||
|
||||
State decisions use the Linux Meter/Admission semantics: AVAILABLE, BUSY,
|
||||
DRAINING, NETWORK_DEGRADED, GPU_UNAVAILABLE, LOW_DISK, THERMAL_THROTTLED, OFFLINE,
|
||||
PAUSED. Status preserves Linux field names (`status`, `disk_free_bytes`,
|
||||
`registration_state`, `last_job_duration_seconds`, `last_job_realtime_factor`,
|
||||
`util`, etc.) and adds `state`, NVENC capability flags, and fleet aliases. QSV
|
||||
flags remain false; a successful NVENC test is never reported as QSV success.
|
||||
The supervisor requires a fresh admission document (<30 seconds), an explicit
|
||||
unpaused control document, and no GPU-refusal marker. Missing/stale admission
|
||||
always triggers the official runner graceful-shutdown request.
|
||||
|
||||
AC policy uses `GetSystemPowerStatus`: battery or unknown AC drains. Thermal
|
||||
zones are read through `root/wmi:MSAcpi_ThermalZoneTemperature` when supported;
|
||||
unavailable CPU sensors remain null. NVIDIA temperature remains a real reading.
|
||||
Defaults: 20 GiB disk reserve, 85 C thermal ceiling, 120 s thermal cooldown,
|
||||
10 Mbps each direction, 1.25x bandwidth recovery margin, three good/two bad
|
||||
samples, 900 s bandwidth expiry, no DERP admission. The existing Tailscale-only
|
||||
server probe (`TAILSCALE_IP:9044`) uses bounded 8 MiB GET download and PUT upload
|
||||
checks, EWMA, and runs while the runner is drained. GPU failure events remain
|
||||
latched until a later real self-test. Boot/driver changes trigger new proof.
|
||||
`powercfg /change standby-timeout-ac 0` is an elevated install step. The laptop
|
||||
must remain on AC; lid-close/Modern Standby behavior still needs owner review.
|
||||
|
||||
## Adapter contract and tests
|
||||
|
||||
The pinned runner requests `libx264`. The executable accepts only reviewed
|
||||
command shapes and rewrites them before executing FFmpeg: CUDA/NVDEC hardware
|
||||
frames -> `scale_cuda` -> `h264_nvenc`, preset p4, preserving bitrate cap/GOP.
|
||||
The original libx264 command is never executed. H.264, SDR HEVC, and VP9 sources
|
||||
have reviewed hardware-decode routes; rotated, HDR, unsupported pixel formats,
|
||||
unknown options, stream codec overrides, extra outputs, and unknown codecs fail
|
||||
closed with exit 78. Audio-only and explicit stream-copy/remux jobs are permitted
|
||||
without video encoding. Static audio-merge images use software image processing
|
||||
and `hwupload_cuda`; video encoding remains NVENC. AV1 input is refused until a
|
||||
reviewed decode route is tested; AV1 capability is not inferred from FFmpeg's
|
||||
encoder list. A local diagnostic can set `PT_NVENC_TARGET=hevc_nvenc`; any other target is refused. Never change the PeerTube service target without a reviewed codec contract. HEVC encoding is proven by the gate but the PeerTube default output
|
||||
remains H.264, matching the pinned runner/server contract.
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s infra/peertube-workers/win/tests -p test_nvenc.py -v
|
||||
```
|
||||
|
||||
A running encode that fails, or lasts >=3 seconds without a readable positive
|
||||
encoder signal, returns failure and latches GPU_UNAVAILABLE. Very short encodes
|
||||
may finish within NVIDIA's utilization averaging window; their explicit NVENC
|
||||
encoder and current boot/driver proof remain required. For a real E2E test record
|
||||
positive encode telemetry, job UUID, server Pending/Processing/Completed states,
|
||||
input/output codec and resolution, wall time, RTF, and playable result. Local
|
||||
synthetic success is not a PeerTube E2E result.
|
||||
|
||||
Failure checks: drain -> runner gracefully exits; start/restart -> reconnect
|
||||
milestone; create `state\gpu-unavailable` -> admission GPU_UNAVAILABLE and shim
|
||||
exit 78; remove marker, drain, rerun self-test, then start. Do not unload the
|
||||
owner's desktop GPU driver to simulate failure. Test reboot recovery and AC loss
|
||||
only after services and the owner's physical AC/lid policy can be exercised.
|
||||
|
||||
## Deployment-specific elevation blocker / worker 2
|
||||
|
||||
The verified route is SSH into WSL using `~/.ssh/devserver`, then native
|
||||
PowerShell/Node through `/mnt/c`. Interop works. The Windows token is medium
|
||||
integrity with administrators marked deny-only. `sc create` fails with
|
||||
OpenSCManager error 5; registering a Highest scheduled task returns access denied.
|
||||
Enabling alternate-port Windows sshd needs the same unavailable elevation and
|
||||
would not remove this blocker. No existing recovery task, UAC policy, driver, or
|
||||
server setting was changed to bypass it. The owner must run the prepared Services
|
||||
phase in an elevated Windows PowerShell. An interactive native process deployment
|
||||
can be validated over the working route but is not a boot-persistent service.
|
||||
|
||||
Worker 2 changes only name, root (if wanted), GPU device index, and local network
|
||||
access. Reuse this server probe and existing authorized token; give each worker a
|
||||
unique runner name and isolated profile. Do not copy worker 1's credential TOML.
|
||||
Require a fresh real hardware gate on that GPU and service context. Start with
|
||||
concurrency 2; benchmark equivalent real workloads at 1–4 before increasing it.
|
||||
The RTX 3090's stock session ceiling is not a two-job constraint; no driver patch
|
||||
is deployed. Its measured limit is 350 W; an optional 200–250 W cap needs driver
|
||||
support and throughput/thermal measurement before adopting it. No power limit or
|
||||
driver was changed during deployment. Hardware throughput/session/codec claims
|
||||
from research remain estimates until measured on the specific worker.
|
||||
|
||||
## Authorized E2E fixture
|
||||
|
||||
`win/e2e-peertube.mjs` is an explicit, privileged server test, never an installer
|
||||
autostart step. Run it only for an authorized private upload, with other workers
|
||||
drained if a specific worker must own the job. Supply a synthetic H.264 file as
|
||||
`/tmp/pt-worker-e2e-source.mp4` inside the PeerTube container, then run the script
|
||||
there through Node stdin with `NODE_DB_LOG=false` and `LOGGER_LEVEL=error`. It
|
||||
requires the inspected PeerTube 7.3.0 model layout. It registers the installed
|
||||
models without the initializer's SQL function/extension mutations, creates one
|
||||
10-minute admin OAuth session through the installed token model, performs one
|
||||
private upload through the HTTP API, records safe state metadata, checks private
|
||||
HLS/media HTTP responses and full decoding, then revokes/removes that session.
|
||||
It does not change the root password or runner registration token. Credentials
|
||||
remain in process memory and the temporary normal OAuth database row; they never
|
||||
appear in command arguments, report JSON, or source. Upstream ORM logging is
|
||||
disabled before the token is created. The private test video remains available
|
||||
for owner review; delete disposable server/container scratch files afterward.
|
||||
|
||||
First upload with the target worker drained to capture Pending, then start the
|
||||
worker while the helper polls. Report JSON goes to `/tmp/pt-worker-e2e-result.json`
|
||||
inside the container; it contains no credentials. This host's successful private
|
||||
fixture is [Windows NVENC worker E2E](https://tube.occupymedia.cc/w/7d3EHVeikq3JZ29XqjKJyd),
|
||||
video UUID `324566b4-3506-417d-9cca-cc2a3b7a14cc`. Jobs 1/2 completed on runner 1.
|
||||
The cold WebVideo encode took 36.359 s with positive 100% NVIDIA encoder samples;
|
||||
its warm HLS encode took 2.594 s. These are observed 5-second-fixture timings,
|
||||
not a concurrency benchmark. The short HLS run sampled 0% in NVIDIA's averaging
|
||||
window; its command was still explicitly NVENC, and the WebVideo job supplies
|
||||
mandatory job-correlated GPU proof. Root OAuth sessions were verified back at 0.
|
||||
135
infra/peertube-workers/bandwidth-probe.py
Normal file
135
infra/peertube-workers/bandwidth-probe.py
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
#!/usr/bin/python3
|
||||
"""Bounded disposable transfer endpoint, bound ONLY to the PeerTube Tailscale IP."""
|
||||
import argparse
|
||||
import http.server
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
|
||||
MAX_BYTES = 16 * 1024**2
|
||||
CHUNK = bytes(range(256)) * 4096
|
||||
SLOTS = threading.BoundedSemaphore(2)
|
||||
LOCK = threading.Lock()
|
||||
RECENT = {}
|
||||
|
||||
|
||||
class Probe(http.server.BaseHTTPRequestHandler):
|
||||
protocol_version = 'HTTP/1.1'
|
||||
|
||||
def setup(self):
|
||||
super().setup()
|
||||
self.connection.settimeout(30)
|
||||
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
def reply(self, code, data):
|
||||
blob = json.dumps(data).encode()
|
||||
self.send_response(code)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(blob)))
|
||||
self.send_header('Cache-Control', 'no-store')
|
||||
self.end_headers()
|
||||
self.wfile.write(blob)
|
||||
|
||||
def admission(self, direction):
|
||||
address = ipaddress.ip_address(self.client_address[0])
|
||||
if address not in ipaddress.ip_network('100.64.0.0/10') and not address.is_loopback:
|
||||
self.reply(403, {'error': 'Tailscale clients only'})
|
||||
return False
|
||||
with LOCK:
|
||||
now = time.monotonic()
|
||||
for key in list(RECENT):
|
||||
if now - RECENT[key] > 60:
|
||||
RECENT.pop(key)
|
||||
key = (str(address), direction)
|
||||
if now - RECENT.get(key, -60) < 10:
|
||||
self.reply(429, {'error': 'probe rate limit'})
|
||||
return False
|
||||
RECENT[key] = now
|
||||
if not SLOTS.acquire(blocking=False):
|
||||
self.reply(503, {'error': 'probe busy'})
|
||||
return False
|
||||
return True
|
||||
|
||||
def do_GET(self):
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
if parsed.path == '/health':
|
||||
return self.reply(200, {'ok': True, 'max_bytes': MAX_BYTES, 'storage': 'discarded'})
|
||||
if parsed.path != '/download':
|
||||
return self.reply(404, {'error': 'unknown endpoint'})
|
||||
try:
|
||||
count = int(urllib.parse.parse_qs(parsed.query).get('bytes', [8 * 1024**2])[0])
|
||||
if not 65536 <= count <= MAX_BYTES:
|
||||
raise ValueError()
|
||||
except ValueError:
|
||||
return self.reply(400, {'error': 'invalid size'})
|
||||
if not self.admission('download'):
|
||||
return
|
||||
try:
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Length', str(count))
|
||||
self.send_header('Content-Type', 'application/octet-stream')
|
||||
self.send_header('Cache-Control', 'no-store')
|
||||
self.end_headers()
|
||||
while count:
|
||||
block = CHUNK[:min(count, len(CHUNK))]
|
||||
self.wfile.write(block)
|
||||
count -= len(block)
|
||||
except (OSError, socket.timeout):
|
||||
self.close_connection = True
|
||||
finally:
|
||||
SLOTS.release()
|
||||
|
||||
def do_PUT(self):
|
||||
if self.path != '/upload':
|
||||
self.close_connection = True
|
||||
return self.reply(404, {'error': 'unknown endpoint'})
|
||||
try:
|
||||
count = int(self.headers.get('Content-Length', '0'))
|
||||
if not 65536 <= count <= MAX_BYTES:
|
||||
raise ValueError()
|
||||
except ValueError:
|
||||
self.close_connection = True
|
||||
return self.reply(413, {'error': 'invalid size'})
|
||||
if not self.admission('upload'):
|
||||
self.close_connection = True
|
||||
return
|
||||
received = 0
|
||||
try:
|
||||
while received < count:
|
||||
data = self.rfile.read(min(1024**2, count - received))
|
||||
if not data:
|
||||
raise OSError('short upload')
|
||||
received += len(data)
|
||||
self.reply(200, {'bytes_received': received})
|
||||
except (OSError, socket.timeout):
|
||||
self.close_connection = True
|
||||
finally:
|
||||
SLOTS.release()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--bind', required=True)
|
||||
parser.add_argument('--port', type=int, default=9044)
|
||||
args = parser.parse_args()
|
||||
address = ipaddress.ip_address(args.bind)
|
||||
if address not in ipaddress.ip_network('100.64.0.0/10') and not address.is_loopback:
|
||||
raise SystemExit('Refusing a public/LAN/wildcard bind: require a Tailscale IP')
|
||||
server = http.server.ThreadingHTTPServer((args.bind, args.port), Probe)
|
||||
notify = os.environ.get('NOTIFY_SOCKET')
|
||||
if notify:
|
||||
address = '\0' + notify[1:] if notify.startswith('@') else notify
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as channel:
|
||||
channel.connect(address)
|
||||
channel.sendall(b'READY=1')
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
182
infra/peertube-workers/ffmpeg-qsv.py
Executable file
182
infra/peertube-workers/ffmpeg-qsv.py
Executable file
|
|
@ -0,0 +1,182 @@
|
|||
#!/usr/bin/python3
|
||||
"""Pinned PeerTube runner 0.2.0 FFmpeg adapter. Never executes a CPU video encoder."""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REAL = '/usr/bin/ffmpeg'
|
||||
PROBE = '/usr/bin/ffprobe'
|
||||
DEVICE = os.environ.get('PT_RENDER_DEVICE', '/dev/dri/renderD128')
|
||||
STATE = Path(os.environ.get('PT_STATE_DIR', '/var/lib/pt-worker/state'))
|
||||
|
||||
|
||||
def event(kind, **fields):
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
data = (json.dumps({'time': time.time(), 'kind': kind, **fields}) + '\n').encode()
|
||||
fd = os.open(STATE / 'events.jsonl', os.O_CREAT | os.O_APPEND | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
os.write(fd, data)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def probe(path):
|
||||
p = subprocess.run([PROBE, '-v', 'error', '-show_streams', '-show_format',
|
||||
'-of', 'json', path], capture_output=True, text=True, timeout=30)
|
||||
if p.returncode:
|
||||
raise ValueError('input probe failed')
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def engines(pid):
|
||||
"""i915 DRM fdinfo: cumulative hardware engine time, deduplicated by client id."""
|
||||
result, clients = {}, set()
|
||||
for f in Path(f'/proc/{pid}/fdinfo').glob('*'):
|
||||
try:
|
||||
text = f.read_text()
|
||||
except OSError:
|
||||
continue
|
||||
cid = re.search(r'^drm-client-id:\s*(\d+)', text, re.M)
|
||||
if not cid or cid[1] in clients:
|
||||
continue
|
||||
clients.add(cid[1])
|
||||
for key, value in re.findall(r'^drm-engine-([\w-]+):\s*(\d+)\s+ns', text, re.M):
|
||||
result[key] = result.get(key, 0) + int(value)
|
||||
return result
|
||||
|
||||
|
||||
def rewrite(args, input_probe=None):
|
||||
# fluent-ffmpeg queries capabilities before building commands. These do no encode.
|
||||
if args in (['-version'], ['-formats'], ['-encoders'], ['-filters'], ['-codecs'], ['-hwaccels']):
|
||||
return args, None
|
||||
pairs = list(zip(args, args[1:]))
|
||||
video_options = [(k, v) for k, v in pairs if k in ('-vcodec', '-c:v', '-codec:v')]
|
||||
if len(video_options) > 1:
|
||||
raise ValueError('multiple video codec options refused')
|
||||
if '-vn' in args and not video_options:
|
||||
return args, None # audio-only VOD renditions have no video encode
|
||||
if not video_options or video_options[0][1] != 'libx264':
|
||||
raise ValueError('unknown runner command or video encoder refused')
|
||||
if '-filter_complex' in args or '-hwaccel' in args or '-init_hw_device' in args:
|
||||
raise ValueError('unreviewed filter/device override refused')
|
||||
inputs = [v for k, v in pairs if k == '-i']
|
||||
if not inputs:
|
||||
raise ValueError('missing video input')
|
||||
data = input_probe if input_probe is not None else probe(inputs[0])
|
||||
stream = next((s for s in data['streams'] if s['codec_type'] == 'video'), None)
|
||||
if stream is None:
|
||||
raise ValueError('missing video stream')
|
||||
codec = stream['codec_name']
|
||||
still = codec in ('png', 'mjpeg', 'webp') and '-loop' in args
|
||||
decoder = {'h264': 'h264_qsv', 'hevc': 'hevc_qsv', 'vp9': 'vp9_qsv', 'av1': 'av1_qsv'}.get(codec)
|
||||
if not decoder and not still:
|
||||
raise ValueError('source codec has no reviewed QSV decode path')
|
||||
if stream.get('pix_fmt') not in ('yuv420p', 'yuv420p10le', 'nv12', 'p010le') and not still:
|
||||
raise ValueError('unsupported pixel format; CPU decode fallback refused')
|
||||
if stream.get('tags', {}).get('rotate') or any(s.get('rotation', 0) for s in stream.get('side_data_list', [])):
|
||||
raise ValueError('rotated source requires reviewed GPU transpose path')
|
||||
width, height = stream['width'], stream['height']
|
||||
duration = float(data.get('format', {}).get('duration', 0) or 0)
|
||||
w, h = width // 2 * 2, height // 2 * 2
|
||||
output = ['-init_hw_device', f'vaapi=va:{DEVICE}', '-init_hw_device', 'qsv=qs@va',
|
||||
'-filter_hw_device', 'qs']
|
||||
i, first_input = 0, True
|
||||
while i < len(args):
|
||||
key = args[i]
|
||||
val = args[i + 1] if i + 1 < len(args) else None
|
||||
if key == '-i' and first_input:
|
||||
if not still:
|
||||
output += ['-hwaccel', 'qsv', '-hwaccel_output_format', 'qsv', '-c:v', decoder, '-noautorotate']
|
||||
output += [key, val]
|
||||
first_input = False
|
||||
i += 2
|
||||
elif key in ('-vcodec', '-c:v', '-codec:v'):
|
||||
output += [key, 'h264_qsv']
|
||||
i += 2
|
||||
elif key in ('-pix_fmt', '-preset', '-preset:v', '-b_strategy', '-bf', '-tune', '-crf'):
|
||||
i += 2
|
||||
elif key in ('-vf', '-filter:v'):
|
||||
if val == 'scale=trunc(iw/2)*2:trunc(ih/2)*2' and still:
|
||||
w, h = width // 2 * 2, height // 2 * 2
|
||||
else:
|
||||
match = re.fullmatch(r'scale=w=(-2|\d+):h=(-2|\d+)', val or '')
|
||||
if not match:
|
||||
raise ValueError('unreviewed software filter refused')
|
||||
w, h = map(int, match.groups())
|
||||
if w == -2 and h > 0:
|
||||
w = max(2, round(width * h / height / 2) * 2)
|
||||
elif h == -2 and w > 0:
|
||||
h = max(2, round(height * w / width / 2) * 2)
|
||||
if w <= 0 or h <= 0 or w % 2 or h % 2:
|
||||
raise ValueError('invalid output dimensions')
|
||||
prefix = 'format=nv12,hwupload=extra_hw_frames=64,' if still else ''
|
||||
output += [key, prefix + f'scale_qsv=w={w}:h={h}:format=nv12']
|
||||
i += 2
|
||||
else:
|
||||
output.append(key)
|
||||
i += 1
|
||||
if '-vf' not in args and '-filter:v' not in args:
|
||||
prefix = 'format=nv12,hwupload=extra_hw_frames=64,' if still else ''
|
||||
output[-1:-1] = ['-vf', prefix + f'scale_qsv=w={width // 2 * 2}:h={height // 2 * 2}:format=nv12']
|
||||
# Keep PeerTube's rate cap and GOP; QSV has no x264 CRF mode.
|
||||
rate = next((v for k, v in pairs if k in ('-maxrate:v', '-maxrate')), None)
|
||||
extra = ['-preset', 'medium', '-bf', '2']
|
||||
if rate and not any(k in ('-b:v', '-vb') for k, _ in pairs):
|
||||
extra += ['-b:v', rate]
|
||||
output[-1:-1] = extra
|
||||
job_uuid = None
|
||||
try:
|
||||
job_uuid = json.loads((STATE / 'inputs' / (Path(inputs[0]).name + '.json')).read_text())['uuid']
|
||||
except (OSError, ValueError, KeyError):
|
||||
pass
|
||||
return output, {'job_uuid': job_uuid, 'source_codec': codec, 'target_codec': 'h264', 'encoder': 'h264_qsv',
|
||||
'source_width': width, 'source_height': height, 'duration': duration,
|
||||
'target_width': w, 'target_height': h,
|
||||
'static_image_upload': still}
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
args, info = rewrite(sys.argv[1:])
|
||||
if info is None:
|
||||
os.execv(REAL, [REAL, *args])
|
||||
if not os.access(DEVICE, os.R_OK | os.W_OK):
|
||||
raise ValueError('render device unavailable')
|
||||
started = time.monotonic()
|
||||
p = subprocess.Popen([REAL, *args], stderr=subprocess.PIPE, text=True)
|
||||
progress = {}
|
||||
def stderr_progress():
|
||||
for line in p.stderr:
|
||||
sys.stderr.write(line)
|
||||
sys.stderr.flush()
|
||||
for key, value in re.findall(r'(frame|fps|speed)=\s*([\d.]+)', line):
|
||||
progress[key] = float(value)
|
||||
reader = threading.Thread(target=stderr_progress, daemon=True)
|
||||
reader.start()
|
||||
maximum = {}
|
||||
while p.poll() is None:
|
||||
for key, value in engines(p.pid).items():
|
||||
maximum[key] = max(maximum.get(key, 0), value)
|
||||
time.sleep(0.1)
|
||||
elapsed = time.monotonic() - started
|
||||
reader.join(timeout=2)
|
||||
event('encode', **info, seconds=elapsed, returncode=p.returncode, engine_ns=maximum,
|
||||
ffmpeg_fps=progress.get('fps'), frames=progress.get('frame'), ffmpeg_speed=progress.get('speed'),
|
||||
realtime_factor=info['duration'] / elapsed if info['duration'] else None)
|
||||
if p.returncode:
|
||||
event('gpu_failure', reason='qsv_transcode_failed')
|
||||
sys.exit(p.returncode)
|
||||
except (ValueError, OSError, subprocess.SubprocessError, KeyError) as e:
|
||||
# Messages are fixed strings; never expose source URLs or runner credentials.
|
||||
event('gpu_failure', reason=type(e).__name__)
|
||||
print('PT-WORKER: QSV adapter refused command; worker must drain', file=sys.stderr)
|
||||
sys.exit(78)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
161
infra/peertube-workers/install-peertube-worker-windows.ps1
Normal file
161
infra/peertube-workers/install-peertube-worker-windows.ps1
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
# Run from an elevated PowerShell for MSI/services. User-level preparation is supported.
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$Root='C:\pt-worker', [string]$Name="$env:COMPUTERNAME-win",
|
||||
[string]$PeerTubeUrl='http://TAILSCALE_IP:9000', [string]$CanonicalOrigin='https://tube.occupymedia.cc',
|
||||
[string]$ServerIp='TAILSCALE_IP', [int]$Device=0,
|
||||
[ValidateSet('Prepare','SelfTest','Enroll','ServiceFiles','Services','All')][string]$Phase='All',
|
||||
[switch]$RegistrationTokenStdin, [switch]$UserLevel
|
||||
)
|
||||
$ErrorActionPreference='Stop';$ProgressPreference='SilentlyContinue'
|
||||
$Source=$PSScriptRoot
|
||||
$Pins=Get-Content (Join-Path $Source 'win\dependencies.json') -Raw | ConvertFrom-Json
|
||||
$admin=([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
|
||||
if($Root -notmatch '^[A-Za-z]:\\' -or $Root -match '[\s"<>]'){throw 'Choose an absolute local path without spaces or XML characters'}
|
||||
$env:PT_ROOT=$Root
|
||||
$env:PT_STATE_DIR=Join-Path $Root 'state'
|
||||
$env:PT_WORKER_CONFIG=Join-Path $Root 'state\worker.json'
|
||||
$env:APPDATA=Join-Path $Root 'profile\Roaming'
|
||||
$env:LOCALAPPDATA=Join-Path $Root 'profile\Local'
|
||||
$env:TMP=Join-Path $Root 'tmp';$env:TEMP=$env:TMP
|
||||
$env:FFMPEG_PATH=Join-Path $Root 'bin\ffmpeg-nvenc.exe'
|
||||
$env:FFPROBE_PATH=Join-Path $Root 'bin\ffprobe.exe'
|
||||
$node=Join-Path $Root 'node\node.exe'
|
||||
$toml=Join-Path $env:APPDATA 'peertube-runner-nodejs\Config\default\config.toml'
|
||||
function CheckExit([string]$What){if($LASTEXITCODE -ne 0){throw "$What failed (exit $LASTEXITCODE)"}}
|
||||
function Fetch($Pin,[string]$File){
|
||||
$out=Join-Path $Root "downloads\$File"
|
||||
if(!(Test-Path $out)){& curl.exe --fail --location --silent --show-error --output $out $Pin.url;CheckExit 'Download'}
|
||||
if((Get-FileHash $out -Algorithm SHA256).Hash.ToLower() -ne $Pin.sha256){throw "SHA256 mismatch for $File; no installation performed"}
|
||||
return $out
|
||||
}
|
||||
function ProtectPaths {
|
||||
$me=[Security.Principal.WindowsIdentity]::GetCurrent().User.Value
|
||||
& icacls.exe $Root /inheritance:r /grant:r "*$($me):(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' '*S-1-5-19:(OI)(CI)RX' | Out-Null;CheckExit 'Root ACL'
|
||||
foreach($dir in @('state','profile','tmp','logs')){
|
||||
& icacls.exe (Join-Path $Root $dir) /inheritance:r /grant:r "*$($me):(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' '*S-1-5-19:(OI)(CI)F' | Out-Null;CheckExit 'Private ACL'
|
||||
}
|
||||
}
|
||||
if($Phase -in @('Prepare','All')){
|
||||
if(Test-Path (Join-Path $Root 'state\runner.json')){
|
||||
$running=Get-Content (Join-Path $Root 'state\runner.json') -Raw | ConvertFrom-Json
|
||||
if($running.running){throw 'Drain and wait for runner_running=false before preparing/upgrading'}
|
||||
}
|
||||
foreach($dir in @('bin','downloads','runner','state','profile','tmp','logs','build')){New-Item -ItemType Directory -Force (Join-Path $Root $dir) | Out-Null}
|
||||
'{"paused":true}' | Set-Content (Join-Path $Root 'state\control.json') -Encoding ASCII
|
||||
ProtectPaths
|
||||
$msi=Fetch $Pins.node_msi 'node.msi'
|
||||
if(!$UserLevel){
|
||||
if(!$admin){throw 'Node MSI requires an elevated Windows PowerShell. Use -UserLevel to prepare isolated binaries.'}
|
||||
$p=Start-Process msiexec.exe -Wait -PassThru -ArgumentList @('/i',$msi,'/qn','/norestart');if($p.ExitCode -notin @(0,3010)){throw "Node MSI exit $($p.ExitCode)"}
|
||||
}
|
||||
# The worker always uses its isolated pinned Node even after a machine-wide MSI.
|
||||
$zip=Fetch $Pins.node_zip 'node.zip'
|
||||
Expand-Archive $zip (Join-Path $Root 'downloads\node') -Force
|
||||
if(!(Test-Path (Join-Path $Root 'node'))){New-Item -ItemType Directory (Join-Path $Root 'node') | Out-Null}
|
||||
Copy-Item (Join-Path $Root "downloads\node\node-v$($Pins.node_version)-win-x64\*") (Join-Path $Root 'node') -Recurse -Force
|
||||
$zip=Fetch $Pins.ffmpeg 'ffmpeg-8.1-20261009.zip'
|
||||
Expand-Archive $zip (Join-Path $Root 'downloads\ffmpeg') -Force
|
||||
$ff=Get-ChildItem (Join-Path $Root 'downloads\ffmpeg') -Filter ffmpeg.exe -Recurse | Select-Object -First 1
|
||||
Copy-Item ($ff.Directory.FullName+'\*') (Join-Path $Root 'bin') -Force
|
||||
Copy-Item (Fetch $Pins.winsw 'winsw.exe') (Join-Path $Root 'bin\WinSW-x64.exe') -Force
|
||||
foreach($file in @('ffmpeg-nvenc.py','pt-worker-nvenc.py','ipc-nvenc.cjs','runner-supervisor.cjs','enroll-server.cjs','service-stop.cjs')){Copy-Item (Join-Path $Source "win\$file") (Join-Path $Root "bin\$file") -Force}
|
||||
Copy-Item (Join-Path $Source 'observe.cjs') (Join-Path $Root 'bin\observe.cjs') -Force
|
||||
Copy-Item (Join-Path $Source 'win\selftest-nvenc.ps1') (Join-Path $Root 'selftest-nvenc.ps1') -Force
|
||||
Copy-Item (Join-Path $Source 'pt-worker-win.ps1') (Join-Path $Root 'pt-worker-win.ps1') -Force
|
||||
Copy-Item (Join-Path $Source 'runtime-package.json') (Join-Path $Root 'runner\package.json') -Force
|
||||
Copy-Item (Join-Path $Source 'runtime-package-lock.json') (Join-Path $Root 'runner\package-lock.json') -Force
|
||||
$env:PATH=(Join-Path $Root 'node')+';'+$env:PATH
|
||||
Push-Location (Join-Path $Root 'runner');try {& (Join-Path $Root 'node\npm.cmd') ci --ignore-scripts;CheckExit 'Pinned runner install'}finally{Pop-Location}
|
||||
# Python is a build prerequisite only. Frozen executables embed their runtime.
|
||||
$python=(Get-Command python.exe -ErrorAction Stop).Source
|
||||
& $python -m venv (Join-Path $Root 'build-env');CheckExit 'Build environment'
|
||||
$builder=Join-Path $Root 'build-env\Scripts\python.exe'
|
||||
& $builder -m pip install --disable-pip-version-check -r (Join-Path $Source 'win\build-requirements.txt');CheckExit 'Pinned builder install'
|
||||
foreach($app in @(@{Name='ffmpeg-nvenc';Source='ffmpeg-nvenc.py'},@{Name='pt-admission';Source='pt-worker-nvenc.py'})){
|
||||
& $builder -m PyInstaller --noconfirm --clean --onefile --name $app.Name --distpath (Join-Path $Root 'bin') --workpath (Join-Path $Root "build\$($app.Name)") --specpath (Join-Path $Root 'build') (Join-Path $Root "bin\$($app.Source)");CheckExit 'Executable build'
|
||||
}
|
||||
if(!(Test-Path (Join-Path $Root 'state\worker.json'))){
|
||||
@{name=$Name;peertube_url=$PeerTubeUrl;canonical_origin=$CanonicalOrigin;tailscale_peer=$ServerIp;probe_url="http://${ServerIp}:9044";peertube_version='7.3.0';concurrency=2;nvenc_device=$Device;require_ac=$true;node_binary=$node;tailscale_binary='C:\Program Files\Tailscale\tailscale.exe';poll_seconds=5;probe_interval_seconds=300} | ConvertTo-Json | Set-Content (Join-Path $Root 'state\worker.json') -Encoding ASCII
|
||||
}
|
||||
if(!(Test-Path $toml)){New-Item -ItemType Directory -Force (Split-Path $toml) | Out-Null;"[jobs]`nconcurrency = 0`n[ffmpeg]`nthreads = 2`nnice = 20`n" | Set-Content $toml -Encoding ASCII}
|
||||
$manifest=@{pins=$Pins;built_at=[DateTime]::UtcNow.ToString('o');binaries=@()}
|
||||
foreach($file in @('ffmpeg.exe','ffprobe.exe','ffmpeg-nvenc.exe','pt-admission.exe','WinSW-x64.exe')){$manifest.binaries+=@{name=$file;sha256=(Get-FileHash (Join-Path $Root "bin\$file")).Hash.ToLower()}}
|
||||
$manifest | ConvertTo-Json -Depth 8 | Set-Content (Join-Path $Root 'state\dependencies.json') -Encoding ASCII
|
||||
}
|
||||
if($Phase -in @('SelfTest','All')){
|
||||
& (Join-Path $Root 'selftest-nvenc.ps1') -Root $Root -Device $Device;CheckExit 'Mandatory NVENC proof'
|
||||
& (Join-Path $Root 'bin\pt-admission.exe') network-test;CheckExit 'Windows bandwidth probe'
|
||||
}
|
||||
if($Phase -in @('Enroll','All')){
|
||||
$cfg=Get-Content (Join-Path $Root 'state\worker.json') -Raw | ConvertFrom-Json
|
||||
$proof=Get-Content (Join-Path $Root 'state\nvenc.json') -Raw | ConvertFrom-Json
|
||||
if(!$proof.nvenc_h264_ok -or !$proof.nvenc_hevc_ok){throw 'Registration refused: NVENC proof absent'}
|
||||
$boot=(Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToUniversalTime().ToString('o')
|
||||
$driver=(& nvidia-smi.exe -i $cfg.nvenc_device --query-gpu=driver_version --format=csv,noheader,nounits).Trim()
|
||||
if($proof.boot_id -ne $boot -or $proof.driver_version -ne $driver -or ([DateTimeOffset]::UtcNow.ToUnixTimeSeconds()-$proof.tested_at) -gt 86400){throw 'Registration refused: stale boot/driver proof'}
|
||||
$text=Get-Content $toml -Raw
|
||||
if($text -match 'runnerToken\s*='){
|
||||
if($text -notmatch [regex]::Escape($cfg.name)){throw 'Other runner exists in profile; inspect before enrollment'}
|
||||
Write-Output 'Already enrolled; registration preserved'
|
||||
}else{
|
||||
if(Test-Path (Join-Path $Root 'state\enrollment.pending')){throw 'Previous registration result uncertain; inspect server before retry'}
|
||||
if(!$RegistrationTokenStdin){throw 'Enrollment requires -RegistrationTokenStdin with the existing secret piped on stdin; never pass the token as an argument'}
|
||||
ProtectPaths
|
||||
& $node (Join-Path $Root 'bin\enroll-server.cjs');CheckExit 'Enrollment listener'
|
||||
Start-Sleep -Seconds 3
|
||||
'Attempted; no secret material' | Set-Content (Join-Path $Root 'state\enrollment.pending')
|
||||
# Node reads inherited stdin directly. No PowerShell variable/file/history holds the token.
|
||||
& $node (Join-Path $Root 'bin\ipc-nvenc.cjs') register;CheckExit 'Official registration IPC'
|
||||
& $node (Join-Path $Root 'bin\ipc-nvenc.cjs') graceful-shutdown;CheckExit 'Enrollment drain'
|
||||
Remove-Item (Join-Path $Root 'state\enrollment.pending')
|
||||
}
|
||||
# Keep official runner credential private, never print TOML or arbitrary logs.
|
||||
[IO.File]::WriteAllText($toml, ((Get-Content $toml -Raw) -replace '(?m)^concurrency\s*=\s*\d+', 'concurrency = 2'))
|
||||
}
|
||||
if($Phase -in @('ServiceFiles','Services','All')){
|
||||
foreach($svc in @('pt-runner','pt-admission')){if(Get-Service $svc -ErrorAction SilentlyContinue){throw 'Existing service requires an explicit drain/upgrade; refusing overwrite'}}
|
||||
New-Item -ItemType Directory -Force (Join-Path $Root 'services'),(Join-Path $Root 'logs') | Out-Null
|
||||
ProtectPaths
|
||||
foreach($svc in @('pt-runner','pt-admission')){
|
||||
$exe=if($svc -eq 'pt-runner'){$node}else{Join-Path $Root 'bin\pt-admission.exe'}
|
||||
$arguments=if($svc -eq 'pt-runner'){Join-Path $Root 'bin\runner-supervisor.cjs'}else{'daemon'}
|
||||
$stop=if($svc -eq 'pt-runner'){"<stopexecutable>$node</stopexecutable><stoparguments>$(Join-Path $Root 'bin\service-stop.cjs')</stoparguments>"}else{''}
|
||||
$xml=@"
|
||||
<service><id>$svc</id><name>$svc</name><description>PeerTube Windows NVENC worker: AC-only admission, VOD-only, no CPU video encoding</description>
|
||||
<executable>$exe</executable><startarguments>$arguments</startarguments><workingdirectory>$Root</workingdirectory>
|
||||
<env name="PT_ROOT" value="$Root"/><env name="PT_STATE_DIR" value="$(Join-Path $Root 'state')"/>
|
||||
<env name="APPDATA" value="$(Join-Path $Root 'profile\Roaming')"/><env name="LOCALAPPDATA" value="$(Join-Path $Root 'profile\Local')"/>
|
||||
<env name="TMP" value="$(Join-Path $Root 'tmp')"/><env name="TEMP" value="$(Join-Path $Root 'tmp')"/>
|
||||
<startmode>Automatic</startmode><delayedAutoStart>true</delayedAutoStart><onfailure action="restart" delay="30 sec"/><resetfailure>1 hour</resetfailure>
|
||||
<serviceaccount><domain>NT AUTHORITY</domain><user>LocalService</user></serviceaccount>
|
||||
<logpath>$(Join-Path $Root 'logs')</logpath><log mode="roll-by-size"><sizeThreshold>1024</sizeThreshold><keepFiles>3</keepFiles></log>
|
||||
<stoptimeout>1 hour</stoptimeout>$stop</service>
|
||||
"@
|
||||
Copy-Item (Join-Path $Root 'bin\WinSW-x64.exe') (Join-Path $Root "services\$svc.exe")
|
||||
$xml | Set-Content (Join-Path $Root "services\$svc.xml") -Encoding UTF8
|
||||
|
||||
}
|
||||
if($Phase -eq 'ServiceFiles'){Write-Output 'WinSW service files prepared; no SCM changes';return}
|
||||
if(!$admin){throw 'Service files prepared. Installation requires elevated Windows PowerShell (OpenSCManager error 5 in filtered SSH/WSL tokens).'}
|
||||
# Transition an interactive validation deployment without duplicate profile owners.
|
||||
'{"paused":true}' | Set-Content (Join-Path $Root 'state\control.json') -Encoding ASCII
|
||||
$deadline=(Get-Date).AddHours(1)
|
||||
while(Test-Path (Join-Path $Root 'state\runner.json')){
|
||||
$r=Get-Content (Join-Path $Root 'state\runner.json') -Raw | ConvertFrom-Json
|
||||
if(!$r.running){break}
|
||||
if((Get-Date) -gt $deadline){throw 'Active jobs still draining; service transition stopped'}
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
if(Test-Path (Join-Path $Root 'state\supervisor-process.json')){
|
||||
'requested' | Set-Content (Join-Path $Root 'state\supervisor-stop') -Encoding ASCII
|
||||
Start-Sleep -Seconds 3
|
||||
}
|
||||
Get-Process pt-admission -ErrorAction SilentlyContinue | Where-Object {$_.Path -eq (Join-Path $Root 'bin\pt-admission.exe')} | Stop-Process
|
||||
foreach($svc in @('pt-runner','pt-admission')){& (Join-Path $Root "services\$svc.exe") install;CheckExit 'Service installation'}
|
||||
& powercfg.exe /change standby-timeout-ac 0;CheckExit 'AC sleep policy'
|
||||
# Force proof from service/session 0 before opening admission there.
|
||||
Remove-Item (Join-Path $Root 'state\nvenc.json') -ErrorAction SilentlyContinue
|
||||
Start-Service pt-admission,pt-runner
|
||||
Write-Output 'Services installed and started PAUSED. Validate session-0 NVENC, then pt-worker-win start.'
|
||||
}
|
||||
196
infra/peertube-workers/install-peertube-worker.sh
Executable file
196
infra/peertube-workers/install-peertube-worker.sh
Executable file
|
|
@ -0,0 +1,196 @@
|
|||
#!/bin/bash
|
||||
# Native Debian 12 / Ubuntu 24.04+ Intel DRM worker. No secrets in argv/history.
|
||||
set -euo pipefail
|
||||
set +x
|
||||
umask 077
|
||||
task_source=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
|
||||
worker_name= peertube_url= token_fd= canonical_origin= tailscale_peer=TAILSCALE_IP
|
||||
while (($#)); do
|
||||
case "$1" in
|
||||
--name) worker_name=${2:?}; shift 2 ;;
|
||||
--peertube-url) peertube_url=${2:?}; shift 2 ;;
|
||||
--tailscale-peer) tailscale_peer=${2:?}; shift 2 ;;
|
||||
--canonical-origin) canonical_origin=${2:?}; shift 2 ;;
|
||||
--registration-token-fd) token_fd=${2:?}; shift 2 ;;
|
||||
*) echo 'Usage: install-peertube-worker.sh --name HOST --peertube-url URL [--tailscale-peer IP] [--registration-token-fd N]' >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[[ $EUID -eq 0 && $worker_name =~ ^[a-zA-Z0-9._-]+$ && $peertube_url =~ ^https?:// ]] || { echo 'Root, valid name and URL required' >&2; exit 2; }
|
||||
exec 9>/run/lock/pt-worker-install.lock
|
||||
flock -n 9 || { echo 'Another installation is running' >&2; exit 1; }
|
||||
[[ $(ps -p 1 -o comm=) == systemd ]] || { echo 'systemd is required; inspect WSL init before onboarding' >&2; exit 1; }
|
||||
[[ -c /dev/dri/renderD128 ]] || { echo 'No Intel DRM render device; /dev/dxg alone is not this Linux QSV path. No packages or services changed.' >&2; exit 1; }
|
||||
[[ $(cat /sys/class/drm/renderD128/device/vendor) == 0x8086 ]] || { echo 'This installer needs an Intel GPU; NVIDIA needs another hardware adapter' >&2; exit 1; }
|
||||
[[ $(basename "$(readlink -f /sys/class/drm/renderD128/device/driver)") == i915 || $(basename "$(readlink -f /sys/class/drm/renderD128/device/driver)") == xe ]] || { echo 'Intel DRM driver unavailable' >&2; exit 1; }
|
||||
command -v tailscale >/dev/null || { echo 'Tailscale must be joined already' >&2; exit 1; }
|
||||
tailscale ping --c=1 --timeout=5s "$tailscale_peer" >/dev/null
|
||||
canonical_origin=$(python3 - "$peertube_url" "$canonical_origin" <<'PY'
|
||||
import json, sys, urllib.request
|
||||
from urllib.parse import urlsplit
|
||||
with urllib.request.urlopen(sys.argv[1].rstrip('/')+'/api/v1/config', timeout=10) as r:
|
||||
c=json.load(r); v=c['serverVersion']
|
||||
if v != '7.3.0': raise SystemExit('This adapter is verified against PeerTube 7.3.0; inspect a different version before installing')
|
||||
if sys.version_info < (3,11): raise SystemExit('Python >=3.11 required')
|
||||
origin=sys.argv[2]
|
||||
if not origin:
|
||||
logos=[x['fileUrl'] for x in c['instance']['logo'] if x.get('isFallback')]
|
||||
if not logos: raise SystemExit('Supply --canonical-origin: no fallback asset URL identifies the instance origin')
|
||||
u=urlsplit(logos[0]);origin=u.scheme+'://'+u.netloc
|
||||
print(origin.rstrip('/'))
|
||||
PY
|
||||
)
|
||||
if [[ -f /etc/pt-worker/worker.json ]]; then
|
||||
python3 - "$worker_name" "$peertube_url" <<'PY'
|
||||
import json, sys
|
||||
c=json.load(open('/etc/pt-worker/worker.json'))
|
||||
if c['name'] != sys.argv[1] or c['peertube_url'].rstrip('/') != sys.argv[2].rstrip('/'):
|
||||
raise SystemExit('Existing name/URL differs: preserve registration and inspect before changing it')
|
||||
PY
|
||||
fi
|
||||
if systemctl is-active --quiet pt-worker.service; then
|
||||
/usr/local/bin/pt-worker drain
|
||||
for _ in $(seq 1 120); do
|
||||
if /usr/local/bin/pt-worker status | python3 -c 'import json,sys;sys.exit(bool(json.load(sys.stdin).get("runner_running")))'; then break; fi
|
||||
sleep 5
|
||||
done
|
||||
/usr/local/bin/pt-worker status | python3 -c 'import json,sys;sys.exit(bool(json.load(sys.stdin).get("runner_running")))' || { echo 'Active jobs still draining; installer stopped without overwriting code' >&2; exit 1; }
|
||||
systemctl stop pt-worker.service
|
||||
fi
|
||||
if [[ -d /opt/pt-worker ]]; then
|
||||
mkdir -p /var/backups/pt-worker
|
||||
task_backup=/var/backups/pt-worker/$(date -u +%Y%m%dT%H%M%SZ).tgz
|
||||
tar -czf "$task_backup" /opt/pt-worker /etc/pt-worker /var/lib/pt-worker/.config
|
||||
chmod 600 "$task_backup"
|
||||
fi
|
||||
. /etc/os-release
|
||||
case "$ID" in debian|ubuntu) ;; *) echo 'Only Debian/Ubuntu package sets verified' >&2; exit 1;; esac
|
||||
apt-get update
|
||||
# Intel media VA driver includes Alder Lake support; do not change kernel/perf policies.
|
||||
task_media_package=intel-media-va-driver
|
||||
if apt-cache show intel-media-va-driver-non-free >/dev/null 2>&1; then task_media_package=intel-media-va-driver-non-free; fi
|
||||
apt-get install -y ffmpeg vainfo "$task_media_package" intel-gpu-tools python3 curl ca-certificates xz-utils pciutils
|
||||
python3 - <<'PY'
|
||||
import re, subprocess
|
||||
s=subprocess.check_output(['/usr/bin/ffmpeg','-version'],text=True)
|
||||
m=re.search(r'ffmpeg version (\d+)\.(\d+)',s)
|
||||
if not m or tuple(map(int,m.groups())) < (4,3): raise SystemExit('FFmpeg >=4.3 required')
|
||||
subprocess.run(['/usr/bin/ffprobe','-version'],check=True,stdout=subprocess.DEVNULL)
|
||||
PY
|
||||
id pt-worker >/dev/null 2>&1 || useradd --system --create-home --home-dir /var/lib/pt-worker --shell /usr/sbin/nologin pt-worker
|
||||
for task_group in render video; do
|
||||
getent group "$task_group" >/dev/null && usermod -a -G "$task_group" pt-worker
|
||||
done
|
||||
task_render_group=$(stat -c %G /dev/dri/renderD128)
|
||||
[[ $task_render_group != root ]] || { echo 'Render device has root group; fix the host udev policy first' >&2; exit 1; }
|
||||
usermod -a -G "$task_render_group" pt-worker
|
||||
install -d -m 755 /opt/pt-worker
|
||||
install -d -m 750 -o root -g pt-worker /etc/pt-worker
|
||||
install -d -m 700 -o pt-worker -g pt-worker /var/lib/pt-worker/state
|
||||
for task_file in pt-worker.py ffmpeg-qsv.py observe.cjs ipc.cjs; do
|
||||
install -m 755 "$task_source/$task_file" "/opt/pt-worker/$task_file"
|
||||
done
|
||||
install -m 755 "$task_source/pt-worker" /usr/local/bin/pt-worker
|
||||
if [[ ! -x /opt/pt-worker/node/bin/node ]]; then
|
||||
task_tmp=$(mktemp -d)
|
||||
trap 'python3 -c '\''import shutil,sys;shutil.rmtree(sys.argv[1])'\'' "$task_tmp"' EXIT
|
||||
case "$(uname -m)" in x86_64) task_arch=x64;; *) echo 'Intel worker requires x86_64' >&2; exit 1;; esac
|
||||
curl -fsSL https://nodejs.org/dist/latest-v22.x/SHASUMS256.txt -o "$task_tmp/SHA256SUMS"
|
||||
task_archive=$(awk '$2 ~ /linux-x64.tar.xz$/ {print $2}' "$task_tmp/SHA256SUMS")
|
||||
[[ $task_archive =~ ^node-v22\.[0-9]+\.[0-9]+-linux-x64.tar.xz$ ]] || exit 1
|
||||
curl -fsSL "https://nodejs.org/dist/latest-v22.x/$task_archive" -o "$task_tmp/$task_archive"
|
||||
(cd "$task_tmp"; awk -v file="$task_archive" '$2==file' SHA256SUMS | sha256sum -c -)
|
||||
mkdir -p /opt/pt-worker/node
|
||||
tar -xJf "$task_tmp/$task_archive" -C /opt/pt-worker/node --strip-components=1
|
||||
fi
|
||||
export PATH=/opt/pt-worker/node/bin:$PATH
|
||||
node -e 'if (+process.versions.node.split(".")[0] < 22) process.exit(1)'
|
||||
install -d -m 755 /opt/pt-worker/runtime
|
||||
install -m 644 "$task_source/runtime-package.json" /opt/pt-worker/runtime/package.json
|
||||
install -m 644 "$task_source/runtime-package-lock.json" /opt/pt-worker/runtime/package-lock.json
|
||||
npm ci --prefix /opt/pt-worker/runtime --omit=optional --ignore-scripts --no-audit --no-fund
|
||||
[[ $(/opt/pt-worker/runtime/node_modules/.bin/peertube-runner --version) == 0.2.0 ]] || exit 1
|
||||
if [[ ! -f /etc/pt-worker/worker.json ]]; then
|
||||
python3 - "$task_source/worker.example.json" "$worker_name" "$peertube_url" "$tailscale_peer" "$canonical_origin" <<'PY'
|
||||
import json,sys
|
||||
c=json.load(open(sys.argv[1]));c.update(name=sys.argv[2],peertube_url=sys.argv[3].rstrip('/'),tailscale_peer=sys.argv[4],canonical_origin=sys.argv[5],probe_url='http://'+sys.argv[4]+':9044')
|
||||
with open('/etc/pt-worker/worker.json','w') as f: json.dump(c,f,indent=2)
|
||||
PY
|
||||
fi
|
||||
chown root:pt-worker /etc/pt-worker/worker.json
|
||||
chmod 640 /etc/pt-worker/worker.json
|
||||
# Real hardware test as the actual service user, with its real device permissions.
|
||||
/usr/local/bin/pt-worker self-test || { echo 'QSV failed or media-engine evidence unavailable: not registering or starting' >&2; exit 1; }
|
||||
task_registered=$(runuser -u pt-worker -- python3 - <<'PY'
|
||||
import tomllib,json
|
||||
from pathlib import Path
|
||||
c=json.load(open('/etc/pt-worker/worker.json')); p=Path('/var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml')
|
||||
d=tomllib.loads(p.read_text()) if p.exists() else {}
|
||||
print(int(any(x['url'].rstrip('/')==c['peertube_url'] and x['runnerName']==c['name'] for x in d.get('registeredInstances',[]))))
|
||||
PY
|
||||
)
|
||||
if [[ $task_registered == 0 ]]; then
|
||||
[[ ! -e /var/lib/pt-worker/state/enrollment.pending ]] || { echo 'Previous registration outcome is uncertain; inspect the named runner in PeerTube before retrying. No duplicate enrollment attempted.' >&2; exit 1; }
|
||||
# Registration requires a running server. concurrency=0 makes enrollment refuse jobs.
|
||||
install -d -m 700 -o pt-worker -g pt-worker /var/lib/pt-worker/.config/peertube-runner-nodejs/default
|
||||
if [[ -e /var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml ]]; then
|
||||
python3 - <<'PY'
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
p=Path('/var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml')
|
||||
c=tomllib.loads(p.read_text())
|
||||
if c.get('registeredInstances'): raise SystemExit('Other registered instances exist: inspect before enrollment')
|
||||
PY
|
||||
fi
|
||||
cat > /var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml <<'TOML'
|
||||
[jobs]
|
||||
concurrency = 0
|
||||
[ffmpeg]
|
||||
threads = 2
|
||||
nice = 20
|
||||
TOML
|
||||
chown pt-worker:pt-worker /var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml
|
||||
# Logs go to /dev/null during enrollment; the official CLI secret flag is deliberately avoided.
|
||||
runuser -u pt-worker -- env HOME=/var/lib/pt-worker PATH="$PATH" \
|
||||
/opt/pt-worker/runtime/node_modules/.bin/peertube-runner server \
|
||||
--enable-job vod-web-video-transcoding --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding >/dev/null 2>&1 &
|
||||
task_enroll_pid=$!
|
||||
trap 'kill "$task_enroll_pid" 2>/dev/null || true; [[ -z ${task_tmp:-} ]] || python3 -c '\''import shutil,sys;shutil.rmtree(sys.argv[1])'\'' "$task_tmp"' EXIT
|
||||
for _ in $(seq 1 30); do
|
||||
[[ -S /var/lib/pt-worker/.local/share/peertube-runner-nodejs/default/peertube-runner.sock ]] && break
|
||||
sleep 1
|
||||
done
|
||||
if [[ -n $token_fd ]]; then
|
||||
[[ $token_fd =~ ^[0-9]+$ ]] || exit 2
|
||||
touch /var/lib/pt-worker/state/enrollment.pending
|
||||
runuser -u pt-worker -- env HOME=/var/lib/pt-worker /opt/pt-worker/node/bin/node /opt/pt-worker/ipc.cjs register <&"$token_fd"
|
||||
else
|
||||
[[ -t 0 ]] || { echo 'Provide a secure token FD or run interactively' >&2; exit 1; }
|
||||
read -r -s -p 'Dedicated runner registration token: ' task_token
|
||||
echo
|
||||
touch /var/lib/pt-worker/state/enrollment.pending
|
||||
printf '%s' "$task_token" | runuser -u pt-worker -- env HOME=/var/lib/pt-worker /opt/pt-worker/node/bin/node /opt/pt-worker/ipc.cjs register
|
||||
unset task_token
|
||||
fi
|
||||
runuser -u pt-worker -- env HOME=/var/lib/pt-worker PATH="$PATH" /opt/pt-worker/runtime/node_modules/.bin/peertube-runner graceful-shutdown >/dev/null
|
||||
wait "$task_enroll_pid"
|
||||
# IPC returns success only after saving the dedicated runner credential to TOML.
|
||||
fi
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
Path('/var/lib/pt-worker/state/enrollment.pending').unlink(missing_ok=True)
|
||||
PY
|
||||
python3 - <<'PY'
|
||||
import re,json
|
||||
from pathlib import Path
|
||||
p=Path('/var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml')
|
||||
c=json.load(open('/etc/pt-worker/worker.json'))
|
||||
s=p.read_text(); s,n=re.subn(r'(?m)^concurrency\s*=\s*\d+', 'concurrency = '+str(c['concurrency']), s,count=1)
|
||||
if n!=1: raise SystemExit('Missing concurrency configuration')
|
||||
p.write_text(s)
|
||||
PY
|
||||
chmod 600 /var/lib/pt-worker/.config/peertube-runner-nodejs/default/config.toml
|
||||
install -m 644 "$task_source/systemd/pt-worker.service" /etc/systemd/system/pt-worker.service
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now pt-worker.service
|
||||
echo 'Registered. Admission starts PAUSED; measure bandwidth or use pt-worker start --calibrate for initial real jobs.'
|
||||
/usr/local/bin/pt-worker status
|
||||
32
infra/peertube-workers/ipc.cjs
Normal file
32
infra/peertube-workers/ipc.cjs
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
// Uses the same IPC library/protocol as the official runner. Registration secret from stdin.
|
||||
const { Client } = require('./runtime/node_modules/@peertube/net-ipc');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const action = process.argv[2];
|
||||
const home = process.env.HOME;
|
||||
const socket = path.join(home, '.local/share/peertube-runner-nodejs/default/peertube-runner.sock');
|
||||
const client = new Client({ path: socket, reconnect: false });
|
||||
const deadline = setTimeout(() => { client.destroy(); process.exit(1); }, 15000);
|
||||
(async () => {
|
||||
try {
|
||||
await client.connect();
|
||||
let request = { type: action };
|
||||
if (action === 'register') {
|
||||
const cfg = JSON.parse(fs.readFileSync(process.env.PT_WORKER_CONFIG || '/etc/pt-worker/worker.json'));
|
||||
const registrationToken = fs.readFileSync(0, 'utf8').trim();
|
||||
if (!registrationToken.startsWith('ptrrt-')) throw new Error('bad token');
|
||||
request = { type: 'register', url: cfg.peertube_url, runnerName: cfg.name, registrationToken,
|
||||
runnerDescription: 'VOD only; QSV required; admission controlled' };
|
||||
} else if (!['list-jobs', 'list-registered', 'graceful-shutdown'].includes(action)) throw new Error('bad action');
|
||||
const result = await client.request(request);
|
||||
if (!result.success) throw new Error('IPC failed');
|
||||
if (action === 'list-jobs') {
|
||||
// Drop payloads (can contain URLs); status consumers only need counts.
|
||||
console.log(JSON.stringify({ concurrency: result.data.concurrency, active_jobs: result.data.processingJobs.length }));
|
||||
} else if (action === 'list-registered') console.log(JSON.stringify(result.data));
|
||||
else console.log(JSON.stringify({ success: true }));
|
||||
} catch { console.error('Runner IPC failed; inspect local service health'); process.exitCode = 1; }
|
||||
finally { clearTimeout(deadline); client.destroy(); }
|
||||
})();
|
||||
107
infra/peertube-workers/observe.cjs
Normal file
107
infra/peertube-workers/observe.cjs
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
'use strict';
|
||||
// Transfer observation and canonical-origin routing. PeerTube retains queue selection and retries.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { AsyncLocalStorage } = require('node:async_hooks');
|
||||
const context = new AsyncLocalStorage();
|
||||
const state = process.env.PT_STATE_DIR || '/var/lib/pt-worker/state';
|
||||
let route;
|
||||
try {
|
||||
const cfg = JSON.parse(fs.readFileSync(process.env.PT_WORKER_CONFIG || '/etc/pt-worker/worker.json'));
|
||||
if (cfg.canonical_origin && cfg.peertube_url) route = { from: new URL(cfg.canonical_origin), to: new URL(cfg.peertube_url) };
|
||||
} catch { /* Enrollment and isolated tests may not yet have a config. */ }
|
||||
const originals = { 'http:': require('node:http').request, 'https:': require('node:https').request };
|
||||
function emit(data) {
|
||||
try {
|
||||
fs.mkdirSync(state, { recursive: true, mode: 0o700 });
|
||||
fs.appendFileSync(path.join(state, 'events.jsonl'), JSON.stringify({ time: Date.now() / 1000, ...data }) + '\n', { mode: 0o600 });
|
||||
} catch { /* Observation never changes job behavior. */ }
|
||||
}
|
||||
const createWriteStream = fs.createWriteStream;
|
||||
fs.createWriteStream = function (filename, ...args) {
|
||||
const pathname = context.getStore();
|
||||
const match = pathname?.match(/\/api\/v1\/runners\/jobs\/([a-f\d-]{36})\/files\//);
|
||||
if (match && typeof filename === 'string') {
|
||||
try {
|
||||
const dir = path.join(state, 'inputs');
|
||||
fs.mkdirSync(dir, { recursive: true, mode: 0o700 });
|
||||
fs.writeFileSync(path.join(dir, path.basename(filename) + '.json'), JSON.stringify({ uuid: match[1] }), { mode: 0o600 });
|
||||
} catch { /* Optional correlation only. */ }
|
||||
}
|
||||
return createWriteStream.call(this, filename, ...args);
|
||||
};
|
||||
require('node:module').syncBuiltinESMExports();
|
||||
for (const name of ['node:http', 'node:https']) {
|
||||
const mod = require(name);
|
||||
const original = mod.request;
|
||||
mod.request = function (...args) {
|
||||
let selected = original;
|
||||
// Source payloads use the public origin. Route only that origin via Tailscale.
|
||||
let options = args[0];
|
||||
if (typeof options === 'string' || options instanceof URL) {
|
||||
const u = new URL(options);
|
||||
options = { protocol: u.protocol, hostname: u.hostname, port: u.port, path: u.pathname + u.search,
|
||||
...(typeof args[1] === 'object' ? args[1] : {}) };
|
||||
}
|
||||
if (route && (options.hostname || options.host) === route.from.hostname &&
|
||||
(options.protocol || (name === 'node:https' ? 'https:' : 'http:')) === route.from.protocol &&
|
||||
String(options.port || (route.from.protocol === 'https:' ? 443 : 80)) === String(route.from.port || (route.from.protocol === 'https:' ? 443 : 80))) {
|
||||
const callback = typeof args[args.length - 1] === 'function' ? args[args.length - 1] : undefined;
|
||||
options = { ...options, protocol: route.to.protocol, hostname: route.to.hostname, port: route.to.port,
|
||||
headers: { ...(options.headers || {}), host: route.from.host } };
|
||||
delete options.host;
|
||||
selected = originals[route.to.protocol];
|
||||
args = callback ? [options, callback] : [options];
|
||||
}
|
||||
let req;
|
||||
if (typeof args[args.length - 1] === 'function') {
|
||||
const callback = args[args.length - 1];
|
||||
args[args.length - 1] = function (...values) {
|
||||
return context.run((req.path || '').split('?')[0], () => callback.apply(this, values));
|
||||
};
|
||||
}
|
||||
req = selected.apply(this, args);
|
||||
// No host, URL, query, headers, body, or tokens are ever recorded.
|
||||
const pathname = (req.path || '').split('?')[0];
|
||||
const match = pathname.match(/\/api\/v1\/runners\/jobs\/([a-f\d-]{36})\/(accept|success|error|abort)/);
|
||||
const download = pathname.includes('/files/');
|
||||
const upload = (match && match[2] === 'success') || (req.method === 'PUT');
|
||||
const begun = performance.now();
|
||||
let sent = 0, received = 0, done = false, requestErrored = false;
|
||||
function size(value, encoding) {
|
||||
if (Buffer.isBuffer(value) || value instanceof Uint8Array) return value.length;
|
||||
if (typeof value === 'string') return Buffer.byteLength(value, typeof encoding === 'string' ? encoding : 'utf8');
|
||||
return 0;
|
||||
}
|
||||
const write = req.write;
|
||||
req.write = function (chunk, ...rest) { sent += size(chunk, rest[0]); return write.call(this, chunk, ...rest); };
|
||||
const end = req.end;
|
||||
req.end = function (chunk, ...rest) { sent += size(chunk, rest[0]); return end.call(this, chunk, ...rest); };
|
||||
function finish(ok, code) {
|
||||
if (done) return;
|
||||
done = true;
|
||||
const seconds = Math.max(0.001, (performance.now() - begun) / 1000);
|
||||
if (download || upload) emit({ kind: 'transfer', direction: download ? 'download' : 'upload',
|
||||
bytes: download ? received : sent, seconds, ok, status_code: code || null });
|
||||
if (match) emit({ kind: 'job_' + match[2], uuid: match[1], ok });
|
||||
}
|
||||
req.on('response', res => {
|
||||
function interrupted() {
|
||||
finish(false, res.statusCode);
|
||||
req.destroy();
|
||||
// A ClientRequest may already be marked destroyed when its response aborts;
|
||||
// destroy(error) then emits nothing. Forward one error to the upstream promise.
|
||||
queueMicrotask(() => {
|
||||
if (!requestErrored) req.emit('error', new Error('Worker transfer stream interrupted'));
|
||||
});
|
||||
}
|
||||
res.on('data', chunk => { received += chunk.length; });
|
||||
res.once('end', () => finish(res.statusCode >= 200 && res.statusCode < 300, res.statusCode));
|
||||
res.once('aborted', interrupted);
|
||||
res.once('error', interrupted);
|
||||
});
|
||||
req.on('error', () => { requestErrored = true; finish(false); });
|
||||
return req;
|
||||
};
|
||||
}
|
||||
require('node:module').syncBuiltinESMExports();
|
||||
13
infra/peertube-workers/pt-worker
Executable file
13
infra/peertube-workers/pt-worker
Executable file
|
|
@ -0,0 +1,13 @@
|
|||
#!/bin/sh
|
||||
set -eu
|
||||
# Control files and GPU tests always run as the dedicated worker user.
|
||||
if [ "$(id -u)" -eq 0 ]; then
|
||||
case "${1:-}" in
|
||||
start|restart) systemctl start pt-worker.service ;;
|
||||
esac
|
||||
exec runuser -u pt-worker -- env HOME=/var/lib/pt-worker \
|
||||
PATH=/opt/pt-worker/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
||||
PT_STATE_DIR=/var/lib/pt-worker/state PT_WORKER_CONFIG=/etc/pt-worker/worker.json \
|
||||
/usr/bin/python3 /opt/pt-worker/pt-worker.py "$@"
|
||||
fi
|
||||
exec /usr/bin/python3 /opt/pt-worker/pt-worker.py "$@"
|
||||
4
infra/peertube-workers/pt-worker-win.ps1
Normal file
4
infra/peertube-workers/pt-worker-win.ps1
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
param([Parameter(Position=0)][ValidateSet('status','start','stop','restart','drain','self-test','network-test')][string]$Command='status',[string]$Root='C:\pt-worker')
|
||||
$env:PT_ROOT=$Root
|
||||
& (Join-Path $Root 'bin\pt-admission.exe') $Command
|
||||
exit $LASTEXITCODE
|
||||
737
infra/peertube-workers/pt-worker.py
Executable file
737
infra/peertube-workers/pt-worker.py
Executable file
|
|
@ -0,0 +1,737 @@
|
|||
#!/usr/bin/python3
|
||||
"""Whole-worker admission and operations for the official PeerTube runner."""
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import fcntl
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import tomllib
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
CONFIG = Path(os.environ.get('PT_WORKER_CONFIG', '/etc/pt-worker/worker.json'))
|
||||
STATE = Path(os.environ.get('PT_STATE_DIR', '/var/lib/pt-worker/state'))
|
||||
HOME = Path(os.environ.get('PT_WORKER_HOME', '/var/lib/pt-worker'))
|
||||
RUNNER_CONFIG = HOME / '.config/peertube-runner-nodejs/default/config.toml'
|
||||
VOD = ['vod-web-video-transcoding', 'vod-hls-transcoding', 'vod-audio-merge-transcoding']
|
||||
spec = importlib.util.spec_from_file_location('qsv', ROOT / 'ffmpeg-qsv.py')
|
||||
qsv = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(qsv)
|
||||
|
||||
|
||||
def run(args, timeout=10):
|
||||
return subprocess.run(args, capture_output=True, text=True, timeout=timeout, check=False)
|
||||
|
||||
|
||||
def atomic(path, data):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_name(path.name + '.new')
|
||||
fd = os.open(temporary, os.O_CREAT | os.O_TRUNC | os.O_WRONLY, 0o600)
|
||||
with os.fdopen(fd, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
f.write('\n')
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def read(path, default=None):
|
||||
try:
|
||||
return json.loads(path.read_text())
|
||||
except (OSError, ValueError):
|
||||
return {} if default is None else default
|
||||
|
||||
|
||||
def config():
|
||||
value = read(CONFIG)
|
||||
if not value.get('peertube_url') or not value.get('tailscale_peer'):
|
||||
raise ValueError('worker.json needs peertube_url and tailscale_peer')
|
||||
return value
|
||||
|
||||
|
||||
def env():
|
||||
return {**os.environ, 'HOME': str(HOME), 'PT_STATE_DIR': str(STATE),
|
||||
'XDG_CONFIG_HOME': str(HOME / '.config'), 'XDG_CACHE_HOME': str(HOME / '.cache'),
|
||||
'XDG_DATA_HOME': str(HOME / '.local/share'),
|
||||
'FFMPEG_PATH': str(ROOT / 'ffmpeg-qsv.py'), 'FFPROBE_PATH': '/usr/bin/ffprobe',
|
||||
'NODE_OPTIONS': '--require=' + str(ROOT / 'observe.cjs')}
|
||||
|
||||
|
||||
def registered(cfg):
|
||||
try:
|
||||
data = tomllib.loads(RUNNER_CONFIG.read_text())
|
||||
return any(r['url'].rstrip('/') == cfg['peertube_url'].rstrip('/') and
|
||||
r['runnerName'] == cfg['name'] and r.get('runnerToken') for r in data.get('registeredInstances', []))
|
||||
except (OSError, ValueError, KeyError):
|
||||
return False
|
||||
|
||||
|
||||
def ipc(action):
|
||||
p = subprocess.run([config().get('node_binary', '/usr/bin/node'), str(ROOT / 'ipc.cjs'), action], env=env(),
|
||||
capture_output=True, text=True, timeout=5)
|
||||
if p.returncode:
|
||||
raise RuntimeError('runner IPC unavailable')
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def network(cfg):
|
||||
result = {'tailscale_ip': None, 'tailscale_path': 'unknown', 'latency_ms': None,
|
||||
'network_ok': False, 'api_ok': False, 'peer_online': False}
|
||||
try:
|
||||
ts = json.loads(run(['tailscale', 'status', '--json'], 5).stdout)
|
||||
result['tailscale_ip'] = next((a for a in ts.get('Self', {}).get('TailscaleIPs', []) if ':' not in a), None)
|
||||
peers = list(ts.get('Peer', {}).values())
|
||||
peer = next((p for p in peers if cfg['tailscale_peer'] in p.get('TailscaleIPs', []) or
|
||||
p.get('HostName') == cfg['tailscale_peer'] or
|
||||
p.get('DNSName', '').split('.')[0] == cfg['tailscale_peer']), None)
|
||||
if ts.get('BackendState') != 'Running' or not peer or not peer.get('Online'):
|
||||
return result
|
||||
result['peer_online'] = True
|
||||
p = run(['tailscale', 'ping', '--c=1', '--until-direct=false', '--timeout=3s', cfg['tailscale_peer']], 5)
|
||||
text = p.stdout
|
||||
if 'pong from' not in text:
|
||||
return result
|
||||
result['tailscale_path'] = 'DERP' if 'DERP(' in text else 'direct'
|
||||
latency = re.search(r'in ([\d.]+)ms', text)
|
||||
if latency:
|
||||
result['latency_ms'] = float(latency[1])
|
||||
started = time.monotonic()
|
||||
with urllib.request.urlopen(cfg['peertube_url'].rstrip('/') + '/api/v1/config', timeout=4) as response:
|
||||
version = json.load(response).get('serverVersion')
|
||||
result['api_ok'] = version == cfg.get('peertube_version', '7.3.0')
|
||||
result['api_latency_ms'] = (time.monotonic() - started) * 1000
|
||||
result['network_ok'] = result['api_ok']
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def transfer_probe(cfg, megabytes=8):
|
||||
url = cfg.get('probe_url')
|
||||
rates = {}
|
||||
if not url:
|
||||
return rates
|
||||
payload = b'pt-worker-probe-' * ((megabytes * 1024**2 // 16) + 1)
|
||||
payload = payload[:megabytes * 1024**2]
|
||||
for direction in ('download', 'upload'):
|
||||
begun, count = time.monotonic(), 0
|
||||
try:
|
||||
req = urllib.request.Request(url.rstrip('/') + ('/download?bytes=' + str(len(payload)) if direction == 'download' else '/upload'),
|
||||
data=payload if direction == 'upload' else None,
|
||||
method='PUT' if direction == 'upload' else 'GET')
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
if direction == 'upload':
|
||||
count = json.load(response)['bytes_received']
|
||||
else:
|
||||
while chunk := response.read(1024**2):
|
||||
count += len(chunk)
|
||||
seconds = time.monotonic() - begun
|
||||
qsv.event('transfer', direction=direction, bytes=count, seconds=seconds, ok=True, source='probe')
|
||||
rates[direction + '_mbps'] = count * 8 / seconds / 1e6
|
||||
except (OSError, ValueError, KeyError):
|
||||
qsv.event('transfer', direction=direction, bytes=count, seconds=time.monotonic() - begun, ok=False, source='probe')
|
||||
rates[direction + '_mbps'] = None
|
||||
return rates
|
||||
|
||||
|
||||
def sensors():
|
||||
temperatures = []
|
||||
for p in Path('/sys/class/thermal').glob('thermal_zone*/temp'):
|
||||
try:
|
||||
temperatures.append(float(p.read_text().strip()) / 1000)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
for p in Path('/sys/class/hwmon').glob('hwmon*/temp*_input'):
|
||||
try:
|
||||
temperatures.append(float(p.read_text().strip()) / 1000)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
ac, batteries = [], []
|
||||
for p in Path('/sys/class/power_supply').glob('*'):
|
||||
try:
|
||||
kind = (p / 'type').read_text().strip()
|
||||
if kind in ('Mains', 'USB', 'USB_C') and (p / 'online').exists():
|
||||
ac.append((p / 'online').read_text().strip() == '1')
|
||||
if kind == 'Battery':
|
||||
batteries.append({'capacity': int((p / 'capacity').read_text()),
|
||||
'status': (p / 'status').read_text().strip()})
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
# Native servers with no battery have an implicit permanent power supply.
|
||||
wsl = 'microsoft' in Path('/proc/sys/kernel/osrelease').read_text().lower()
|
||||
on_ac = any(ac) if ac else (None if wsl or batteries else True)
|
||||
throttles = {}
|
||||
for p in Path('/sys/devices/system/cpu').glob('cpu*/thermal_throttle/*_throttle_count'):
|
||||
try:
|
||||
throttles[str(p)] = int(p.read_text())
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
mem = {}
|
||||
for line in Path('/proc/meminfo').read_text().splitlines():
|
||||
key, value = line.split(':', 1)
|
||||
if key in ('MemTotal', 'MemAvailable'):
|
||||
mem[key] = int(value.strip().split()[0]) * 1024
|
||||
return {'temperature_c': max(temperatures, default=None), 'temperatures_c': temperatures,
|
||||
'on_ac': on_ac, 'battery': batteries, 'throttle_counts': throttles,
|
||||
'ram_total_bytes': mem.get('MemTotal'), 'ram_available_bytes': mem.get('MemAvailable'),
|
||||
'load_average': list(os.getloadavg()), 'wsl2': wsl}
|
||||
|
||||
|
||||
def hardware():
|
||||
cpu = next((l.split(':', 1)[1].strip() for l in Path('/proc/cpuinfo').read_text().splitlines()
|
||||
if l.startswith('model name')), 'unknown')
|
||||
gpu = []
|
||||
for p in Path('/sys/class/drm').glob('renderD*/device'):
|
||||
try:
|
||||
gpu.append({'render': p.parent.name, 'vendor': (p / 'vendor').read_text().strip(),
|
||||
'device': (p / 'device').read_text().strip(),
|
||||
'driver': (p / 'driver').resolve().name})
|
||||
except OSError:
|
||||
pass
|
||||
return {'cpu_model': cpu, 'gpu_model': gpu}
|
||||
|
||||
|
||||
class Meter:
|
||||
def __init__(self):
|
||||
self.data = read(STATE / 'meter.json', {'offset': 0, 'jobs_completed': 0, 'jobs_failed': 0,
|
||||
'transfer_failures': 0, 'active': {}})
|
||||
|
||||
def invalidate(self):
|
||||
for key in ('download', 'upload'):
|
||||
self.data.pop(key, None)
|
||||
|
||||
def update(self):
|
||||
p = STATE / 'events.jsonl'
|
||||
if not p.exists():
|
||||
return
|
||||
with p.open() as f:
|
||||
if p.stat().st_size < self.data.get('offset', 0):
|
||||
self.data['offset'] = 0
|
||||
f.seek(self.data.get('offset', 0))
|
||||
while True:
|
||||
position = f.tell()
|
||||
line = f.readline()
|
||||
if not line:
|
||||
break
|
||||
if not line.endswith('\n'):
|
||||
f.seek(position)
|
||||
break
|
||||
try:
|
||||
self.consume(json.loads(line))
|
||||
except (ValueError, KeyError, TypeError):
|
||||
pass
|
||||
self.data['offset'] = f.tell()
|
||||
atomic(STATE / 'meter.json', self.data)
|
||||
|
||||
def consume(self, e):
|
||||
kind, stamp = e['kind'], e['time']
|
||||
if kind == 'transfer':
|
||||
if not e['ok']:
|
||||
self.data['transfer_failures'] = self.data.get('transfer_failures', 0) + 1
|
||||
self.data['last_transfer_failure'] = stamp
|
||||
elif e['bytes'] >= 65536 and e['seconds'] > 0:
|
||||
direction = e['direction']
|
||||
rate = e['bytes'] * 8 / e['seconds'] / 1e6
|
||||
old = self.data.get(direction, {}).get('mbps', rate)
|
||||
self.data[direction] = {'mbps': .3 * rate + .7 * old, 'time': stamp,
|
||||
'bytes': e['bytes'], 'seconds': e['seconds']}
|
||||
elif kind == 'gpu_failure':
|
||||
self.data['gpu_failure_time'] = stamp
|
||||
elif kind == 'encode':
|
||||
self.data['last_encode'] = e
|
||||
if e.get('job_uuid'):
|
||||
self.data.setdefault('encodes', {})[e['job_uuid']] = e
|
||||
elif kind == 'job_accept' and e['ok']:
|
||||
self.data.setdefault('active', {})[e['uuid']] = stamp
|
||||
elif kind in ('job_success', 'job_error', 'job_abort') and e['ok']:
|
||||
begun = self.data.setdefault('active', {}).pop(e['uuid'], None)
|
||||
if kind == 'job_success':
|
||||
self.data['jobs_completed'] = self.data.get('jobs_completed', 0) + 1
|
||||
elapsed = stamp - begun if begun else None
|
||||
self.data['last_job_duration_seconds'] = elapsed
|
||||
self.data['last_job_uuid'] = e['uuid']
|
||||
encoded = self.data.setdefault('encodes', {}).pop(e['uuid'], {})
|
||||
duration = encoded.get('duration')
|
||||
self.data['last_job_realtime_factor'] = duration / elapsed if duration and elapsed else None
|
||||
self.data['last_job_encode'] = encoded or None
|
||||
elif kind == 'job_error':
|
||||
self.data['jobs_failed'] = self.data.get('jobs_failed', 0) + 1
|
||||
for path in (STATE / 'inputs').glob('*.json'):
|
||||
if read(path).get('uuid') == e['uuid']:
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
def rates(self, now, ttl):
|
||||
return {f'recent_{k}_mbps': self.data.get(k, {}).get('mbps')
|
||||
if now - self.data.get(k, {}).get('time', 0) < ttl else None for k in ('download', 'upload')}
|
||||
|
||||
|
||||
class Admission:
|
||||
def __init__(self, cfg):
|
||||
self.cfg, self.good, self.bad = cfg, 0, 0
|
||||
self.admitted, self.path, self.throttles = False, None, {}
|
||||
|
||||
def decide(self, sample, metric, control, now):
|
||||
cfg = self.cfg
|
||||
def refuse(state):
|
||||
self.admitted, self.good = False, 0
|
||||
return state
|
||||
if self.path and sample['tailscale_path'] != self.path:
|
||||
self.admitted = False
|
||||
self.good = 0
|
||||
metric.invalidate()
|
||||
self.path = sample['tailscale_path']
|
||||
rates = metric.rates(now, cfg.get('bandwidth_ttl_seconds', 900))
|
||||
sample.update(rates)
|
||||
if control.get('paused', True):
|
||||
return refuse('PAUSED')
|
||||
if not sample['registered']:
|
||||
return refuse('PAUSED')
|
||||
gpu_failed = metric.data.get('gpu_failure_time', 0) >= sample.get('gpu_tested_at', 0)
|
||||
if not sample['gpu_accessible'] or not sample.get('gpu_ready') or gpu_failed:
|
||||
return refuse('GPU_UNAVAILABLE')
|
||||
if cfg.get('require_ac', True) and sample['on_ac'] is not True:
|
||||
return refuse('DRAINING')
|
||||
if sample['disk_free_bytes'] < cfg.get('min_disk_free_bytes', 20 * 1024**3):
|
||||
return refuse('LOW_DISK')
|
||||
rising = any(v > self.throttles.get(k, v) for k, v in sample['throttle_counts'].items())
|
||||
self.throttles = sample['throttle_counts']
|
||||
if rising or (sample['temperature_c'] is not None and sample['temperature_c'] >= cfg.get('max_temperature_c', 85)):
|
||||
self.thermal_until = now + cfg.get('thermal_cooldown_seconds', 120)
|
||||
if now < getattr(self, 'thermal_until', 0):
|
||||
return refuse('THERMAL_THROTTLED')
|
||||
if not sample['network_ok']:
|
||||
self.admitted, self.good = False, 0
|
||||
return 'OFFLINE'
|
||||
calibrating = control.get('calibrate_until', 0) > now
|
||||
factor = cfg.get('recovery_factor', 1.25) if not self.admitted else 1
|
||||
bad = sample['latency_ms'] is None or sample['latency_ms'] > cfg.get('max_latency_ms', 150)
|
||||
bad |= sample['tailscale_path'] == 'DERP' and not cfg.get('allow_derp', False)
|
||||
if not calibrating:
|
||||
bad |= any(rates[f'recent_{key}_mbps'] is None or rates[f'recent_{key}_mbps'] <
|
||||
cfg.get(f'min_{key}_mbps', 10) * factor for key in ('download', 'upload'))
|
||||
bad |= now - metric.data.get('last_transfer_failure', 0) < cfg.get('transfer_failure_cooldown_seconds', 60)
|
||||
if bad:
|
||||
self.good, self.bad = 0, self.bad + 1
|
||||
if self.bad >= cfg.get('bad_samples', 2) or not self.admitted:
|
||||
self.admitted = False
|
||||
return 'NETWORK_DEGRADED'
|
||||
else:
|
||||
self.bad, self.good = 0, self.good + 1
|
||||
if self.good >= cfg.get('good_samples', 3):
|
||||
self.admitted = True
|
||||
if not self.admitted:
|
||||
return 'NETWORK_DEGRADED'
|
||||
return 'BUSY' if sample['active_jobs'] else 'AVAILABLE'
|
||||
|
||||
|
||||
class Utilization:
|
||||
def __init__(self):
|
||||
self.cpu, self.gpu, self.at = None, {}, time.monotonic()
|
||||
|
||||
def sample(self):
|
||||
cpu = list(map(int, Path('/proc/stat').read_text().splitlines()[0].split()[1:9]))
|
||||
gpu = {}
|
||||
for p in Path('/proc').glob('[0-9]*'):
|
||||
try:
|
||||
if p.stat().st_uid == os.getuid() and (p / 'comm').read_text().strip() == 'ffmpeg':
|
||||
for engine, value in qsv.engines(p.name).items():
|
||||
gpu[(p.name, engine)] = value
|
||||
except OSError:
|
||||
pass
|
||||
now = time.monotonic()
|
||||
elapsed, result = now - self.at, {'cpu': None, 'gpu': None}
|
||||
if self.cpu:
|
||||
total = sum(cpu) - sum(self.cpu)
|
||||
idle = cpu[3] + cpu[4] - self.cpu[3] - self.cpu[4]
|
||||
result['cpu'] = (1 - idle / total) * 100 if total else None
|
||||
if gpu:
|
||||
by_engine = {}
|
||||
for key, value in gpu.items():
|
||||
if key in self.gpu:
|
||||
delta = max(0, value - self.gpu[key])
|
||||
by_engine[key[1]] = by_engine.get(key[1], 0) + delta / (elapsed * 1e9) * 100
|
||||
result['gpu'] = by_engine or None
|
||||
self.cpu, self.gpu, self.at = cpu, gpu, now
|
||||
return result
|
||||
|
||||
|
||||
def log_runner(pipe):
|
||||
# Upstream logs contain job tokens and sometimes runner tokens: retain only safe milestones.
|
||||
patterns = [r'Processing job of type ([\w-]+): ([a-f\d-]{36})',
|
||||
r'Finished processing job of type ([\w-]+): ([a-f\d-]{36})']
|
||||
for line in pipe:
|
||||
for pattern in patterns:
|
||||
match = re.search(pattern, line)
|
||||
if match:
|
||||
qsv.event('runner_milestone', phase='completed' if 'Finished' in pattern else 'processing',
|
||||
job_type=match[1], uuid=match[2])
|
||||
if 'Cannot process job' in line:
|
||||
qsv.event('runner_error', reason='process_job_failed')
|
||||
|
||||
|
||||
def daemon():
|
||||
cfg = config()
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
lock = (STATE / 'daemon.lock').open('w')
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
meter, admission, utilization = Meter(), Admission(cfg), Utilization()
|
||||
child, draining, stopping, restart_after, last_probe, previous_path = None, False, False, 0, 0, None
|
||||
def stop(*_):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
signal.signal(signal.SIGINT, stop)
|
||||
hw = hardware()
|
||||
proof = read(STATE / 'qsv.json')
|
||||
boot_id = Path('/proc/sys/kernel/random/boot_id').read_text().strip()
|
||||
if proof.get('boot_id') != boot_id and os.access(cfg.get('render_device', '/dev/dri/renderD128'), os.R_OK | os.W_OK):
|
||||
self_test()
|
||||
while True:
|
||||
now = time.time()
|
||||
meter.update()
|
||||
control = read(STATE / 'control.json', {'paused': True})
|
||||
if stopping:
|
||||
control['paused'] = True
|
||||
if child and child.poll() is not None:
|
||||
restart_after = now + 15
|
||||
child, draining = None, False
|
||||
sample = {**network(cfg), **sensors(), **hw, 'hostname': cfg['name'], 'updated_at': now,
|
||||
'registered': registered(cfg), 'registration_state': 'registered' if registered(cfg) else 'unregistered',
|
||||
'concurrency': cfg.get('concurrency', 2), 'active_jobs': 0 if not child else None,
|
||||
'qsv': read(STATE / 'qsv.json'), 'draining': draining}
|
||||
gpu_health = read(Path(cfg.get('gpu_health_file', str(STATE / 'qsv.json'))))
|
||||
sample.update(gpu_ready=gpu_health.get('hardware_video_ok', False), gpu_tested_at=gpu_health.get('tested_at', 0))
|
||||
device = cfg.get('gpu_device', cfg.get('render_device', '/dev/dri/renderD128'))
|
||||
sample['gpu_accessible'] = os.access(device, os.R_OK | os.W_OK)
|
||||
disk = os.statvfs(HOME)
|
||||
sample['disk_free_bytes'] = disk.f_bavail * disk.f_frsize
|
||||
ipc_ok = not child
|
||||
if child:
|
||||
try:
|
||||
sample.update(ipc('list-jobs'))
|
||||
ipc_ok = True
|
||||
except (OSError, ValueError, subprocess.SubprocessError, RuntimeError):
|
||||
pass
|
||||
if previous_path and previous_path != sample['tailscale_path']:
|
||||
meter.invalidate()
|
||||
last_probe = 0
|
||||
admission.path = None
|
||||
admission.admitted, admission.good = False, 0
|
||||
previous_path = sample['tailscale_path']
|
||||
probe_due = cfg.get('probe_url') and not control.get('paused', True) and sample['network_ok'] and \
|
||||
sample['on_ac'] is True and sample['tailscale_path'] == 'direct' and sample['active_jobs'] == 0 and \
|
||||
now - last_probe >= cfg.get('probe_interval_seconds', 300)
|
||||
if probe_due and child and not draining:
|
||||
try:
|
||||
result = subprocess.run([cfg['runner_binary'], 'graceful-shutdown'], env=env(),
|
||||
capture_output=True, text=True, timeout=5)
|
||||
draining = result.returncode == 0
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
pass
|
||||
if probe_due and not child:
|
||||
transfer_probe(cfg)
|
||||
last_probe = now
|
||||
meter.update()
|
||||
sample.update(network(cfg))
|
||||
sample.update(sensors())
|
||||
if sample['tailscale_path'] != previous_path:
|
||||
meter.invalidate()
|
||||
admission.admitted, admission.good, admission.path = False, 0, None
|
||||
last_probe = 0
|
||||
now = time.time()
|
||||
desired = admission.decide(sample, meter, control, now)
|
||||
if not ipc_ok:
|
||||
desired = 'OFFLINE'
|
||||
allow = desired in ('AVAILABLE', 'BUSY') and not stopping
|
||||
if child and not allow and not draining:
|
||||
try:
|
||||
# SIGTERM upstream aborts jobs. Use its official graceful-shutdown command instead.
|
||||
result = subprocess.run([cfg['runner_binary'], 'graceful-shutdown'], env=env(),
|
||||
capture_output=True, text=True, timeout=5)
|
||||
draining = result.returncode == 0
|
||||
if draining:
|
||||
qsv.event('drain', reason=desired)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
pass
|
||||
if not child and allow and now >= restart_after:
|
||||
child = subprocess.Popen([cfg['runner_binary'], 'server', *sum([['--enable-job', j] for j in VOD], [])],
|
||||
env=env(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
|
||||
start_new_session=True)
|
||||
threading.Thread(target=log_runner, args=(child.stdout,), daemon=True).start()
|
||||
sample.update({'status': 'DRAINING' if draining and desired in ('AVAILABLE', 'BUSY') else desired,
|
||||
'draining': draining, 'runner_running': child is not None,
|
||||
'qsv_h264_ok': sample['qsv'].get('qsv_h264_ok'),
|
||||
'qsv_hevc_ok': sample['qsv'].get('qsv_hevc_ok'),
|
||||
'util': utilization.sample(),
|
||||
**{k: meter.data.get(k) for k in ('jobs_completed', 'jobs_failed', 'transfer_failures',
|
||||
'last_job_uuid', 'last_job_duration_seconds', 'last_job_realtime_factor', 'last_job_encode', 'last_encode')}})
|
||||
# Null is intentional until the corresponding measurement has been made.
|
||||
atomic(STATE / 'status.json', sample)
|
||||
if stopping and child is None:
|
||||
return
|
||||
# After suspend, wall-clock gaps force a fresh path/throughput evaluation.
|
||||
before = time.time()
|
||||
time.sleep(cfg.get('poll_seconds', 5))
|
||||
if time.time() - before > cfg.get('poll_seconds', 5) * 3:
|
||||
meter.invalidate()
|
||||
admission.admitted, admission.good = False, 0
|
||||
|
||||
|
||||
def measured(command, log):
|
||||
started = time.monotonic()
|
||||
maximum, samples = {}, []
|
||||
with log.open('w') as f:
|
||||
p = subprocess.Popen(command, stdout=f, stderr=subprocess.STDOUT)
|
||||
while p.poll() is None:
|
||||
for k, v in qsv.engines(p.pid).items():
|
||||
maximum[k] = max(maximum.get(k, 0), v)
|
||||
samples.append(sensors())
|
||||
if time.monotonic() - started > 180:
|
||||
p.kill()
|
||||
p.wait()
|
||||
break
|
||||
time.sleep(.1)
|
||||
return {'returncode': p.returncode, 'seconds': time.monotonic() - started,
|
||||
'engine_ns': maximum, 'max_temperature_c': max((s['temperature_c'] for s in samples
|
||||
if s['temperature_c'] is not None), default=None),
|
||||
'min_available_ram_bytes': min((s['ram_available_bytes'] for s in samples), default=None)}
|
||||
|
||||
|
||||
def self_test():
|
||||
cfg = config()
|
||||
device = cfg.get('render_device', '/dev/dri/renderD128')
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
report = {**hardware(), 'tested_at': time.time(), 'boot_id': Path('/proc/sys/kernel/random/boot_id').read_text().strip(),
|
||||
'qsv_h264_ok': False, 'qsv_hevc_ok': False,
|
||||
'av1_decode_ok': None, 'av1_encode_ok': None, 'tests': {}, 'device': device}
|
||||
diagnostics = STATE / 'self-test'
|
||||
diagnostics.mkdir(exist_ok=True)
|
||||
for name, command in [('ffmpeg-version', ['/usr/bin/ffmpeg', '-version']),
|
||||
('hwaccels', ['/usr/bin/ffmpeg', '-hwaccels']),
|
||||
('encoders', ['/usr/bin/ffmpeg', '-encoders']),
|
||||
('vainfo', ['vainfo', '--display', 'drm', '--device', device])]:
|
||||
try:
|
||||
p = run(command, 20)
|
||||
(diagnostics / (name + '.log')).write_text(p.stdout + p.stderr)
|
||||
report[name + '_exit'] = p.returncode
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
report[name + '_exit'] = None
|
||||
if not os.access(device, os.R_OK | os.W_OK):
|
||||
report['reason'] = 'render device missing or inaccessible; hardware tests did not run'
|
||||
atomic(STATE / 'qsv.json', report)
|
||||
print(json.dumps(report, indent=2))
|
||||
return 1
|
||||
with tempfile.TemporaryDirectory(prefix='selftest-', dir=STATE) as directory:
|
||||
tmp = Path(directory)
|
||||
source = tmp / 'source.mp4'
|
||||
# CPU x264 is used ONLY to create synthetic test input, outside the runner.
|
||||
p = run(['/usr/bin/ffmpeg', '-v', 'error', '-f', 'lavfi', '-i', 'testsrc2=size=1920x1080:rate=30',
|
||||
'-f', 'lavfi', '-i', 'sine=frequency=1000:sample_rate=48000', '-t', '10', '-c:v', 'libx264',
|
||||
'-threads', '2', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', str(source)], 90)
|
||||
if p.returncode:
|
||||
report['reason'] = 'test fixture creation failed'
|
||||
else:
|
||||
base = ['/usr/bin/ffmpeg', '-v', 'verbose', '-init_hw_device', f'vaapi=va:{device}',
|
||||
'-init_hw_device', 'qsv=qs@va', '-filter_hw_device', 'qs', '-hwaccel', 'qsv',
|
||||
'-hwaccel_output_format', 'qsv', '-c:v', 'h264_qsv', '-i', str(source),
|
||||
'-vf', 'scale_qsv=w=1280:h=720:format=nv12', '-an']
|
||||
for codec, key in [('h264', 'qsv_h264_ok'), ('hevc', 'qsv_hevc_ok'), ('av1', 'av1_encode_ok')]:
|
||||
out = tmp / (codec + '.mp4')
|
||||
result = measured([*base, '-c:v', codec + '_qsv', '-b:v', '2M', str(out)], diagnostics / (codec + '.log'))
|
||||
result['hardware_engine_evidence'] = any(v > 0 for k, v in result['engine_ns'].items() if k.startswith('video'))
|
||||
report['tests'][codec] = result
|
||||
report[key] = result['returncode'] == 0 and result['hardware_engine_evidence']
|
||||
if result['returncode'] == 0:
|
||||
report['tests'][codec]['output_probe'] = qsv.probe(str(out))
|
||||
if report['qsv_hevc_ok']:
|
||||
command = base[:base.index('-c:v')] + ['-re', '-c:v', 'hevc_qsv', '-i', str(tmp / 'hevc.mp4'),
|
||||
'-vf', 'scale_qsv=w=640:h=360:format=nv12', '-an', '-c:v', 'h264_qsv', '-f', 'null', '-']
|
||||
result = measured(command, diagnostics / 'hevc-decode.log')
|
||||
report['tests']['hevc_decode'] = result
|
||||
report['qsv_hevc_decode_ok'] = result['returncode'] == 0 and any(v > 0 for k, v in result['engine_ns'].items() if k.startswith('video'))
|
||||
av1_source = tmp / 'av1-source.mkv'
|
||||
p = run(['/usr/bin/ffmpeg', '-v', 'error', '-f', 'lavfi', '-i', 'testsrc2=size=640x360:rate=30',
|
||||
'-t', '2', '-c:v', 'libaom-av1', '-threads', '2', '-cpu-used', '8', str(av1_source)], 90)
|
||||
if p.returncode == 0:
|
||||
command = base[:base.index('-c:v')] + ['-re', '-c:v', 'av1_qsv', '-i', str(av1_source),
|
||||
'-vf', 'scale_qsv=w=640:h=360:format=nv12', '-an', '-c:v', 'h264_qsv', '-f', 'null', '-']
|
||||
result = measured(command, diagnostics / 'av1-decode.log')
|
||||
report['tests']['av1_decode'] = result
|
||||
report['av1_decode_ok'] = result['returncode'] == 0 and any(v > 0 for k, v in result['engine_ns'].items() if k.startswith('video'))
|
||||
# Exercise the adapter itself; compare with the direct QSV tests.
|
||||
command = [str(ROOT / 'ffmpeg-qsv.py'), '-i', str(source), '-vcodec', 'libx264',
|
||||
'-vf', 'scale=w=-2:h=720', '-pix_fmt', 'yuv420p', '-preset', 'veryfast',
|
||||
'-maxrate:v', '2000000', '-bufsize:v', '4000000', '-b_strategy', '1', '-bf', '16',
|
||||
'-g:v', '60', '-r', '30', '-c:a', 'aac', str(tmp / 'adapter.mp4')]
|
||||
p = subprocess.run(command, env=env(), capture_output=True, text=True, timeout=90)
|
||||
(diagnostics / 'adapter.log').write_text(p.stderr)
|
||||
report['adapter_exit'] = p.returncode
|
||||
report['qsv_h264_ok'] &= p.returncode == 0
|
||||
# The admission controller compares this timestamp to failures emitted by earlier tests.
|
||||
report['tested_at'] = time.time()
|
||||
report['hardware_video_ok'] = report['qsv_h264_ok']
|
||||
report['backend'] = 'intel_qsv'
|
||||
atomic(STATE / 'qsv.json', report)
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0 if report['qsv_h264_ok'] else 1
|
||||
|
||||
|
||||
def network_test(args):
|
||||
"""Real HTTP throughput against explicit disposable operator-supplied endpoints."""
|
||||
cfg = config()
|
||||
report = network(cfg)
|
||||
if not args.download_url or not args.upload_url:
|
||||
if cfg.get('probe_url'):
|
||||
rates = transfer_probe(cfg, min(args.megabytes, 16))
|
||||
print(json.dumps({**report, **rates, 'throughput_measured': all(v is not None for v in rates.values())}, indent=2))
|
||||
return 0 if all(v is not None for v in rates.values()) else 1
|
||||
print(json.dumps({**report, 'throughput_measured': False,
|
||||
'reason': 'supply --download-url and --upload-url for disposable fixtures; API ping is not bandwidth'}, indent=2))
|
||||
return 1
|
||||
rates = {}
|
||||
for direction, url in [('download', args.download_url), ('upload', args.upload_url)]:
|
||||
# Test URLs must remain within the same PeerTube/storage route.
|
||||
if urllib.parse.urlparse(url).hostname != urllib.parse.urlparse(cfg['peertube_url']).hostname:
|
||||
raise ValueError('test URL must use configured PeerTube host')
|
||||
count, begun = 0, time.monotonic()
|
||||
try:
|
||||
req = urllib.request.Request(url)
|
||||
if direction == 'upload':
|
||||
req = urllib.request.Request(url, data=os.urandom(args.megabytes * 1024**2), method='PUT')
|
||||
count = args.megabytes * 1024**2
|
||||
with urllib.request.urlopen(req, timeout=60) as response:
|
||||
while chunk := response.read(1024**2):
|
||||
if direction == 'download':
|
||||
count += len(chunk)
|
||||
seconds = time.monotonic() - begun
|
||||
qsv.event('transfer', direction=direction, bytes=count, seconds=seconds, ok=True)
|
||||
rates[direction + '_mbps'] = count * 8 / seconds / 1e6
|
||||
except OSError:
|
||||
qsv.event('transfer', direction=direction, bytes=count, seconds=time.monotonic() - begun, ok=False)
|
||||
rates[direction + '_mbps'] = None
|
||||
print(json.dumps({**report, **rates, 'throughput_measured': all(v is not None for v in rates.values())}, indent=2))
|
||||
return 0 if all(v is not None for v in rates.values()) else 1
|
||||
|
||||
|
||||
def benchmark(args):
|
||||
status = read(STATE / 'status.json')
|
||||
if status.get('runner_running'):
|
||||
raise ValueError('drain worker and wait for runner_running=false before benchmarking')
|
||||
source = Path(args.source).resolve()
|
||||
probe = qsv.probe(str(source))
|
||||
duration = float(probe.get('format', {}).get('duration', 0))
|
||||
records = []
|
||||
with tempfile.TemporaryDirectory(prefix='benchmark-', dir=STATE) as directory:
|
||||
tmp = Path(directory)
|
||||
for concurrency in (1, 2, 3, 4):
|
||||
def job(index):
|
||||
out = tmp / f'{concurrency}-{index}.mp4'
|
||||
cmd = [str(ROOT / 'ffmpeg-qsv.py'), '-i', str(source), '-vcodec', 'libx264',
|
||||
'-vf', 'scale=w=-2:h=720', '-pix_fmt', 'yuv420p', '-maxrate:v', '2000000',
|
||||
'-bufsize:v', '4000000', '-r', '30', '-an', str(out)]
|
||||
start = time.monotonic()
|
||||
p = subprocess.run(cmd, env=env(), capture_output=True, text=True, timeout=600)
|
||||
elapsed = time.monotonic() - start
|
||||
out.unlink(missing_ok=True)
|
||||
return {'seconds': elapsed, 'realtime_factor': duration / elapsed,
|
||||
'fps': duration * 30 / elapsed, 'returncode': p.returncode}
|
||||
start, samples, util_samples = time.monotonic(), [], []
|
||||
utilization = Utilization()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
|
||||
futures = [executor.submit(job, i) for i in range(concurrency)]
|
||||
while not all(f.done() for f in futures):
|
||||
samples.append(sensors())
|
||||
util_samples.append(utilization.sample())
|
||||
time.sleep(.5)
|
||||
jobs = [f.result() for f in futures]
|
||||
wall = time.monotonic() - start
|
||||
records.append({'concurrency': concurrency, 'wall_seconds': wall, 'jobs': jobs,
|
||||
'aggregate_media_seconds_per_second': concurrency * duration / wall,
|
||||
'max_temperature_c': max((s['temperature_c'] for s in samples
|
||||
if s['temperature_c'] is not None), default=None),
|
||||
'min_available_ram_bytes': min((s['ram_available_bytes'] for s in samples), default=None),
|
||||
'utilization_samples': util_samples,
|
||||
'throttle_counts_start': samples[0]['throttle_counts'] if samples else {},
|
||||
'throttle_counts_end': samples[-1]['throttle_counts'] if samples else {},
|
||||
'recent_download_mbps': None, 'recent_upload_mbps': None})
|
||||
report = {'scope': 'local encode diagnostic; choose production concurrency from real end-to-end jobs',
|
||||
'source_duration': duration, 'results': records, 'production_concurrency_selected': None}
|
||||
atomic(STATE / 'benchmark.json', report)
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0 if all(j['returncode'] == 0 for r in records for j in r['jobs']) else 1
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
sub = parser.add_subparsers(dest='command', required=True)
|
||||
for name in ('daemon', 'status', 'start', 'stop', 'restart', 'drain', 'self-test'):
|
||||
p = sub.add_parser(name)
|
||||
if name == 'start':
|
||||
p.add_argument('--calibrate', action='store_true', help='explicit 10-minute initial job measurement window')
|
||||
if name == 'status':
|
||||
p.add_argument('--prometheus', action='store_true')
|
||||
p = sub.add_parser('network-test')
|
||||
p.add_argument('--download-url')
|
||||
p.add_argument('--upload-url')
|
||||
p.add_argument('--megabytes', type=int, default=16)
|
||||
p = sub.add_parser('benchmark')
|
||||
p.add_argument('--source', required=True)
|
||||
args = parser.parse_args()
|
||||
if args.command == 'daemon':
|
||||
return daemon()
|
||||
if args.command == 'status':
|
||||
empty = {k: None for k in ('tailscale_ip', 'tailscale_path', 'registration_state', 'cpu_model', 'gpu_model',
|
||||
'qsv_h264_ok', 'qsv_hevc_ok', 'active_jobs', 'concurrency', 'util', 'temperature_c', 'temperatures_c',
|
||||
'disk_free_bytes', 'recent_download_mbps', 'recent_upload_mbps', 'jobs_completed', 'jobs_failed',
|
||||
'last_job_duration_seconds', 'last_job_realtime_factor')}
|
||||
value = read(STATE / 'status.json', {**empty, 'hostname': socket.gethostname(), 'status': 'OFFLINE'})
|
||||
if time.time() - value.get('updated_at', 0) > 60:
|
||||
value['status'] = 'OFFLINE'
|
||||
value['stale'] = True
|
||||
if args.prometheus:
|
||||
print('pt_worker_available ' + str(int(value.get('status') in ('AVAILABLE', 'BUSY'))))
|
||||
for key in ('active_jobs', 'concurrency', 'disk_free_bytes', 'temperature_c', 'recent_download_mbps',
|
||||
'recent_upload_mbps', 'jobs_completed', 'jobs_failed'):
|
||||
if isinstance(value.get(key), (int, float)):
|
||||
print('pt_worker_' + key + ' ' + str(value[key]))
|
||||
else:
|
||||
print(json.dumps(value, indent=2))
|
||||
return 0
|
||||
if args.command in ('start', 'stop', 'restart', 'drain'):
|
||||
control = {'paused': args.command in ('stop', 'drain'),
|
||||
'calibrate_until': time.time() + 600 if getattr(args, 'calibrate', False) else 0}
|
||||
atomic(STATE / 'control.json', control)
|
||||
if args.command == 'restart':
|
||||
# Drain first; the controller restarts the child after it exits, preserving registration.
|
||||
try:
|
||||
subprocess.run([config()['runner_binary'], 'graceful-shutdown'], env=env(),
|
||||
capture_output=True, timeout=5, check=True)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
pass
|
||||
print(json.dumps({'requested': args.command, **control}))
|
||||
return 0
|
||||
if args.command == 'self-test':
|
||||
if read(STATE / 'status.json').get('runner_running'):
|
||||
raise ValueError('drain and wait for runner_running=false before self-test')
|
||||
return self_test()
|
||||
if args.command == 'network-test':
|
||||
return network_test(args)
|
||||
if args.command == 'benchmark':
|
||||
return benchmark(args)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
sys.exit(main() or 0)
|
||||
except (ValueError, RuntimeError, OSError, subprocess.SubprocessError) as error:
|
||||
print('pt-worker: ' + str(error), file=sys.stderr)
|
||||
sys.exit(1)
|
||||
172
infra/peertube-workers/runtime-package-lock.json
generated
Normal file
172
infra/peertube-workers/runtime-package-lock.json
generated
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
{
|
||||
"name": "pt-worker-runtime",
|
||||
"version": "1.0.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "pt-worker-runtime",
|
||||
"version": "1.0.0",
|
||||
"dependencies": {
|
||||
"@peertube/net-ipc": "2.2.0",
|
||||
"@peertube/peertube-runner": "0.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@msgpackr-extract/msgpackr-extract-darwin-arm64": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-darwin-arm64/-/msgpackr-extract-darwin-arm64-3.0.4.tgz",
|
||||
"integrity": "sha512-LCkGo6JDfaBhgST7UpPWgNgLINpcpabaHfyz5OBx75nUYxBsaEPxjnyNjWpeb/xBup/682QnBfRBy2/LvPutZQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
]
|
||||
},
|
||||
"node_modules/@msgpackr-extract/msgpackr-extract-darwin-x64": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-darwin-x64/-/msgpackr-extract-darwin-x64-3.0.4.tgz",
|
||||
"integrity": "sha512-zExlW9zUJKZH/tOtVMttwjKa4Xm/3KcNjnE3dPN92uCktwavMxpgCA3MoJK/DOnTWsQgo224OaST27/mPNAf+w==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
]
|
||||
},
|
||||
"node_modules/@msgpackr-extract/msgpackr-extract-linux-arm": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-linux-arm/-/msgpackr-extract-linux-arm-3.0.4.tgz",
|
||||
"integrity": "sha512-Tg3yX65f5GbtXLkrYEHE5oibZG9epyYWas7FogTTEJeDEF9JlXJzKgXaNhT3UXlTOeA+AfZpYZYZ0uPj7Cfquw==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@msgpackr-extract/msgpackr-extract-linux-arm64": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-linux-arm64/-/msgpackr-extract-linux-arm64-3.0.4.tgz",
|
||||
"integrity": "sha512-dgX0P/9wGPJeHFBG+ZmhgE6bmtMt7NP5CRBGyyktpopdk/mW4POnrpQsSLtKI1dwpc+pPLuXHDh6vvskyQE/sw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@msgpackr-extract/msgpackr-extract-linux-x64": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-linux-x64/-/msgpackr-extract-linux-x64-3.0.4.tgz",
|
||||
"integrity": "sha512-8TNXMEjJc3QEy7R/x1INhgiU+XakDAFUzBhaz7+Rbrs8NH5UQeHQxxmzsSBJGyV6I1jW79undiQm8tOI+D+8FQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@msgpackr-extract/msgpackr-extract-win32-x64": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-win32-x64/-/msgpackr-extract-win32-x64-3.0.4.tgz",
|
||||
"integrity": "sha512-CmCXPQrkbwExx3j946/PtHWHbYJiCRBRDl4BlkRQcJB/YOwQxJRTpoo7aTsortjgoJ1x7opzTSxn7C+ASSLVjQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
]
|
||||
},
|
||||
"node_modules/@peertube/net-ipc": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@peertube/net-ipc/-/net-ipc-2.2.0.tgz",
|
||||
"integrity": "sha512-fcKrTuYRlpUpwX//eaU6t11Boeqa/RY9yzCuIw/4e/jygcTd6ZQ3jTOUv6ABm06yiQTI+W/Tx9fyebFSLaTf9A==",
|
||||
"deprecated": "Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.",
|
||||
"engines": {
|
||||
"node": ">=16.0.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"fast-zlib": "^2.0.1",
|
||||
"msgpackr": "^1.3.2"
|
||||
}
|
||||
},
|
||||
"node_modules/@peertube/peertube-runner": {
|
||||
"version": "0.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@peertube/peertube-runner/-/peertube-runner-0.2.0.tgz",
|
||||
"integrity": "sha512-xawEdU9oGKNtMGyRdypEcNg4dyNCvYtg0MHfxA6tB81qWFhc5U2Of3DfYq5iz9UDIj0DVFxVdMijPtc5pkfSCw==",
|
||||
"bin": {
|
||||
"peertube-runner": "dist/peertube-runner.js"
|
||||
}
|
||||
},
|
||||
"node_modules/detect-libc": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
|
||||
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/fast-zlib": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/fast-zlib/-/fast-zlib-2.0.1.tgz",
|
||||
"integrity": "sha512-DCoYgNagM2Bt1VIpXpdGnRx4LzqJeYG0oh6Nf/7cWo6elTXkFGMw9CrRCYYUIapYNrozYMoyDRflx9mgT3Awyw==",
|
||||
"optional": true,
|
||||
"funding": {
|
||||
"type": "patreon",
|
||||
"url": "https://patreon.com/timotejroiko"
|
||||
}
|
||||
},
|
||||
"node_modules/msgpackr": {
|
||||
"version": "1.12.1",
|
||||
"resolved": "https://registry.npmjs.org/msgpackr/-/msgpackr-1.12.1.tgz",
|
||||
"integrity": "sha512-4EUH9tQHnMmEgzW/MdAP0KIfa1T9AF+htl0ffe2n5vb2EKn9y2co8ccpgWko6S52Jy1PQZKwRnx5/KkYjtd9MQ==",
|
||||
"optional": true,
|
||||
"optionalDependencies": {
|
||||
"msgpackr-extract": "^3.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/msgpackr-extract": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/msgpackr-extract/-/msgpackr-extract-3.0.4.tgz",
|
||||
"integrity": "sha512-4kmO/MdyUIkLIvTPr8VHLil4AtoKIoniWPIEk5+CDy0xnWC84azhSFmuJ7PxZdsYtiP5kEeQsORAVIeMgxT+Hw==",
|
||||
"hasInstallScript": true,
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"node-gyp-build-optional-packages": "5.2.2"
|
||||
},
|
||||
"bin": {
|
||||
"download-msgpackr-prebuilds": "bin/download-prebuilds.js"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@msgpackr-extract/msgpackr-extract-darwin-arm64": "3.0.4",
|
||||
"@msgpackr-extract/msgpackr-extract-darwin-x64": "3.0.4",
|
||||
"@msgpackr-extract/msgpackr-extract-linux-arm": "3.0.4",
|
||||
"@msgpackr-extract/msgpackr-extract-linux-arm64": "3.0.4",
|
||||
"@msgpackr-extract/msgpackr-extract-linux-x64": "3.0.4",
|
||||
"@msgpackr-extract/msgpackr-extract-win32-x64": "3.0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/node-gyp-build-optional-packages": {
|
||||
"version": "5.2.2",
|
||||
"resolved": "https://registry.npmjs.org/node-gyp-build-optional-packages/-/node-gyp-build-optional-packages-5.2.2.tgz",
|
||||
"integrity": "sha512-s+w+rBWnpTMwSFbaE0UXsRlg7hU4FjekKU4eyAih5T8nJuNZT1nNsskXpxmeqSK9UzkBl6UgRlnKc8hz8IEqOw==",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"detect-libc": "^2.0.1"
|
||||
},
|
||||
"bin": {
|
||||
"node-gyp-build-optional-packages": "bin.js",
|
||||
"node-gyp-build-optional-packages-optional": "optional.js",
|
||||
"node-gyp-build-optional-packages-test": "build-test.js"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
9
infra/peertube-workers/runtime-package.json
Normal file
9
infra/peertube-workers/runtime-package.json
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"name": "pt-worker-runtime",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"dependencies": {
|
||||
"@peertube/peertube-runner": "0.2.0",
|
||||
"@peertube/net-ipc": "2.2.0"
|
||||
}
|
||||
}
|
||||
32
infra/peertube-workers/systemd/pt-worker-netprobe.service
Normal file
32
infra/peertube-workers/systemd/pt-worker-netprobe.service
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
[Unit]
|
||||
Description=Bounded PeerTube worker throughput probe (Tailscale only)
|
||||
After=tailscaled.service network-online.target
|
||||
Wants=tailscaled.service network-online.target
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
NotifyAccess=main
|
||||
TimeoutStartSec=45
|
||||
User=pt-netprobe
|
||||
Group=pt-netprobe
|
||||
ExecStart=/usr/bin/python3 /opt/pt-worker-netprobe/bandwidth-probe.py --bind 100.92.231.98 --port 9044
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
PrivateDevices=true
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
CapabilityBoundingSet=
|
||||
RestrictAddressFamilies=AF_INET AF_UNIX
|
||||
MemoryMax=96M
|
||||
TasksMax=16
|
||||
CPUQuota=20%
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
37
infra/peertube-workers/systemd/pt-worker.service
Normal file
37
infra/peertube-workers/systemd/pt-worker.service
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
[Unit]
|
||||
Description=PeerTube VOD runner with hardware and network admission
|
||||
Wants=network-online.target tailscaled.service
|
||||
After=network-online.target tailscaled.service
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=pt-worker
|
||||
Group=pt-worker
|
||||
Environment=HOME=/var/lib/pt-worker
|
||||
Environment=PATH=/opt/pt-worker/node/bin:/usr/local/bin:/usr/bin:/bin
|
||||
Environment=PT_STATE_DIR=/var/lib/pt-worker/state
|
||||
Environment=PT_WORKER_CONFIG=/etc/pt-worker/worker.json
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 /opt/pt-worker/pt-worker.py daemon
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
# Controller uses runner graceful-shutdown; do not send SIGTERM to FFmpeg on initial stop.
|
||||
KillMode=mixed
|
||||
TimeoutStopSec=infinity
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/var/lib/pt-worker
|
||||
PrivateTmp=true
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictSUIDSGID=true
|
||||
LockPersonality=true
|
||||
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
|
||||
CapabilityBoundingSet=
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
128
infra/peertube-workers/tests/runner_smoke.py
Normal file
128
infra/peertube-workers/tests/runner_smoke.py
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
#!/usr/bin/python3
|
||||
"""Integration test with published runner against a loopback API; no production calls."""
|
||||
import argparse
|
||||
import http.server
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
UUID = '22222222-2222-4222-8222-222222222222'
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--node', required=True)
|
||||
parser.add_argument('--runtime', required=True, help='directory containing installed runtime node_modules')
|
||||
args = parser.parse_args()
|
||||
with tempfile.TemporaryDirectory(prefix='pt-runner-smoke-') as directory:
|
||||
tmp = Path(directory)
|
||||
home, state = tmp / 'home', tmp / 'state'
|
||||
home.mkdir(); state.mkdir()
|
||||
(tmp / 'runtime').symlink_to(Path(args.runtime).resolve(), target_is_directory=True)
|
||||
shutil.copy(ROOT / 'ipc.cjs', tmp / 'ipc.cjs')
|
||||
fixture = tmp / 'source.mp4'
|
||||
subprocess.run(['/usr/bin/ffmpeg', '-v', 'error', '-f', 'lavfi', '-i', 'testsrc2=size=64x64:rate=2',
|
||||
'-t', '1', '-c:v', 'libx264', '-threads', '1', str(fixture)], check=True)
|
||||
requests, accepted, failures = [], threading.Event(), threading.Event()
|
||||
registered_count = 0
|
||||
offered = False
|
||||
class API(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *_): pass
|
||||
def do_POST(self):
|
||||
nonlocal registered_count, offered
|
||||
length = int(self.headers.get('Content-Length', 0))
|
||||
self.rfile.read(length) # discarded; never log credentials
|
||||
requests.append(self.path)
|
||||
if self.path == '/api/v1/runners/register':
|
||||
registered_count += 1
|
||||
return self.reply({'runnerToken': 'ptrt-local-test-only'})
|
||||
if self.path == '/api/v1/runners/jobs/request':
|
||||
jobs = [] if offered or not registered_count else [{'uuid': UUID, 'type': 'vod-web-video-transcoding',
|
||||
'payload': {'input': {'videoFileUrl': f'https://canonical.invalid/api/v1/runners/jobs/{UUID}/files/videos/source/max-quality',
|
||||
'separatedAudioFileUrl': []}, 'output': {'resolution': 64, 'fps': 2}}}]
|
||||
return self.reply({'availableJobs': jobs})
|
||||
if self.path.endswith('/accept'):
|
||||
offered = True
|
||||
accepted.set()
|
||||
return self.reply({'job': {'uuid': UUID, 'type': 'vod-web-video-transcoding', 'jobToken': 'local-job-secret',
|
||||
'payload': {'input': {'videoFileUrl': f'https://canonical.invalid/api/v1/runners/jobs/{UUID}/files/videos/source/max-quality',
|
||||
'separatedAudioFileUrl': []}, 'output': {'resolution': 64, 'fps': 2}}}})
|
||||
if '/files/' in self.path:
|
||||
blob = fixture.read_bytes()
|
||||
self.send_response(200); self.send_header('Content-Length', str(len(blob))); self.end_headers()
|
||||
self.wfile.write(blob)
|
||||
return
|
||||
if self.path.endswith('/error'):
|
||||
failures.set()
|
||||
self.send_response(204); self.end_headers()
|
||||
def reply(self, data):
|
||||
blob = json.dumps(data).encode()
|
||||
self.send_response(200); self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(blob))); self.end_headers(); self.wfile.write(blob)
|
||||
server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), API)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
url = f'http://127.0.0.1:{server.server_port}'
|
||||
cfg = tmp / 'worker.json'
|
||||
cfg.write_text(json.dumps({'name': 'smoke-worker', 'peertube_url': url, 'canonical_origin': 'https://canonical.invalid'}))
|
||||
env = {**os.environ, 'HOME': str(home), 'XDG_CONFIG_HOME': str(home / '.config'),
|
||||
'XDG_CACHE_HOME': str(home / '.cache'), 'XDG_DATA_HOME': str(home / '.local/share'),
|
||||
'PT_WORKER_CONFIG': str(cfg), 'PT_STATE_DIR': str(state),
|
||||
'PT_RENDER_DEVICE': str(tmp / 'missing-render'), 'FFMPEG_PATH': str(ROOT / 'ffmpeg-qsv.py'),
|
||||
'FFPROBE_PATH': '/usr/bin/ffprobe', 'NODE_OPTIONS': '--require=' + str(ROOT / 'observe.cjs')}
|
||||
runner = Path(args.runtime) / 'node_modules/@peertube/peertube-runner/dist/peertube-runner.js'
|
||||
runner_config = home / '.config/peertube-runner-nodejs/default/config.toml'
|
||||
runner_config.parent.mkdir(parents=True)
|
||||
runner_config.write_text('[jobs]\nconcurrency=0\n[ffmpeg]\nthreads=2\nnice=20\n')
|
||||
socket = home / '.local/share/peertube-runner-nodejs/default/peertube-runner.sock'
|
||||
children = []
|
||||
def start():
|
||||
child = subprocess.Popen([args.node, str(runner), 'server', '--enable-job', 'vod-web-video-transcoding'],
|
||||
env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
children.append(child)
|
||||
for _ in range(100):
|
||||
if socket.exists():
|
||||
p = subprocess.run([args.node, str(tmp / 'ipc.cjs'), 'list-jobs'], env=env, capture_output=True, timeout=3)
|
||||
if p.returncode == 0: return child
|
||||
if child.poll() is not None: raise AssertionError('runner exited')
|
||||
time.sleep(.1)
|
||||
raise AssertionError('IPC socket unavailable')
|
||||
def call(command, data=None):
|
||||
return subprocess.run([args.node, str(tmp / 'ipc.cjs'), command], input=data,
|
||||
env=env, capture_output=True, text=True, timeout=20, check=True)
|
||||
try:
|
||||
child = start()
|
||||
call('register', 'ptrrt-local-test-only')
|
||||
assert registered_count == 1
|
||||
assert json.loads(call('list-jobs').stdout)['active_jobs'] == 0
|
||||
call('graceful-shutdown'); child.wait(timeout=10)
|
||||
assert not accepted.is_set(), 'enrollment accepted a job at concurrency=0'
|
||||
runner_config.write_text(runner_config.read_text().replace('concurrency = 0', 'concurrency = 1'))
|
||||
socket.unlink(missing_ok=True)
|
||||
child = start()
|
||||
assert accepted.wait(15), 'job not accepted'
|
||||
assert failures.wait(15), 'missing QSV did not report job failure'
|
||||
call('graceful-shutdown'); child.wait(timeout=10)
|
||||
assert registered_count == 1, 'restart duplicated registration'
|
||||
transcoding = home / '.cache/peertube-runner-nodejs/default/transcoding'
|
||||
assert not list(transcoding.iterdir()), 'temporary files not cleaned'
|
||||
events = (state / 'events.jsonl').read_text()
|
||||
assert 'gpu_failure' in events
|
||||
assert 'local-job-secret' not in events and 'ptrt-local-test-only' not in events
|
||||
assert not any(p.endswith('/success') for p in requests), 'failed QSV uploaded output'
|
||||
assert any('/files/' in p for p in requests), 'canonical source was not routed to loopback'
|
||||
print('PASS: official runner 0.2.0 registration via secret stdin, concurrency=0 enrollment, restart without re-registration, source routing, QSV failure reporting, graceful shutdown, tmp cleanup')
|
||||
finally:
|
||||
for child in children:
|
||||
if child.poll() is None:
|
||||
child.kill(); child.wait()
|
||||
server.shutdown(); server.server_close()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
62
infra/peertube-workers/tests/test_observer.cjs
Normal file
62
infra/peertube-workers/tests/test_observer.cjs
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
'use strict';
|
||||
// Real loopback transfers exercise routing, byte counts, file/job correlation, and secret exclusion.
|
||||
const fs = require('node:fs');
|
||||
const http = require('node:http');
|
||||
const path = require('node:path');
|
||||
const os = require('node:os');
|
||||
const assert = require('node:assert/strict');
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'pt-observer-'));
|
||||
const uuid = '11111111-1111-4111-8111-111111111111';
|
||||
const source = Buffer.alloc(300000, 7);
|
||||
let uploaded = 0;
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.url.includes('/files/abort')) {
|
||||
res.writeHead(200, { 'Content-Length': '1000000' });
|
||||
res.write(Buffer.alloc(1000)); setTimeout(() => res.destroy(), 20);
|
||||
} else if (req.url.includes('/files/')) { res.writeHead(200); res.end(source); }
|
||||
else {
|
||||
req.on('data', chunk => { uploaded += chunk.length; });
|
||||
req.on('end', () => { res.writeHead(204); res.end(); });
|
||||
}
|
||||
});
|
||||
server.listen(0, '127.0.0.1', async () => {
|
||||
const port = server.address().port;
|
||||
const cfg = path.join(tmp, 'worker.json');
|
||||
fs.writeFileSync(cfg, JSON.stringify({ canonical_origin: 'https://canonical.invalid', peertube_url: `http://127.0.0.1:${port}` }));
|
||||
process.env.PT_WORKER_CONFIG = cfg;
|
||||
process.env.PT_STATE_DIR = tmp;
|
||||
require('../observe.cjs');
|
||||
const https = require('node:https');
|
||||
try {
|
||||
await new Promise((ok, fail) => {
|
||||
const req = https.request(`https://canonical.invalid/api/v1/runners/jobs/${uuid}/files/videos/source?secret=NEVER_RECORD_ME`, res => {
|
||||
const file = fs.createWriteStream(path.join(tmp, 'source'));
|
||||
res.pipe(file); file.on('finish', ok);
|
||||
});
|
||||
req.on('error', fail); req.end('runnerToken=NEVER_RECORD_ME');
|
||||
});
|
||||
await new Promise((ok, fail) => {
|
||||
const req = http.request({ hostname: '127.0.0.1', port, path: `/api/v1/runners/jobs/${uuid}/success`, method: 'POST' }, res => {
|
||||
res.resume(); res.on('end', ok);
|
||||
});
|
||||
req.on('error', fail);
|
||||
req.write(Buffer.alloc(100000, 1)); req.end(Buffer.alloc(200000, 2));
|
||||
});
|
||||
const events = fs.readFileSync(path.join(tmp, 'events.jsonl'), 'utf8');
|
||||
assert(!events.includes('NEVER_RECORD_ME'));
|
||||
const parsed = events.trim().split('\n').map(JSON.parse);
|
||||
assert.equal(parsed.find(e => e.kind === 'transfer' && e.direction === 'download').bytes, source.length);
|
||||
assert.equal(parsed.find(e => e.kind === 'transfer' && e.direction === 'upload').bytes, uploaded);
|
||||
assert.equal(uploaded, 300000);
|
||||
assert.equal(JSON.parse(fs.readFileSync(path.join(tmp, 'inputs/source.json'))).uuid, uuid);
|
||||
await new Promise((ok, fail) => {
|
||||
const timeout = setTimeout(() => fail(new Error('interrupted transfer did not fail promptly')), 2000);
|
||||
const req = http.request({ hostname: '127.0.0.1', port, path: `/api/v1/runners/jobs/${uuid}/files/abort` }, res => res.resume());
|
||||
req.on('error', () => { clearTimeout(timeout); ok(); }); req.end();
|
||||
});
|
||||
const failed = fs.readFileSync(path.join(tmp, 'events.jsonl'), 'utf8').trim().split('\n').map(JSON.parse);
|
||||
assert(failed.some(e => e.kind === 'transfer' && !e.ok));
|
||||
console.log('PASS: real download/upload counts, Tailscale-origin routing, UUID correlation, interrupted-transfer failure, no secrets');
|
||||
} catch (e) { console.error(e); process.exitCode = 1; }
|
||||
finally { server.close(); fs.rmSync(tmp, { recursive: true, force: true }); }
|
||||
});
|
||||
157
infra/peertube-workers/tests/test_worker.py
Normal file
157
infra/peertube-workers/tests/test_worker.py
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
spec = importlib.util.spec_from_file_location('worker', ROOT / 'pt-worker.py')
|
||||
worker = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(worker)
|
||||
|
||||
|
||||
class PolicyTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.old_state = worker.STATE
|
||||
worker.STATE = Path(self.tmp.name)
|
||||
self.meter = worker.Meter()
|
||||
self.now = time.time()
|
||||
for direction in ('download', 'upload'):
|
||||
self.meter.consume({'kind': 'transfer', 'time': self.now, 'ok': True,
|
||||
'direction': direction, 'bytes': 10_000_000, 'seconds': 1})
|
||||
self.policy = worker.Admission({})
|
||||
self.sample = {'tailscale_path': 'direct', 'latency_ms': 5, 'network_ok': True,
|
||||
'registered': True, 'qsv': {'qsv_h264_ok': True, 'tested_at': self.now},
|
||||
'gpu_ready': True, 'gpu_tested_at': self.now,
|
||||
'gpu_accessible': True, 'on_ac': True, 'disk_free_bytes': 100 * 1024**3,
|
||||
'temperature_c': 50, 'throttle_counts': {}, 'active_jobs': 0}
|
||||
self.control = {'paused': False}
|
||||
|
||||
def tearDown(self):
|
||||
worker.STATE = self.old_state
|
||||
self.tmp.cleanup()
|
||||
|
||||
def decide(self):
|
||||
return self.policy.decide(self.sample, self.meter, self.control, self.now)
|
||||
|
||||
def available(self):
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
self.assertEqual(self.decide(), 'AVAILABLE')
|
||||
|
||||
def test_recovery_hysteresis_and_low_bandwidth_drain(self):
|
||||
self.available()
|
||||
self.meter.data['download']['mbps'] = 1
|
||||
self.assertEqual(self.decide(), 'AVAILABLE')
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
self.meter.data['download']['mbps'] = 11 # insufficient recovery margin
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
self.meter.data['download']['mbps'] = 20
|
||||
self.available()
|
||||
|
||||
def test_path_transition_invalidates_real_bandwidth(self):
|
||||
self.available()
|
||||
self.sample['tailscale_path'] = 'DERP'
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
self.assertIsNone(self.sample['recent_download_mbps'])
|
||||
self.sample['tailscale_path'] = 'direct'
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
|
||||
def test_battery_gpu_disk_and_network_refuse(self):
|
||||
self.available()
|
||||
self.sample['on_ac'] = False
|
||||
self.assertEqual(self.decide(), 'DRAINING')
|
||||
self.sample['on_ac'] = True
|
||||
self.sample['gpu_accessible'] = False
|
||||
self.assertEqual(self.decide(), 'GPU_UNAVAILABLE')
|
||||
self.sample['gpu_accessible'] = True
|
||||
self.sample['disk_free_bytes'] = 0
|
||||
self.assertEqual(self.decide(), 'LOW_DISK')
|
||||
self.sample['disk_free_bytes'] = 100 * 1024**3
|
||||
self.sample['network_ok'] = False
|
||||
self.assertEqual(self.decide(), 'OFFLINE')
|
||||
|
||||
def test_gpu_failure_latched_until_later_selftest(self):
|
||||
self.meter.consume({'kind': 'gpu_failure', 'time': self.now + 1})
|
||||
self.assertEqual(self.decide(), 'GPU_UNAVAILABLE')
|
||||
self.sample['qsv']['tested_at'] = self.now + 2
|
||||
self.sample['gpu_tested_at'] = self.now + 2
|
||||
self.available()
|
||||
|
||||
def test_transfer_failure_cooldown_and_stale_metrics(self):
|
||||
self.meter.consume({'kind': 'transfer', 'time': self.now, 'ok': False})
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
self.now += 61
|
||||
self.available()
|
||||
self.now += 900
|
||||
self.assertEqual(self.decide(), 'AVAILABLE')
|
||||
self.assertEqual(self.decide(), 'NETWORK_DEGRADED')
|
||||
|
||||
def test_thermal_throttle_and_pause(self):
|
||||
self.available()
|
||||
self.sample['throttle_counts'] = {'cpu0': 1}
|
||||
self.decide()
|
||||
self.sample['throttle_counts'] = {'cpu0': 2}
|
||||
self.assertEqual(self.decide(), 'THERMAL_THROTTLED')
|
||||
self.sample['temperature_c'] = 50
|
||||
self.now += 121
|
||||
self.available()
|
||||
self.control['paused'] = True
|
||||
self.assertEqual(self.decide(), 'PAUSED')
|
||||
|
||||
def test_job_correlation_uses_uuid_under_concurrency(self):
|
||||
for uuid in ('a', 'b'):
|
||||
self.meter.consume({'kind': 'job_accept', 'time': self.now, 'ok': True, 'uuid': uuid})
|
||||
self.meter.consume({'kind': 'encode', 'time': self.now + 2, 'job_uuid': 'b', 'duration': 200})
|
||||
self.meter.consume({'kind': 'encode', 'time': self.now + 3, 'job_uuid': 'a', 'duration': 100})
|
||||
self.meter.consume({'kind': 'job_success', 'time': self.now + 10, 'uuid': 'b', 'ok': True})
|
||||
self.assertEqual(self.meter.data['last_job_realtime_factor'], 20)
|
||||
|
||||
|
||||
class AdapterTests(unittest.TestCase):
|
||||
def probe(self, codec='h264', rotation=False):
|
||||
return {'streams': [{'codec_type': 'video', 'codec_name': codec, 'pix_fmt': 'yuv420p',
|
||||
'width': 1920, 'height': 1080,
|
||||
'side_data_list': [{'rotation': 90}] if rotation else []}],
|
||||
'format': {'duration': '10'}}
|
||||
|
||||
def test_qsv_decode_scale_encode_and_no_x264_flags(self):
|
||||
args, info = worker.qsv.rewrite(['-i', '/tmp/in', '-vcodec', 'libx264', '-vf', 'scale=w=-2:h=720',
|
||||
'-pix_fmt', 'yuv420p', '-preset', 'veryfast', '-b_strategy', '1', '-bf', '16', '/tmp/out'], self.probe())
|
||||
self.assertIn('h264_qsv', args)
|
||||
self.assertIn('scale_qsv=w=1280:h=720:format=nv12', args)
|
||||
self.assertNotIn('libx264', args)
|
||||
self.assertNotIn('veryfast', args)
|
||||
self.assertNotIn('-b_strategy', args)
|
||||
self.assertEqual(info['target_height'], 720)
|
||||
|
||||
def test_unknown_decode_rotation_and_cpu_encoder_fail(self):
|
||||
for probe in (self.probe('mpeg4'), self.probe(rotation=True)):
|
||||
with self.assertRaises(ValueError):
|
||||
worker.qsv.rewrite(['-i', '/tmp/in', '-c:v', 'libx264', '/tmp/out'], probe)
|
||||
with self.assertRaises(ValueError):
|
||||
worker.qsv.rewrite(['-i', '/tmp/in', '-c:v', 'libx265', '/tmp/out'], self.probe())
|
||||
|
||||
def test_unreviewed_software_filter_refused(self):
|
||||
with self.assertRaises(ValueError):
|
||||
worker.qsv.rewrite(['-i', '/tmp/in', '-c:v', 'libx264', '-vf', 'scale=640:360,eq=contrast=2', '/tmp/out'], self.probe())
|
||||
|
||||
def test_real_missing_qsv_refuses_without_cpu_output(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src, out = Path(tmp) / 'src.mp4', Path(tmp) / 'out.mp4'
|
||||
subprocess.run(['/usr/bin/ffmpeg', '-v', 'error', '-f', 'lavfi', '-i', 'testsrc2=size=64x64:rate=2',
|
||||
'-t', '1', '-c:v', 'libx264', '-threads', '1', str(src)], check=True)
|
||||
env = {**os.environ, 'PT_STATE_DIR': tmp, 'PT_RENDER_DEVICE': tmp + '/missing-device'}
|
||||
p = subprocess.run(['/usr/bin/python3', str(ROOT / 'ffmpeg-qsv.py'), '-i', str(src), '-c:v',
|
||||
'libx264', str(out)], env=env, capture_output=True)
|
||||
self.assertEqual(p.returncode, 78)
|
||||
self.assertFalse(out.exists())
|
||||
self.assertIn('gpu_failure', (Path(tmp) / 'events.jsonl').read_text())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
7
infra/peertube-workers/win/build-requirements.txt
Normal file
7
infra/peertube-workers/win/build-requirements.txt
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
pyinstaller==6.16.0
|
||||
pyinstaller-hooks-contrib==2026.8
|
||||
altgraph==0.17.5
|
||||
packaging==26.3
|
||||
pefile==2023.2.7
|
||||
pywin32-ctypes==0.2.3
|
||||
setuptools==84.0.0
|
||||
24
infra/peertube-workers/win/dependencies.json
Normal file
24
infra/peertube-workers/win/dependencies.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"node_version": "22.23.3",
|
||||
"node_msi": {
|
||||
"url": "https://nodejs.org/dist/v22.23.3/node-v22.23.3-x64.msi",
|
||||
"sha256": "1c0efc8449987e7da5d184786a0a96da83ffa11d334421201e5c09b93017cb8d"
|
||||
},
|
||||
"node_zip": {
|
||||
"url": "https://nodejs.org/dist/v22.23.3/node-v22.23.3-win-x64.zip",
|
||||
"sha256": "2b0ff57b049cda1bbcea2240eec20467018713c1efe1f7360c2681859b90ed71"
|
||||
},
|
||||
"ffmpeg": {
|
||||
"url": "https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-10-09-14-16/ffmpeg-n8.1.3-16-ge0a878dd70-win64-gpl-8.1.zip",
|
||||
"sha256": "3c1efebafb60dae9aa1322a1662595d2ee516b352af2a2f6493923998bbee612",
|
||||
"version": "n8.1.3-16-ge0a878dd70-20261009",
|
||||
"validated_original_zip_sha256": "75c912eb2c8253a9efd5e8bf75f03d4124ce1f94a3af41218f0b9da7c3986d76"
|
||||
},
|
||||
"winsw": {
|
||||
"url": "https://github.com/winsw/winsw/releases/download/v2.12.0/WinSW-x64.exe",
|
||||
"sha256": "05b82d46ad331cc16bdc00de5c6332c1ef818df8ceefcd49c726553209b3a0da",
|
||||
"version": "2.12.0"
|
||||
},
|
||||
"runner": "0.2.0",
|
||||
"net_ipc": "2.2.0"
|
||||
}
|
||||
86
infra/peertube-workers/win/e2e-peertube.mjs
Normal file
86
infra/peertube-workers/win/e2e-peertube.mjs
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
// Authorized private-video E2E. No credential is accepted in argv or printed.
|
||||
import fs from 'node:fs';
|
||||
import crypto from 'node:crypto';
|
||||
import {execFileSync} from 'node:child_process';
|
||||
const emit=x=>process.stdout.write(JSON.stringify(x)+'\n');
|
||||
const sleep=ms=>new Promise(r=>setTimeout(r,ms));
|
||||
let stage='model-registration', session, db, auth, video, result={states:[]};
|
||||
try {
|
||||
if(JSON.parse(fs.readFileSync('/app/package.json','utf8')).version !== '7.3.0') throw Error('unreviewed server version');
|
||||
const dbURL=new URL('file:///app/dist/core/initializers/database.js');
|
||||
db=await import(dbURL.href);
|
||||
const {logger}=await import('file:///app/dist/core/helpers/logger.js');logger.silent=true;
|
||||
// Register the shipped models without invoking initDatabaseModels: that helper
|
||||
// recreates a SQL function and is outside this task's authorized server scope.
|
||||
const source=fs.readFileSync(dbURL,'utf8');
|
||||
const names=source.match(/sequelizeTypescript\.addModels\(\[([\s\S]*?)\]\)/)[1].split(',').map(x=>x.trim()).filter(Boolean);
|
||||
const imports=[...source.matchAll(/import \{ (\w+) \} from '(\.\.\/models\/[^']+)';/g)];
|
||||
const models=[];
|
||||
for(const name of names){const item=imports.find(m=>m[1]===name);if(!item)throw Error('model not found');models.push((await import(new URL(item[2],dbURL).href))[name]);}
|
||||
db.sequelizeTypescript.addModels(models);
|
||||
await db.sequelizeTypescript.authenticate();
|
||||
const {OAuthTokenModel}=await import('file:///app/dist/core/models/oauth/oauth-token.js');
|
||||
const {OAuthClientModel}=await import('file:///app/dist/core/models/oauth/oauth-client.js');
|
||||
const {UserModel}=await import('file:///app/dist/core/models/user/user.js');
|
||||
const {RunnerJobModel}=await import('file:///app/dist/core/models/runner/runner-job.js');
|
||||
const {RUNNER_JOB_STATES}=await import('file:///app/dist/core/initializers/constants.js');
|
||||
const {Op}=await import('sequelize');
|
||||
const user=await UserModel.findOne({where:{username:'root',role:0}}), client=await OAuthClientModel.loadFirstClient();
|
||||
if(!user||!client)throw Error('admin/client absent');
|
||||
stage='temporary-admin-session';
|
||||
const expires=new Date(Date.now()+10*60*1000);
|
||||
session=await OAuthTokenModel.create({accessToken:crypto.randomBytes(32).toString('hex'),refreshToken:crypto.randomBytes(32).toString('hex'),accessTokenExpiresAt:expires,refreshTokenExpiresAt:expires,userId:user.id,oAuthClientId:client.id,loginDevice:'authorized-pt-worker-e2e',loginIP:'127.0.0.1',loginDate:new Date()});
|
||||
auth={Authorization:'Bearer '+session.accessToken};
|
||||
const base='http://127.0.0.1:9000/api/v1';
|
||||
async function api(path,options={}) {const r=await fetch(base+path,{...options,headers:{...auth,...options.headers},signal:AbortSignal.timeout(30000)});if(!r.ok){result.http_status=r.status;throw Error('API request failed');}return r.status===204?null:r.json();}
|
||||
stage='admin-auth-check';
|
||||
const me=await api('/users/me');
|
||||
const channel=me.videoChannels?.[0];if(!channel)throw Error('root channel absent');
|
||||
const baseline=await RunnerJobModel.max('id')||0;
|
||||
stage='private-upload';
|
||||
const form=new FormData();
|
||||
for(const [k,v] of Object.entries({channelId:String(channel.id),name:'Windows NVENC worker E2E 2026-10-09',privacy:'3',nsfw:'false',commentsEnabled:'false',downloadEnabled:'true',waitTranscoding:'true'}))form.set(k,v);
|
||||
form.set('videofile',new Blob([fs.readFileSync('/tmp/pt-worker-e2e-source.mp4')],{type:'video/mp4'}),'nvenc-e2e.mp4');
|
||||
const upload=await api('/videos/upload',{method:'POST',body:form});video=upload.video;
|
||||
result.video={id:video.id,uuid:video.uuid,shortUUID:video.shortUUID,privacy:'private',watch_url:'https://tube.occupymedia.cc/w/'+video.shortUUID};
|
||||
emit({phase:'uploaded',video:result.video});
|
||||
stage='remote-job-states';
|
||||
const end=Date.now()+5*60*1000, seen=new Map();let completed;
|
||||
while(Date.now()<end){
|
||||
const jobs=await RunnerJobModel.findAll({attributes:['id','uuid','type','state','runnerId','createdAt','startedAt','updatedAt'],where:{id:{[Op.gt]:baseline}},raw:true});
|
||||
for(const j of jobs){const label=RUNNER_JOB_STATES[j.state];if(seen.get(j.uuid)!==label){seen.set(j.uuid,label);const item={...j,state:label,observed_at:new Date().toISOString()};result.states.push(item);emit({phase:'job-state',...item});}if(label==='Completed')completed=j;if(label==='Errored')throw Error('remote job errored');}
|
||||
if(completed && jobs.some(j=>j.type==='vod-hls-transcoding' && RUNNER_JOB_STATES[j.state]==='Completed'))break;
|
||||
await sleep(100);
|
||||
}
|
||||
if(!completed)throw Error('remote job deadline');
|
||||
result.job={id:completed.id,uuid:completed.uuid,type:completed.type,runner_id:completed.runnerId};
|
||||
stage='result-media-api';
|
||||
let details;
|
||||
for(let n=0;n<60;n++){details=await api('/videos/'+video.uuid);if(details.streamingPlaylists?.[0]?.files?.length)break;await sleep(500);}
|
||||
const playlist=details.streamingPlaylists?.[0];if(!playlist?.files?.length)throw Error('HLS result absent');
|
||||
let fileToken;
|
||||
async function media(url){let u=new URL(url);u.protocol='http:';u.hostname='127.0.0.1';u.port='9000';if(fileToken)u.searchParams.set('videoFileToken',fileToken);let r=await fetch(u,{headers:auth,signal:AbortSignal.timeout(15000)});if(!r.ok&&!fileToken){try{const t=await api('/videos/'+video.uuid+'/token',{method:'POST'});fileToken=t.files?.token||t.token||t.videoFileToken;}catch{}if(fileToken){u.searchParams.set('videoFileToken',fileToken);r=await fetch(u,{headers:auth,signal:AbortSignal.timeout(15000)});}}if(!r.ok){result.media_http_status=r.status;throw Error('private media fetch failed');}return r;}
|
||||
stage='playlist-playback';
|
||||
const master=await (await media(playlist.playlistUrl)).text();
|
||||
const renditionLine=master.split(/\r?\n/).find(l=>l.trim()&&!l.startsWith('#'));
|
||||
const renditionURL=new URL(renditionLine,playlist.playlistUrl).href;
|
||||
const rendition=await (await media(renditionURL)).text();
|
||||
const assetLine=rendition.split(/\r?\n/).find(l=>l.trim()&&!l.startsWith('#'));
|
||||
const assetURL=new URL(assetLine,renditionURL).href;
|
||||
const blob=Buffer.from(await (await media(assetURL)).arrayBuffer());
|
||||
const path='/tmp/pt-worker-e2e-result.mp4';fs.writeFileSync(path,blob,{mode:0o600});
|
||||
const probe=JSON.parse(execFileSync('ffprobe',['-v','error','-show_streams','-show_format','-of','json',path],{encoding:'utf8',timeout:30000}));
|
||||
stage='playback-decode';execFileSync('ffmpeg',['-v','error','-nostdin','-i',path,'-f','null','-'],{stdio:'pipe',timeout:30000});
|
||||
const stream=probe.streams.find(s=>s.codec_type==='video');
|
||||
result.playback={master_http:200,rendition_http:200,asset_http:200,decode_exit:0,bytes:blob.length,codec:stream.codec_name,width:stream.width,height:stream.height,duration:probe.format.duration};
|
||||
result.success=true;emit({phase:'e2e-completed',job:result.job,playback:result.playback});
|
||||
}catch(e){result.success=false;result.failed_stage=stage;result.error_type=e.name;emit({phase:'e2e-failed',stage,error_type:e.name,http_status:result.http_status,media_http_status:result.media_http_status});}
|
||||
finally{
|
||||
if(session){
|
||||
try{await fetch('http://127.0.0.1:9000/api/v1/users/revoke-token',{method:'POST',headers:auth,signal:AbortSignal.timeout(10000)});await sleep(300);}catch{}
|
||||
try{await session.destroy();result.temporary_admin_session_removed=true;}catch{result.temporary_admin_session_removed=false;}
|
||||
}
|
||||
fs.writeFileSync('/tmp/pt-worker-e2e-result.json',JSON.stringify(result,null,2),{mode:0o600});
|
||||
emit({phase:'final-e2e-record',...result});
|
||||
if(db)await db.sequelizeTypescript.close();
|
||||
}
|
||||
11
infra/peertube-workers/win/enroll-server.cjs
Normal file
11
infra/peertube-workers/win/enroll-server.cjs
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
'use strict';
|
||||
// Enrollment listener has concurrency zero and discards upstream logs.
|
||||
const fs = require('node:fs'), path = require('node:path'), {spawn} = require('node:child_process');
|
||||
const root=process.env.PT_ROOT || 'C:\\pt-worker';
|
||||
const cfg=JSON.parse(fs.readFileSync(path.join(root,'state','worker.json'),'utf8').replace(/^\uFEFF/,''));
|
||||
const env={...process.env, PT_ROOT:root, APPDATA:path.join(root,'profile','Roaming'), LOCALAPPDATA:path.join(root,'profile','Local'), FFMPEG_PATH:path.join(root,'bin','ffmpeg-nvenc.exe'),FFPROBE_PATH:path.join(root,'bin','ffprobe.exe'), PT_STATE_DIR:path.join(root,'state'),PT_WORKER_CONFIG:path.join(root,'state','worker.json')};
|
||||
const toml=fs.readFileSync(path.join(env.APPDATA,'peertube-runner-nodejs','Config','default','config.toml'),'utf8');
|
||||
if(!/^concurrency\s*=\s*0\s*$/m.test(toml)) throw Error('Enrollment requires concurrency zero');
|
||||
const child=spawn(cfg.node_binary,[path.join(root,'runner','node_modules','@peertube','peertube-runner','dist','peertube-runner.js'),'server','--enable-job','vod-web-video-transcoding','--enable-job','vod-hls-transcoding','--enable-job','vod-audio-merge-transcoding'],{env,detached:true,windowsHide:true,stdio:'ignore'});
|
||||
fs.writeFileSync(path.join(root,'state','enroll-process.json'),JSON.stringify({pid:child.pid}));
|
||||
child.unref(); console.log(JSON.stringify({enrollment_listener_started:true,concurrency:0}));
|
||||
210
infra/peertube-workers/win/ffmpeg-nvenc.py
Normal file
210
infra/peertube-workers/win/ffmpeg-nvenc.py
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
"""Fail-closed PeerTube 0.2.0 adapter; freeze with PyInstaller as a real PE .exe.
|
||||
Video encoding is always NVENC. CPU image/audio processing is permitted.
|
||||
Only reviewed argument shapes are accepted, including explicit copy-only remux.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(sys.executable if getattr(sys, 'frozen', False) else __file__).resolve().parent
|
||||
REAL = Path(os.environ.get('PT_FFMPEG_EXE', str(HERE / 'ffmpeg.exe')))
|
||||
PROBE = Path(os.environ.get('PT_FFPROBE_EXE', str(HERE / 'ffprobe.exe')))
|
||||
STATE = Path(os.environ.get('PT_STATE_DIR', r'C:\pt-worker\state'))
|
||||
DEVICE = os.environ.get('PT_NVENC_DEVICE', '0')
|
||||
TARGET = os.environ.get('PT_NVENC_TARGET', 'h264_nvenc')
|
||||
QUERIES = ('-version', '-formats', '-encoders', '-filters', '-codecs', '-hwaccels')
|
||||
FLAGS = {'-y', '-n', '-nostdin', '-hide_banner', '-an', '-vn', '-shortest', '-copyts'}
|
||||
VALUES = {'-i', '-vcodec', '-c:v', '-codec:v', '-acodec', '-c:a', '-codec:a', '-c',
|
||||
'-vf', '-filter:v', '-pix_fmt', '-preset', '-preset:v', '-b_strategy', '-bf',
|
||||
'-tune', '-crf', '-maxrate', '-maxrate:v', '-bufsize', '-bufsize:v',
|
||||
'-b:v', '-vb', '-b:a', '-ab', '-q:a', '-g', '-g:v', '-r', '-r:v',
|
||||
'-threads', '-f', '-movflags', '-map', '-map_metadata', '-max_muxing_queue_size',
|
||||
'-channel_layout', '-ac', '-ar', '-t', '-loop', '-hls_time', '-hls_list_size',
|
||||
'-hls_playlist_type', '-hls_segment_filename', '-hls_segment_type', '-hls_flags',
|
||||
'-bsf:a', '-loglevel', '-v', '-sc_threshold'}
|
||||
|
||||
|
||||
def event(kind, **fields):
|
||||
try:
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
with (STATE / 'events.jsonl').open('a', encoding='utf8') as f:
|
||||
f.write(json.dumps({'time': time.time(), 'kind': kind, **fields}) + '\n')
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def probe(filename):
|
||||
p = subprocess.run([str(PROBE), '-v', 'error', '-show_streams', '-show_format',
|
||||
'-of', 'json', filename], capture_output=True, timeout=30)
|
||||
if p.returncode:
|
||||
raise ValueError('probe failed')
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def parse(args):
|
||||
if len(args) < 2 or args[-1].startswith('-'):
|
||||
raise ValueError('missing output')
|
||||
entries, i = [], 0
|
||||
while i < len(args) - 1:
|
||||
key = args[i]
|
||||
if key in FLAGS:
|
||||
entries.append((key, None)); i += 1
|
||||
elif key in VALUES and i + 1 < len(args) - 1:
|
||||
entries.append((key, args[i + 1])); i += 2
|
||||
else:
|
||||
raise ValueError('unknown option or extra output')
|
||||
return entries
|
||||
|
||||
|
||||
def rewrite(args, input_probe=None):
|
||||
if len(args) == 1 and args[0] in QUERIES:
|
||||
return args, None
|
||||
if TARGET not in ('h264_nvenc', 'hevc_nvenc') or not DEVICE.isdigit() or (STATE / 'gpu-unavailable').exists():
|
||||
raise ValueError('GPU disabled')
|
||||
entries = parse(args)
|
||||
# No codec aliases, extra output streams, or global software encoders can escape validation.
|
||||
video = [(k, v) for k, v in entries if k in ('-vcodec', '-c:v', '-codec:v')]
|
||||
generic = [v for k, v in entries if k == '-c']
|
||||
audio = [v for k, v in entries if k in ('-acodec', '-c:a', '-codec:a')]
|
||||
if any(v not in ('copy', 'aac', 'libfdk_aac', 'libmp3lame') for v in audio):
|
||||
raise ValueError('unknown audio encoder')
|
||||
if generic:
|
||||
if generic != ['copy'] or video or audio:
|
||||
raise ValueError('unreviewed global codec')
|
||||
return args, None
|
||||
if len(video) > 1:
|
||||
raise ValueError('multiple video encoders')
|
||||
if '-vn' in args:
|
||||
if video:
|
||||
raise ValueError('contradictory video options')
|
||||
return args, None
|
||||
if video and video[0][1] == 'copy':
|
||||
return args, None
|
||||
if len(video) != 1 or video[0][1] != 'libx264':
|
||||
raise ValueError('unknown video encoder')
|
||||
inputs = [v for k, v in entries if k == '-i']
|
||||
if not inputs or len(inputs) > 2:
|
||||
raise ValueError('unreviewed inputs')
|
||||
data = input_probe if input_probe is not None else probe(inputs[0])
|
||||
stream = next(s for s in data['streams'] if s['codec_type'] == 'video')
|
||||
codec = stream['codec_name']
|
||||
still = codec in ('png', 'mjpeg', 'webp') and '-loop' in args
|
||||
if not still and codec not in ('h264', 'hevc', 'vp9'):
|
||||
raise ValueError('source has no reviewed NVDEC path')
|
||||
if not still and stream.get('pix_fmt') not in ('yuv420p', 'yuv420p10le', 'nv12', 'p010le'):
|
||||
raise ValueError('unsupported pixel format')
|
||||
if stream.get('color_transfer') in ('smpte2084', 'arib-std-b67'):
|
||||
raise ValueError('HDR needs reviewed tone mapping')
|
||||
if stream.get('tags', {}).get('rotate') or any(s.get('rotation', 0) for s in stream.get('side_data_list', [])):
|
||||
raise ValueError('rotation needs reviewed GPU path')
|
||||
width, height = stream['width'], stream['height']
|
||||
w, h = width // 2 * 2, height // 2 * 2
|
||||
filters = [v for k, v in entries if k in ('-vf', '-filter:v')]
|
||||
if len(filters) > 1:
|
||||
raise ValueError('multiple filters')
|
||||
if filters:
|
||||
value = filters[0]
|
||||
if value == 'scale=trunc(iw/2)*2:trunc(ih/2)*2' and still:
|
||||
pass
|
||||
else:
|
||||
match = re.fullmatch(r'scale=w=(-2|\d+):h=(-2|\d+)', value)
|
||||
if not match:
|
||||
raise ValueError('unreviewed filter')
|
||||
w, h = map(int, match.groups())
|
||||
if w == -2 and h > 0:
|
||||
w = max(2, round(width * h / height / 2) * 2)
|
||||
elif h == -2 and w > 0:
|
||||
h = max(2, round(height * w / width / 2) * 2)
|
||||
if w <= 0 or h <= 0 or w % 2 or h % 2 or w > 7680 or h > 4320:
|
||||
raise ValueError('invalid dimensions')
|
||||
out = ['-init_hw_device', f'cuda=nv:{DEVICE}', '-filter_hw_device', 'nv']
|
||||
first = True
|
||||
for key, value in entries:
|
||||
if key == '-i' and first:
|
||||
if not still:
|
||||
out += ['-hwaccel', 'cuda', '-hwaccel_device', DEVICE, '-hwaccel_output_format', 'cuda', '-noautorotate']
|
||||
first = False
|
||||
if key in ('-pix_fmt', '-preset', '-preset:v', '-b_strategy', '-bf', '-tune', '-crf', '-vf', '-filter:v'):
|
||||
continue
|
||||
if key in ('-vcodec', '-c:v', '-codec:v'):
|
||||
value = TARGET
|
||||
out.append(key)
|
||||
if value is not None:
|
||||
out.append(value)
|
||||
prefix = 'format=nv12,hwupload_cuda,' if still else ''
|
||||
out += ['-vf', prefix + f'scale_cuda=w={w}:h={h}:format=nv12',
|
||||
'-preset', 'p4', '-gpu', DEVICE, '-bf', '2', '-rc', 'vbr']
|
||||
rate = next((v for k, v in entries if k in ('-maxrate:v', '-maxrate')), None)
|
||||
if rate and not any(k in ('-b:v', '-vb') for k, _ in entries):
|
||||
out += ['-b:v', rate]
|
||||
out.append(args[-1])
|
||||
uuid = None
|
||||
try:
|
||||
uuid = json.loads((STATE / 'inputs' / (Path(inputs[0]).name + '.json')).read_text())['uuid']
|
||||
except (OSError, ValueError, KeyError):
|
||||
pass
|
||||
return out, {'job_uuid': uuid, 'source_codec': codec, 'target_codec': 'hevc' if TARGET == 'hevc_nvenc' else 'h264', 'encoder': TARGET,
|
||||
'source_width': width, 'source_height': height, 'target_width': w, 'target_height': h,
|
||||
'duration': float(data.get('format', {}).get('duration', 0) or 0), 'static_image_upload': still}
|
||||
|
||||
|
||||
def encoder_util():
|
||||
try:
|
||||
p = subprocess.run(['nvidia-smi.exe', '-i', DEVICE, '--query-gpu=utilization.encoder',
|
||||
'--format=csv,noheader,nounits'], capture_output=True, text=True, timeout=3)
|
||||
return float(p.stdout.strip()) if p.returncode == 0 else None
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
args, info = rewrite(sys.argv[1:])
|
||||
if not REAL.is_file():
|
||||
raise ValueError('ffmpeg missing')
|
||||
if info is None:
|
||||
sys.exit(subprocess.call([str(REAL), *args]))
|
||||
start = time.monotonic()
|
||||
# Do not persist ffmpeg command lines, media paths, stderr, or signed URLs.
|
||||
p = subprocess.Popen([str(REAL), *args], stdout=sys.stdout, stderr=subprocess.PIPE,
|
||||
text=True, encoding='utf8', errors='replace')
|
||||
progress = {}
|
||||
def consume():
|
||||
for line in p.stderr:
|
||||
sys.stderr.write(line); sys.stderr.flush()
|
||||
for k, v in re.findall(r'(frame|fps|speed)=\s*([\d.]+)', line):
|
||||
progress[k] = float(v)
|
||||
reader = threading.Thread(target=consume, daemon=True); reader.start()
|
||||
samples = []
|
||||
while p.poll() is None:
|
||||
value = encoder_util()
|
||||
if value is not None:
|
||||
samples.append(value)
|
||||
time.sleep(.2)
|
||||
reader.join(timeout=2)
|
||||
seconds = time.monotonic() - start
|
||||
maximum = max(samples) if samples else None
|
||||
# Short jobs can end inside the utilization averaging interval. They still
|
||||
# explicitly use NVENC; boot/driver self-test provides mandatory GPU proof.
|
||||
no_signal = seconds >= 3 and (maximum is None or maximum == 0)
|
||||
rc = p.returncode or (78 if no_signal else 0)
|
||||
event('encode', **info, seconds=seconds, returncode=rc, ffmpeg_returncode=p.returncode,
|
||||
engine_busy_max=maximum, engine_samples=len(samples), engine_counters_readable=bool(samples),
|
||||
frames=progress.get('frame'), ffmpeg_fps=progress.get('fps'), ffmpeg_speed=progress.get('speed'),
|
||||
realtime_factor=info['duration'] / seconds if info['duration'] else None)
|
||||
if rc:
|
||||
event('gpu_failure', reason='nvenc_transcode_failed' if p.returncode else 'no_encoder_signal')
|
||||
sys.exit(rc)
|
||||
except (ValueError, OSError, subprocess.SubprocessError, KeyError, StopIteration):
|
||||
event('gpu_failure', reason='unreviewed_or_unavailable_nvenc')
|
||||
print('PT-WORKER: NVENC adapter refused command; drain required', file=sys.stderr)
|
||||
sys.exit(78)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
253
infra/peertube-workers/win/ffmpeg-qsv.py
Normal file
253
infra/peertube-workers/win/ffmpeg-qsv.py
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Pinned PeerTube runner 0.2.0 FFmpeg adapter (native Windows).
|
||||
|
||||
Port of infra/peertube-workers/ffmpeg-qsv.py from the validated Linux QSV worker.
|
||||
Same contract: rewrite known runner command shapes into explicit QSV decode ->
|
||||
hardware frames -> scale_qsv -> h264_qsv; never execute a CPU video encoder;
|
||||
fail closed (exit 78) on anything unreviewed.
|
||||
|
||||
Windows deltas from the Linux adapter:
|
||||
- real ffmpeg/ffprobe are sibling .exe files of this shim (the runner resolves
|
||||
FFMPEG_PATH with fs.exists, then spawns it without a shell, so the shim
|
||||
MUST be a real .exe: a .cmd/.bat wrapper is rejected by Node >= 18.20/20.12
|
||||
with EINVAL, and fluent-ffmpeg passes no `shell` option).
|
||||
- QSV device init is `-init_hw_device qsv=qs` (D3D11 child device by default;
|
||||
the libmfx/oneVPL dispatcher DLLs must sit next to ffmpeg.exe).
|
||||
- "hardware really did it" telemetry is the PDH GPU Engine counter
|
||||
\\GPU Engine(pid_<pid>_*)\\Utilization Percentage over engtype_VideoEncode
|
||||
instances, sampled via typeperf (i915 DRM fdinfo does not exist on Windows).
|
||||
If the counters are readable and stay exactly 0 for the whole run, the run
|
||||
is recorded as a GPU failure even on exit 0.
|
||||
- capability queries (-version/-encoders/...) are passed through untouched.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
REAL = Path(os.environ.get('PT_FFMPEG_EXE', str(HERE / 'ffmpeg.exe')))
|
||||
PROBE = Path(os.environ.get('PT_FFPROBE_EXE', str(HERE / 'ffprobe.exe')))
|
||||
# Optional child-device spec for the QSV device (e.g. 'd3d11va', 'd3d11va:1').
|
||||
# Empty means the default adapter. Verify on first hardware contact.
|
||||
DEVICE = os.environ.get('PT_QSV_DEVICE', '').strip()
|
||||
STATE = Path(os.environ.get('PT_STATE_DIR', r'C:\pt-worker\state'))
|
||||
|
||||
PASSTHROUGH = (['-version'], ['-formats'], ['-encoders'], ['-filters'],
|
||||
['-codecs'], ['-hwaccels'])
|
||||
|
||||
|
||||
def event(kind, **fields):
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
line = json.dumps({'time': time.time(), 'kind': kind, **fields}) + '\n'
|
||||
try:
|
||||
with open(STATE / 'events.jsonl', 'a', encoding='utf8') as handle:
|
||||
handle.write(line)
|
||||
except OSError:
|
||||
pass # telemetry loss must never fail a job
|
||||
|
||||
|
||||
def probe(path):
|
||||
p = subprocess.run([str(PROBE), '-v', 'error', '-show_streams', '-show_format',
|
||||
'-of', 'json', str(path)], capture_output=True, text=True, timeout=30)
|
||||
if p.returncode:
|
||||
raise ValueError('input probe failed')
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def csv_cells(line):
|
||||
return [cell.strip('"') for cell in line.split('","')]
|
||||
|
||||
|
||||
def as_float(text):
|
||||
try:
|
||||
return float(text)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
try: # locale decimal-comma fallback
|
||||
return float(text.replace('.', '').replace(',', '.'))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def video_encode_busy(pid):
|
||||
"""Sum of PDH \\GPU Engine(pid_<pid>_*)\\Utilization Percentage over
|
||||
engtype_VideoEncode instances, one 1-second typeperf sample.
|
||||
|
||||
Returns (busy_or_None, readable). Counter names and decimal points are
|
||||
locale-sensitive; unparseable samples count as not-readable, never as zero.
|
||||
"""
|
||||
counter = rf'\GPU Engine(pid_{pid}_*)\Utilization Percentage'
|
||||
try:
|
||||
p = subprocess.run(['typeperf', counter, '-sc', '1'], capture_output=True,
|
||||
text=True, timeout=15)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None, False
|
||||
if p.returncode:
|
||||
return None, False
|
||||
rows = [line for line in (p.stdout or '').splitlines() if line.startswith('"')]
|
||||
if len(rows) < 2:
|
||||
return None, False
|
||||
headers = list(csv_cells(rows[0]))
|
||||
values = list(csv_cells(rows[1]))
|
||||
total, seen = 0.0, False
|
||||
for index, header in enumerate(headers):
|
||||
if 'engtype_VideoEncode' not in header.lower() or index >= len(values):
|
||||
continue
|
||||
number = as_float(values[index])
|
||||
if number is None:
|
||||
return None, False
|
||||
seen = True
|
||||
total += number
|
||||
return (total if seen else None), seen
|
||||
|
||||
|
||||
def rewrite(args, input_probe=None):
|
||||
# fluent-ffmpeg queries capabilities before building commands. These do no encode.
|
||||
if list(args) in [list(p) for p in PASSTHROUGH]:
|
||||
return args, None
|
||||
pairs = list(zip(args, args[1:]))
|
||||
video_options = [(k, v) for k, v in pairs if k in ('-vcodec', '-c:v', '-codec:v')]
|
||||
if len(video_options) > 1:
|
||||
raise ValueError('multiple video codec options refused')
|
||||
if '-vn' in args and not video_options:
|
||||
return args, None # audio-only VOD renditions have no video encode
|
||||
if not video_options or video_options[0][1] != 'libx264':
|
||||
raise ValueError('unknown runner command or video encoder refused')
|
||||
if '-filter_complex' in args or '-hwaccel' in args or '-init_hw_device' in args:
|
||||
raise ValueError('unreviewed filter/device override refused')
|
||||
inputs = [v for k, v in pairs if k == '-i']
|
||||
if not inputs:
|
||||
raise ValueError('missing video input')
|
||||
data = input_probe if input_probe is not None else probe(inputs[0])
|
||||
stream = next((s for s in data['streams'] if s['codec_type'] == 'video'), None)
|
||||
if stream is None:
|
||||
raise ValueError('missing video stream')
|
||||
codec = stream['codec_name']
|
||||
still = codec in ('png', 'mjpeg', 'webp') and '-loop' in args
|
||||
decoder = {'h264': 'h264_qsv', 'hevc': 'hevc_qsv', 'vp9': 'vp9_qsv', 'av1': 'av1_qsv'}.get(codec)
|
||||
if not decoder and not still:
|
||||
raise ValueError('source codec has no reviewed QSV decode path')
|
||||
if stream.get('pix_fmt') not in ('yuv420p', 'yuv420p10le', 'nv12', 'p010le') and not still:
|
||||
raise ValueError('unsupported pixel format; CPU decode fallback refused')
|
||||
if stream.get('tags', {}).get('rotate') or any(s.get('rotation', 0) for s in stream.get('side_data_list', [])):
|
||||
raise ValueError('rotated source requires reviewed GPU transpose path')
|
||||
width, height = stream['width'], stream['height']
|
||||
duration = float(data.get('format', {}).get('duration', 0) or 0)
|
||||
w, h = width // 2 * 2, height // 2 * 2
|
||||
output = ['-init_hw_device', f'qsv=qs:{DEVICE}' if DEVICE else 'qsv=qs',
|
||||
'-filter_hw_device', 'qs']
|
||||
i, first_input = 0, True
|
||||
while i < len(args):
|
||||
key = args[i]
|
||||
val = args[i + 1] if i + 1 < len(args) else None
|
||||
if key == '-i' and first_input:
|
||||
if not still:
|
||||
output += ['-hwaccel', 'qsv', '-hwaccel_output_format', 'qsv', '-c:v', decoder, '-noautorotate']
|
||||
output += [key, val]
|
||||
first_input = False
|
||||
i += 2
|
||||
elif key in ('-vcodec', '-c:v', '-codec:v'):
|
||||
output += [key, 'h264_qsv']
|
||||
i += 2
|
||||
elif key in ('-pix_fmt', '-preset', '-preset:v', '-b_strategy', '-bf', '-tune', '-crf'):
|
||||
i += 2
|
||||
elif key in ('-vf', '-filter:v'):
|
||||
if val == 'scale=trunc(iw/2)*2:trunc(ih/2)*2' and still:
|
||||
w, h = width // 2 * 2, height // 2 * 2
|
||||
else:
|
||||
match = re.fullmatch(r'scale=w=(-2|\d+):h=(-2|\d+)', val or '')
|
||||
if not match:
|
||||
raise ValueError('unreviewed software filter refused')
|
||||
w, h = map(int, match.groups())
|
||||
if w == -2 and h > 0:
|
||||
w = max(2, round(width * h / height / 2) * 2)
|
||||
elif h == -2 and w > 0:
|
||||
h = max(2, round(height * w / width / 2) * 2)
|
||||
if w <= 0 or h <= 0 or w % 2 or h % 2:
|
||||
raise ValueError('invalid output dimensions')
|
||||
prefix = 'format=nv12,hwupload=extra_hw_frames=64,' if still else ''
|
||||
output += [key, prefix + f'scale_qsv=w={w}:h={h}:format=nv12']
|
||||
i += 2
|
||||
else:
|
||||
output.append(key)
|
||||
i += 1
|
||||
if '-vf' not in args and '-filter:v' not in args:
|
||||
prefix = 'format=nv12,hwupload=extra_hw_frames=64,' if still else ''
|
||||
output[-1:-1] = ['-vf', prefix + f'scale_qsv=w={width // 2 * 2}:h={height // 2 * 2}:format=nv12']
|
||||
# Keep PeerTube's rate cap and GOP; QSV has no x264 CRF mode.
|
||||
rate = next((v for k, v in pairs if k in ('-maxrate:v', '-maxrate')), None)
|
||||
extra = ['-preset', 'medium', '-bf', '2']
|
||||
if rate and not any(k in ('-b:v', '-vb') for k, _ in pairs):
|
||||
extra += ['-b:v', rate]
|
||||
output[-1:-1] = extra
|
||||
job_uuid = None
|
||||
try:
|
||||
job_uuid = json.loads((STATE / 'inputs' / (Path(inputs[0]).name + '.json')).read_text(encoding='utf8'))['uuid']
|
||||
except (OSError, ValueError, KeyError):
|
||||
pass
|
||||
return output, {'job_uuid': job_uuid, 'source_codec': codec, 'target_codec': 'h264', 'encoder': 'h264_qsv',
|
||||
'source_width': width, 'source_height': height, 'duration': duration,
|
||||
'target_width': w, 'target_height': h,
|
||||
'static_image_upload': still}
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
if not REAL.is_file():
|
||||
raise ValueError('real ffmpeg.exe not found next to shim')
|
||||
args, info = rewrite(sys.argv[1:])
|
||||
if info is None:
|
||||
os.execv(str(REAL), [str(REAL), *args])
|
||||
started = time.monotonic()
|
||||
p = subprocess.Popen([str(REAL), *args], stderr=subprocess.PIPE, text=True,
|
||||
encoding='utf8', errors='replace')
|
||||
progress = {}
|
||||
def stderr_progress():
|
||||
for line in p.stderr:
|
||||
sys.stderr.write(line)
|
||||
sys.stderr.flush()
|
||||
for key, value in re.findall(r'(frame|fps|speed)=\s*([\d.]+)', line):
|
||||
try:
|
||||
progress[key] = float(value)
|
||||
except ValueError:
|
||||
pass
|
||||
reader = threading.Thread(target=stderr_progress, daemon=True)
|
||||
reader.start()
|
||||
maximum, samples, readable = None, 0, False
|
||||
while p.poll() is None:
|
||||
busy, ok = video_encode_busy(p.pid)
|
||||
readable |= ok
|
||||
if ok:
|
||||
samples += 1
|
||||
if busy is not None:
|
||||
maximum = busy if maximum is None else max(maximum, busy)
|
||||
time.sleep(1.0)
|
||||
elapsed = time.monotonic() - started
|
||||
reader.join(timeout=2)
|
||||
event('encode', **info, seconds=elapsed, returncode=p.returncode,
|
||||
engine_busy_max=maximum, engine_samples=samples,
|
||||
engine_counters_readable=readable,
|
||||
ffmpeg_fps=progress.get('fps'), frames=progress.get('frame'),
|
||||
ffmpeg_speed=progress.get('speed'),
|
||||
realtime_factor=info['duration'] / elapsed if info['duration'] else None)
|
||||
if p.returncode:
|
||||
event('gpu_failure', reason='qsv_transcode_failed')
|
||||
elif readable and samples >= 3 and maximum == 0:
|
||||
# Exit 0 with a readable VideoEncode counter pinned at 0 is NOT
|
||||
# proof of hardware encoding; fail closed like the Linux fdinfo gate.
|
||||
event('gpu_failure', reason='no_video_encode_utilization')
|
||||
sys.exit(p.returncode)
|
||||
except (ValueError, OSError, subprocess.SubprocessError, KeyError) as e:
|
||||
# Messages are fixed strings; never expose source URLs or runner credentials.
|
||||
event('gpu_failure', reason=type(e).__name__)
|
||||
print('PT-WORKER: QSV adapter refused command; worker must drain', file=sys.stderr)
|
||||
sys.exit(78)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
55
infra/peertube-workers/win/ipc-nvenc.cjs
Normal file
55
infra/peertube-workers/win/ipc-nvenc.cjs
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
// Same IPC protocol/library as the official runner 0.2.0; Windows named-pipe variant
|
||||
// of infra/peertube-workers/ipc.cjs. Registration secret comes from stdin, never argv.
|
||||
//
|
||||
// Pipe-name derivation (verified in the shipped 0.2.0 bundle):
|
||||
// ConfigManager id defaults to "default"; socket path is
|
||||
// join(envPaths("peertube-runner", suffix "nodejs").data, id, "peertube-runner.sock")
|
||||
// = %LOCALAPPDATA%\peertube-runner-nodejs\Data\default\peertube-runner.sock
|
||||
// The bundled @peertube/net-ipc rewrites any path on win32:
|
||||
// `\\\\.\\pipe\\${path.replace(/^\//, "").replace(/\//g, "-")}`
|
||||
// so we pass the RAW Windows path and never build the \\.\pipe\ prefix ourselves.
|
||||
const root = process.argv[3] || process.env.PT_ROOT || 'C:\\pt-worker';
|
||||
process.env.APPDATA = require('node:path').join(root, 'profile', 'Roaming');
|
||||
process.env.LOCALAPPDATA = require('node:path').join(root, 'profile', 'Local');
|
||||
const { Client } = require(require('node:path').join(root, 'runner', 'node_modules', '@peertube', 'net-ipc'));
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const action = process.argv[2];
|
||||
const base = process.env.LOCALAPPDATA || path.join(os.homedir(), 'AppData', 'Local');
|
||||
const id = process.env.PT_RUNNER_ID || 'default';
|
||||
const sock = path.join(base, 'peertube-runner-nodejs', 'Data', id, 'peertube-runner.sock');
|
||||
let client = new Client({ path: sock, reconnect: false });
|
||||
const deadline = setTimeout(() => { client.destroy(); process.exit(1); }, 15000);
|
||||
(async () => {
|
||||
try {
|
||||
let connected = false;
|
||||
for (let attempt = 0; attempt < 20; attempt++) {
|
||||
try { await client.connect(); connected = true; break; }
|
||||
catch { client.destroy(); await new Promise(resolve => setTimeout(resolve, 500)); client = new Client({path: sock, reconnect: false}); }
|
||||
}
|
||||
if (!connected) throw new Error('IPC unavailable');
|
||||
let request = { type: action };
|
||||
if (action === 'register') {
|
||||
const cfg = JSON.parse(fs.readFileSync(process.env.PT_WORKER_CONFIG || path.join(root, 'state', 'worker.json'), 'utf8').replace(/^\uFEFF/, ''));
|
||||
const proof = JSON.parse(fs.readFileSync(path.join(root, 'state', 'nvenc.json'), 'utf8').replace(/^\uFEFF/, ''));
|
||||
const identity = require('node:child_process').execFileSync('nvidia-smi.exe', ['-i',String(cfg.nvenc_device || 0),'--query-gpu=uuid,driver_version','--format=csv,noheader,nounits'],{timeout:5000,encoding:'utf8'}).trim().split(/,\s*/);
|
||||
if (identity[0] !== proof.gpu_uuid || identity[1] !== proof.driver_version) throw new Error('changed GPU');
|
||||
if (!proof.nvenc_h264_ok || !proof.nvenc_hevc_ok || Date.now()/1000 - proof.tested_at > 86400) throw new Error('proof missing');
|
||||
const registrationToken = fs.readFileSync(0, 'utf8').trim();
|
||||
if (!registrationToken.startsWith('ptrrt-')) throw new Error('bad token');
|
||||
request = { type: 'register', url: cfg.peertube_url, runnerName: cfg.name, registrationToken,
|
||||
runnerDescription: 'Windows NVENC only; VOD; AC-only admission; concurrency 2' };
|
||||
} else if (!['list-jobs', 'list-registered', 'graceful-shutdown'].includes(action)) throw new Error('bad action');
|
||||
const result = await client.request(request);
|
||||
if (!result.success) throw new Error('IPC failed');
|
||||
if (action === 'list-jobs') {
|
||||
// Drop payloads (can contain URLs); status consumers only need counts.
|
||||
console.log(JSON.stringify({ concurrency: result.data.concurrency, active_jobs: result.data.processingJobs.length }));
|
||||
} else if (action === 'list-registered') console.log(JSON.stringify({ registered_count: result.data.registeredInstances?.length || result.data.instances?.length || 0 }));
|
||||
else console.log(JSON.stringify({ success: true }));
|
||||
} catch { console.error('Runner IPC failed; inspect local service health'); process.exitCode = 1; }
|
||||
finally { clearTimeout(deadline); client.destroy(); }
|
||||
})();
|
||||
43
infra/peertube-workers/win/ipc-win.cjs
Normal file
43
infra/peertube-workers/win/ipc-win.cjs
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
// Same IPC protocol/library as the official runner 0.2.0; Windows named-pipe variant
|
||||
// of infra/peertube-workers/ipc.cjs. Registration secret comes from stdin, never argv.
|
||||
//
|
||||
// Pipe-name derivation (verified in the shipped 0.2.0 bundle):
|
||||
// ConfigManager id defaults to "default"; socket path is
|
||||
// join(envPaths("peertube-runner", suffix "nodejs").data, id, "peertube-runner.sock")
|
||||
// = %LOCALAPPDATA%\peertube-runner-nodejs\Data\default\peertube-runner.sock
|
||||
// The bundled @peertube/net-ipc rewrites any path on win32:
|
||||
// `\\\\.\\pipe\\${path.replace(/^\//, "").replace(/\//g, "-")}`
|
||||
// so we pass the RAW Windows path and never build the \\.\pipe\ prefix ourselves.
|
||||
const { Client } = require('./runtime/node_modules/@peertube/net-ipc');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const action = process.argv[2];
|
||||
const base = process.env.LOCALAPPDATA || path.join(os.homedir(), 'AppData', 'Local');
|
||||
const id = process.env.PT_RUNNER_ID || 'default';
|
||||
const sock = path.join(base, 'peertube-runner-nodejs', 'Data', id, 'peertube-runner.sock');
|
||||
const client = new Client({ path: sock, reconnect: false });
|
||||
const deadline = setTimeout(() => { client.destroy(); process.exit(1); }, 15000);
|
||||
(async () => {
|
||||
try {
|
||||
await client.connect();
|
||||
let request = { type: action };
|
||||
if (action === 'register') {
|
||||
const cfg = JSON.parse(fs.readFileSync(process.env.PT_WORKER_CONFIG || 'C:\\pt-worker\\state\\worker.json', 'utf8'));
|
||||
const registrationToken = fs.readFileSync(0, 'utf8').trim();
|
||||
if (!registrationToken.startsWith('ptrrt-')) throw new Error('bad token');
|
||||
request = { type: 'register', url: cfg.peertube_url, runnerName: cfg.name, registrationToken,
|
||||
runnerDescription: 'VOD only; QSV required; admission controlled' };
|
||||
} else if (!['list-jobs', 'list-registered', 'graceful-shutdown'].includes(action)) throw new Error('bad action');
|
||||
const result = await client.request(request);
|
||||
if (!result.success) throw new Error('IPC failed');
|
||||
if (action === 'list-jobs') {
|
||||
// Drop payloads (can contain URLs); status consumers only need counts.
|
||||
console.log(JSON.stringify({ concurrency: result.data.concurrency, active_jobs: result.data.processingJobs.length }));
|
||||
} else if (action === 'list-registered') console.log(JSON.stringify(result.data));
|
||||
else console.log(JSON.stringify({ success: true }));
|
||||
} catch { console.error('Runner IPC failed; inspect local service health'); process.exitCode = 1; }
|
||||
finally { clearTimeout(deadline); client.destroy(); }
|
||||
})();
|
||||
565
infra/peertube-workers/win/pt-worker-nvenc.py
Normal file
565
infra/peertube-workers/win/pt-worker-nvenc.py
Normal file
|
|
@ -0,0 +1,565 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Native Windows NVENC admission controller. Linux Meter/Admission semantics,
|
||||
Windows power/thermal/GPU sensors, isolated profile, and official named-pipe IPC.
|
||||
The supervisor owns runner processes; admission only publishes fail-closed gates.
|
||||
"""
|
||||
import argparse
|
||||
import ctypes
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
import tomllib
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(os.environ.get('PT_ROOT', r'C:\pt-worker'))
|
||||
os.environ.update(APPDATA=str(ROOT / 'profile' / 'Roaming'), LOCALAPPDATA=str(ROOT / 'profile' / 'Local'), PT_STATE_DIR=str(ROOT / 'state'), PT_WORKER_CONFIG=str(ROOT / 'state' / 'worker.json'))
|
||||
CONFIG = ROOT / 'state' / 'worker.json'
|
||||
STATE = ROOT / 'state'
|
||||
RUNNER_CONFIG = Path(os.environ.get('APPDATA', str(Path.home() / 'AppData' / 'Roaming'))) / 'peertube-runner-nodejs' / 'Config' / 'default' / 'config.toml'
|
||||
|
||||
DEFAULTS = {
|
||||
'require_ac': True, 'min_disk_free_bytes': 20 * 1024 ** 3, 'max_temperature_c': 85,
|
||||
'min_download_mbps': 10, 'min_upload_mbps': 10, 'max_latency_ms': 150,
|
||||
'allow_derp': False, 'recovery_factor': 1.25, 'good_samples': 3, 'bad_samples': 2,
|
||||
'bandwidth_ttl_seconds': 900, 'transfer_failure_cooldown_seconds': 60,
|
||||
'thermal_cooldown_seconds': 120, 'poll_seconds': 5,
|
||||
}
|
||||
# Last-resort resource gates that hold even while admission policy is disabled.
|
||||
# Same thresholds as the Linux installer defaults (15 GiB RAM floor for a
|
||||
# 2-concurrency QSV worker, 6 GiB min per-job scratch); no CPU fallback exists.
|
||||
HARD_FLOORS = {'ram_total_bytes': 15 * 1024 ** 3, 'min_scratch_bytes': 6 * 1024 ** 3}
|
||||
|
||||
|
||||
def run(args, timeout=10):
|
||||
return subprocess.run(args, capture_output=True, text=True, timeout=timeout)
|
||||
|
||||
|
||||
def read(path, default=None):
|
||||
try:
|
||||
return json.loads(Path(path).read_text(encoding='utf-8-sig'))
|
||||
except (OSError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def config():
|
||||
value = read(CONFIG)
|
||||
if not value or not value.get('peertube_url') or not value.get('tailscale_peer'):
|
||||
raise ValueError('worker.json needs peertube_url and tailscale_peer')
|
||||
return {**DEFAULTS, **value}
|
||||
|
||||
|
||||
def ipc(action, node_binary='node'):
|
||||
"""Official runner IPC via ipc-win.cjs; the runner must be running in server mode."""
|
||||
env = {**os.environ, 'PT_STATE_DIR': str(STATE)}
|
||||
p = run([node_binary, str(ROOT / 'bin' / 'ipc-nvenc.cjs'), action], 15)
|
||||
if p.returncode:
|
||||
raise RuntimeError('runner IPC unavailable')
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def registered(cfg):
|
||||
try:
|
||||
data = tomllib.loads(RUNNER_CONFIG.read_text(encoding='utf-8-sig'))
|
||||
return any(r['url'].rstrip('/') == cfg['peertube_url'].rstrip('/') and
|
||||
r['runnerName'] == cfg['name'] and r.get('runnerToken')
|
||||
for r in data.get('registeredInstances', []))
|
||||
except (OSError, ValueError, KeyError):
|
||||
return False
|
||||
|
||||
|
||||
def network(cfg):
|
||||
result = {'tailscale_ip': None, 'tailscale_path': 'unknown', 'latency_ms': None,
|
||||
'network_ok': False, 'api_ok': False, 'peer_online': False}
|
||||
try:
|
||||
ts = json.loads(run([cfg.get('tailscale_binary', r'C:\Program Files\Tailscale\tailscale.exe'), 'status', '--json'], 5).stdout)
|
||||
result['tailscale_ip'] = next((a for a in ts.get('Self', {}).get('TailscaleIPs', []) if ':' not in a), None)
|
||||
peers = list(ts.get('Peer', {}).values())
|
||||
peer = next((p for p in peers if cfg['tailscale_peer'] in p.get('TailscaleIPs', []) or
|
||||
p.get('HostName') == cfg['tailscale_peer'] or
|
||||
p.get('DNSName', '').split('.')[0] == cfg['tailscale_peer']), None)
|
||||
if ts.get('BackendState') != 'Running' or not peer or not peer.get('Online'):
|
||||
return result
|
||||
result['peer_online'] = True
|
||||
p = run([cfg.get('tailscale_binary', r'C:\Program Files\Tailscale\tailscale.exe'), 'ping', '--c=1', '--until-direct=false', '--timeout=3s', cfg['tailscale_peer']], 5)
|
||||
if 'pong from' not in p.stdout:
|
||||
return result
|
||||
result['tailscale_path'] = 'DERP' if 'DERP(' in p.stdout else 'direct'
|
||||
latency = re.search(r'in ([\d.]+)ms', p.stdout)
|
||||
if latency:
|
||||
result['latency_ms'] = float(latency[1])
|
||||
started = time.monotonic()
|
||||
with urllib.request.urlopen(cfg['peertube_url'].rstrip('/') + '/api/v1/config', timeout=4) as response:
|
||||
version = json.load(response).get('serverVersion')
|
||||
result['api_ok'] = version == cfg.get('peertube_version', '7.3.0')
|
||||
result['network_ok'] = result['api_ok']
|
||||
except (OSError, ValueError, subprocess.SubprocessError, urllib.error.URLError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def transfer_probe(cfg, megabytes=8):
|
||||
"""Bounded probe against pt-worker-netprobe (same server-side probe as Linux)."""
|
||||
url = cfg.get('probe_url')
|
||||
rates = {}
|
||||
if not url:
|
||||
return rates
|
||||
payload = os.urandom(megabytes * 1024 * 1024)
|
||||
for direction, request in (('download', urllib.request.Request(url.rstrip('/') + '/download',
|
||||
data=None, method='GET')),
|
||||
('upload', None)):
|
||||
try:
|
||||
started = time.monotonic()
|
||||
if direction == 'download':
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
seen = 0
|
||||
while chunk := response.read(1 << 20):
|
||||
seen += len(chunk)
|
||||
if seen >= megabytes * 1024 * 1024:
|
||||
break
|
||||
seconds = time.monotonic() - started
|
||||
if seen < 65536:
|
||||
continue
|
||||
else:
|
||||
req = urllib.request.Request(url.rstrip('/') + '/upload', data=payload, method='PUT')
|
||||
urllib.request.urlopen(req, timeout=30).read()
|
||||
seconds = time.monotonic() - started
|
||||
rates[direction] = {'mbps': megabytes * 1024 * 1024 * 8 / 1e6 / seconds, 'time': time.time(),
|
||||
'bytes': megabytes * 1024 * 1024, 'seconds': seconds}
|
||||
except (OSError, urllib.error.URLError, ValueError):
|
||||
rates[direction + '_failed'] = True
|
||||
return rates
|
||||
|
||||
|
||||
def get_system_power_status():
|
||||
"""Win32 GetSystemPowerStatus; dict or None when unsupported/unknown."""
|
||||
class PLS(ctypes.Structure):
|
||||
_fields_ = [('ACLineStatus', ctypes.c_byte), ('BatteryFlag', ctypes.c_byte),
|
||||
('BatteryLifePercent', ctypes.c_byte), ('Reserved1', ctypes.c_byte),
|
||||
('BatteryLifeTime', ctypes.c_uint32), ('BatteryFullLifeTime', ctypes.c_uint32)]
|
||||
status = PLS()
|
||||
if not ctypes.windll.kernel32.GetSystemPowerStatus(ctypes.byref(status)):
|
||||
return None
|
||||
ac = {0: False, 1: True}.get(status.ACLineStatus) # 255 == unknown -> None
|
||||
percentage = status.BatteryLifePercent
|
||||
return {'on_ac': ac, 'battery_flag': status.BatteryFlag,
|
||||
'battery_percent': None if percentage in (0, 255) else percentage}
|
||||
|
||||
|
||||
_thermal_cache = {'time': 0, 'temperature_c': None, 'temperatures_c': []}
|
||||
|
||||
def sensors():
|
||||
result = {'temperature_c': None, 'temperatures_c': [], 'on_ac': None, 'battery': [],
|
||||
'throttle_counts': {}, 'ram_total_bytes': None, 'ram_available_bytes': None,
|
||||
'load_average': None, 'wsl2': False}
|
||||
try:
|
||||
if time.time() - _thermal_cache['time'] < 300:
|
||||
raise LookupError('cached')
|
||||
_thermal_cache['time'] = time.time()
|
||||
p = run(['powershell.exe', '-NoProfile', '-NonInteractive', '-Command',
|
||||
'(Get-CimInstance -Namespace root/wmi -ClassName MSAcpi_ThermalZoneTemperature -ErrorAction Stop | '
|
||||
'Measure-Object -Property CurrentTemperature -Maximum).Maximum'], 12)
|
||||
if p.returncode == 0:
|
||||
value = float(p.stdout.strip()) / 10.0 - 273.15
|
||||
if 0 < value < 120:
|
||||
result['temperature_c'] = round(value, 1)
|
||||
result['temperatures_c'] = [result['temperature_c']]
|
||||
_thermal_cache.update(temperature_c=result['temperature_c'], temperatures_c=result['temperatures_c'])
|
||||
except (OSError, ValueError, LookupError, subprocess.SubprocessError):
|
||||
result.update(temperature_c=_thermal_cache['temperature_c'], temperatures_c=_thermal_cache['temperatures_c'])
|
||||
power = get_system_power_status()
|
||||
if power:
|
||||
result['on_ac'] = power['on_ac']
|
||||
if power['battery_percent'] is not None:
|
||||
result['battery'] = [{'capacity': power['battery_percent'],
|
||||
'status': 'on_ac' if power['on_ac'] is True else 'discharging' if power['on_ac'] is False else 'unknown'}]
|
||||
try:
|
||||
import ctypes.wintypes # noqa: F401 (MEMORYSTATUSEX needs no extra types)
|
||||
class MEMORYSTATUSEX(ctypes.Structure):
|
||||
_fields_ = [('dwLength', ctypes.c_uint32), ('dwMemoryLoad', ctypes.c_uint32),
|
||||
('ullTotalPhys', ctypes.c_uint64), ('ullAvailPhys', ctypes.c_uint64),
|
||||
('ullTotalPageFile', ctypes.c_uint64), ('ullAvailPageFile', ctypes.c_uint64),
|
||||
('ullTotalVirtual', ctypes.c_uint64), ('ullAvailVirtual', ctypes.c_uint64),
|
||||
('ullAvailExtendedVirtual', ctypes.c_uint64)]
|
||||
memory = MEMORYSTATUSEX()
|
||||
memory.dwLength = ctypes.sizeof(MEMORYSTATUSEX)
|
||||
if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(memory)):
|
||||
result['ram_total_bytes'] = memory.ullTotalPhys
|
||||
result['ram_available_bytes'] = memory.ullAvailPhys
|
||||
except (OSError, AttributeError, ValueError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def disk_free(path=str(ROOT)):
|
||||
try:
|
||||
free = ctypes.c_ulonglong()
|
||||
if ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(path), None, None, ctypes.byref(free)):
|
||||
return free.value
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def gpu_identity():
|
||||
try:
|
||||
p = run(['powershell.exe', '-NoProfile', '-NonInteractive', '-Command',
|
||||
'Get-CimInstance Win32_VideoController | Select-Object Name,DriverVersion,Status | ConvertTo-Json'], 12)
|
||||
data = json.loads(p.stdout or '[]')
|
||||
if isinstance(data, dict):
|
||||
data = [data]
|
||||
return data
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
return []
|
||||
|
||||
|
||||
class Meter:
|
||||
"""Byte-identical semantics to the Linux controller's Meter."""
|
||||
|
||||
def __init__(self):
|
||||
self.data = read(STATE / 'meter.json', {'offset': 0, 'jobs_completed': 0, 'jobs_failed': 0,
|
||||
'transfer_failures': 0, 'active': {}}) or {}
|
||||
|
||||
def invalidate(self):
|
||||
for key in ('download', 'upload'):
|
||||
self.data.pop(key, None)
|
||||
|
||||
def update(self):
|
||||
p = STATE / 'events.jsonl'
|
||||
if not p.exists():
|
||||
return
|
||||
with p.open(encoding='utf-8-sig') as f:
|
||||
if p.stat().st_size < self.data.get('offset', 0):
|
||||
self.data['offset'] = 0
|
||||
f.seek(self.data.get('offset', 0))
|
||||
while True:
|
||||
position = f.tell()
|
||||
line = f.readline()
|
||||
if not line:
|
||||
break
|
||||
if not line.endswith('\n'):
|
||||
f.seek(position)
|
||||
break
|
||||
try:
|
||||
self.consume(json.loads(line))
|
||||
except (ValueError, KeyError, TypeError):
|
||||
pass
|
||||
self.data['offset'] = f.tell()
|
||||
try:
|
||||
(STATE / 'meter.json').write_text(json.dumps(self.data), encoding='utf-8-sig')
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def consume(self, e):
|
||||
kind, stamp = e['kind'], e['time']
|
||||
if kind == 'transfer':
|
||||
if not e['ok']:
|
||||
self.data['transfer_failures'] = self.data.get('transfer_failures', 0) + 1
|
||||
self.data['last_transfer_failure'] = stamp
|
||||
elif e['bytes'] >= 65536 and e['seconds'] > 0:
|
||||
direction = e['direction']
|
||||
rate = e['bytes'] * 8 / e['seconds'] / 1e6
|
||||
old = self.data.get(direction, {}).get('mbps', rate)
|
||||
self.data[direction] = {'mbps': .3 * rate + .7 * old, 'time': stamp,
|
||||
'bytes': e['bytes'], 'seconds': e['seconds']}
|
||||
elif kind == 'gpu_failure':
|
||||
self.data['gpu_failure_time'] = stamp
|
||||
elif kind == 'encode':
|
||||
self.data['last_encode'] = e
|
||||
if e.get('job_uuid'):
|
||||
self.data.setdefault('encodes', {})[e['job_uuid']] = e
|
||||
elif kind == 'job_accept' and e['ok']:
|
||||
self.data.setdefault('active', {})[e['uuid']] = stamp
|
||||
elif kind in ('job_success', 'job_error', 'job_abort') and e['ok']:
|
||||
begun = self.data.setdefault('active', {}).pop(e['uuid'], None)
|
||||
if kind == 'job_success':
|
||||
self.data['jobs_completed'] = self.data.get('jobs_completed', 0) + 1
|
||||
elapsed = stamp - begun if begun else None
|
||||
self.data['last_job_duration_seconds'] = elapsed
|
||||
self.data['last_job_uuid'] = e['uuid']
|
||||
encoded = self.data.setdefault('encodes', {}).pop(e['uuid'], {})
|
||||
duration = encoded.get('duration')
|
||||
self.data['last_job_realtime_factor'] = duration / elapsed if duration and elapsed else None
|
||||
self.data['last_job_encode'] = encoded or None
|
||||
elif kind == 'job_error':
|
||||
self.data['jobs_failed'] = self.data.get('jobs_failed', 0) + 1
|
||||
for path in (STATE / 'inputs').glob('*.json'):
|
||||
if read(path) and read(path).get('uuid') == e['uuid']:
|
||||
try:
|
||||
path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def rates(self, now, ttl):
|
||||
return {f'recent_{k}_mbps': self.data.get(k, {}).get('mbps')
|
||||
if now - self.data.get(k, {}).get('time', 0) < ttl else None
|
||||
for k in ('download', 'upload')}
|
||||
|
||||
|
||||
class Admission:
|
||||
"""Decision gates: PAUSED, GPU_UNAVAILABLE, DRAINING (power), LOW_DISK,
|
||||
THERMAL_THROTTLED, OFFLINE, NETWORK_DEGRADED, BUSY, AVAILABLE.
|
||||
Hysteresis and thresholds identical to the Linux controller."""
|
||||
|
||||
def __init__(self, cfg):
|
||||
self.cfg, self.good, self.bad = cfg, 0, 0
|
||||
self.admitted, self.path, self.throttles = False, None, {}
|
||||
|
||||
def decide(self, sample, metric, control, now):
|
||||
cfg = self.cfg
|
||||
def refuse(state):
|
||||
self.admitted, self.good = False, 0
|
||||
return state
|
||||
if self.path and sample['tailscale_path'] != self.path:
|
||||
self.admitted = False
|
||||
self.good = 0
|
||||
metric.invalidate()
|
||||
self.path = sample['tailscale_path']
|
||||
rates = metric.rates(now, cfg.get('bandwidth_ttl_seconds', 900))
|
||||
sample.update(rates)
|
||||
if control.get('paused', True):
|
||||
return refuse('PAUSED')
|
||||
if not sample['registered']:
|
||||
return refuse('PAUSED')
|
||||
gpu_failed = metric.data.get('gpu_failure_time', 0) >= sample.get('gpu_tested_at', 0)
|
||||
if not sample['gpu_accessible'] or not sample.get('gpu_ready') or gpu_failed:
|
||||
return refuse('GPU_UNAVAILABLE')
|
||||
if cfg.get('require_ac', True) and sample['on_ac'] is not True:
|
||||
return refuse('DRAINING')
|
||||
if sample['disk_free_bytes'] is None or sample['disk_free_bytes'] < cfg.get('min_disk_free_bytes', 20 * 1024 ** 3):
|
||||
return refuse('LOW_DISK')
|
||||
rising = any(v > self.throttles.get(k, v) for k, v in sample['throttle_counts'].items())
|
||||
self.throttles = sample['throttle_counts']
|
||||
if rising or (sample['temperature_c'] is not None and sample['temperature_c'] >= cfg.get('max_temperature_c', 85)):
|
||||
self.thermal_until = now + cfg.get('thermal_cooldown_seconds', 120)
|
||||
if now < getattr(self, 'thermal_until', 0):
|
||||
return refuse('THERMAL_THROTTLED')
|
||||
if not sample['network_ok']:
|
||||
self.admitted, self.good = False, 0
|
||||
return 'OFFLINE'
|
||||
calibrating = control.get('calibrate_until', 0) > now
|
||||
factor = cfg.get('recovery_factor', 1.25) if not self.admitted else 1
|
||||
bad = sample['latency_ms'] is None or sample['latency_ms'] > cfg.get('max_latency_ms', 150)
|
||||
bad |= sample['tailscale_path'] == 'DERP' and not cfg.get('allow_derp', False)
|
||||
if not calibrating:
|
||||
bad |= any(rates[f'recent_{key}_mbps'] is None or rates[f'recent_{key}_mbps'] <
|
||||
cfg.get(f'min_{key}_mbps', 10) * factor for key in ('download', 'upload'))
|
||||
bad |= now - metric.data.get('last_transfer_failure', 0) < cfg.get('transfer_failure_cooldown_seconds', 60)
|
||||
if bad:
|
||||
self.good, self.bad = 0, self.bad + 1
|
||||
if self.bad >= cfg.get('bad_samples', 2) or not self.admitted:
|
||||
self.admitted = False
|
||||
return 'NETWORK_DEGRADED'
|
||||
else:
|
||||
self.bad, self.good = 0, self.good + 1
|
||||
if self.good >= cfg.get('good_samples', 3):
|
||||
self.admitted = True
|
||||
if not self.admitted:
|
||||
return 'NETWORK_DEGRADED'
|
||||
return 'BUSY' if sample['active_jobs'] else 'AVAILABLE'
|
||||
|
||||
|
||||
def atomic(path, value):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix('.tmp-' + str(os.getpid()))
|
||||
tmp.write_text(json.dumps(value), encoding='utf8')
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def gpu_snapshot(cfg):
|
||||
p = run(['nvidia-smi.exe', '-i', str(cfg.get('nvenc_device', 0)),
|
||||
'--query-gpu=name,uuid,driver_version,temperature.gpu,utilization.encoder',
|
||||
'--format=csv,noheader,nounits'], 5)
|
||||
if p.returncode:
|
||||
raise RuntimeError('GPU query failed')
|
||||
name, uuid, driver, temp, util = [v.strip() for v in p.stdout.strip().split(',')]
|
||||
return {'gpu_model': name, 'gpu_uuid': uuid, 'driver_version': driver,
|
||||
'temperature_c': float(temp), 'encoder_util': float(util)}
|
||||
|
||||
|
||||
def boot_hardware():
|
||||
p = run(['powershell.exe', '-NoProfile', '-NonInteractive', '-Command',
|
||||
"[pscustomobject]@{boot_id=(Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToUniversalTime().ToString('o');cpu_model=(Get-CimInstance Win32_Processor | Select-Object -First 1).Name} | ConvertTo-Json -Compress"], 30)
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def collect(cfg, meter, admission, hw):
|
||||
now = time.time()
|
||||
meter.update()
|
||||
sensor = sensors()
|
||||
proof = read(STATE / 'nvenc.json', {}) or {}
|
||||
gpu = {}
|
||||
try:
|
||||
gpu = gpu_snapshot(cfg)
|
||||
except (OSError, ValueError, RuntimeError, subprocess.SubprocessError):
|
||||
pass
|
||||
ready = bool(proof.get('nvenc_h264_ok') and proof.get('nvenc_hevc_ok') and
|
||||
proof.get('boot_id') == hw.get('boot_id') and
|
||||
proof.get('driver_version') == gpu.get('driver_version') and
|
||||
proof.get('gpu_uuid') == gpu.get('gpu_uuid') and
|
||||
proof.get('device') == cfg.get('nvenc_device', 0) and
|
||||
0 <= now - proof.get('tested_at', 0) < 7 * 86400)
|
||||
runner = read(STATE / 'runner.json', {}) or {}
|
||||
running = runner.get('running', False) and now - runner.get('updated_at', 0) < 30
|
||||
jobs, ipc_ok = {'active_jobs': 0, 'concurrency': cfg.get('concurrency', 2)}, not running
|
||||
if running:
|
||||
try:
|
||||
jobs = ipc('list-jobs', cfg['node_binary'])
|
||||
ipc_ok = True
|
||||
except (OSError, ValueError, RuntimeError, subprocess.SubprocessError):
|
||||
jobs['active_jobs'] = None
|
||||
temps = sensor['temperatures_c'] + ([gpu['temperature_c']] if gpu else [])
|
||||
sample = {**network(cfg), **sensor, **hw, 'hostname': cfg['name'], 'updated_at': now,
|
||||
'registered': registered(cfg), 'registration_state': 'registered' if registered(cfg) else 'unregistered',
|
||||
**jobs, 'gpu_accessible': bool(gpu) and not (STATE / 'gpu-unavailable').exists(),
|
||||
'gpu_ready': ready, 'gpu_tested_at': proof.get('tested_at', 0),
|
||||
'disk_free_bytes': disk_free(), 'temperature_c': max(temps, default=None),
|
||||
'temperatures_c': temps, 'throttle_counts': {}, 'qsv': {'qsv_h264_ok': False, 'qsv_hevc_ok': False},
|
||||
'qsv_h264_ok': False, 'qsv_hevc_ok': False,
|
||||
'nvenc_h264_ok': proof.get('nvenc_h264_ok', False), 'nvenc_hevc_ok': proof.get('nvenc_hevc_ok', False),
|
||||
'av1_decode_ok': proof.get('av1_decode_ok'), 'av1_encode_ok': False, 'backend': 'nvidia_nvenc',
|
||||
'gpu_model': gpu.get('gpu_model'), 'driver_version': gpu.get('driver_version'),
|
||||
'tested_at': proof.get('tested_at'), 'runner_running': bool(running),
|
||||
'draining': runner.get('draining', False), 'util': {'cpu': None, 'gpu': {'encoder': gpu.get('encoder_util')}}}
|
||||
if sample['disk_free_bytes'] is None:
|
||||
sample['disk_free_bytes'] = 0
|
||||
state = admission.decide(sample, meter, read(STATE / 'control.json', {'paused': True}), now)
|
||||
if not ipc_ok:
|
||||
state = 'OFFLINE'
|
||||
sample.update(status=state, state=state, admission_allowed=state in ('AVAILABLE', 'BUSY'),
|
||||
jobs_completed=meter.data.get('jobs_completed', 0), jobs_failed=meter.data.get('jobs_failed', 0),
|
||||
last_job_duration_seconds=meter.data.get('last_job_duration_seconds'),
|
||||
last_job_realtime_factor=meter.data.get('last_job_realtime_factor'))
|
||||
sample.update({k: meter.data.get(k) for k in ('transfer_failures', 'last_job_uuid', 'last_job_encode', 'last_encode')})
|
||||
# Linux status keys remain intact; research/fleet aliases are additive.
|
||||
sample.update(registration=sample['registered'], gpu_util=gpu.get('encoder_util'), temps=temps,
|
||||
disk_free=sample['disk_free_bytes'], last_job_duration=sample['last_job_duration_seconds'],
|
||||
last_job_rtf=sample['last_job_realtime_factor'], updated_at=time.time())
|
||||
return sample
|
||||
|
||||
|
||||
def self_test_marker_stale(hw, proof):
|
||||
return not proof.get('tested_at') or time.time() - proof['tested_at'] > 7 * 86400 or proof.get('boot_id') != hw.get('boot_id') or proof.get('driver_version') != hw.get('driver_version')
|
||||
|
||||
|
||||
def daemon():
|
||||
import msvcrt
|
||||
import signal
|
||||
lock = (STATE / 'admission.lock').open('a+b')
|
||||
lock.seek(0); lock.write(b'0'); lock.flush(); lock.seek(0)
|
||||
msvcrt.locking(lock.fileno(), msvcrt.LK_NBLCK, 1)
|
||||
cfg = config()
|
||||
meter, admission = Meter(), Admission(cfg)
|
||||
while True:
|
||||
try:
|
||||
hw = boot_hardware()
|
||||
break
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
atomic(STATE / 'status.json', {'hostname': cfg['name'], 'status': 'OFFLINE', 'state': 'OFFLINE', 'updated_at': time.time(), 'admission_allowed': False})
|
||||
time.sleep(5)
|
||||
atomic(STATE / 'hardware.json', hw)
|
||||
stopped = False
|
||||
def stop(*_):
|
||||
nonlocal stopped
|
||||
stopped = True
|
||||
signal.signal(signal.SIGINT, stop)
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
last_probe, previous_path = 0, None
|
||||
attempted_proof = None
|
||||
while not stopped:
|
||||
try:
|
||||
proof = read(STATE / 'nvenc.json', {}) or {}
|
||||
gpu = gpu_snapshot(cfg)
|
||||
identity = (hw.get('boot_id'), gpu.get('driver_version'))
|
||||
if self_test_marker_stale(hw | {'driver_version': gpu.get('driver_version')}, proof) and identity != attempted_proof and not read(STATE / 'runner.json', {}).get('running'):
|
||||
attempted_proof = identity
|
||||
atomic(STATE / 'status.json', {'updated_at': time.time(), 'status': 'GPU_UNAVAILABLE', 'state': 'GPU_UNAVAILABLE', 'admission_allowed': False})
|
||||
if get_system_power_status() and get_system_power_status()['on_ac'] is True:
|
||||
subprocess.run(['powershell.exe', '-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Bypass', '-File', str(ROOT / 'selftest-nvenc.ps1'), '-Root', str(ROOT), '-Device', str(cfg.get('nvenc_device', 0))], capture_output=True, timeout=90)
|
||||
sample = collect(cfg, meter, admission, hw)
|
||||
if previous_path and previous_path != sample['tailscale_path']:
|
||||
meter.invalidate(); last_probe = 0
|
||||
admission.admitted, admission.good, admission.path = False, 0, None
|
||||
sample['admission_allowed'] = False
|
||||
sample['state'] = sample['status'] = 'NETWORK_DEGRADED'
|
||||
previous_path = sample['tailscale_path']
|
||||
atomic(STATE / 'status.json', sample)
|
||||
control = read(STATE / 'control.json', {'paused': True})
|
||||
if not control.get('paused', True) and sample['network_ok'] and sample['on_ac'] is True and \
|
||||
sample['tailscale_path'] == 'direct' and sample['active_jobs'] == 0 and \
|
||||
time.time() - last_probe >= cfg.get('probe_interval_seconds', 300):
|
||||
# Close admission and wait for the official graceful drain before measuring.
|
||||
sample.update(admission_allowed=False, state='DRAINING', status='DRAINING')
|
||||
atomic(STATE / 'status.json', sample)
|
||||
if not sample['runner_running']:
|
||||
rates = transfer_probe(cfg)
|
||||
for direction in ('download', 'upload'):
|
||||
value = rates.get(direction)
|
||||
meter.consume({'kind': 'transfer', 'time': time.time(), 'direction': direction,
|
||||
'ok': bool(value), 'bytes': value['bytes'] if value else 0,
|
||||
'seconds': value['seconds'] if value else 1})
|
||||
atomic(STATE / 'meter.json', meter.data)
|
||||
last_probe = time.time()
|
||||
except Exception:
|
||||
# Suppress exception strings: upstream data may contain credentials/URLs.
|
||||
atomic(STATE / 'status.json', {'hostname': cfg['name'], 'status': 'OFFLINE', 'state': 'OFFLINE',
|
||||
'updated_at': time.time(), 'admission_allowed': False})
|
||||
before_sleep = time.time()
|
||||
time.sleep(cfg.get('poll_seconds', 5))
|
||||
if time.time() - before_sleep > cfg.get('poll_seconds', 5) * 3:
|
||||
meter.invalidate()
|
||||
admission.admitted, admission.good = False, 0
|
||||
last_probe = 0
|
||||
atomic(STATE / 'status.json', {'hostname': cfg['name'], 'status': 'OFFLINE', 'state': 'OFFLINE',
|
||||
'updated_at': time.time(), 'admission_allowed': False})
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('command', choices=['daemon', 'status', 'network-test', 'start', 'stop', 'drain', 'restart', 'self-test'])
|
||||
args = parser.parse_args()
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
if args.command == 'daemon':
|
||||
daemon()
|
||||
elif args.command == 'status':
|
||||
value = read(STATE / 'status.json', {'hostname': config()['name'], 'status': 'OFFLINE', 'state': 'OFFLINE'})
|
||||
if time.time() - value.get('updated_at', 0) > 30:
|
||||
value.update(status='OFFLINE', state='OFFLINE', stale=True, admission_allowed=False)
|
||||
print(json.dumps(value, indent=2))
|
||||
elif args.command == 'network-test':
|
||||
meter = Meter(); meter.update()
|
||||
rates = transfer_probe(config())
|
||||
for direction in ('download', 'upload'):
|
||||
r = rates.get(direction)
|
||||
meter.consume({'kind': 'transfer', 'time': time.time(), 'direction': direction, 'ok': bool(r),
|
||||
'bytes': r['bytes'] if r else 0, 'seconds': r['seconds'] if r else 1})
|
||||
atomic(STATE / 'meter.json', meter.data)
|
||||
print(json.dumps(rates))
|
||||
return 0 if all(k in rates for k in ('download', 'upload')) else 1
|
||||
elif args.command == 'self-test':
|
||||
if read(STATE / 'runner.json', {}).get('running'):
|
||||
raise ValueError('drain before self-test')
|
||||
return subprocess.call(['powershell.exe', '-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Bypass',
|
||||
'-File', str(ROOT / 'selftest-nvenc.ps1'), '-Root', str(ROOT),
|
||||
'-Device', str(config().get('nvenc_device', 0))])
|
||||
else:
|
||||
paused = args.command in ('stop', 'drain')
|
||||
atomic(STATE / 'control.json', {'paused': paused})
|
||||
if args.command == 'restart':
|
||||
try:
|
||||
ipc('graceful-shutdown', config()['node_binary'])
|
||||
except (OSError, RuntimeError, subprocess.SubprocessError):
|
||||
pass
|
||||
print(json.dumps({'requested': args.command, 'paused': paused}))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (OSError, ValueError, RuntimeError, subprocess.SubprocessError):
|
||||
print('pt-worker-win: command failed; check service health', file=__import__('sys').stderr)
|
||||
raise SystemExit(1)
|
||||
459
infra/peertube-workers/win/pt-worker-win.py
Normal file
459
infra/peertube-workers/win/pt-worker-win.py
Normal file
|
|
@ -0,0 +1,459 @@
|
|||
#!/usr/bin/env python3
|
||||
"""pt-worker admission controller, native Windows port (validator for the design
|
||||
in windows-worker-result.md).
|
||||
|
||||
Ported from infra/peertube-workers/pt-worker.py: identical Meter / Admission
|
||||
state machine and policy defaults (AC-only, 20 GiB reserve, 85 C ceiling,
|
||||
10 Mbps floors with 1.25x recovery factor, DERP disallowed, 3-good/2-bad
|
||||
hysteresis, 120 s thermal cooldown, 60 s transfer-failure cooldown, 900 s
|
||||
bandwidth TTL, 5 s poll). Windows deltas:
|
||||
- power/battery via GetSystemPowerStatus (ctypes; None == unknown == drain);
|
||||
Modern Standby is mitigated with `powercfg /change standby-timeout-ac 0`
|
||||
during install, not here.
|
||||
- temperatures via MSAcpi_ThermalZoneTemperature; unsupported sensors yield
|
||||
None (unsupported measurements are never invented).
|
||||
- GPU encode telemetry from PDH GPU Engine (engtype_VideoEncode) counters;
|
||||
DRM fdinfo does not exist on Windows. Node >=18.20/20.12: `os.loadavg()`
|
||||
returns [0,0,0] on Windows and is not emitted.
|
||||
- cross-process daemon lock is a win32 mutex via msvcrt (fcntl is POSIX).
|
||||
- drain is delivered through the official runner named-pipe IPC.
|
||||
- hardware re-proof triggers on driver_version change in addition to boot_id
|
||||
(Windows Update can silently change graphics drivers).
|
||||
|
||||
Runs on CPython >= 3.11 with psutil (optional: improves disk/memory collection).
|
||||
Emitting a status JSON is safe to run standalone on Windows or elsewhere.
|
||||
"""
|
||||
import argparse
|
||||
import ctypes
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
import tomllib
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(r'C:\pt-worker')
|
||||
CONFIG = ROOT / 'state' / 'worker.json'
|
||||
STATE = ROOT / 'state'
|
||||
RUNNER_CONFIG = Path(os.environ.get('APPDATA', str(Path.home() / 'AppData' / 'Roaming'))) / 'peertube-runner-nodejs' / 'Config' / 'config.toml'
|
||||
|
||||
DEFAULTS = {
|
||||
'require_ac': True, 'min_disk_free_bytes': 20 * 1024 ** 3, 'max_temperature_c': 85,
|
||||
'min_download_mbps': 10, 'min_upload_mbps': 10, 'max_latency_ms': 150,
|
||||
'allow_derp': False, 'recovery_factor': 1.25, 'good_samples': 3, 'bad_samples': 2,
|
||||
'bandwidth_ttl_seconds': 900, 'transfer_failure_cooldown_seconds': 60,
|
||||
'thermal_cooldown_seconds': 120, 'poll_seconds': 5,
|
||||
}
|
||||
# Last-resort resource gates that hold even while admission policy is disabled.
|
||||
# Same thresholds as the Linux installer defaults (15 GiB RAM floor for a
|
||||
# 2-concurrency QSV worker, 6 GiB min per-job scratch); no CPU fallback exists.
|
||||
HARD_FLOORS = {'ram_total_bytes': 15 * 1024 ** 3, 'min_scratch_bytes': 6 * 1024 ** 3}
|
||||
|
||||
|
||||
def run(args, timeout=10):
|
||||
return subprocess.run(args, capture_output=True, text=True, timeout=timeout)
|
||||
|
||||
|
||||
def read(path, default=None):
|
||||
try:
|
||||
return json.loads(Path(path).read_text(encoding='utf8'))
|
||||
except (OSError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def config():
|
||||
value = read(CONFIG)
|
||||
if not value or not value.get('peertube_url') or not value.get('tailscale_peer'):
|
||||
raise ValueError('worker.json needs peertube_url and tailscale_peer')
|
||||
return {**DEFAULTS, **value}
|
||||
|
||||
|
||||
def ipc(action, node_binary='node'):
|
||||
"""Official runner IPC via ipc-win.cjs; the runner must be running in server mode."""
|
||||
env = {**os.environ, 'PT_STATE_DIR': str(STATE)}
|
||||
p = run([node_binary, str(ROOT / 'bin' / 'ipc-win.cjs'), action], 15)
|
||||
if p.returncode:
|
||||
raise RuntimeError('runner IPC unavailable')
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def registered(cfg):
|
||||
try:
|
||||
data = tomllib.loads(RUNNER_CONFIG.read_text(encoding='utf8'))
|
||||
return any(r['url'].rstrip('/') == cfg['peertube_url'].rstrip('/') and
|
||||
r['runnerName'] == cfg['name'] and r.get('runnerToken')
|
||||
for r in data.get('registeredInstances', []))
|
||||
except (OSError, ValueError, KeyError):
|
||||
return False
|
||||
|
||||
|
||||
def network(cfg):
|
||||
result = {'tailscale_ip': None, 'tailscale_path': 'unknown', 'latency_ms': None,
|
||||
'network_ok': False, 'api_ok': False, 'peer_online': False}
|
||||
try:
|
||||
ts = json.loads(run(['tailscale', 'status', '--json'], 5).stdout)
|
||||
result['tailscale_ip'] = next((a for a in ts.get('Self', {}).get('TailscaleIPs', []) if ':' not in a), None)
|
||||
peers = list(ts.get('Peer', {}).values())
|
||||
peer = next((p for p in peers if cfg['tailscale_peer'] in p.get('TailscaleIPs', []) or
|
||||
p.get('HostName') == cfg['tailscale_peer'] or
|
||||
p.get('DNSName', '').split('.')[0] == cfg['tailscale_peer']), None)
|
||||
if ts.get('BackendState') != 'Running' or not peer or not peer.get('Online'):
|
||||
return result
|
||||
result['peer_online'] = True
|
||||
p = run(['tailscale', 'ping', '--c=1', '--until-direct=false', '--timeout=3s', cfg['tailscale_peer']], 5)
|
||||
if 'pong from' not in p.stdout:
|
||||
return result
|
||||
result['tailscale_path'] = 'DERP' if 'DERP(' in p.stdout else 'direct'
|
||||
latency = re.search(r'in ([\d.]+)ms', p.stdout)
|
||||
if latency:
|
||||
result['latency_ms'] = float(latency[1])
|
||||
started = time.monotonic()
|
||||
with urllib.request.urlopen(cfg['peertube_url'].rstrip('/') + '/api/v1/config', timeout=4) as response:
|
||||
version = json.load(response).get('serverVersion')
|
||||
result['api_ok'] = version == cfg.get('peertube_version', '7.3.0')
|
||||
result['network_ok'] = result['api_ok']
|
||||
except (OSError, ValueError, subprocess.SubprocessError, urllib.error.URLError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def transfer_probe(cfg, megabytes=8):
|
||||
"""Bounded probe against pt-worker-netprobe (same server-side probe as Linux)."""
|
||||
url = cfg.get('probe_url')
|
||||
rates = {}
|
||||
if not url:
|
||||
return rates
|
||||
payload = os.urandom(megabytes * 1024 * 1024)
|
||||
for direction, request in (('download', urllib.request.Request(url.rstrip('/') + '/download',
|
||||
data=None, method='GET')),
|
||||
('upload', None)):
|
||||
try:
|
||||
started = time.monotonic()
|
||||
if direction == 'download':
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
seen = 0
|
||||
while len(response.read(1 << 20)):
|
||||
seen += 1 << 20
|
||||
if seen >= megabytes * 1024 * 1024:
|
||||
break
|
||||
seconds = time.monotonic() - started
|
||||
if seen < 65536:
|
||||
continue
|
||||
else:
|
||||
req = urllib.request.Request(url.rstrip('/') + '/upload', data=payload, method='POST')
|
||||
urllib.request.urlopen(req, timeout=30).read()
|
||||
seconds = time.monotonic() - started
|
||||
rates[direction] = {'mbps': megabytes * 8 / seconds, 'time': time.time(),
|
||||
'bytes': megabytes * 1024 * 1024, 'seconds': seconds}
|
||||
except (OSError, urllib.error.URLError, ValueError):
|
||||
rates[direction + '_failed'] = True
|
||||
return rates
|
||||
|
||||
|
||||
def get_system_power_status():
|
||||
"""Win32 GetSystemPowerStatus; dict or None when unsupported/unknown."""
|
||||
class PLS(ctypes.Structure):
|
||||
_fields_ = [('ACLineStatus', ctypes.c_byte), ('BatteryFlag', ctypes.c_byte),
|
||||
('BatteryLifePercent', ctypes.c_byte), ('Reserved1', ctypes.c_byte),
|
||||
('BatteryLifeTime', ctypes.c_uint32), ('BatteryFullLifeTime', ctypes.c_uint32)]
|
||||
status = PLS()
|
||||
if not ctypes.windll.kernel32.GetSystemPowerStatus(ctypes.byref(status)):
|
||||
return None
|
||||
ac = {0: False, 1: True}.get(status.ACLineStatus) # 255 == unknown -> None
|
||||
percentage = status.BatteryLifePercent
|
||||
return {'on_ac': ac, 'battery_flag': status.BatteryFlag,
|
||||
'battery_percent': None if percentage in (0, 255) else percentage}
|
||||
|
||||
|
||||
def sensors():
|
||||
result = {'temperature_c': None, 'temperatures_c': [], 'on_ac': None, 'battery': [],
|
||||
'throttle_counts': {}, 'ram_total_bytes': None, 'ram_available_bytes': None,
|
||||
'load_average': None}
|
||||
try:
|
||||
p = run(['powershell', '-NoProfile', '-NonInteractive', '-Command',
|
||||
'(Get-CimInstance -ClassName MSAcpi_ThermalZoneTemperature -ErrorAction Stop | '
|
||||
'Measure-Object -Property CurrentTemperature -Maximum).Maximum'], 12)
|
||||
if p.returncode == 0:
|
||||
value = float(p.stdout.strip()) / 10.0 - 273.15
|
||||
if 0 < value < 120:
|
||||
result['temperature_c'] = round(value, 1)
|
||||
result['temperatures_c'] = [result['temperature_c']]
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
pass
|
||||
power = get_system_power_status()
|
||||
if power:
|
||||
result['on_ac'] = power['on_ac']
|
||||
if power['battery_percent'] is not None:
|
||||
result['battery'] = [{'capacity': power['battery_percent'],
|
||||
'status': {0: 'unknown', 1: 'discharging', 2: 'on_ac'}.get(power['battery_flag'], 'other')}]
|
||||
try:
|
||||
import ctypes.wintypes # noqa: F401 (MEMORYSTATUSEX needs no extra types)
|
||||
class MEMORYSTATUSEX(ctypes.Structure):
|
||||
_fields_ = [('dwLength', ctypes.c_uint32), ('dwMemoryLoad', ctypes.c_uint32),
|
||||
('ullTotalPhys', ctypes.c_uint64), ('ullAvailPhys', ctypes.c_uint64),
|
||||
('ullTotalPageFile', ctypes.c_uint64), ('ullAvailPageFile', ctypes.c_uint64),
|
||||
('ullTotalVirtual', ctypes.c_uint64), ('ullAvailVirtual', ctypes.c_uint64),
|
||||
('ullAvailExtendedVirtual', ctypes.c_uint64)]
|
||||
memory = MEMORYSTATUSEX()
|
||||
memory.dwLength = ctypes.sizeof(MEMORYSTATUSEX)
|
||||
if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(memory)):
|
||||
result['ram_total_bytes'] = memory.ullTotalPhys
|
||||
result['ram_available_bytes'] = memory.ullAvailPhys
|
||||
except (OSError, AttributeError, ValueError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def disk_free(path=str(ROOT)):
|
||||
try:
|
||||
free = ctypes.c_ulonglong()
|
||||
if ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(path), None, None, ctypes.byref(free)):
|
||||
return free.value
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def gpu_identity():
|
||||
try:
|
||||
p = run(['powershell', '-NoProfile', '-NonInteractive', '-Command',
|
||||
'Get-CimInstance Win32_VideoController | Select-Object Name,DriverVersion,Status | ConvertTo-Json'], 12)
|
||||
data = json.loads(p.stdout or '[]')
|
||||
if isinstance(data, dict):
|
||||
data = [data]
|
||||
return data
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
return []
|
||||
|
||||
|
||||
class Meter:
|
||||
"""Byte-identical semantics to the Linux controller's Meter."""
|
||||
|
||||
def __init__(self):
|
||||
self.data = read(STATE / 'meter.json', {'offset': 0, 'jobs_completed': 0, 'jobs_failed': 0,
|
||||
'transfer_failures': 0, 'active': {}}) or {}
|
||||
|
||||
def invalidate(self):
|
||||
for key in ('download', 'upload'):
|
||||
self.data.pop(key, None)
|
||||
|
||||
def update(self):
|
||||
p = STATE / 'events.jsonl'
|
||||
if not p.exists():
|
||||
return
|
||||
with p.open(encoding='utf8') as f:
|
||||
if p.stat().st_size < self.data.get('offset', 0):
|
||||
self.data['offset'] = 0
|
||||
f.seek(self.data.get('offset', 0))
|
||||
while True:
|
||||
position = f.tell()
|
||||
line = f.readline()
|
||||
if not line:
|
||||
break
|
||||
if not line.endswith('\n'):
|
||||
f.seek(position)
|
||||
break
|
||||
try:
|
||||
self.consume(json.loads(line))
|
||||
except (ValueError, KeyError, TypeError):
|
||||
pass
|
||||
self.data['offset'] = f.tell()
|
||||
try:
|
||||
(STATE / 'meter.json').write_text(json.dumps(self.data), encoding='utf8')
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def consume(self, e):
|
||||
kind, stamp = e['kind'], e['time']
|
||||
if kind == 'transfer':
|
||||
if not e['ok']:
|
||||
self.data['transfer_failures'] = self.data.get('transfer_failures', 0) + 1
|
||||
self.data['last_transfer_failure'] = stamp
|
||||
elif e['bytes'] >= 65536 and e['seconds'] > 0:
|
||||
direction = e['direction']
|
||||
rate = e['bytes'] * 8 / e['seconds'] / 1e6
|
||||
old = self.data.get(direction, {}).get('mbps', rate)
|
||||
self.data[direction] = {'mbps': .3 * rate + .7 * old, 'time': stamp,
|
||||
'bytes': e['bytes'], 'seconds': e['seconds']}
|
||||
elif kind == 'gpu_failure':
|
||||
self.data['gpu_failure_time'] = stamp
|
||||
elif kind == 'encode':
|
||||
self.data['last_encode'] = e
|
||||
if e.get('job_uuid'):
|
||||
self.data.setdefault('encodes', {})[e['job_uuid']] = e
|
||||
elif kind == 'job_accept' and e['ok']:
|
||||
self.data.setdefault('active', {})[e['uuid']] = stamp
|
||||
elif kind in ('job_success', 'job_error', 'job_abort') and e['ok']:
|
||||
begun = self.data.setdefault('active', {}).pop(e['uuid'], None)
|
||||
if kind == 'job_success':
|
||||
self.data['jobs_completed'] = self.data.get('jobs_completed', 0) + 1
|
||||
elapsed = stamp - begun if begun else None
|
||||
self.data['last_job_duration_seconds'] = elapsed
|
||||
self.data['last_job_uuid'] = e['uuid']
|
||||
encoded = self.data.setdefault('encodes', {}).pop(e['uuid'], {})
|
||||
duration = encoded.get('duration')
|
||||
self.data['last_job_realtime_factor'] = duration / elapsed if duration and elapsed else None
|
||||
self.data['last_job_encode'] = encoded or None
|
||||
elif kind == 'job_error':
|
||||
self.data['jobs_failed'] = self.data.get('jobs_failed', 0) + 1
|
||||
for path in (STATE / 'inputs').glob('*.json'):
|
||||
if read(path) and read(path).get('uuid') == e['uuid']:
|
||||
try:
|
||||
path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def rates(self, now, ttl):
|
||||
return {f'recent_{k}_mbps': self.data.get(k, {}).get('mbps')
|
||||
if now - self.data.get(k, {}).get('time', 0) < ttl else None
|
||||
for k in ('download', 'upload')}
|
||||
|
||||
|
||||
class Admission:
|
||||
"""Decision gates: PAUSED, GPU_UNAVAILABLE, DRAINING (power), LOW_DISK,
|
||||
THERMAL_THROTTLED, OFFLINE, NETWORK_DEGRADED, BUSY, AVAILABLE.
|
||||
Hysteresis and thresholds identical to the Linux controller."""
|
||||
|
||||
def __init__(self, cfg):
|
||||
self.cfg, self.good, self.bad = cfg, 0, 0
|
||||
self.admitted, self.path, self.throttles = False, None, {}
|
||||
|
||||
def decide(self, sample, metric, control, now):
|
||||
cfg = self.cfg
|
||||
def refuse(state):
|
||||
self.admitted, self.good = False, 0
|
||||
return state
|
||||
if self.path and sample['tailscale_path'] != self.path:
|
||||
self.admitted = False
|
||||
self.good = 0
|
||||
metric.invalidate()
|
||||
self.path = sample['tailscale_path']
|
||||
rates = metric.rates(now, cfg.get('bandwidth_ttl_seconds', 900))
|
||||
sample.update(rates)
|
||||
if control.get('paused', True):
|
||||
return refuse('PAUSED')
|
||||
if not sample['registered']:
|
||||
return refuse('PAUSED')
|
||||
gpu_failed = metric.data.get('gpu_failure_time', 0) >= sample.get('gpu_tested_at', 0)
|
||||
if not sample['gpu_accessible'] or not sample.get('gpu_ready') or gpu_failed:
|
||||
return refuse('GPU_UNAVAILABLE')
|
||||
if cfg.get('require_ac', True) and sample['on_ac'] is not True:
|
||||
return refuse('DRAINING')
|
||||
if sample['disk_free_bytes'] is None or sample['disk_free_bytes'] < cfg.get('min_disk_free_bytes', 20 * 1024 ** 3):
|
||||
return refuse('LOW_DISK')
|
||||
rising = any(v > self.throttles.get(k, v) for k, v in sample['throttle_counts'].items())
|
||||
self.throttles = sample['throttle_counts']
|
||||
if rising or (sample['temperature_c'] is not None and sample['temperature_c'] >= cfg.get('max_temperature_c', 85)):
|
||||
self.thermal_until = now + cfg.get('thermal_cooldown_seconds', 120)
|
||||
if now < getattr(self, 'thermal_until', 0):
|
||||
return refuse('THERMAL_THROTTLED')
|
||||
if not sample['network_ok']:
|
||||
self.admitted, self.good = False, 0
|
||||
return 'OFFLINE'
|
||||
calibrating = control.get('calibrate_until', 0) > now
|
||||
factor = cfg.get('recovery_factor', 1.25) if not self.admitted else 1
|
||||
bad = sample['latency_ms'] is None or sample['latency_ms'] > cfg.get('max_latency_ms', 150)
|
||||
bad |= sample['tailscale_path'] == 'DERP' and not cfg.get('allow_derp', False)
|
||||
if not calibrating:
|
||||
bad |= any(rates[f'recent_{key}_mbps'] is None or rates[f'recent_{key}_mbps'] <
|
||||
cfg.get(f'min_{key}_mbps', 10) * factor for key in ('download', 'upload'))
|
||||
bad |= now - metric.data.get('last_transfer_failure', 0) < cfg.get('transfer_failure_cooldown_seconds', 60)
|
||||
if bad:
|
||||
self.good, self.bad = 0, self.bad + 1
|
||||
if self.bad >= cfg.get('bad_samples', 2) or not self.admitted:
|
||||
self.admitted = False
|
||||
return 'NETWORK_DEGRADED'
|
||||
else:
|
||||
self.bad, self.good = 0, self.good + 1
|
||||
if self.good >= cfg.get('good_samples', 3):
|
||||
self.admitted = True
|
||||
if not self.admitted:
|
||||
return 'NETWORK_DEGRADED'
|
||||
return 'BUSY' if sample['active_jobs'] else 'AVAILABLE'
|
||||
|
||||
|
||||
def status():
|
||||
"""Read-only status document (same schema as the Linux worker's status JSON;
|
||||
names preserved so central fleet tooling needs no OS branches)."""
|
||||
cfg = config()
|
||||
now = time.time()
|
||||
meter = Meter()
|
||||
meter.update()
|
||||
admission = Admission(cfg)
|
||||
sensors_data = sensors()
|
||||
hw = read(STATE / 'hardware.json', {}) or {}
|
||||
gpu = hw.get('gpu_model') or gpu_identity()
|
||||
proof = read(STATE / 'qsv.json', {}) or {}
|
||||
proof_current = bool(proof.get('qsv_h264_ok')) and proof.get('tested_at', 0) > now - 7 * 86400 and \
|
||||
(proof.get('boot_id') == hw.get('boot_id') and proof.get('driver_version') == hw.get('driver_version'))
|
||||
control = read(STATE / 'control.json', {}) or {}
|
||||
active = {}
|
||||
try:
|
||||
active = ipc('list-jobs', cfg.get('node_binary', 'node'))
|
||||
except (RuntimeError, ValueError, OSError):
|
||||
active = {}
|
||||
sample = {**network(cfg), 'registered': registered(cfg),
|
||||
'gpu_accessible': bool(proof_current), 'gpu_ready': bool(proof_current),
|
||||
'gpu_tested_at': proof.get('tested_at', 0),
|
||||
'on_ac': sensors_data['on_ac'], 'disk_free_bytes': disk_free(),
|
||||
'temperature_c': sensors_data['temperature_c'],
|
||||
'throttle_counts': sensors_data['throttle_counts'],
|
||||
'active_jobs': active.get('active_jobs'), 'concurrency': active.get('concurrency')}
|
||||
state = admission.decide(sample, meter, control, now)
|
||||
rates = meter.rates(now, cfg.get('bandwidth_ttl_seconds', 900))
|
||||
last = meter.data.get('last_encode') or {}
|
||||
return {'hostname': os.environ.get('COMPUTERNAME', 'unknown'), 'state': state,
|
||||
'tailscale_ip': sample['tailscale_ip'], 'tailscale_path': sample['tailscale_path'],
|
||||
'registration': sample['registered'], 'cpu_model': hw.get('cpu_model'),
|
||||
'gpu_model': gpu, 'gpu_driver_version': hw.get('driver_version'),
|
||||
'qsv_h264_ok': proof.get('qsv_h264_ok'), 'qsv_hevc_ok': proof.get('qsv_hevc_ok'),
|
||||
'av1_decode_ok': proof.get('av1_decode_ok'), 'av1_encode_ok': proof.get('av1_encode_ok'),
|
||||
'active_jobs': sample['active_jobs'], 'concurrency': sample['concurrency'],
|
||||
'gpu_util': last.get('engine_busy_max'), 'temps': sensors_data['temperatures_c'],
|
||||
'disk_free': sample['disk_free_bytes'],
|
||||
'ram_total_bytes': sensors_data['ram_total_bytes'],
|
||||
'ram_available_bytes': sensors_data['ram_available_bytes'],
|
||||
'on_ac': sample['on_ac'], 'battery': sensors_data['battery'],
|
||||
'recent_download_mbps': rates['recent_download_mbps'],
|
||||
'recent_upload_mbps': rates['recent_upload_mbps'],
|
||||
'jobs_completed': meter.data.get('jobs_completed', 0),
|
||||
'jobs_failed': meter.data.get('jobs_failed', 0),
|
||||
'last_job_duration': meter.data.get('last_job_duration_seconds'),
|
||||
'last_job_rtf': meter.data.get('last_job_realtime_factor'),
|
||||
'tested_at': proof.get('tested_at'), 'boot_id': hw.get('boot_id'),
|
||||
'driver_version': hw.get('driver_version')}
|
||||
|
||||
|
||||
def self_test_marker_stale(hw, proof):
|
||||
"""Re-proof triggers: boot change, GPU driver change, or 7-day proof age.
|
||||
(Extension of the Linux boot-id rule: Windows Update changes graphics
|
||||
drivers without a boot.)"""
|
||||
if not proof.get('tested_at') or time.time() - proof['tested_at'] > 7 * 86400:
|
||||
return True
|
||||
return proof.get('boot_id') != hw.get('boot_id') or proof.get('driver_version') != hw.get('driver_version')
|
||||
|
||||
|
||||
def daemon():
|
||||
raise SystemExit('daemon loop is a validation item; run status/network-test here, '
|
||||
'see windows-worker-result.md section 6 before production enablement')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('command', choices=['status', 'network-test', 'drain'])
|
||||
args = parser.parse_args()
|
||||
if args.command == 'status':
|
||||
print(json.dumps(status(), indent=1))
|
||||
elif args.command == 'network-test':
|
||||
cfg = config()
|
||||
rates = transfer_probe(cfg)
|
||||
print(json.dumps(rates, indent=1))
|
||||
elif args.command == 'drain':
|
||||
cfg = config()
|
||||
print(json.dumps(ipc('graceful-shutdown', cfg.get('node_binary', 'node'))))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
35
infra/peertube-workers/win/register-from-ssh.py
Normal file
35
infra/peertube-workers/win/register-from-ssh.py
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
"""Pipe the existing registration secret directly between SSH sessions. No file/argv/log."""
|
||||
import argparse,json,re,shlex,subprocess,tempfile
|
||||
from pathlib import Path
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--worker',required=True,help='SSH target user@WSL-host')
|
||||
parser.add_argument('--name',required=True)
|
||||
parser.add_argument('--root',default=r'C:\pt-worker')
|
||||
parser.add_argument('--server',default='root@100.92.231.98')
|
||||
parser.add_argument('--server-key',default=str(Path.home()/'.ssh/id_rsa'))
|
||||
parser.add_argument('--worker-key',default=str(Path.home()/'.ssh/devserver'))
|
||||
args=parser.parse_args()
|
||||
assert re.fullmatch(r'[A-Za-z0-9_-]+',args.name),'Use a simple unique runner name'
|
||||
assert re.fullmatch(r'[A-Za-z]:\\[^\r\n]+',args.root),'Require local Windows drive root'
|
||||
SSH=['ssh','-o','BatchMode=yes','-o','StrictHostKeyChecking=yes']
|
||||
server=SSH+['-i',args.server_key,args.server]
|
||||
worker=SSH+['-i',args.worker_key,args.worker]
|
||||
query='SELECT "registrationToken" FROM "runnerRegistrationToken" WHERE id=1;'
|
||||
command="docker exec -i occ-tube-peertube-postgres-1 sh -c 'exec psql -X -v ON_ERROR_STOP=1 -U \"$POSTGRES_USER\" -d peertube -At'"
|
||||
# metadata check is independent of, and never prints, token material.
|
||||
p=subprocess.run(server+[command],input=f"SELECT (SELECT count(*) FROM \"runnerRegistrationToken\"),(SELECT count(*) FROM \"runner\" WHERE name='{args.name}');",text=True,capture_output=True,check=True)
|
||||
assert p.stdout.strip()=='1|0','Expected existing token and no duplicate runner'
|
||||
marker=Path(tempfile.gettempdir())/('pt-worker-'+args.name+'-enrollment.pending')
|
||||
assert not marker.exists(),'Previous result uncertain; inspect server before retrying'
|
||||
marker.write_text('Registration attempted; contains no secrets.\n')
|
||||
source=subprocess.Popen(server+[command],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
|
||||
source.stdin.write(query.encode());source.stdin.close()
|
||||
node='/mnt/'+args.root[0].lower()+args.root[2:].replace('\\','/')+'/node/node.exe'
|
||||
remote=shlex.join([node,args.root+r'\bin\ipc-nvenc.cjs','register',args.root])
|
||||
receiver=subprocess.run(worker+[remote],stdin=source.stdout,capture_output=True)
|
||||
source.stdout.close();source.wait(timeout=20)
|
||||
if source.returncode or receiver.returncode:
|
||||
raise SystemExit('Registration failed; secret-bearing outputs suppressed, inspect safe metadata before retry')
|
||||
assert json.loads(receiver.stdout)['success'] is True
|
||||
marker.unlink()
|
||||
print(json.dumps({'runner_name':args.name,'registration_success':True,'secret_transport':'SSH stdout->SSH stdin->official named-pipe IPC','secret_saved':False}))
|
||||
65
infra/peertube-workers/win/runner-supervisor.cjs
Normal file
65
infra/peertube-workers/win/runner-supervisor.cjs
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
'use strict';
|
||||
// Both WinSW and interactive validation use this supervisor. Only official IPC
|
||||
// drains the runner; loss of admission never kills an in-flight video job.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { spawn, execFile } = require('node:child_process');
|
||||
const root = process.env.PT_ROOT || 'C:\\pt-worker';
|
||||
const state = path.join(root, 'state');
|
||||
const cfg = JSON.parse(fs.readFileSync(path.join(state, 'worker.json'), 'utf8').replace(/^\uFEFF/, ''));
|
||||
const env = { ...process.env, PT_ROOT: root, PT_STATE_DIR: state,
|
||||
PT_WORKER_CONFIG: path.join(state, 'worker.json'),
|
||||
APPDATA: path.join(root, 'profile', 'Roaming'), LOCALAPPDATA: path.join(root, 'profile', 'Local'),
|
||||
TMP: path.join(root, 'tmp'), TEMP: path.join(root, 'tmp'),
|
||||
FFMPEG_PATH: path.join(root, 'bin', 'ffmpeg-nvenc.exe'),
|
||||
FFPROBE_PATH: path.join(root, 'bin', 'ffprobe.exe'), PT_NVENC_DEVICE: String(cfg.nvenc_device || 0) };
|
||||
let child = null, draining = false, asking = false, stopping = false, restartAfter = 0;
|
||||
function read(name) { try { return JSON.parse(fs.readFileSync(path.join(state, name), 'utf8').replace(/^\uFEFF/, '')); } catch { return {}; } }
|
||||
function emit(data) { fs.appendFileSync(path.join(state, 'events.jsonl'), JSON.stringify({ time: Date.now()/1000, ...data }) + '\n'); }
|
||||
function atomic(name, data) { const file = path.join(state, name), tmp = file+'.'+process.pid+'.tmp'; fs.writeFileSync(tmp, JSON.stringify(data)); fs.renameSync(tmp, file); }
|
||||
function drain() {
|
||||
if (!child || asking || draining) return;
|
||||
const target = child;
|
||||
asking = true;
|
||||
execFile(cfg.node_binary, [path.join(root, 'bin', 'ipc-nvenc.cjs'), 'graceful-shutdown'],
|
||||
{ env, timeout: 17000, windowsHide: true }, err => { asking = false; if (!err && child === target) draining = true; });
|
||||
}
|
||||
function tick() {
|
||||
if (fs.existsSync(path.join(state, 'supervisor-stop'))) stopping = true;
|
||||
const now = Date.now()/1000, admission = read('status.json'), control = read('control.json');
|
||||
const allow = !stopping && control.paused === false && admission.admission_allowed === true &&
|
||||
now - admission.updated_at < 30 && ['AVAILABLE', 'BUSY'].includes(admission.status) &&
|
||||
!fs.existsSync(path.join(state, 'gpu-unavailable'));
|
||||
if (child && !allow) drain();
|
||||
if (!child && allow && now >= restartAfter) {
|
||||
draining = false;
|
||||
child = spawn(cfg.node_binary, ['--require', path.join(root, 'bin', 'observe.cjs'),
|
||||
path.join(root, 'runner', 'node_modules', '@peertube', 'peertube-runner', 'dist', 'peertube-runner.js'),
|
||||
'server', '--enable-job', 'vod-web-video-transcoding', '--enable-job', 'vod-hls-transcoding',
|
||||
'--enable-job', 'vod-audio-merge-transcoding'], { env, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
// Upstream logs can contain tokens. Retain only known safe milestones.
|
||||
let buffer = '';
|
||||
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => {
|
||||
buffer += chunk.toString();
|
||||
const lines = buffer.split(/\r?\n/); buffer = lines.pop().slice(-8192);
|
||||
for (const line of lines) {
|
||||
const match = line.match(/(Finished processing|Processing) job of type ([\w-]+): ([a-f\d-]{36})/);
|
||||
if (match) emit({ kind: 'runner_milestone', phase: match[1].startsWith('Finished') ? 'completed' : 'processing', job_type: match[2], uuid: match[3] });
|
||||
if (line.includes('Connected to ') && line.includes('/runners')) emit({kind: 'runner_connected'});
|
||||
if (line.includes('Cannot process job')) emit({kind: 'runner_error', reason: 'process_job_failed'});
|
||||
}
|
||||
});
|
||||
child.once('error', () => { emit({kind: 'runner_error', reason: 'spawn_failed'}); });
|
||||
child.once('exit', code => {
|
||||
emit({kind: 'runner_exit', code, graceful_requested: draining || asking});
|
||||
child = null; draining = false; asking = false; restartAfter = Date.now()/1000 + 15;
|
||||
});
|
||||
emit({kind: 'runner_start', concurrency: cfg.concurrency});
|
||||
}
|
||||
atomic('runner.json', {running: !!child, draining, updated_at: now, pid: child?.pid || null});
|
||||
if (stopping && !child) process.exit(0);
|
||||
}
|
||||
fs.rmSync(path.join(state, 'supervisor-stop'), {force: true});
|
||||
process.on('SIGINT', () => { stopping = true; drain(); });
|
||||
process.on('SIGTERM', () => { stopping = true; drain(); });
|
||||
setInterval(tick, 1000); tick();
|
||||
42
infra/peertube-workers/win/selftest-nvenc.ps1
Normal file
42
infra/peertube-workers/win/selftest-nvenc.ps1
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
param([string]$Root='C:\pt-worker', [int]$Device=0)
|
||||
$ErrorActionPreference='Stop'
|
||||
$ProgressPreference='SilentlyContinue'
|
||||
$ffmpeg=Join-Path $Root 'bin\ffmpeg.exe'
|
||||
$state=Join-Path $Root 'state'
|
||||
New-Item -ItemType Directory -Force $state,(Join-Path $Root 'tmp') | Out-Null
|
||||
$gpu=& nvidia-smi.exe -i $Device --query-gpu=name,uuid,driver_version --format=csv,noheader,nounits
|
||||
if($LASTEXITCODE){throw 'nvidia-smi identity failed'}
|
||||
$identity=$gpu -split ',\s*'
|
||||
$boot=(Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToUniversalTime().ToString('o')
|
||||
$proof=[ordered]@{tested_at=[DateTimeOffset]::UtcNow.ToUnixTimeSeconds();boot_id=$boot;driver_version=$identity[2];gpu_model=$identity[0];gpu_uuid=$identity[1];device=$Device;nvenc_h264_ok=$false;nvenc_hevc_ok=$false;av1_encode_ok=$false;av1_decode_ok=$null;tests=@()}
|
||||
foreach($codec in @('h264','hevc')){
|
||||
$out=Join-Path $Root "tmp\selftest-$codec.mp4"
|
||||
$err=Join-Path $state "selftest-$codec.stderr"
|
||||
$stdout=Join-Path $state "selftest-$codec.stdout"
|
||||
$argv=@('-hide_banner','-nostdin','-y','-re','-f','lavfi','-i','testsrc2=size=1920x1080:rate=30','-t','5','-an','-c:v',"${codec}_nvenc",'-gpu',"$Device",'-preset','p4','-pix_fmt','yuv420p',$out)
|
||||
$start=Get-Date
|
||||
$p=Start-Process $ffmpeg -ArgumentList $argv -PassThru -NoNewWindow -RedirectStandardError $err -RedirectStandardOutput $stdout
|
||||
$null=$p.Handle
|
||||
$samples=@(); $sessions=@()
|
||||
while(!( $p.HasExited ) -and ((Get-Date)-$start).TotalSeconds -lt 30){
|
||||
$value=& nvidia-smi.exe -i $Device --query-gpu=utilization.encoder,encoder.stats.sessionCount,encoder.stats.averageFps --format=csv,noheader,nounits
|
||||
if($LASTEXITCODE -eq 0){
|
||||
$cells=$value -split ',\s*'
|
||||
$n=0.0
|
||||
if([double]::TryParse($cells[0],[ref]$n)){$samples+=$n}
|
||||
if($cells.Count -ge 3){$sessions+=@{count=$cells[1];fps=$cells[2]}}
|
||||
}
|
||||
Start-Sleep -Milliseconds 200
|
||||
$p.Refresh()
|
||||
}
|
||||
if(!$p.HasExited){$p.Kill();throw 'NVENC self-test exceeded 30 second deadline'}
|
||||
$p.WaitForExit()
|
||||
$max=if($samples.Count){($samples | Measure-Object -Maximum).Maximum}else{$null}
|
||||
$ok=$p.ExitCode -eq 0 -and $null -ne $max -and $max -gt 0
|
||||
$proof["nvenc_${codec}_ok"]=$ok
|
||||
$proof.tests+=@{codec=$codec;encoder="${codec}_nvenc";width=1920;height=1080;duration=5;exit_code=$p.ExitCode;seconds=((Get-Date)-$start).TotalSeconds;encoder_util_max=$max;encoder_util_samples=$samples;encoder_stats=$sessions;stderr_file=$err}
|
||||
if(!$ok){$proof.failure=if($p.ExitCode -ne 0){Get-Content $err -Raw}else{'No positive NVIDIA encoder utilization: hardware proof refused'};break}
|
||||
}
|
||||
$proof | ConvertTo-Json -Depth 8 | Set-Content (Join-Path $state 'nvenc.json') -Encoding UTF8
|
||||
$proof | ConvertTo-Json -Depth 8
|
||||
if(!$proof.nvenc_h264_ok -or !$proof.nvenc_hevc_ok){exit 78}
|
||||
159
infra/peertube-workers/win/selftest-qsv.ps1
Normal file
159
infra/peertube-workers/win/selftest-qsv.ps1
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
# Windows QSV self-test for the PeerTube transcode worker (validator).
|
||||
#
|
||||
# Gates registration. A real encode MUST produce a positive PDH
|
||||
# \GPU Engine\Utilization Percentage (engtype_VideoEncode) signal; exit 0 alone
|
||||
# is insufficient. No CPU fallback exists anywhere in this worker.
|
||||
#
|
||||
# Usage (PowerShell 5.1+, from C:\pt-worker\bin after the ffmpeg zip is unpacked):
|
||||
# powershell -ExecutionPolicy Bypass -File selftest-qsv.ps1
|
||||
# Writes C:\pt-worker\state\qsv.json on completion.
|
||||
|
||||
$ErrorActionPreference = 'Continue'
|
||||
$State = 'C:\pt-worker\state'
|
||||
$Bin = 'C:\pt-worker\bin'
|
||||
$FFmpeg = Join-Path $Bin 'ffmpeg.exe'
|
||||
$FFprobe = Join-Path $Bin 'ffprobe.exe'
|
||||
New-Item -ItemType Directory -Force -Path $State | Out-Null
|
||||
|
||||
function HexToBytes([string]$hex) {
|
||||
$bytes = New-Object byte[] ($hex.Length / 2)
|
||||
for ($i = 0; $i -lt $bytes.Length; $i++) { $bytes[$i] = [Convert]::ToByte($hex.Substring($i * 2, 2), 16) }
|
||||
return ,$bytes
|
||||
}
|
||||
|
||||
# 5-second synthetic 1080p30 clip (testsrc2 -> x264 -> mp4). No network, no libx264 use at encode time.
|
||||
$TestClip = Join-Path $State 'self-test-source.mp4'
|
||||
& $FFmpeg -hide_banner -loglevel error -y -f lavfi -i 'testsrc2=size=1920x1080:rate=30' -t 5 -c:v libx264 -pix_fmt yuv420p $TestClip
|
||||
if ($LASTEXITCODE -ne 0) { Write-Error 'failed to build self-test source clip'; exit 2 }
|
||||
|
||||
function Get-VideoEncodeBusy([int]$ProcId, [double]$Seconds = 3.0) {
|
||||
# Sum of engtype_VideoEncode instances of \GPU Engine(pid_<id>_*)\Utilization Percentage.
|
||||
# Locale-sensitive counter names/decimal separators: unparseable => -1 (unknown, never 0).
|
||||
$out = & typeperf "\GPU Engine(pid_$ProcId`_*)\Utilization Percentage" -sc 1 2>$null
|
||||
$rows = @($out | Where-Object { $_ -like '"*' })
|
||||
if ($rows.Count -lt 2) { return -1 }
|
||||
$headers = ($rows[0] -replace '^"|"$','').split('","')
|
||||
$values = ($rows[1] -replace '^"|"$','').split('","')
|
||||
$total = 0.0; $seen = $false
|
||||
for ($i = 0; $i -lt $headers.Count; $i++) {
|
||||
if ($headers[$i] -notmatch 'engtype_VideoEncode') { continue }
|
||||
if ($i -ge $values.Count) { return -1 }
|
||||
$raw = $values[$i]
|
||||
$num = 0.0
|
||||
if (-not [double]::TryParse($raw, [Globalization.NumberStyles]::Float,
|
||||
[Globalization.CultureInfo]::InvariantCulture, [ref]$num)) {
|
||||
$raw = $raw -replace '\.', '' -replace ',', '.'
|
||||
if (-not [double]::TryParse($raw, [Globalization.NumberStyles]::Float,
|
||||
[Globalization.CultureInfo]::InvariantCulture, [ref]$num)) { return -1 }
|
||||
}
|
||||
$total += $num; $seen = $true
|
||||
}
|
||||
if (-not $seen) { return -1 }
|
||||
return $total
|
||||
}
|
||||
|
||||
function Test-RealEncode([string]$Name, [string[]]$ExtraArgs, [int]$ProcId) {
|
||||
$log = Join-Path $State "selftest-$Name.log"
|
||||
$proc = Start-Process -FilePath $FFmpeg -ArgumentList $ExtraArgs -NoNewWindow -PassThru `
|
||||
-RedirectStandardError $log -RedirectStandardOutput (Join-Path $State "selftest-$Name.out")
|
||||
$maxBusy = 0.0; $samples = 0; $readable = $false
|
||||
while (-not $proc.HasExited) {
|
||||
$busy = Get-VideoEncodeBusy -ProcId ($proc.Id)
|
||||
if ($busy -ge 0) { $readable = $true; $samples++; if ($busy -gt $maxBusy) { $maxBusy = $busy } }
|
||||
Start-Sleep -Milliseconds 800
|
||||
}
|
||||
$proc.WaitForExit() | Out-Null
|
||||
return @{ name = $Name; exit = $proc.ExitCode; max_busy = $maxBusy;
|
||||
samples = $samples; counters_readable = $readable; log = $log }
|
||||
}
|
||||
|
||||
$report = [ordered]@{
|
||||
schema = 'pt-worker-qsv-selftest/1'
|
||||
tested_at = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds()
|
||||
gpu = @(); driver_version = $null
|
||||
dispatcher_dlls = @{ libmfx = (Test-Path (Join-Path $Bin 'libmfx64-gen.dll'));
|
||||
libvpl = (Test-Path (Join-Path $Bin 'libvpl64-1.dll')) }
|
||||
ffmpeg_version = $null; qsv_h264_ok = $false; qsv_hevc_ok = $false
|
||||
h264_mf_ok = $false; av1_decode_ok = $null; av1_encode_ok = $false
|
||||
tests = @{}; failures = @(); boot_id = (Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToString('o')
|
||||
}
|
||||
|
||||
$gpus = @(); try {
|
||||
$gpus = @(Get-CimInstance Win32_VideoController | Select-Object Name, DriverVersion, Status)
|
||||
} catch { }
|
||||
$report.gpu = $gpus
|
||||
$intel = $gpus | Where-Object { $_.Name -match 'Intel.*(?:Iris|UHD|Graphics)' } | Select-Object -First 1
|
||||
if (-not $intel) { $report.failures += 'no Intel graphics adapter found via WMI' }
|
||||
else { $report.driver_version = $intel.DriverVersion }
|
||||
$report.ffmpeg_version = (& $FFmpeg -version | Select-Object -First 1)
|
||||
|
||||
# --- capability listing (necessary, not sufficient) ---
|
||||
$encoders = (& $FFmpeg -hide_banner -encoders) -join "`n"
|
||||
foreach ($codec in @('h264_qsv', 'hevc_qsv', 'h264_mf', 'av1_qsv')) {
|
||||
$report.tests["encoder_$codec"] = ($encoders -match [regex]::Escape($codec))
|
||||
}
|
||||
if ($report.tests.encoder_h264_qsv -ne $true) { $report.failures += 'h264_qsv encoder missing from ffmpeg build' }
|
||||
|
||||
# --- session init proof (device open) ---
|
||||
$initLog = Join-Path $State 'selftest-qsv-init.log'
|
||||
& $FFmpeg -hide_banner -v verbose -y -init_hw_device qsv=hw -f lavfi -i 'testsrc2=size=1280x720:rate=30' `
|
||||
-frames:v 10 -c:v h264_qsv -f null - 2> $initLog
|
||||
$initExit = $LASTEXITCODE
|
||||
$initText = Get-Content $initLog -Raw
|
||||
$report.tests.qsv_device_init_exit = $initExit
|
||||
if ($initExit -ne 0) { $report.failures += "h264_qsv session init failed (exit $initExit); see selftest-qsv-init.log" }
|
||||
|
||||
# --- REAL H.264 1080p encode with mandatory VideoEncode counter proof ---
|
||||
$h264 = Test-RealEncode -Name 'qsv-h264' -ProcId 0 -ExtraArgs @(
|
||||
'-hide_banner', '-nostdin', '-v', 'info',
|
||||
'-init_hw_device', 'qsv=qs', '-filter_hw_device', 'qs',
|
||||
'-hwaccel', 'qsv', '-hwaccel_output_format', 'qsv', '-c:v', 'h264_qsv',
|
||||
'-i', $TestClip,
|
||||
'-vf', 'scale_qsv=w=1280:h=720:format=nv12',
|
||||
'-c:v', 'h264_qsv', '-preset', 'medium', '-b:v', '2M', '-maxrate', '2M', '-bufsize', '4M', '-bf', '2',
|
||||
'-an', '-f', 'null', '-'
|
||||
)
|
||||
$report.tests.h264 = $h264
|
||||
if ($h264.exit -eq 0 -and $h264.counters_readable -and $h264.samples -ge 3 -and $h264.max_busy -gt 0) {
|
||||
$report.qsv_h264_ok = $true
|
||||
} elseif ($h264.exit -eq 0) {
|
||||
$report.failures += 'h264_qsv exited 0 without a positive VideoEncode counter; not accepted as hardware proof'
|
||||
}
|
||||
|
||||
# --- HEVC encode attempt (capability discovery; not enabled in production) ---
|
||||
$hevc = Test-RealEncode -Name 'qsv-hevc' -ProcId 0 -ExtraArgs @(
|
||||
'-hide_banner', '-nostdin', '-v', 'info',
|
||||
'-init_hw_device', 'qsv=qs', '-filter_hw_device', 'qs',
|
||||
'-hwaccel', 'qsv', '-hwaccel_output_format', 'qsv', '-c:v', 'hevc_qsv',
|
||||
'-i', $TestClip,
|
||||
'-vf', 'scale_qsv=w=1280:h=720:format=nv12',
|
||||
'-c:v', 'hevc_qsv', '-preset', 'medium', '-b:v', '2M',
|
||||
'-an', '-f', 'null', '-'
|
||||
)
|
||||
$report.tests.hevc = $hevc
|
||||
$report.qsv_hevc_ok = ($hevc.exit -eq 0 -and $hevc.counters_readable -and $hevc.samples -ge 3 -and $hevc.max_busy -gt 0)
|
||||
|
||||
# --- AV1: decode-only expectation on Alder Lake; verify, never assume ---
|
||||
# Build an AV1 source only if an AV1 encoder (software) exists in this build;
|
||||
# decoding AV1 hardware-wise is the capability under test.
|
||||
$av1DecodeOk = $false
|
||||
$av1Source = Join-Path $State 'self-test-source-av1.mp4'
|
||||
& $FFmpeg -hide_banner -loglevel error -y -f lavfi -i 'testsrc2=size=320x180:rate=30' -t 1 -c:v libaom-av1 $av1Source 2>$null
|
||||
if ($LASTEXITCODE -eq 0) {
|
||||
$av1Log = Join-Path $State 'selftest-av1-decode.log'
|
||||
& $FFmpeg -hide_banner -v error -y -c:v av1_qsv -i $av1Source -frames:v 5 -f null - 2> $av1Log
|
||||
$av1DecodeOk = ($LASTEXITCODE -eq 0)
|
||||
}
|
||||
$report.av1_decode_ok = $av1DecodeOk
|
||||
$encAv1 = ($encoders -match 'av1_qsv')
|
||||
$report.av1_encode_ok = $false
|
||||
$report.tests.av1_encoder_present = $encAv1
|
||||
|
||||
# --- Record + verdict. NO CPU fallback exists; registration is blocked on failure. ---
|
||||
$report | ConvertTo-Json -Depth 5 | Set-Content -Encoding UTF8 (Join-Path $State 'qsv.json')
|
||||
if ($report.failures.Count -gt 0 -or -not $report.qsv_h264_ok) {
|
||||
Write-Host 'SELF-TEST FAILED:' ($report.failures -join '; ')
|
||||
exit 1
|
||||
}
|
||||
Write-Host 'SELF-TEST PASSED: h264_qsv with positive VideoEncode counter proof'
|
||||
exit 0
|
||||
8
infra/peertube-workers/win/service-stop.cjs
Normal file
8
infra/peertube-workers/win/service-stop.cjs
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
'use strict';
|
||||
const fs=require('node:fs'),path=require('node:path');
|
||||
const root=process.env.PT_ROOT || 'C:\\pt-worker', state=path.join(root,'state');
|
||||
// The supervisor owns graceful shutdown; only stop after all processing jobs exit.
|
||||
fs.writeFileSync(path.join(state,'supervisor-stop'), 'requested');
|
||||
const timer=setInterval(()=>{
|
||||
try {const r=JSON.parse(fs.readFileSync(path.join(state,'runner.json'),'utf8'));if(!r.running){clearInterval(timer);process.exit(0);}}catch{}
|
||||
},1000);
|
||||
87
infra/peertube-workers/win/tests/test_nvenc.py
Normal file
87
infra/peertube-workers/win/tests/test_nvenc.py
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
WIN = Path(__file__).resolve().parents[1]
|
||||
ROOT = WIN.parent
|
||||
|
||||
def load(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
|
||||
return m
|
||||
|
||||
shim = load('nvenc', WIN / 'ffmpeg-nvenc.py')
|
||||
worker = load('nvenc_worker', WIN / 'pt-worker-nvenc.py')
|
||||
linux = load('linux_worker', ROOT / 'pt-worker.py')
|
||||
PROBE = {'streams': [{'codec_type': 'video', 'codec_name': 'h264', 'pix_fmt': 'yuv420p', 'width': 1920, 'height': 1080}], 'format': {'duration': '10'}}
|
||||
|
||||
class AdapterTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(); self.old = shim.STATE
|
||||
shim.STATE = Path(self.temp.name)
|
||||
def tearDown(self):
|
||||
shim.STATE = self.old; self.temp.cleanup()
|
||||
def test_runner_default_becomes_nvdec_cuda_scale_nvenc(self):
|
||||
args, info = shim.rewrite(['-i','source.mp4','-vcodec','libx264','-vf','scale=w=-2:h=720','-pix_fmt','yuv420p','-preset','veryfast','-maxrate:v','2000000','-bufsize:v','4000000','-b_strategy','1','-bf','16','-g:v','60','-r','30','-c:a','aac','-movflags','faststart','-f','mp4','out.mp4'], PROBE)
|
||||
self.assertIn('h264_nvenc', args); self.assertIn('cuda', args)
|
||||
self.assertIn('scale_cuda=w=1280:h=720:format=nv12', args)
|
||||
self.assertIn('p4', args); self.assertNotIn('libx264', args)
|
||||
self.assertNotIn('-b_strategy', args); self.assertEqual(info['target_height'],720)
|
||||
def test_unknown_args_aliases_codecs_filters_and_extra_outputs_refused(self):
|
||||
base=['-i','a','-c:v','libx264']
|
||||
for extra in (['-unknown','1'],['-c:v:0','libx264'],['-c','libx264'],['-filter_complex','null'],['-vf','scale=640:360,eq=contrast=2'],['-hwaccel','none'],['-x264-params','no-scenecut=1'],['other.mp4'],['-c:v','libx265']):
|
||||
with self.subTest(extra=extra),self.assertRaises(ValueError):
|
||||
shim.rewrite(base+extra+['out'],PROBE)
|
||||
def test_source_rotation_hdr_unknown_codec_refused(self):
|
||||
for change in ({'codec_name':'mpeg4'},{'side_data_list':[{'rotation':90}]},{'color_transfer':'smpte2084'}):
|
||||
p={**PROBE,'streams':[{**PROBE['streams'][0],**change}]}
|
||||
with self.assertRaises(ValueError):shim.rewrite(['-i','a','-c:v','libx264','out'],p)
|
||||
def test_hevc_sdr_source_cuda_decode(self):
|
||||
p={**PROBE,'streams':[{**PROBE['streams'][0],'codec_name':'hevc','pix_fmt':'yuv420p10le'}]}
|
||||
args,_=shim.rewrite(['-i','a','-c:v','libx264','out'],p)
|
||||
self.assertIn('h264_nvenc',args);self.assertIn('cuda',args)
|
||||
def test_hevc_target_is_hardware_only_and_unknown_target_refused(self):
|
||||
old=shim.TARGET
|
||||
try:
|
||||
shim.TARGET='hevc_nvenc'
|
||||
args,info=shim.rewrite(['-i','a','-c:v','libx264','out'],PROBE)
|
||||
self.assertIn('hevc_nvenc',args);self.assertEqual(info['target_codec'],'hevc')
|
||||
shim.TARGET='libx265'
|
||||
with self.assertRaises(ValueError):shim.rewrite(['-i','a','-c:v','libx264','out'],PROBE)
|
||||
finally:shim.TARGET=old
|
||||
def test_gpu_unavailable_marker_refuses_video_and_remux(self):
|
||||
(shim.STATE/'gpu-unavailable').touch()
|
||||
for args in (['-i','a','-c:v','libx264','out'],['-i','a','-c','copy','out']):
|
||||
with self.assertRaises(ValueError):shim.rewrite(args,PROBE)
|
||||
def test_queries_audio_and_copy_are_explicit(self):
|
||||
for args in (['-version'],['-encoders'],['-i','a','-vn','-c:a','aac','out'],['-i','a','-c','copy','out']):
|
||||
out,info=shim.rewrite(args,PROBE);self.assertEqual(out,args);self.assertIsNone(info)
|
||||
with self.assertRaises(ValueError):shim.rewrite(['-i','a','-vn','-c:v','libx264','out'],PROBE)
|
||||
|
||||
class AdmissionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.old=(linux.STATE,worker.STATE)
|
||||
linux.STATE=worker.STATE=Path(self.temp.name);self.now=time.time()
|
||||
def tearDown(self):linux.STATE,worker.STATE=self.old;self.temp.cleanup()
|
||||
def test_linux_windows_decisions_match_for_entire_sequence(self):
|
||||
overrides=[{}, {}, {}, {'active_jobs':2}, {'on_ac':False}, {'on_ac':None}, {}, {}, {}, {'temperature_c':90}, {'temperature_c':50}, {'gpu_ready':False}, {'disk_free_bytes':0}, {'network_ok':False}, {'tailscale_path':'DERP'}]
|
||||
traces=[]
|
||||
for mod in (linux,worker):
|
||||
meter=mod.Meter();policy=mod.Admission({})
|
||||
for d in ('download','upload'):meter.consume({'kind':'transfer','time':self.now,'ok':True,'direction':d,'bytes':10_000_000,'seconds':1})
|
||||
trace=[]
|
||||
for i,o in enumerate(overrides):
|
||||
sample={'tailscale_path':'direct','latency_ms':5,'network_ok':True,'registered':True,'gpu_ready':True,'gpu_accessible':True,'gpu_tested_at':self.now,'on_ac':True,'disk_free_bytes':100*1024**3,'temperature_c':50,'throttle_counts':{},'active_jobs':0,**o}
|
||||
trace.append(policy.decide(sample,meter,{'paused':False},self.now+i*130))
|
||||
traces.append(trace)
|
||||
self.assertEqual(traces[0],traces[1]);self.assertEqual(traces[0][:6],['NETWORK_DEGRADED','NETWORK_DEGRADED','AVAILABLE','BUSY','DRAINING','DRAINING'])
|
||||
def test_gpu_failure_latched_until_later_proof(self):
|
||||
policy=worker.Admission({});meter=worker.Meter()
|
||||
meter.consume({'kind':'gpu_failure','time':self.now+1})
|
||||
sample={'tailscale_path':'direct','latency_ms':1,'network_ok':True,'registered':True,'gpu_ready':True,'gpu_accessible':True,'gpu_tested_at':self.now,'on_ac':True,'disk_free_bytes':100*1024**3,'temperature_c':None,'throttle_counts':{},'active_jobs':0}
|
||||
self.assertEqual(policy.decide(sample,meter,{'paused':False},self.now+2),'GPU_UNAVAILABLE')
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
247
infra/peertube-workers/win/tests/test_win_worker.py
Normal file
247
infra/peertube-workers/win/tests/test_win_worker.py
Normal file
|
|
@ -0,0 +1,247 @@
|
|||
"""Tests for the Windows worker port (win/). Runs on Linux; no GPU, no Windows host.
|
||||
|
||||
Covers the three pieces the wave doc says must hold before first hardware contact:
|
||||
1. ffmpeg-qsv.py argument rewriting matches the validated Linux adapter's
|
||||
contract and fails closed on unreviewed commands (exit 78).
|
||||
2. the capability-query passthrough list matches what the shipped runner
|
||||
actually generates (verified in the 0.2.0 bundle).
|
||||
3. pt-worker-win.py Meter/Admission semantics are byte-identical to the
|
||||
Linux controller's for every decision input (states, hysteresis, floors).
|
||||
"""
|
||||
import importlib.util
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
WIN = Path(__file__).resolve().parents[2] / 'win'
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def load(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
qsv = load('qsv_win', WIN / 'ffmpeg-qsv.py')
|
||||
win = load('worker_win', WIN / 'pt-worker-win.py')
|
||||
linux = load('worker_linux', ROOT / 'pt-worker.py')
|
||||
|
||||
PROBE = {'streams': [{'codec_type': 'video', 'codec_name': 'h264', 'pix_fmt': 'yuv420p',
|
||||
'width': 1920, 'height': 1080}],
|
||||
'format': {'duration': '10.0'}}
|
||||
|
||||
|
||||
class RewriteTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.old_state, self.old_probe = qsv.STATE, qsv.probe
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
qsv.STATE = Path(self.tmp.name)
|
||||
def stub_probe(path): # no real ffprobe.exe on this Linux host
|
||||
name = str(path).lower()
|
||||
if name.endswith(('.png', '.mjpeg', '.webp')):
|
||||
return {'streams': [{**PROBE['streams'][0], 'codec_name': 'png',
|
||||
'pix_fmt': 'rgba'}], 'format': {'duration': '0'}}
|
||||
return PROBE
|
||||
qsv.probe = stub_probe
|
||||
|
||||
def tearDown(self):
|
||||
qsv.STATE, qsv.probe = self.old_state, self.old_probe
|
||||
self.tmp.cleanup()
|
||||
|
||||
def rewritten(self, args, data=PROBE):
|
||||
out, _ = qsv.rewrite(args, input_probe=data)
|
||||
return out
|
||||
|
||||
def test_h264_web_video_becomes_qsv_pipeline(self):
|
||||
out = self.rewritten(['-i', 'in.mp4', '-vf', 'scale=w=-2:h=720', '-c:v', 'libx264',
|
||||
'-preset', 'veryfast', '-crf', '18', '-maxrate', '1500K',
|
||||
'-bufsize', '3000K', '-g', '96', '-x264-params', 'nofast-pskip=1',
|
||||
'-movflags', '+faststart', 'out.mp4'])
|
||||
text = ' '.join(out)
|
||||
self.assertIn('-hwaccel qsv', text)
|
||||
self.assertIn('-hwaccel_output_format qsv', text)
|
||||
self.assertIn('-c:v h264_qsv', text)
|
||||
self.assertIn('scale_qsv=w=1280:h=720', text)
|
||||
self.assertIn('-preset medium', text)
|
||||
self.assertIn('-b:v 1500K', text) # runner's rate cap kept, CRF dropped
|
||||
self.assertIn('-g 96', text) # runner's GOP kept
|
||||
self.assertNotIn('libx264', text)
|
||||
self.assertNotIn('-crf', text)
|
||||
|
||||
def test_hevc_source_uses_hevc_qsv_decode(self):
|
||||
data = {**PROBE, 'streams': [{**PROBE['streams'][0], 'codec_name': 'hevc', 'pix_fmt': 'yuv420p10le'}]}
|
||||
out = self.rewritten(['-i', 'in.mkv', '-c:v', 'libx264', 'out.mp4'], data)
|
||||
self.assertIn('-c:v hevc_qsv', ' '.join(out))
|
||||
|
||||
def test_no_vf_gets_default_scale_qsv(self):
|
||||
out = self.rewritten(['-i', 'in.mp4', '-c:v', 'libx264', 'out.mp4'])
|
||||
self.assertIn('scale_qsv=w=1920:h=1080', ' '.join(out))
|
||||
|
||||
def test_audio_only_passthrough(self):
|
||||
out, info = qsv.rewrite(['-i', 'in.mp4', '-vn', '-c:a', 'aac', 'out.m4a'])
|
||||
self.assertEqual(info, None)
|
||||
|
||||
def test_capability_queries_passthrough(self):
|
||||
for args in (['-version'], ['-encoders'], ['-hwaccels']):
|
||||
out, info = qsv.rewrite(args)
|
||||
self.assertEqual((out, info), (args, None))
|
||||
|
||||
def test_fail_closed_cpu_or_unknown(self):
|
||||
# Note: -vf contents are validated after the (stubbed) probe, matching
|
||||
# the Linux adapter, which needs the probe to resolve -2 dimensions.
|
||||
for args in (['-i', 'in.mp4', '-c:v', 'mpeg4', 'out.mp4'],
|
||||
['-i', 'in.mp4', '-c:v', 'libx264', '-filter_complex', 'x', 'out.mp4'],
|
||||
['-i', 'in.mp4', '-c:v', 'libx264', '-vf', 'scale=w=-2:h=720,hue=s=0', 'out.mp4'],
|
||||
['-i', 'in.png', '-c:v', 'libx264', '-vf', 'scale=w=-2:h=720', 'out.mp4'],
|
||||
['-i', 'in.mp4', '-c:v', 'libx264', '-vf', 'scale=w=0:h=720', 'out.mp4']):
|
||||
with self.assertRaises(ValueError):
|
||||
qsv.rewrite(args)
|
||||
|
||||
def test_refusal_process_exit_78(self):
|
||||
import os
|
||||
env = {**os.environ, 'PT_STATE_DIR': str(self.tmp.name), 'PT_FFPROBE_EXE': 'ffprobe.exe'}
|
||||
p = subprocess.run([sys.executable, str(WIN / 'ffmpeg-qsv.py'),
|
||||
'-i', 'x.mp4', '-c:v', 'libx264', 'o.mp4'],
|
||||
capture_output=True, text=True, env=env)
|
||||
self.assertEqual(p.returncode, 78)
|
||||
|
||||
def test_events_have_fixed_reasons_only(self):
|
||||
import os
|
||||
env = {**os.environ, 'PT_STATE_DIR': str(self.tmp.name), 'PT_FFPROBE_EXE': 'ffprobe.exe'}
|
||||
p = subprocess.run([sys.executable, str(WIN / 'ffmpeg-qsv.py'),
|
||||
'-i', 'x.mp4', '-c:v', 'libx264', 'o.mp4'],
|
||||
capture_output=True, text=True, env=env)
|
||||
events = [json.loads(line) for line in
|
||||
(Path(self.tmp.name) / 'events.jsonl').read_text().splitlines()]
|
||||
self.assertEqual(events[0]['kind'], 'gpu_failure')
|
||||
self.assertEqual(events[0]['reason'], 'ValueError')
|
||||
self.assertNotIn('x.mp4', p.stderr)
|
||||
|
||||
|
||||
class RunnerContractTests(unittest.TestCase):
|
||||
"""Verified directly in the shipped runner 0.2.0 bundle this session."""
|
||||
|
||||
def test_queries_actually_issued_by_runner(self):
|
||||
bundle = Path('/tmp/peertube-worker-inspect/runtime/node_modules/@peertube/peertube-runner'
|
||||
'/dist/peertube-runner.js')
|
||||
if not bundle.exists():
|
||||
self.skipTest('pinned runner bundle not present')
|
||||
text = bundle.read_text(errors='replace')
|
||||
# Exactly the queries fluent-ffmpeg's capability probe issues in 0.2.0
|
||||
# (-hwaccels is NOT issued; the shim passes it through anyway, harmlessly).
|
||||
for query in ('-version', '-formats', '-encoders', '-filters', '-codecs'):
|
||||
self.assertIn(query, text)
|
||||
self.assertNotIn('-hwaccels', text)
|
||||
self.assertIn('encodersToTry = this.availableEncoders.encodersToTry', text)
|
||||
self.assertIn('"libx264"', text)
|
||||
|
||||
def test_spawn_without_shell_and_ffprobe_sibling(self):
|
||||
bundle = Path('/tmp/peertube-worker-inspect/runtime/node_modules/@peertube/peertube-runner'
|
||||
'/dist/peertube-runner.js')
|
||||
if not bundle.exists():
|
||||
self.skipTest('pinned runner bundle not present')
|
||||
text = bundle.read_text(errors='replace')
|
||||
self.assertNotIn('shell: true', text) # EINVAL on .cmd/.bat shims
|
||||
self.assertIn('utils.isWindows ? "ffprobe.exe" : "ffprobe"', text)
|
||||
self.assertIn('!utils.isWindows', text) # `nice` skipped on win32
|
||||
|
||||
|
||||
class AdmissionParityTests(unittest.TestCase):
|
||||
"""The Windows controller must decide identically to the Linux controller."""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.old_win, self.old_lin = win.STATE, linux.STATE
|
||||
win.STATE = linux.STATE = Path(self.tmp.name)
|
||||
self.now = time.time()
|
||||
|
||||
def tearDown(self):
|
||||
win.STATE, linux.STATE = self.old_win, self.old_lin
|
||||
self.tmp.cleanup()
|
||||
|
||||
def meter_with_transfers(self, module, mbps):
|
||||
meter = module.Meter()
|
||||
for direction in ('download', 'upload'):
|
||||
meter.consume({'kind': 'transfer', 'time': self.now, 'ok': True, 'direction': direction,
|
||||
'bytes': int(mbps * 1e6 / 8), 'seconds': 1})
|
||||
return meter
|
||||
|
||||
def sample(self, **overrides):
|
||||
base = {'tailscale_path': 'direct', 'latency_ms': 5, 'network_ok': True,
|
||||
'registered': True, 'gpu_ready': True, 'gpu_tested_at': self.now,
|
||||
'gpu_accessible': True, 'on_ac': True, 'disk_free_bytes': 100 * 1024 ** 3,
|
||||
'temperature_c': 50, 'throttle_counts': {}, 'active_jobs': 0}
|
||||
base.update(overrides)
|
||||
return base
|
||||
|
||||
def parity(self, overrides, control=None, decisions=1):
|
||||
control = control or {'paused': False}
|
||||
outputs = []
|
||||
for module in (linux, win):
|
||||
policy = module.Admission({})
|
||||
meter = self.meter_with_transfers(module, 50)
|
||||
for _ in range(decisions):
|
||||
outputs.append(policy.decide(self.sample(**overrides), meter, control, self.now))
|
||||
self.assertEqual(outputs[0], outputs[1],
|
||||
f'Linux and Windows controllers diverge on {overrides}: {outputs}')
|
||||
return outputs[0]
|
||||
|
||||
def test_happy_path(self):
|
||||
self.parity({}, decisions=4)
|
||||
|
||||
def test_power_loss_drains_identically(self):
|
||||
self.assertEqual(self.parity({'on_ac': False}), 'DRAINING')
|
||||
|
||||
def test_unknown_power_drains_identically(self):
|
||||
self.assertEqual(self.parity({'on_ac': None}), 'DRAINING')
|
||||
|
||||
def test_missing_gpu_proof_blocks_identically(self):
|
||||
self.assertEqual(self.parity({'gpu_ready': False, 'gpu_accessible': False}), 'GPU_UNAVAILABLE')
|
||||
|
||||
def test_gpu_failure_event_after_proof_blocks_identically(self):
|
||||
outputs = []
|
||||
for module in (linux, win):
|
||||
policy = module.Admission({})
|
||||
meter = self.meter_with_transfers(module, 50)
|
||||
meter.consume({'kind': 'gpu_failure', 'time': self.now + 1})
|
||||
outputs.append(policy.decide(self.sample(gpu_tested_at=self.now), meter,
|
||||
{'paused': False}, self.now + 2))
|
||||
self.assertEqual(outputs[0], 'GPU_UNAVAILABLE')
|
||||
self.assertEqual(outputs[0], outputs[1])
|
||||
|
||||
def test_low_disk_and_thermal(self):
|
||||
self.assertEqual(self.parity({'disk_free_bytes': 10 * 1024 ** 3}), 'LOW_DISK')
|
||||
self.assertEqual(self.parity({'temperature_c': 90}), 'THERMAL_THROTTLED')
|
||||
|
||||
def test_derp_disallowed_and_latency_ceiling(self):
|
||||
self.assertEqual(self.parity({'tailscale_path': 'DERP'}), 'NETWORK_DEGRADED')
|
||||
self.assertEqual(self.parity({'latency_ms': 500}), 'NETWORK_DEGRADED')
|
||||
|
||||
def test_bandwidth_floor_hysteresis(self):
|
||||
# 5 Mbps < 10 Mbps floor: every sample is bad, both controllers degrade.
|
||||
for module in (linux, win):
|
||||
policy = module.Admission({})
|
||||
meter = self.meter_with_transfers(module, 5)
|
||||
outputs = [policy.decide(self.sample(), meter, {'paused': False}, self.now)
|
||||
for _ in range(3)]
|
||||
self.assertEqual(outputs, ['NETWORK_DEGRADED'] * 3, module.__name__)
|
||||
|
||||
def test_recovery_factor(self):
|
||||
outputs = []
|
||||
for module in (linux, win):
|
||||
policy = module.Admission({})
|
||||
meter = self.meter_with_transfers(module, 11)
|
||||
outputs.append(policy.decide(self.sample(), meter, {'paused': False}, self.now))
|
||||
outputs.append(policy.decide(self.sample(), meter, {'paused': False}, self.now))
|
||||
self.assertEqual(outputs[0], 'NETWORK_DEGRADED')
|
||||
self.assertEqual(outputs[0], outputs[1])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=1)
|
||||
27
infra/peertube-workers/worker.example.json
Normal file
27
infra/peertube-workers/worker.example.json
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
{
|
||||
"name": "worker-name",
|
||||
"peertube_url": "http://TAILSCALE_IP:9000",
|
||||
"canonical_origin": "https://tube.occupymedia.cc",
|
||||
"probe_url": "http://TAILSCALE_IP:9044",
|
||||
"probe_interval_seconds": 300,
|
||||
"peertube_version": "7.3.0",
|
||||
"tailscale_peer": "TAILSCALE_IP",
|
||||
"runner_binary": "/opt/pt-worker/runtime/node_modules/.bin/peertube-runner",
|
||||
"node_binary": "/opt/pt-worker/node/bin/node",
|
||||
"render_device": "/dev/dri/renderD128",
|
||||
"concurrency": 2,
|
||||
"require_ac": true,
|
||||
"poll_seconds": 5,
|
||||
"min_disk_free_bytes": 21474836480,
|
||||
"max_temperature_c": 85,
|
||||
"thermal_cooldown_seconds": 120,
|
||||
"min_download_mbps": 10,
|
||||
"min_upload_mbps": 10,
|
||||
"max_latency_ms": 150,
|
||||
"allow_derp": false,
|
||||
"recovery_factor": 1.25,
|
||||
"good_samples": 3,
|
||||
"bad_samples": 2,
|
||||
"bandwidth_ttl_seconds": 900,
|
||||
"transfer_failure_cooldown_seconds": 60
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue