feat: add remote worker system for distributed translation
- Add remote translator worker (Docker) that connects via WebSocket - Add backend Go handlers for remote worker CRUD (REST + WebSocket) - Add frontend Admin panel for managing remote workers - Add ProtectedRoute component to secure admin routes - Add PostgreSQL schema for remote_workers table - Fix duplicate Load function in config.go - Fix frontend duplicate fetchStatus function - Fix Dockerfiles missing go.sum (use go mod download + tidy) Features: - Workers connect to server via WebSocket (no ports needed) - API key authentication per worker - Job assigner automatically distributes translation jobs - Worker status tracking (online/offline/disabled) - Timeout handling for stuck jobs
This commit is contained in:
parent
b8e48f3199
commit
8093cc1e40
21 changed files with 2065 additions and 58 deletions
50
Dockerfile.remote-worker
Normal file
50
Dockerfile.remote-worker
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
FROM python:3.11-slim-bookworm
|
||||
|
||||
SHELL ["/bin/bash", "-c"]
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
patchelf libpq-dev gcc git curl wget bash \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
||||
TOKENIZERS_PARALLELISM=false \
|
||||
HF_HOME=/root/.cache/huggingface \
|
||||
DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir --upgrade pip
|
||||
|
||||
RUN pip install --no-cache-dir torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cpu
|
||||
|
||||
RUN pip install --no-cache-dir \
|
||||
ctranslate2==3.24.0 \
|
||||
sentencepiece \
|
||||
transformers==4.36.0 \
|
||||
protobuf==3.20.3 \
|
||||
"numpy<2" \
|
||||
psycopg2-binary \
|
||||
langdetect \
|
||||
websocket-client
|
||||
|
||||
RUN find /usr/local/lib/python3.11/site-packages/ctranslate2* \
|
||||
-name "libctranslate2-*.so.*" -o -name "libctranslate2.so*" | \
|
||||
xargs -I {} patchelf --clear-execstack {} || true
|
||||
|
||||
COPY workers/ ./workers/
|
||||
COPY init-db/ ./init-db/
|
||||
COPY migrations/ ./migrations/
|
||||
|
||||
ENV DB_HOST=db
|
||||
ENV DB_PORT=5432
|
||||
ENV DB_NAME=rss
|
||||
ENV DB_USER=rss
|
||||
ENV DB_PASS=x
|
||||
|
||||
ENV WORKER_SERVER=ws://localhost:8080/ws/worker
|
||||
ENV CT2_DEVICE=cpu
|
||||
ENV CT2_COMPUTE_TYPE=int8
|
||||
|
||||
CMD ["python", "-m", "workers.remote_translator_worker"]
|
||||
Loading…
Add table
Add a link
Reference in a new issue