# AGENTS.md - RSS2 Development Guide ## 🛠️ Build & Test Commands ### Backend (Go) ```bash cd backend && go mod tidy cd backend && go build -o ../bin/server ./cmd/server cd rss-ingestor-go && go build -o ../bin/rss-ingestor . # Single test cd backend && go test ./internal/handlers -v -run TestLogin cd backend && go test ./internal/auth -v -run TestGenerateToken ``` ### Frontend ```bash cd frontend && npm install cd frontend && npm run dev # Development cd frontend && npm run build # Production build cd frontend && npm test # Run tests cd frontend && npm run test:ui # UI mode ``` ### Makefile ```bash make build # Build all binaries make clean # Remove binaries make docker-build # Build Docker images ``` --- ## 📁 Project Structure ``` rss2/ ├── backend/ # Go API server + Workers │ ├── cmd/ │ │ ├── server/main.go # API REST principal (Gin) │ │ ├── wiki_worker/main.go # Wikipedia integration │ │ ├── qdrant/main.go # Vector indexing worker │ │ ├── related/main.go # Related news worker │ │ ├── topics/main.go # Country/topic matcher │ │ ├── scraper/main.go # Deep scraping worker │ │ ├── discovery/main.go # RSS feed discovery │ │ └── topics/main.go # Topic matching │ └── internal/ │ ├── handlers/ # HTTP endpoints │ ├── models/ # Data models │ ├── auth/ # JWT authentication │ ├── middleware/ # CORS, Auth middleware │ ├── services/ # ML services (Translate, Embeddings, NER, Semantic Search) │ ├── db/ # PostgreSQL connection │ ├── cache/ # Redis connection │ └── config/ # Configuration loading ├── frontend/ # React + TypeScript + Vite │ └── src/ │ ├── pages/ # Home, News, Search, Admin, Feeds, Stats │ ├── components/ # Layout, UI components │ └── services/ # API client ├── workers/ # Python workers │ ├── ctranslator_worker.py # NLLB-200 translation (CTranslate2) │ ├── ner_worker.py # Spacy NER + Topic extraction │ ├── embeddings_worker.py # Sentence transformers embeddings │ ├── cluster_worker.py # News clustering │ ├── langdetect_worker.py # Language detection │ ├── llm_categorizer_worker.py # Ollama LLM categorization │ ├── simple_categorizer_worker.py │ ├── simple_translator.py │ ├── simple_translator_worker.py │ ├── translation_worker.py │ ├── translation_scheduler.py │ └── remote_translator_worker.py # Remote GPU worker via WebSocket ├── data/ # PostgreSQL, Redis, Qdrant data ├── models/ # ML models (nllb-ct2) ├── hf_cache/ # HuggingFace cache ├── init-db/ # SQL migrations ├── monitoring/ # Prometheus + Grafana config ├── rss-ingestor-go/ # RSS crawler (Go) └── docker-compose.yml # Full stack orchestration ``` --- ## 🚀 Servicios del Sistema ### Capa de Acceso y API (Puerto 8888) | Servicio | Tecnología | Descripción | |---------|------------|-------------| | **nginx** | Nginx Alpine | Gateway y Proxy Inverso | | **rss2_frontend** | React + Vite | Interfaz web responsiva | | **backend-go** | Go + Gin | API REST principal | ### Ingesta y Descubrimiento (Go) | Servicio | Tecnología | Descripción | |---------|------------|-------------| | **rss-ingestor-go** | Go | Crawler RSS de alto rendimiento | | **scraper** | Go | Scraper profundo con sanitización HTML | | **discovery** | Go | Agente de descubrimiento de feeds RSS | ### Procesamiento de Datos e IA | Servicio | Tecnología | Descripción | |---------|------------|-------------| | **translator** | NLLB-200 (CPU) | Traducción neuronal CTranslate2 | | **translator-gpu** | NLLB-200 (GPU) | Traducción acelerada CUDA | | **remote-translator** | WebSocket | Worker GPU remoto | | **embeddings** | S-Transformers | Generación de vectores semánticos | | **ner** | Spacy + BERT | Reconocimiento de entidades (PER, ORG, LOC) | | **llm-categorizer** | Ollama/Mistral | Clasificación con modelos de lenguaje | | **wiki-worker** | Go | Integración Wikipedia + thumbnails | | **topics** | Go | Matcher de países y temas | | **related** | Go | Detección de noticias relacionadas | | **qdrant-worker** | Go | Vectorización + búsqueda semántica | | **cluster** | Python | Agrupación de noticias | | **langdetect** | Python | Detección de idioma | | **translation-scheduler** | Python | Creador de tareas de traducción | ### Capa de Almacenamiento | Servicio | Tecnología | Descripción | |---------|------------|-------------| | **db** | PostgreSQL 18 | Base de datos relacional | | **qdrant** | Qdrant | Base de datos vectorial | | **redis** | Redis 7 | Cache y colas de mensajes | ### Monitoreo | Servicio | Tecnología | Descripción | |---------|------------|-------------| | **prometheus** | Prometheus | Métricas del sistema | | **grafana** | Grafana | Dashboard (puerto 3001) | | **cadvisor** | cAdvisor | Monitoreo Docker | --- ## 📝 Code Style Guidelines ### Backend (Go) #### Imports ```go package handlers import ( "net/http" "github.com/gin-gonic/gin" "github.com/rss2/backend/internal/auth" "github.com/rss2/backend/internal/models" ) ``` **Order:** Standard → Third-party → Local packages **Use:** `goimports` to auto-format #### Naming - Packages: lowercase (`handlers`, `services`) - Functions/Methods: camelCase (`GetNews`, `CreateUser`) - Variables: camelCase (`userId`, `newsList`) - Constants: UPPER_SNAKE_CASE (`MaxPageSize`) - Types: PascalCase (`NewsResponse`, `User`) #### Error Handling ```go func CreateResource(c *gin.Context) { var req CreateRequest if err := c.ShouldBindJSON(&req); err != nil { c.JSON(http.StatusBadRequest, models.ErrorResponse{ Error: "Invalid request", Message: err.Error(), }) return } // ... } ``` **Always return errors** | **Use `models.ErrorResponse`** #### Database ```go err := db.GetPool().QueryRow(ctx, "SELECT * FROM users WHERE id = $1", id).Scan(&user) if err != nil { c.JSON(http.StatusNotFound, models.ErrorResponse{Error: "User not found"}) return } ``` **Use context** | **Named parameters ($1, $2)** #### Tests ```go func TestLoginInvalidRequest(t *testing.T) { router := gin.New() router.POST("/auth/login", Login) body := []byte(`{}`) req, _ := http.NewRequest("POST", "/auth/login", bytes.NewBuffer(body)) req.Header.Set("Content-Type", "application/json") w := httptest.NewRecorder() router.ServeHTTP(w, req) if w.Code != http.StatusBadRequest { t.Errorf("expected 400, got %d", w.Code) } } ``` **Use `gin.TestMode`** | **Test error paths** --- ### Frontend (TypeScript + React) #### Imports ```tsx import React, { useState, useEffect } from 'react' import { Routes, Route } from 'react-router-dom' import { Layout } from './components/layout/Layout' import { api } from './services/api' ``` **Order:** React → Router → Components → Services → Utils #### Components ```tsx function NewsList() { const [news, setNews] = useState([]) const [loading, setLoading] = useState(true) const [error, setError] = useState(null) useEffect(() => { fetchNews(); }, []) const fetchNews = async () => { try { const res = await api.get('/news') setNews(res.data) } catch (err) { setError(err.message) } } return (
{loading && } {error && }
) } ``` #### Types ```tsx interface News { id: number title: string summary: string url: string publishedAt: string } ``` #### Styling - Use Tailwind CSS utility classes - Avoid inline styles - Use `clsx` for conditional classes #### TypeScript Rules - Use `strict: true` mode - Avoid `any` - define interfaces - Use type guards for narrowing - Prefer optional chaining `?.` --- ## 🔒 Security Guidelines 1. **Never commit secrets** - Use `.env` in `.gitignore` 2. **Validate all inputs** - Use `go-playground/validator` 3. **Prepared statements** - Prevent SQL injection 4. **Rate limiting** - On sensitive endpoints 5. **HTTPS only** - Enforce in production ### Variables de Entorno Críticas ```bash POSTGRES_PASSWORD # Contraseña PostgreSQL REDIS_PASSWORD # Contraseña Redis DB_PASS # Contraseña para workers SECRET_KEY # Key JWT GRAFANA_PASSWORD # Dashboard password ``` --- ## 🧪 Testing Best Practices ### Backend - Test error cases and edge cases - Use `gin.TestMode` for HTTP tests ### Frontend - Test component rendering - Test API integration - Test error states - Use vitest --- ## 📦 Deployment ```bash # Generar credenciales seguras ./pre-deploy.sh --generate # Validar y desplegar ./pre-deploy.sh # O manualmente docker compose up -d # Escalar workers de traducción docker compose up -d --scale translator-gpu=4 ``` ### Escalado de Workers GPU ```bash # 1 worker GPU (8GB+ VRAM) docker compose up -d --scale translator-gpu=1 # 2 workers GPU (16GB+ VRAM) docker compose up -d --scale translator-gpu=2 # 4 workers GPU (32GB+ VRAM) docker compose up -d --scale translator-gpu=4 ``` --- ## 🔧 Configuración de Workers ### Environment Variables Principales | Variable | Descripción | Default | |----------|-------------|---------| | `DB_HOST` | Host PostgreSQL | localhost | | `DB_PORT` | Puerto PostgreSQL | 5432 | | `DB_NAME` | Nombre base de datos | rss | | `DB_USER` | Usuario PostgreSQL | rss | | `DB_PASS` | Contraseña PostgreSQL | - | | `TARGET_LANGS` | Idiomas destino | es | | `TRANSLATOR_BATCH` | Tamaño de batch | 32 | | `CT2_DEVICE` | Dispositivo (cpu/cuda) | cpu | | `CT2_COMPUTE_TYPE` | Tipo (int8/float16) | int8 | | `NER_BATCH` | Batch NER | 64 | | `EMB_BATCH` | Batch embeddings | 64 | --- ## 📊 Endpoints de API Principales ### News - `GET /api/news` - Listar noticias (paginado, filtros) - `GET /api/news/:id` - Ver noticia con entidades - `DELETE /api/news/:id` - Eliminar noticia (admin) ### Feeds - `GET /api/feeds` - Listar feeds - `POST /api/feeds` - Crear feed (auth) - `PUT /api/feeds/:id` - Actualizar feed (auth) - `DELETE /api/feeds/:id` - Eliminar feed (auth) ### Search - `GET /api/search?q=...` - Búsqueda texto - `GET /api/search?q=...&semantic=true` - Búsqueda semántica ### Entities - `GET /api/entities?tipo=persona` - Listar entidades (PER, ORG, LOC) ### Admin - `GET /api/admin/backup` - Backup SQL completo - `GET /api/admin/backup/news` - Backup noticias (ZIP) - `GET /api/admin/users` - Listar usuarios - `POST /api/admin/workers/start` - Iniciar workers traducción - `POST /api/admin/workers/stop` - Detener workers traducción - `GET /api/admin/workers/status` - Estado de workers ### Auth - `POST /api/auth/login` - Iniciar sesión - `POST /api/auth/register` - Registrarse - `GET /api/auth/me` - Usuario actual (auth) ### Stats - `GET /api/stats` - Estadísticas globales --- ## 🎯 Capabilities del Sistema 1. **Enriquecimiento Wikipedia**: Detecta personas/orgs, descarga biografías e imágenes 2. **Categorización LLM**: Clasificación con Mistral-7B vía Ollama 3. **Búsqueda Semántica**: Qdrant vector search con mxbai-embed-large 4. **Traducción Neuronal**: NLLB-200 (600M-1.3B params) CPU/GPU 5. **NER**: Spacy es_core_news_lg para entidades nombradas 6. **Noticias Relacionadas**: Similitud coseno entre embeddings 7. **Detección de Idioma**: langdetect 8. **Clustering**: Agrupación automática de noticias 9. **WebSocket Workers**: Workers GPU remotos conectados por WS 10. **Backup Automático**: pg_dump con compresión ZIP --- ## 📖 Documentación Adicional - [README.md](../README.md) - Guía de despliegue completo - [DEPLOY.md](./DEPLOY.md) - Instrucciones de producción - [SECURITY_GUIDE.md](./SECURITY_GUIDE.md) - Guía de seguridad - [QUICKSTART_LLM.md](./QUICKSTART_LLM.md) - Configuración LLM - [remote-worker.md](./remote-worker.md) - Workers remotos