T-HUB/docs/design/2026-10-11-content-search-architecture.md

Report saved to research/ocis-content-search-architecture.md. Full findings below — everything marked "verified" was checked against primary sources today (2026-10-11).

---

Recommended architecture (TL;DR)

Layer	Choice	Why
Metadata + lexical search	oCIS search + Tika extractor	3 env vars, works today
Embeddings	bge-m3 on TEI v1.9.4 (86-1.9 image, RTX 3090)	MIT, 8192 ctx, dense+sparse, strong Spanish
Vector store	pgvector in existing Postgres (halfvec, HNSW)	Fewest moving parts; limits stated below
Deep PDF/scans	Docling v2.137 worker, phase 2	MIT, RAG-oriented, verified OCR engines
Fusion	Postgres FTS (spanish/simple) + pgvector + RRF in SQL	Competitive at this scale

1. oCIS native search — verified from v8.2.1 source

• Default (basic extractor): metadata only — name, size, mime, mtime, tags. Engine is bleve, query language KQL, exposed via WebDAV REPORT (ADR-0018 chose WebDAV over Graph API; it explicitly lists "no server-side result filtering" as a consequence).
• Tika adds full-text content for PDF/ODT/DOCX/TXT/MD/EML email/HTML (~1000 MIME types). Env vars verified from source + docs:

  SEARCH_EXTRACTOR_TYPE=tika
  SEARCH_EXTRACTOR_TIKA_TIKA_URL=http://tika:9998
  FRONTEND_FULL_TEXT_SEARCH_ENABLED=true   # frontend svc — enables WebUI full-text
  SEARCH_CONTENT_EXTRACTION_SIZE_LIMIT=20971520   # default 20MB, from source
  

  Also SEARCH_ENGINE_BLEVE_DATA_PATH (docs: keep the index on its own filesystem — it grows unbounded) and SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS (default true). oCIS ships a ready compose snippet (deployments/examples/ocis_full/tika.yml).
• Cost: Tika is a JVM — budget 1–1.5 GB RAM (my estimate). It downloads each file through oCIS during indexing → with S3NG that means R2 fetches (egress free, cost is latency). Quality caveat: Tika is weak on PDF layout (tables/multi-column).
• Results exposure: WebDAV SEARCH REPORT on remote.php/dav (same as WebUI) + file IDs you can join to chat_media_archive. Filter client-side/Postgres-side.

2. Embeddings — verified numbers from model cards today

Model	Dim	Retrieval quality	Ctx	License	Verdict
bge-m3	1024	MIRACL avg 69.2, MTEB-M ret. 54.60	8192	MIT	Winner — balanced EN/ES, dense+sparse in one pass
Qwen3-Embedding-0.6B	1024 (MRL)	MTEB-M ret. 64.64	32K	Apache-2.0	A/B challenger — best scores, needs query instructions
mE5-large-instruct	1024	MTEB-M ret. 57.12	8192 (practical ≤512)	MIT	Solid fallback; needs query:/passage: prefixes
nomic-embed-text-v2-moe	768	MIRACL 65.80, BEIR 52.86	2048	Apache-2.0	Good, shorter ctx
jina-embeddings-v3	1024	high	8192	CC-BY-NC-4.0	Excluded — non-commercial license
gte-modernbert	768	English-centric (not re-verified today)	8192	Apache-2.0	Wrong tool for EN+ES

VRAM: any of these run in ~2–3 GB fp16 on the 3090 — the 24 GB is mostly headroom for OCR later. Throughput ~1–3k chunks/min at 512 tokens (my estimate; no 3090-specific published benchmark found — flagged).

Serving: TEI v1.9.4 (2026-09-15) — its README explicitly lists XLM-RoBERTa (bge-m3/mE5), Qwen3, ModernBERT, NomicBERT, JinaBERT architectures, and ships a dedicated Ampere-8.6 image for the 3090. Over Ollama (GGUF quantization path; fine as the native-Windows fallback), over Infinity (good, MIT, less battle-tested for these archs), over llama.cpp (weakest embedding tooling).

docker run -d --gpus all -p 8090:80 -v /srv/tei:/data \
  ghcr.io/huggingface/text-embeddings-inference:86-1.9 \
  --model-id BAAI/bge-m3 --max-client-requests 8

⚠️ TEI is Linux-only → WSL2/Docker Desktop on the Windows node, or use a Linux GPU node, or Ollama (bge-m3 is in the library — verified).

3. pgvector — yes, honestly stated

pgvector v0.8.7 (current tag). Real limits: vector indexes cap at 2,000 dims (halfvec 4,000) — 1024 fits; HNSW defaults m=16/ef_construction=64, raise m≈24–32 + ef_search 40–100 for recall; filters apply after the index scan, so selective space_id+group_id filters can return fewer than LIMIT rows → fix with 0.8+ iterative index scans, per-space partial indexes (you'll have few spaces), or partitioning. At ~10⁵ chunks (50k chunks × halfvec ≈ 100 MB) a separate Qdrant/Weaviate/Milvus is pure ops overhead: second DB, second backup path, second consistency story. Revisit Qdrant at ~1M+ vectors or heavy multi-tenant filtering. NestJS note: use raw SQL for the vector path; TypeORM has no native pgvector type.

4. Extraction: two-tier

• Tier 1 (phase 1): Tika — covered above; zero glue code, good enough for text-layer PDFs/ODT/DOCX/EML.
• Tier 2 (phase 2): Docling v2.137.0 (MIT, extremely active) on a GPU node for scanned/complex PDFs → hierarchical chunks → TEI. Verified OCR engine list: tesseract/tesserocr (fast CPU, OK Spanish with spa.traineddata), EasyOCR (better on noisy scans, GPU), RapidOCR, Nemotron-OCR. PaddleOCR remains fine as a standalone subprocess but isn't in docling's current integration list. marker (v2.0.0) and Unstructured (0.27.25) don't beat docling for your format mix; watch marker's model-weight license terms if you ever adopt it.
• Embeddings always come from the Hub-side worker, never from the oCIS index — you need chunks aligned to your metadata model, not Tika's whole-file dump.

5. Hybrid search at your scale

No published head-to-head benchmark exists for "Postgres FTS + pgvector + RRF vs dedicated engines" at 10⁴–10⁵ docs (flagged as not found). The strongest evidence the pattern is mainstream: ParadeDB built exactly this (pg_search v0.26.1, hybrid BM25+vector via RRF, documented as the fix for incomparable score scales). Dedicated engines earn their keep at millions of vectors/exotic filters — not at tens of thousands of files. Postgres ships native spanish + english dictionaries; bleve has a Spanish analyzer too (I could not verify oCIS's internal bleve language mapping — flagged). RRF SQL sketch is in the saved report.

6. Change propagation — the verified event contract

From v8.2.1 source, the search service consumes: ItemTrashed, ItemRestored, ItemMoved, ContainerCreated, FileTouched, FileVersionRestored, TagsAdded, TagsRemoved, SpaceRenamed + UploadReady (async/decomposed-storage path — the one that fires after the blob lands in R2; skip Failed) or FileUploaded (sync). Every mutation triggers a per-space debounced reindex (1000 ms default → SEARCH_EVENTS_REINDEX_DEBOUNCE_DURATION).

No webhooks exist in oCIS (verified — nothing in the repo tree). So: subscribe to NATS directly (OCIS_EVENTS_ENDPOINT, auth vars exist since 5.0) from your NestJS worker — reva's bus uses a main-queue topic with per-consumer durable groups, so your consumer is independent of the search service — plus a nightly WebDAV PROPFIND reconciliation, because missed events are otherwise silent drift.

Phased plan

Phase 1 (this week, no embeddings): add Tika container + 3 env vars → keyword search EN/ES in WebUI + WebDAV REPORT API; NestJS joins hits to chat_media_archive by file ID; nightly reconciliation; sample 20 real PDFs to measure extraction quality.

Phase 2 (semantic): CREATE EXTENSION vector → TEI+bge-m3 on GPU → NATS worker (UploadReady/ItemMoved/ItemTrashed/ItemRestored) → download via WebDAV service account → route scanned→Docling/OCR, else fast path → chunk 512/64 → embed → pgvector → RRF endpoint. Backfill ~100k chunks in hours on the 3090 (estimate). A/B bge-m3 vs Qwen3-0.6B on a 25–30-query internal EN+ES eval set before freezing. Full DDL (with halfvec(1024), HNSW m=24, GIN tsv, per-space partial index pattern) is in the saved report.

Honest gaps and risks

1. Informal ES/EN text is out-of-distribution for every model above — MIRACL is Wikipedia-formal. The internal eval set is mandatory, not optional.
2. OCR on real activist scans (low dpi, photocopies) will be the weak link; tesseract-ES degrades hard below ~150 dpi.
3. oCIS reindex granularity is whole-space debounced — big spaces × Tika downloads = lag under burst; NATS downtime needs the reconciliation poll as a backstop.
4. Tika's layout extraction will make phase-1 semantic search mediocre on tables/columns — Docling is the fix, and it's GPU-hungry during OCR.
5. Unverified/estimates flagged in report: 3090 throughput, current leaderboard ranks, bleve's Spanish analyzer mapping, pgvector version inside your Postgres image (check pg_available_extensions).

Key sources (all fetched today): oCIS v8.2.1 source (services/search/pkg/config/*, pkg/search/events.go, ADR-0018, tika.yml) · doc.owncloud.com/ocis/8.2 search docs · reva pkg/events · model cards (bge-m3, Qwen3-Embedding-0.6B, mE5-large-instruct, nomic-v2, jina-v3) · TEI README · pgvector · pg_search hybrid/RRF · Docling OCR docs.