Add content-search architecture research (group cloud indexing)
Research: Tika full-text via 3 oCIS env vars (phase 1); bge-m3 on TEI + pgvector halfvec/HNSW in the existing Postgres with Postgres FTS + RRF hybrid (phase 2); Docling tier for scanned PDFs; NATS event subscription (UploadReady et al) + nightly PROPFIND reconciliation for change propagation. jina-v3 excluded (non-commercial). Bilingual EN+ES eval set flagged as mandatory before freezing models.
This commit is contained in:
parent
43035a94e8
commit
fc5f94400f
1 changed files with 85 additions and 0 deletions
85
docs/design/2026-10-11-content-search-architecture.md
Normal file
85
docs/design/2026-10-11-content-search-architecture.md
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
Report saved to `research/ocis-content-search-architecture.md`. Full findings below — everything marked "verified" was checked against primary sources today (2026-10-11).
|
||||
|
||||
---
|
||||
|
||||
# Recommended architecture (TL;DR)
|
||||
|
||||
| Layer | Choice | Why |
|
||||
|---|---|---|
|
||||
| Metadata + lexical search | **oCIS search + Tika extractor** | 3 env vars, works today |
|
||||
| Embeddings | **bge-m3 on TEI v1.9.4** (`86-1.9` image, RTX 3090) | MIT, 8192 ctx, dense+sparse, strong Spanish |
|
||||
| Vector store | **pgvector in existing Postgres** (`halfvec`, HNSW) | Fewest moving parts; limits stated below |
|
||||
| Deep PDF/scans | **Docling v2.137** worker, phase 2 | MIT, RAG-oriented, verified OCR engines |
|
||||
| Fusion | **Postgres FTS (spanish/simple) + pgvector + RRF in SQL** | Competitive at this scale |
|
||||
|
||||
# 1. oCIS native search — verified from v8.2.1 source
|
||||
|
||||
- **Default (`basic` extractor): metadata only** — name, size, mime, mtime, tags. Engine is **bleve**, query language **KQL**, exposed via **WebDAV REPORT** (ADR-0018 chose WebDAV over Graph API; it explicitly lists "no server-side result filtering" as a consequence).
|
||||
- **Tika adds full-text content** for PDF/ODT/DOCX/TXT/MD/**EML email**/HTML (~1000 MIME types). Env vars verified from source + docs:
|
||||
```
|
||||
SEARCH_EXTRACTOR_TYPE=tika
|
||||
SEARCH_EXTRACTOR_TIKA_TIKA_URL=http://tika:9998
|
||||
FRONTEND_FULL_TEXT_SEARCH_ENABLED=true # frontend svc — enables WebUI full-text
|
||||
SEARCH_CONTENT_EXTRACTION_SIZE_LIMIT=20971520 # default 20MB, from source
|
||||
```
|
||||
Also `SEARCH_ENGINE_BLEVE_DATA_PATH` (docs: keep the index on its own filesystem — it grows unbounded) and `SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS` (default true). oCIS ships a ready compose snippet (`deployments/examples/ocis_full/tika.yml`).
|
||||
- **Cost**: Tika is a JVM — budget 1–1.5 GB RAM (my estimate). It downloads each file through oCIS during indexing → with S3NG that means R2 fetches (egress free, cost is latency). Quality caveat: Tika is weak on PDF layout (tables/multi-column).
|
||||
- **Results exposure**: WebDAV SEARCH REPORT on `remote.php/dav` (same as WebUI) + file IDs you can join to `chat_media_archive`. Filter client-side/Postgres-side.
|
||||
|
||||
# 2. Embeddings — verified numbers from model cards today
|
||||
|
||||
| Model | Dim | Retrieval quality | Ctx | License | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| **bge-m3** | 1024 | MIRACL avg **69.2**, MTEB-M ret. 54.60 | **8192** | **MIT** | **Winner** — balanced EN/ES, dense+sparse in one pass |
|
||||
| Qwen3-Embedding-0.6B | 1024 (MRL) | MTEB-M ret. **64.64** | 32K | Apache-2.0 | **A/B challenger** — best scores, needs query instructions |
|
||||
| mE5-large-instruct | 1024 | MTEB-M ret. 57.12 | 8192 (practical ≤512) | MIT | Solid fallback; needs `query:`/`passage:` prefixes |
|
||||
| nomic-embed-text-v2-moe | 768 | MIRACL 65.80, BEIR 52.86 | 2048 | Apache-2.0 | Good, shorter ctx |
|
||||
| jina-embeddings-v3 | 1024 | high | 8192 | **CC-BY-NC-4.0** | **Excluded — non-commercial license** |
|
||||
| gte-modernbert | 768 | English-centric (not re-verified today) | 8192 | Apache-2.0 | Wrong tool for EN+ES |
|
||||
|
||||
VRAM: any of these run in ~2–3 GB fp16 on the 3090 — the 24 GB is mostly headroom for OCR later. Throughput ~1–3k chunks/min at 512 tokens (my estimate; no 3090-specific published benchmark found — flagged).
|
||||
|
||||
**Serving: TEI v1.9.4 (2026-09-15)** — its README explicitly lists XLM-RoBERTa (bge-m3/mE5), Qwen3, ModernBERT, NomicBERT, JinaBERT architectures, and ships a dedicated Ampere-8.6 image for the 3090. Over Ollama (GGUF quantization path; fine as the **native-Windows fallback**), over Infinity (good, MIT, less battle-tested for these archs), over llama.cpp (weakest embedding tooling).
|
||||
|
||||
```bash
|
||||
docker run -d --gpus all -p 8090:80 -v /srv/tei:/data \
|
||||
ghcr.io/huggingface/text-embeddings-inference:86-1.9 \
|
||||
--model-id BAAI/bge-m3 --max-client-requests 8
|
||||
```
|
||||
⚠️ TEI is Linux-only → WSL2/Docker Desktop on the Windows node, or use a Linux GPU node, or Ollama (`bge-m3` is in the library — verified).
|
||||
|
||||
# 3. pgvector — yes, honestly stated
|
||||
|
||||
pgvector **v0.8.7** (current tag). Real limits: `vector` indexes cap at **2,000 dims** (`halfvec` 4,000) — 1024 fits; HNSW defaults m=16/ef_construction=64, raise m≈24–32 + `ef_search` 40–100 for recall; **filters apply after the index scan**, so selective `space_id`+`group_id` filters can return fewer than LIMIT rows → fix with 0.8+ iterative index scans, per-space partial indexes (you'll have few spaces), or partitioning. At ~10⁵ chunks (50k chunks × halfvec ≈ 100 MB) a separate Qdrant/Weaviate/Milvus is pure ops overhead: second DB, second backup path, second consistency story. Revisit Qdrant at ~1M+ vectors or heavy multi-tenant filtering. NestJS note: use raw SQL for the vector path; TypeORM has no native pgvector type.
|
||||
|
||||
# 4. Extraction: two-tier
|
||||
|
||||
- **Tier 1 (phase 1): Tika** — covered above; zero glue code, good enough for text-layer PDFs/ODT/DOCX/EML.
|
||||
- **Tier 2 (phase 2): Docling v2.137.0 (MIT, extremely active)** on a GPU node for scanned/complex PDFs → hierarchical chunks → TEI. Verified OCR engine list: **tesseract/tesserocr** (fast CPU, OK Spanish with `spa.traineddata`), **EasyOCR** (better on noisy scans, GPU), **RapidOCR**, Nemotron-OCR. PaddleOCR remains fine as a standalone subprocess but isn't in docling's current integration list. marker (v2.0.0) and Unstructured (0.27.25) don't beat docling for your format mix; watch marker's model-weight license terms if you ever adopt it.
|
||||
- **Embeddings always come from the Hub-side worker**, never from the oCIS index — you need chunks aligned to your metadata model, not Tika's whole-file dump.
|
||||
|
||||
# 5. Hybrid search at your scale
|
||||
|
||||
No published head-to-head benchmark exists for "Postgres FTS + pgvector + RRF vs dedicated engines" at 10⁴–10⁵ docs (flagged as not found). The strongest evidence the pattern is mainstream: **ParadeDB built exactly this** (pg_search v0.26.1, hybrid BM25+vector via RRF, documented as the fix for incomparable score scales). Dedicated engines earn their keep at millions of vectors/exotic filters — not at tens of thousands of files. Postgres ships native `spanish` + `english` dictionaries; bleve has a Spanish analyzer too (I could not verify oCIS's internal bleve language mapping — flagged). RRF SQL sketch is in the saved report.
|
||||
|
||||
# 6. Change propagation — the verified event contract
|
||||
|
||||
From v8.2.1 source, the search service consumes: `ItemTrashed, ItemRestored, ItemMoved, ContainerCreated, FileTouched, FileVersionRestored, TagsAdded, TagsRemoved, SpaceRenamed` + **`UploadReady`** (async/decomposed-storage path — the one that fires after the blob lands in R2; skip `Failed`) or `FileUploaded` (sync). Every mutation triggers a **per-space debounced reindex** (1000 ms default → `SEARCH_EVENTS_REINDEX_DEBOUNCE_DURATION`).
|
||||
|
||||
**No webhooks exist in oCIS** (verified — nothing in the repo tree). So: **subscribe to NATS directly** (`OCIS_EVENTS_ENDPOINT`, auth vars exist since 5.0) from your NestJS worker — reva's bus uses a `main-queue` topic with per-consumer durable groups, so your consumer is independent of the search service — **plus a nightly WebDAV PROPFIND reconciliation**, because missed events are otherwise silent drift.
|
||||
|
||||
# Phased plan
|
||||
|
||||
**Phase 1 (this week, no embeddings):** add Tika container + 3 env vars → keyword search EN/ES in WebUI + WebDAV REPORT API; NestJS joins hits to `chat_media_archive` by file ID; nightly reconciliation; sample 20 real PDFs to measure extraction quality.
|
||||
|
||||
**Phase 2 (semantic):** `CREATE EXTENSION vector` → TEI+bge-m3 on GPU → NATS worker (UploadReady/ItemMoved/ItemTrashed/ItemRestored) → download via WebDAV service account → route scanned→Docling/OCR, else fast path → chunk 512/64 → embed → pgvector → RRF endpoint. Backfill ~100k chunks in hours on the 3090 (estimate). A/B bge-m3 vs Qwen3-0.6B on a 25–30-query internal EN+ES eval set before freezing. Full DDL (with `halfvec(1024)`, HNSW m=24, GIN tsv, per-space partial index pattern) is in the saved report.
|
||||
|
||||
# Honest gaps and risks
|
||||
|
||||
1. **Informal ES/EN text is out-of-distribution** for every model above — MIRACL is Wikipedia-formal. The internal eval set is mandatory, not optional.
|
||||
2. **OCR on real activist scans** (low dpi, photocopies) will be the weak link; tesseract-ES degrades hard below ~150 dpi.
|
||||
3. **oCIS reindex granularity is whole-space debounced** — big spaces × Tika downloads = lag under burst; NATS downtime needs the reconciliation poll as a backstop.
|
||||
4. Tika's layout extraction will make phase-1 semantic search mediocre on tables/columns — Docling is the fix, and it's GPU-hungry during OCR.
|
||||
5. Unverified/estimates flagged in report: 3090 throughput, current leaderboard ranks, bleve's Spanish analyzer mapping, pgvector version inside your Postgres image (check `pg_available_extensions`).
|
||||
|
||||
**Key sources** (all fetched today): oCIS v8.2.1 source (`services/search/pkg/config/*`, `pkg/search/events.go`, ADR-0018, tika.yml) · [doc.owncloud.com/ocis/8.2 search docs](https://doc.owncloud.com/ocis/8.2/admin/conf-examples/search/configure-search.html) · [reva pkg/events](https://github.com/owncloud/reva/blob/main/pkg/events/files.go) · model cards ([bge-m3](https://huggingface.co/BAAI/bge-m3), [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), [mE5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct), [nomic-v2](https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe), [jina-v3](https://huggingface.co/jinaai/jina-embeddings-v3)) · [TEI README](https://github.com/huggingface/text-embeddings-inference) · [pgvector](https://github.com/pgvector/pgvector) · [pg_search hybrid/RRF](https://github.com/paradedb/paradedb/blob/main/docs/concepts/hybrid/overview.mdx) · [Docling OCR docs](https://github.com/docling-project/docling/blob/main/docs/concepts/OCR.md).
|
||||
Loading…
Add table
Add a link
Reference in a new issue