lean

A dockerized MCP server driven by YAML domain manifests. Built-in domains ship out of the box: PDF (marker-pdf + OCR + markitdown + optional VLM), markdown/source-file corpus, and URL corpus. Adding a fourth domain = drop a YAML in configs/ + a tools.py in src/lean/domains/. PDF-like domains also need adapters.py, metadata.py, and optionally parser.py

• a prompt file — see "Adding a new domain" below.

Uses marker-pdf for high-quality PDF extraction (with optional remote GPU acceleration), Vision-Language Model enrichment for charts and figures, section-aware chunking, GPU-accelerated embeddings, provenance-tracked storage, and hybrid BM25 + vector search via pgvector.

Features

• YAML-driven domains — every command (ingest, search, reingest, eval, health, db-init, mcp-serve, api-serve) is selected via --config <yaml>. Built-in domains: lean-pdf-lss, lean-code, lean-web. Domain-specific tools register via shared @mcp_tool / @rest_route / @cli_command decorators in lean.core.adapters.
• Marker-pdf extraction (primary) — datalab-to/marker (surya OCR + texify) with proper table/equation/heading formatting and figure extraction. Runs locally on CPU or on a remote GPU server via HTTP (up to 44× faster on a 29-page PDF; single-machine benchmark — see docs/marker-server-deployment.md for the rig). Falls back to Unlimited-OCR (remote transformers) then markitdown (pure Python) when unavailable.
• VLM chart/image enrichment — Vision-Language Model (MiniMax M3 default, Ollama Qwen3.5 fallback) describes every chart, diagram, and figure at ingest time. Descriptions are structured (title, chart_type, axis_labels, key_data_points, description) and embedded alongside text, making visual content searchable. ~3.5s/image, ~$0.004/image via MiniMax M3 API. Ships disabled-by-default (vlm.enabled: false) in all 3 example configs.
• Provenance metadata — every chunk tracks embedding_model, embedding_dim; every image chunk tracks image_hash, provenance_model (which VLM described it). Enables model-version auditing and future dedup.
• Chunk-type filter — search returns text and image chunks; filter by chunk_type="image" to surface only charts/figures.
• Section-aware chunking — mistune AST parser splits markdown by headings, then a recursive tiktoken-based splitter bounds chunks to a target token window
• GPU-accelerated embeddings — LiquidAI/LFM2.5-Embedding-350M (1024-dim) served via Ollama on the GPU server, with automatic local CPU fallback
• Hybrid search — BM25 full-text (PostgreSQL tsvector) fused with pgvector cosine similarity via Reciprocal Rank Fusion (RRF, k=60)
• Cross-encoder reranking — fetch wide candidate set, rerank with ms-marco-MiniLM-L-6-v2, return top-k
• MCP server — tools exposed over stdio or HTTP, configured per YAML manifest. Built-in domains ship tools + matching REST routes via shared decorators.
• Retrieval evaluation — lean eval command computing hit_rate@k, MRR@k, NDCG@k, Recall@k. Default mode measures self-similarity, not real-world retrieval — see docs/evaluation.md. Use --dataset for curated mode or --mode full for the full search pipeline.
• Optional LLM sidecar — Contextual Retrieval, HyDE, multi-query generation — all opt-in, pipeline works without LLM
• Security hardening — corpus-root path confinement, API key validation (min 16 chars, change-me rejected), HuggingFace model revisions pinned to SHA hashes, non-root Docker user, multi-stage build
• Optional ML deps — torch/transformers/sentence-transformers only when local CPU embeddings are needed (uv sync --extra local-models); marker-pdf for high-quality extraction (uv sync --extra marker); trafilatura for web extraction (uv sync --extra web)

Tech Stack

Prerequisites

• Python 3.12+ with uv
• Docker (for local Supabase)
• Remote GPU server (NVIDIA, 12GB+ VRAM) running:
  • Marker server: pure-Python HTTP wrapper around marker's PdfConverter on port 8000 (44× faster than CPU). Configure at settings.marker.remote_url in the YAML, or MARKER_REMOTE_URL env.
  • Ollama: lfm2.5-embed-32k model (LFM2.5-Embedding-350M, 32K context) on port 11434
  • (optional) Unlimited-OCR server: baidu/Unlimited-OCR via transformers on port 8001 (secondary extraction fallback)

All GPU services are optional — lean falls back to local marker-pdf (CPU), local CPU embeddings, and markitdown when remote servers are not configured. VLM enrichment is also optional (disable via vlm.enabled: false at the top level of the YAML).

Quick Start

git clone <repo-url> && cd lean

# 1. Install dependencies
uv sync --all-groups
uv sync --extra marker          # marker-pdf for high-quality extraction
# (optional) Local CPU embeddings + reranker (~2GB torch):
uv sync --extra local-models
# (optional) URL extraction for the lean-web domain:
uv sync --extra web

# 2. Install git hooks
make hooks-install

# 3. Configure secrets
cp .env.example .env  # set SUPABASE_DB_URL, LEAN_MCP_API_KEY (min 16 chars, not 'change-me')
# (optional) VLM_API_KEY for MiniMax, MINIMAX_API_KEY for LLM features

# 4. Start local database
docker compose up -d supabase-db
make db-init

# 5. Pick a domain and ingest
make CONFIG=configs/lean-pdf-lss.yaml ingest-all   # PDFs
make CONFIG=configs/lean-code.yaml ingest-one FILE=README.md   # markdown
make CONFIG=configs/lean-web.yaml ingest https://example.com    # URL

# 6. Search (text + image chunks)
make search QUERY="What is DMAIC?"
make search QUERY="Pareto chart of defects"   # surfaces VLM-described images

For the remote GPU server (marker + Ollama + optional OCR setup), see docs/ocr-server-deployment.md.

Commands

CLI (lean)

All commands support --json for structured output. Use -v / --verbose for debug logging.

Every command requires --config <yaml> (or LEAN_CONFIG env var).

Domain-specific commands (auto-registered from the domain's tools.py):

Search options (all domains): --k, --doc-id, --section, --author, --year-min, --year-max, --min-score, --chunk-type text\|image.

Makefile

Usage

MCP Client (Claude Desktop, opencode, etc.)

{
  "mcpServers": {
    "lean": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/lean", "lean", "--config",
               "configs/lean-pdf-lss.yaml", "mcp-serve"]
    }
  }
}

Tools exposed by each domain (via shared @mcp_tool / @rest_route / @cli_command decorators in lean.core.adapters):

• lean-pdf-lss — ingest_pdf, search, get_chunk, list_documents, get_document_markdown, delete_document, reingest, corpus_stats
• lean-code — ingest_file, search, get_chunk, list_documents, corpus_stats, get_document_markdown, delete_document
• lean-web — ingest_url, search, list_documents, corpus_stats

Resources and prompts are not auto-shipped. Add them by writing @mcp.resource / @mcp.prompt functions in your domain's tools.py.

REST API

make api-serve   # http://localhost:8766

curl -H "Authorization: Bearer $LEAN_MCP_API_KEY" \
     "http://localhost:8766/search?query=What+is+DMAIC%3F&k=5"

Domain errors map to HTTP codes: ValueError → 400, PermissionError → 403, FileNotFoundError → 404. See docs/architecture.md for the universal error-handling contract.

Adding a new domain

1. Create src/lean/domains/<my_domain>/ with at minimum tools.py (MCP/REST/CLI tool declarations). PDF-like domains also need adapters.py (extraction pipeline wiring), metadata.py (custom metadata extraction), and optionally parser.py + a prompt file (e.g. VLM chart-extraction prompts).
2. Drop a YAML manifest in configs/<my-domain>.yaml.
3. Run uv run lean --config configs/<my-domain>.yaml --help to verify the surface.

The shared decorators in lean.core.adapters register the same function across MCP, REST, and CLI surfaces — write once, expose everywhere.

Documentation

• docs/ — reference docs (configuration, operations, architecture, evaluation, limitations)
• docs/configuration.md — every Settings field, defaults, validators, gotchas
• docs/operations.md — Docker ports, healthcheck semantics, post-ingest reindex, reingest semantics
• docs/architecture.md — pipeline, directory layout, transport tier pattern
• docs/limitations.md — known caveats
• docs/decisions/ — Architecture Decision Records
• AGENTS.md — operational rules for agents

License

MIT for project code. See model licenses for third-party weights (datalab-to/marker, baidu/Unlimited-OCR, LiquidAI/LFM2.5-Embedding-350M, MiniMax M3).

For known caveats and runtime behavior, see docs/limitations.md and AGENTS.md.