self-docs

A self-hosted documentation RAG pipeline for LLM agents — crawl static docs sites, embed them locally with pgvector, and serve semantic search over the Model Context Protocol.

self-docs gives your coding agents (Cursor, Claude Code, Antigravity, or any MCP client) a private, always-current reference library. It crawls upstream documentation sites, chunks and embeds them locally — no third-party embedding API — and exposes hybrid semantic search as MCP tools over streamable HTTP.

Contents

• Why self-docs
• Architecture
• Quickstart — Local Development
• Go CLI & Progressive Disclosure Skill (doc-cli)
• Quickstart — Production (Home-Lab + Traefik)
• MCP Tools & REST Endpoints
• Managing Sources
• Documentation
• Development
• License

Why self-docs

• Local-first embeddings. FastEmbed runs in-process on CPU (ONNX, no GPU/torch); documentation never leaves your network. The model is selectable from a registry (config/models.yaml) — make configure derives the vector dimension and container memory limits from your choice. Default: mixedbread-ai/mxbai-embed-large-v1 (1024-dim).
• Hybrid retrieval. Vector similarity + per-source-language Postgres full-text search over pgvector, so exact terms and semantic matches both surface.
• Efficient re-crawling. Sources can prefer a site's llms.txt index over HTML crawling, and re-syncs use HTTP conditional GET (ETag/If-Modified-Since) to skip unchanged pages before download — see ADR-003.
• Agent-native. Ships as MCP tools (search_docs, list_doc_sources, propose_doc_source) over streamable HTTP — wire it into any MCP client.
• Operator-friendly. Crawl targets live in the database, managed through a loopback-only admin UI or proposed by agents for human approval.
• Self-hostable. One docker compose stack; a Traefik overlay for home-lab ingress.

Architecture

  Cursor ──┐            ┌─────────┐   ┌──────────────┐
  Claude ──┼─ HTTP ──▶  │ Traefik │──▶│ FastMCP srv  │──┐
  Antigrav ┘  /mcp      └─────────┘   │ (search_docs,│  │ SQL
                               │      │  propose_    │  │
  doc-cli (Go CLI / Skill) ───▶│      │  doc_source) │  ▼
  (/api/v1/search, /get)       │      └──────────────┘ ┌────────┐
  operator ── loopback ───────▶│      │ Ingestion    │▶│ pg16 + │
  (/admin UI, 127.0.0.1:8080)  └─────▶│ svc (FastAPI)│ │pgvector│
  internal scheduler ────────────────▶└──────────────┘ └────────┘
  (opt-in, per-source cron)

Source configuration (crawl targets, URL prefixes, schedule) lives in the doc_sources table — not a YAML file. Sources are managed through the loopback-only admin UI at /admin, or proposed by an agent via the propose_doc_source MCP tool (which queues a pending row for human approval and never crawls on its own). The ingestion service includes an opt-in in-process cron scheduler (app.scheduler) for automated re-crawling; see the Runbook for configuration details.

Quickstart — Local Development

cp .env.example .env        # fill in real values
make configure              # optional — pick an embedding model (see below)
make up                     # db + ingestion (:8080) + mcp-server (:8081)
make sync                   # trigger the initial documentation sync

make configure is optional: with no .env overrides both services already use the registry default. Run it to choose a different model — make configure MODEL=BAAI/bge-base-en-v1.5 — and it resolves that model's vector dimension, query/passage prompts, and per-service memory limits into .env, then re-renders db/init/01_schema.sql. Switching models on an existing deployment requires a re-embed; see Runbook → switch the embedding model.

Point local MCP clients at http://127.0.0.1:8081/mcp (streamable HTTP). The server requires an Authorization: Bearer <MCP_TOKEN> header — see Client Setup for per-client configuration.

Go CLI & Progressive Disclosure Skill (doc-cli)

doc-cli is a high-performance Go CLI and progressive disclosure skill that allows terminal AI agents (and human operators) to query self-docs efficiently over the REST API (/api/v1/*).

# Install binary (~/.local/bin/doc-cli) and register global AI skill (~/.gemini/config/skills/doc-cli/SKILL.md)
make install

Agent Progressive Disclosure Protocol (3-Step Workflow)

1. Search First (Token-Efficient Candidate Fetch):

   hljs language-bash

   Copy

   doc-cli search "fastapi dependency injection" --limit 3
   

   Returns candidate chunk IDs, heading paths, relevance scores, and 1-line snippets.

2. Inspect Candidate IDs: Agent evaluates the candidate IDs and heading paths returned.

3. Targeted Fetch by ID:

   hljs language-bash

   Copy

   doc-cli get 42
   

   Fetches exact markdown content for the specified chunk ID.

Skill Diagnostics & Management

doc-cli skill status         # Check global/project skill installation and API health
doc-cli skill install        # Install skill globally (~/.gemini/config/skills/doc-cli/SKILL.md)
doc-cli skill install --project # Install skill locally (.agents/skills/doc-cli/SKILL.md)

Quickstart — Production (Home-Lab + Traefik)

Deploy behind Traefik ingress on a home-lab server:

cp .env.example .env                    # set credentials + DOCS_MCP_HOSTNAME
export MCP_TOKEN=$(openssl rand -hex 32)  # required — persist this in .env
make up-prod                            # applies docker-compose.prod.yml overlay
make sync                               # trigger the initial documentation sync

[!IMPORTANT] MCP_TOKEN is mandatory. If it is missing from .env, mcp-server fails fast on startup and restart-loops. When upgrading an existing deployment, update every client config with the Authorization header before or alongside restarting mcp-server. Follow the MCP_TOKEN upgrade checklist in the runbook.

MCP Tools & REST Endpoints

MCP Server Tools (Streamable HTTP)

Ingestion Service REST Endpoints (/api/v1/*)

Managing Sources

Documentation

Development

# Start an isolated db for testing
docker compose -f docker-compose.yml -f docker-compose.test.yml up -d db

# Run the full suite (unit + integration + e2e)
make test

# Run the retrieval-quality eval (requires a synced db)
make eval

# Lint and static type checks (also enforced in CI)
make lint
make typecheck

Backup and restore are available via make backup, make backup-prune, and make restore FILE=backups/docs_<timestamp>.dump — see the Runbook for the full procedure.

License

Private — not published. All rights reserved; see LICENSE.