Textrawl

Your second brain, wired for AI.

Textrawl is a personal knowledge server with persistent memory, searchable documents, and proactive insights. Import your emails, PDFs, and notes — then ask questions, recall past conversations, and discover connections you missed. Access it all through MCP, the web dashboard, or the REST API.

How It Works

┌──────────────┐         ┌───────────────────────────────────────────┐
│              │         │           Your Second Brain               │
│  MCP Client  │◄───────►│                                           │
│  (Claude,    │   MCP   │   Documents    Memory     Conversations   │
│   ChatGPT)   │         │   ┌────────┐  ┌───────┐  ┌────────────┐  │
└──────────────┘         │   │ Emails │  │ Facts │  │Past sessions│  │
                         │   │ PDFs   │  │People │  │ Summaries  │  │
┌──────────────┐         │   │ Notes  │  │ Links │  │ Context    │  │
│   Dashboard  │◄───────►│   └───┬────┘  └──┬────┘  └─────┬──────┘  │
│  (Web UI)    │  REST   │       └──────────┼──────────────┘         │
└──────────────┘         │                  ▼                        │
                         │     ┌───────────────────────────┐         │
                         │     │  Hybrid Search + Fusion   │         │
                         │     └───────────────────────────┘         │
                         │                  │                        │
                         │                  ▼                        │
                         │      Insights · Daily Briefing            │
                         └───────────────────────────────────────────┘
                                            ▲
                                            │
                                 ┌──────────┴──────────┐
                                 │                     │
                            Desktop App            CLI Tools
                           (drag & drop)        (batch import)

Why Textrawl?

Beyond keyword search. Most search tools only match exact words. Textrawl combines semantic understanding (finds "automobile" when you search "car") with traditional keyword matching — so you get relevant results without missing exact phrases.

Your data, your choice. Use OpenAI's embeddings for best accuracy, Google AI for multimodal support, or run locally with Ollama and local Postgres to keep document text and embeddings on your machine.

Import everything. Emails from Gmail exports, PDFs from your research, saved web pages, images, audio files, Google Takeout archives — Textrawl converts them into searchable knowledge where the relevant converter/provider is configured.

Features

Privacy Model

Textrawl is self-hosted, but data leaves your machine when you configure cloud services. Document text, chunks, embeddings, extracted memories, conversation summaries, images, or audio may be sent to providers such as OpenAI/Google embeddings, Anthropic/OpenAI/Google extraction, Neon/Supabase/RDS, Cloud Run, or GCS. For sensitive data, prefer Ollama/local Postgres and disable cloud LLM extraction/insights.

Quick Start

1. Set Up the Server

git clone https://github.com/jeffgreendesign/textrawl.git
cd textrawl
pnpm install
pnpm setup    # Interactive setup for credentials
pnpm dev      # Start the server

2. Set Up Your Database

1. Create a free project at neon.tech

2. Copy the pooled connection string from the Neon dashboard into DATABASE_URL in .env

3. Run the base schema against your database:

   hljs language-bash

   Copy

   psql $DATABASE_URL -f scripts/setup-db.sql
   

   (Use setup-db-ollama.sql for Ollama / setup-db-ollama-v2.sql for nomic-embed-text-v2-moe / setup-db-google.sql for Google AI)

4. (Optional) For memory tools: psql $DATABASE_URL -f scripts/setup-db-memory.sql

5. (Optional) For conversation tools: psql $DATABASE_URL -f scripts/setup-db-conversation.sql

6. (Optional) For Row Level Security hardening: psql $DATABASE_URL -f scripts/security-rls.sql

3. Connect Claude Desktop

Add to your Claude config (~/Library/Application Support/Claude/claude_desktop_config.json). Create this file if it doesn't exist:

{
  "mcpServers": {
    "textrawl": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://localhost:3000/mcp",
        "--header",
        "Accept: application/json, text/event-stream"
      ]
    }
  }
}

Note: Requires Node.js 22+. If using nvm, ensure your default is set: nvm alias default 22

If you've set API_BEARER_TOKEN in .env, add the auth header:

"--header",
"Authorization: Bearer <your-token>"

Restart Claude Desktop - you'll now see Textrawl's tools available.

3b. Connect ChatGPT Desktop (Alternative)

ChatGPT Desktop supports MCP servers natively (Pro/Plus required):

1. Open Settings → Connectors → Advanced → Developer mode
2. Add a new connector with your server URL: http://localhost:3000/mcp
3. If using auth, add the Authorization: Bearer <your-token> header

See OpenAI MCP documentation for details.

4. Add Your Documents

Imported documents, extracted memories, and conversation summaries are stored in your configured database/storage until deleted. Treat a Textrawl server as single-tenant unless you have added your own user isolation. Set API_BEARER_TOKEN, restrict CORS with ALLOWED_ORIGINS, and avoid importing third-party or private data without consent. Use forget_entity and delete_conversation to remove memory/conversation data, and list_documents/update_document to audit imported documents.

Option A: Desktop App (easiest)

pnpm desktop:dev

Drag files onto the window to convert and upload.

Option B: CLI (for batch imports)

pnpm convert -- mbox ~/Mail/archive.mbox
pnpm upload -- ./converted/

Documentation

Configuration

Tool surfaces (compact vs full)

Textrawl advertises one of three tool surfaces via the MCP_TOOLSET environment variable. The default normal surface is a small set of workflow tools — the recommended interface for personal/family assistants. This follows the current MCP maintainer guidance (Anthropic, Writing effective tools for AI agents): fewer tools, consolidated by workflow into distinct, typed, well-named tools — not a single "intent" dispatcher (which would discard per-tool schemas, annotations, and the name signals models use to pick tools). The MCP spec defines no standard tool-filtering primitive, so MCP_TOOLSET is a server-local convention; host/harness lazy tool loading remains the canonical way to surface fewer tools at a time.

Recommended exposure: personal/family bots → MCP_TOOLSET=normal and EXPOSE_ADMIN_TOOLS=false (so destructive forget_entity/delete_conversation are never reachable); admin/dev contexts → MCP_TOOLSET=full.

Model guidance (GPT-5.5, Claude Sonnet, Claude Haiku): express intent through the workflow tool that matches the job — ask to answer a question across sources, search for raw matches, capture to save a note or URL, remember to store facts, timeline/daily_briefing for temporal/briefing views. Use audience to scope who an answer is for; family_shared/public_safe automatically exclude private memory, conversations, and insights.

MCP Tools

Read-only tools (search, get_document, list_documents, query_memory, query_conversations, get_stats, health_check) include outputSchema and return structuredContent for programmatic consumption alongside the text content response.

Workflow Tools (compact normal surface)

ask, search, get_document, daily_briefing, and timeline (documented below) complete the workflow surface. remember is only advertised when ENABLE_MEMORY=true (omitted otherwise). ask/search accept audience + allowCrossProfile for privacy scoping.

Document Tools

Memory Tools (Persistent Memory)

Enable with ENABLE_MEMORY=true (default). Requires scripts/setup-db-memory.sql or setup-db-memory-ollama.sql.

Conversation Tools (Conversation Memory)

Enable with ENABLE_CONVERSATIONS=true (default). Requires running one of the conversation schema scripts:

• scripts/setup-db-conversation.sql (OpenAI embeddings, 1536d)
• scripts/setup-db-conversation-ollama.sql (Ollama v1 - nomic-embed-text, 1024d)
• scripts/setup-db-conversation-ollama-v2.sql (Ollama v2 - nomic-embed-text-v2-moe, 768d)
• scripts/setup-db-conversation-google.sql (Google AI - gemini-embedding-2-preview, 3072d)

Insight Tools (Proactive Discovery)

Enable with ENABLE_INSIGHTS=true (default).

Stats

Unified Tools

Postgres Analysis Tools

Enabled when DATABASE_URL is configured. Connects directly to Postgres.

Search Parameters

REST API

Upload Documents

curl -X POST http://localhost:3000/api/upload \
  -H "Authorization: Bearer <your-token>" \
  -F "file=@document.pdf" \
  -F "title=Optional Title" \
  -F "tags=tag1,tag2"

Limits: 10MB max file size, 10 uploads/min Formats: .pdf, .docx, .txt, .md

Response:

{
  "success": true,
  "documentId": "uuid",
  "title": "Document Title",
  "tags": ["tag1", "tag2"],
  "chunksCreated": 12
}

Health Checks

• GET /health - Basic health
• GET /health/ready - Readiness probe (checks DB)
• GET /health/live - Liveness probe

Deployment

Docker Compose

docker-compose up -d
docker-compose logs -f

Google Cloud Run

# Create secrets in Secret Manager first
export GCP_PROJECT_ID=your-project-id
./scripts/deploy.sh

Development

pnpm dev            # Watch mode
pnpm build          # Production build
pnpm start          # Run production
pnpm typecheck      # Type check
pnpm lint           # Biome lint check
pnpm quality        # Lint + typecheck combined
pnpm inspector      # MCP Inspector
pnpm setup          # Generate .env with secure token
pnpm desktop:dev    # Run desktop app
pnpm docs:dev       # Run docs site

Local Database (Optional)

Run PostgreSQL + pgvector locally:

# Start local Postgres with pgvector
docker-compose -f docker-compose.local.yml up -d

# Initialize the database schema
docker exec -i textrawl-postgres psql -U postgres -d textrawl < scripts/setup-db.sql

# Optional: Start pgAdmin at http://localhost:5050
docker-compose -f docker-compose.local.yml --profile tools up -d

Local Embeddings with Ollama (No API Key Required)

Run embeddings locally with Ollama instead of OpenAI:

# Start Postgres + Ollama
docker-compose -f docker-compose.local.yml --profile ollama up -d

# Pull the embedding model (~274MB)
docker exec textrawl-ollama ollama pull nomic-embed-text

# Use the Ollama-specific schema (1024 dimensions)
docker exec -i textrawl-postgres psql -U postgres -d textrawl < scripts/setup-db-ollama.sql

Set in .env:

EMBEDDING_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=nomic-embed-text

Supported Ollama models: nomic-embed-text (1024d), nomic-embed-text-v2-moe (768d, recommended for new installs), mxbai-embed-large (1024d)

Note: Each provider uses different embedding dimensions: OpenAI 1536d, Ollama 1024d (or 768d for v2-moe), Google AI 3072d. Use the matching schema: setup-db.sql (OpenAI), setup-db-ollama.sql (Ollama 1024d), setup-db-ollama-v2.sql (Ollama 768d), or setup-db-google.sql (Google AI). You cannot mix providers without re-embedding all documents.

Troubleshooting

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

License

MIT - see LICENSE