Agent-K

Every claim backed by evidence. Every action gated by code.
Nothing is trust the model. Everything is prove it in telemetry.

Agent K is a code enforced incident response agent for a FastAPI RAG support answering service. When a SigNoz alert fires, Agent K investigates the failure through the SigNoz MCP server, publishes only evidence backed root cause claims, executes a sandboxed rollback only when a code based policy allows it, and records its own cost and behavior as telemetry so every decision is auditable in SigNoz.

Built for the Agents of SigNoz hackathon (WeMakeDevs x SigNoz), Track 01: AI and Agent Observability (July 20 to 26, 2026).

Why Agent K

AI applications can fail in unusual ways: slow, expensive, inaccurate, or stuck in retry loops, without producing a normal server error. An engineer usually has to comb through traces, metrics, logs, and deployment history to understand what happened, and an AI agent that investigates can itself make unsupported guesses or take unsafe actions.

Agent K makes every investigation traceable, every conclusion verifiable, and every action controlled by code.

The Three Laws (enforced by software, not prompts)

How it works

flowchart TD
    A[SigNoz alert fires] --> B[Agent K receives alert]
    B --> C[Collect traces, metrics, logs via SigNoz MCP]
    C --> D[Investigate with fixed query set]
    D --> E[Form root cause hypotheses]
    E --> F[Attach SigNoz evidence to each claim]
    F --> G{Law 2 policy gate}
    G -->|All checks pass| H[Execute sandboxed rollback]
    G -->|Any check fails| I[Evidence linked escalation to human]
    H --> J[Run verification query]
    J --> K[Produce auditable incident report]
    I --> K
    K --> L[Record own cost, queries, decisions as telemetry]
    L --> M[Investigation complete]

    classDef law fill:#1a1a2e,stroke:#e94560,color:#fff
    classDef safe fill:#0f3460,stroke:#00b894,color:#fff
    classDef deny fill:#2d0a0a,stroke:#e17055,color:#fff
    classDef done fill:#16213e,stroke:#0fbcf9,color:#fff
    class G law
    class H safe
    class I deny
    class M done

What is in this repo

The monitored application

A FastAPI RAG support answering service with:

The four demonstration incidents

Each incident is deliberately seeded via a feature flag or config change and disclosed as a controlled test failure.

Quick start

Prerequisites

1. Stand up SigNoz via Foundry

curl -fsSL https://signoz.io/foundry.sh | bash      # install foundryctl (once)
foundryctl gauge -f casting.yaml                    # validate prerequisites
foundryctl forge -f casting.yaml -p ./pours         # generate compose plus lockfile
foundryctl cast -f casting.yaml                     # full pipeline: gauge + forge + up

Confirm the stack is up, then complete the required first run setup. See docs/SIGNOZ-RUNBOOK.md for the exact steps, including the first run org and register step that binds the OTLP receivers.

2. Start the RAG service datastore

docker compose up -d                # rag postgres (pgvector/pgvector:pg16)

3. Configure environment

cp .env.example .env
# fill in GROQ_API_KEY (and or CEREBRAS_API_KEY / GEMINI_API_KEY)

4. Install, migrate, seed, run

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

alembic upgrade head                 # create documents table plus vector extension
python -m scripts.seed_corpus        # embed and upsert the 72 doc corpus
uvicorn app.main:app --reload        # http://localhost:8000

Smoke test:

curl localhost:8000/healthz          # {"status":"ok"}
curl -X POST localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question":"How do I rotate my API key?"}'

Testing

SigNoz features used

Stack

AI assistance disclosure

This project was built with AI coding assistance (Claude Code, GitHub Copilot), disclosed here per the hackathon rules. A human reviewed and directed all work. All product claims are grounded in the project planning documents, and demonstration results are described honestly as four out of four on four controlled scenarios, not as general production accuracy.

License

MIT. See LICENSE for the full text.

Contributing

See docs/CONTRIBUTING.md for how to set up, run tests, and submit changes.