k8s-gpu-mcp-server

Just-in-Time SRE Diagnostic Agent for NVIDIA GPU Clusters on Kubernetes

Overview

k8s-gpu-mcp-server is an ephemeral diagnostic agent that provides surgical, real-time NVIDIA GPU hardware introspection for Kubernetes clusters via the Model Context Protocol (MCP).

Unlike traditional monitoring systems, this agent is designed for AI-assisted troubleshooting by SREs debugging complex hardware failures that standard Kubernetes APIs cannot detect.

✨ Key Features

🎯 Low-Footprint, Always-Available - Persistent HTTP server (~15-20MB idle) performs GPU work only when tools are invoked
🔌 HTTP Transport - JSON-RPC 2.0 over HTTP/SSE (production default)
🔍 Deep Hardware Access - Direct NVML integration for GPU diagnostics
🤖 AI-Native - Built for Claude Desktop, Cursor, and MCP-compatible hosts
📋 MCP Prompts - Pre-built GPU diagnostic workflows for guided troubleshooting
🔒 Secure by Default - Read-only operations with explicit operator mode
⚡ Production Ready - Real Tesla T4 testing, 550+ tests passing

🚀 Quick Start

One-Click Install

Click the button above to install automatically in Cursor.

One-Line Installation

hljs language-bash

# Using npx (recommended)
npx k8s-gpu-mcp-server@latest

# Or install globally
npm install -g k8s-gpu-mcp-server

📋 Manual Configuration: Cursor / VS Code

Add to ~/.cursor/mcp.json (Cursor) or VS Code MCP config:

hljs language-json

{
  "mcpServers": {
    "k8s-gpu-mcp": {
      "command": "npx",
      "args": ["-y", "k8s-gpu-mcp-server@latest"]
    }
  }
}

📋 Manual Configuration: Claude Desktop

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

hljs language-json

{
  "mcpServers": {
    "k8s-gpu-mcp": {
      "command": "npx",
      "args": ["-y", "k8s-gpu-mcp-server@latest"]
    }
  }
}

Install from Source

hljs language-bash

# Clone and build
git clone https://github.com/ArangoGutierrez/k8s-gpu-mcp-server.git
cd k8s-gpu-mcp-server
make agent

# Test with mock GPUs (no hardware required)
cat examples/gpu_inventory.json | ./bin/agent --nvml-mode=mock

# Test with real GPU (requires NVIDIA driver)
cat examples/gpu_inventory.json | ./bin/agent --nvml-mode=real

Deploy to Kubernetes

hljs language-bash

# Deploy with Helm OCI (recommended)
helm install k8s-gpu-mcp-server \
  oci://ghcr.io/arangogutierrez/charts/k8s-gpu-mcp-server \
  --namespace gpu-diagnostics --create-namespace

# Or from local chart
helm install k8s-gpu-mcp-server ./deployment/helm/k8s-gpu-mcp-server \
  --namespace gpu-diagnostics --create-namespace

# Find agent pod on target node
NODE_NAME=<node-name>
POD=$(kubectl get pods -n gpu-diagnostics \
  -l app.kubernetes.io/name=k8s-gpu-mcp-server \
  --field-selector spec.nodeName=$NODE_NAME \
  -o jsonpath='{.items[0].metadata.name}')

# Start diagnostic session
kubectl exec -it -n gpu-diagnostics $POD -- /agent --mode=read-only

Note: GPU access requires runtimeClassName: nvidia configured by GPU Operator or nvidia-ctk. For clusters without RuntimeClass, use fallback: --set gpu.runtimeClass.enabled=false --set gpu.resourceRequest.enabled=true

Configure Claude Desktop with kubectl (Advanced)

For deployed agents, add to your Claude Desktop configuration:

hljs language-json

{
  "mcpServers": {
    "k8s-gpu-agent": {
      "command": "kubectl",
      "args": ["exec", "-i", "deploy/k8s-gpu-mcp-server", "-n", "gpu-diagnostics", "--", "/agent"]
    }
  }
}

Then ask Claude: "What's the temperature of the GPUs?"

📖 Full Quick Start Guide → | Kubernetes Deployment →

📊 Architecture

hljs language-scss

┌─────────────────────────────────────────────────────────────────────┐
│                    MCP Client (Claude/Cursor)                        │
└────────────────────────────┬────────────────────────────────────────┘
                             │ stdio / HTTP
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Gateway Pod (:8080)                               │
│       Router → Circuit Breaker → HTTP Client                         │
└────────────────────────────┬────────────────────────────────────────┘
                             │ HTTP (pod-to-pod)
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│  Agent (Node 1) │  │  Agent (Node 2) │  │  Agent (Node N) │
│  9 MCP Tools    │  │  9 MCP Tools    │  │  9 MCP Tools    │
│  NVML → GPU     │  │  NVML → GPU     │  │  NVML → GPU     │
└─────────────────┘  └─────────────────┘  └─────────────────┘

Design Principles:

HTTP-First: Gateway routes via HTTP to agent pods (~50ms latency)
Low Footprint: Persistent HTTP server, ~15-20MB memory
Observable: Circuit breaker, Prometheus metrics, distributed tracing
Interface Abstraction: Testable, flexible, portable (538 tests)

📖 Architecture Documentation →

🛠️ Available Tools

Tool	Description	Category	Status
`get_gpu_inventory`	Hardware inventory + telemetry	NVML	✅ Available
`get_gpu_health`	GPU health monitoring with scoring	NVML	✅ Available
`analyze_xid_errors`	Parse GPU XID error codes from kernel logs	NVML	✅ Available
`get_nvlink_topology`	NVLink interconnect topology and health	NVML	✅ Available
`get_gpu_timeline`	Historical GPU metrics from flight recorder	NVML + Blackbox	✅ Available
`describe_gpu_node`	Node-level GPU diagnostics with K8s metadata	K8s + NVML	✅ Available
`get_pod_gpu_allocation`	GPU-to-Pod correlation via resource requests	K8s	✅ Available
`explain_failure`	Root cause analysis for failed GPU workloads	K8s + Incidents	✅ Available
`get_incident_report`	Detailed incident report with timeline and snapshots	K8s + Incidents	✅ Available
`kill_gpu_process`	Terminate GPU process	Operator	🚧 M4 (Operator)
`reset_gpu`	GPU reset	Operator	🚧 M4 (Operator)

📋 Available Prompts

MCP Prompts provide guided diagnostic workflows that orchestrate multiple tools. See pkg/prompts/prompts.go for prompt definitions.

Prompt	Description
`gpu-health-check`	Comprehensive GPU health assessment with recommendations
`diagnose-xid-errors`	Analyze NVIDIA XID errors with remediation guidance
`gpu-triage`	Standard SRE triage workflow: inventory → health → XID analysis

Example usage with Claude:

hljs language-vbnet

You: "Run the GPU triage workflow for node gpu-worker-5"

Claude: [Executes gpu-triage prompt]
        → Calls get_gpu_inventory, get_gpu_health, analyze_xid_errors
        → Returns structured triage report with recommendations

🔒 Operation Modes

Mode	Flag	Description
Read-Only (default)	`--mode=read-only`	All diagnostic tools, no mutations
Operator	`--mode=operator`	Enables future mutating operations (kill process, reset GPU)

Read-only mode is the default and recommended for most use cases. Operator mode enables future M4 tools that perform write operations on GPUs.

📼 Flight Recorder

The agent includes a built-in flight recorder (pkg/blackbox) that continuously captures GPU telemetry (temperature, power, utilization, memory) into per-GPU ring buffers. This enables tools like get_gpu_timeline and get_incident_report to query historical GPU metrics around the time of a failure.

The flight recorder starts automatically with the agent and requires no additional configuration. Data is retained in memory for the configured window (default: 30 minutes).

📖 MCP Usage Guide →

📈 Project Status

Current Milestone: M3: Kubernetes Integration

Progress: ~90% Complete (HTTP Transport ✅, Gateway ✅, K8s Tools ✅)

Completed Milestones

✅ M1: Foundation & API - Completed Jan 3, 2026
✅ M2: Hardware Introspection - Completed Jan 10, 2026
- Real NVML integration, tested on Tesla T4
- GPU health monitoring, XID error analysis
- npm/Helm distribution

Recent Updates (Jan 2026)

Jan 17: MCP Prompts support - 3 built-in GPU diagnostic workflows
Jan 16: Documentation 360 review for external contributors
Jan 15: K8s tools complete (describe_gpu_node, get_pod_gpu_allocation)
Jan 14: HTTP Transport Epic complete - 150× latency improvement
Jan 14: Cross-node networking fix (Calico VXLAN)
Jan 13: Gateway mode with circuit breaker & Prometheus metrics

📊 View All Milestones →

🧪 Testing

Unit Tests (No GPU Required)

hljs language-bash

make test                   # Run all unit tests (538 tests passing)
make coverage               # Generate coverage report
make coverage-html          # View coverage in browser

Integration Tests (Requires GPU)

hljs language-bash

make test-integration       # Run on GPU hardware
# Or manually:
go test -tags=integration -v ./pkg/nvml/

Latest Test Results:

hljs language-yaml

✓ 538 total tests passing
✓ Race detector enabled (-race)
✓ Coverage: 58-80% by package

Integration tested on Tesla T4:
  - GPU: Tesla T4 (15GB)
  - Temperature: 29°C
  - Power: 13.9W
  - All NVML operations verified

🏗️ Build

hljs language-bash

# Build for local platform
make agent

# Build for Linux (with real NVML)
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 make agent

# Build container image
make image

# Multi-arch release builds
make dist

Binary Sizes:

Mock mode: 4.3MB (CGO disabled)
Real mode: 7.9MB (CGO enabled)

📦 Installation

Using npm (Recommended)

hljs language-bash

# Run directly with npx
npx k8s-gpu-mcp-server@latest

# Or install globally
npm install -g k8s-gpu-mcp-server

From Source

hljs language-bash

git clone https://github.com/ArangoGutierrez/k8s-gpu-mcp-server.git
cd k8s-gpu-mcp-server
make agent
sudo mv bin/agent /usr/local/bin/k8s-gpu-mcp-server

Using Go

hljs language-bash

go install github.com/ArangoGutierrez/k8s-gpu-mcp-server/cmd/agent@latest

Container Image

hljs language-bash

docker pull ghcr.io/arangogutierrez/k8s-gpu-mcp-server:latest

Helm Chart (OCI)

hljs language-bash

# Install from GHCR OCI registry
helm install k8s-gpu-mcp-server \
  oci://ghcr.io/arangogutierrez/charts/k8s-gpu-mcp-server \
  --namespace gpu-diagnostics --create-namespace

🤝 Contributing

We welcome contributions! Please see our Development Guide for details.

Quick Contribution Guide

Check open issues
Fork and create feature branch: git checkout -b feat/my-feature
Make changes, add tests
Run checks: make all
Commit with DCO: git commit -s -S -m "feat(scope): description"
Open PR with labels and milestone

📖 Full Development Guide →

📚 Documentation

Quick Start Guide - Get running in 5 minutes
Kubernetes Deployment - K8s deployment and configuration
Architecture - System design and components
Security Model - RBAC and security configuration
MCP Usage - How to consume the MCP server
Development Guide - Contributing guidelines
Examples - Sample JSON-RPC requests

🔧 Technology Stack

Language: Go 1.25+ (latest stable)
MCP Protocol: mcp-go v0.43.2
GPU Library: go-nvml v0.13.0-1
Testing: testify v1.10.0
Container: Distroless Debian 12 (coming in M3)

🎯 Use Cases

1. Debugging Stuck Training Jobs

hljs language-vbnet

SRE: "Why is the training job on node-5 stuck?"
Claude → k8s-gpu-mcp-server → Detects XID 48 (ECC Error)
Claude: "Node-5 has uncorrectable memory errors. Drain immediately."

2. Thermal Management

hljs language-vbnet

SRE: "Are any GPUs thermal throttling?"
Claude → k8s-gpu-mcp-server → Checks temps and throttle status
Claude: "GPU 3 is at 86°C and thermal throttling. Check cooling."

3. Topology Validation

hljs language-vbnet

SRE: "Is NVLink properly configured for multi-GPU training?"
Claude → k8s-gpu-mcp-server → Inspects NVLink topology
Claude: "All 8 GPUs connected via NVLink, 600GB/s bandwidth."

4. Zombie Process Hunting

hljs language-vbnet

SRE: "GPU memory is full but no pods are running"
Claude → k8s-gpu-mcp-server → Lists GPU processes
Claude: "Found zombie process PID 12345 using 8GB. Kill it?"

🏆 Achievements

✅ Go 1.25 - Latest Go version
✅ Real NVML - Tested on Tesla T4
✅ 550+ Tests Passing - Race detector enabled, 58-80% coverage
✅ HTTP-First Architecture - 150× faster than exec routing
✅ Gateway + Circuit Breaker - Production-grade reliability
✅ MCP Prompts - Guided diagnostic workflows for SRE troubleshooting
✅ Prometheus Metrics - Per-node latency tracking
✅ ~8MB Binary - 84% under 50MB target
✅ MCP 2025-06-18 - Latest protocol version

📄 License

Apache License 2.0 - See LICENSE for details.

🙏 Acknowledgments

NVIDIA NVML - GPU Management Library
Model Context Protocol - MCP Specification
mcp-go - MCP Go Implementation
Anthropic Claude - AI Assistant
Cursor - AI-Powered IDE

📞 Contact

Maintainer: @ArangoGutierrez
Issues: GitHub Issues
Discussions: GitHub Discussions

⭐ Star us on GitHub — it helps!

Report Bug · Request Feature · View Roadmap

k8s-gpu-mcp-server

k8s-gpu-mcp-server

Overview

✨ Key Features

🚀 Quick Start

One-Click Install

One-Line Installation

Install from Source

Deploy to Kubernetes

Configure Claude Desktop with kubectl (Advanced)

📊 Architecture

🛠️ Available Tools

📋 Available Prompts

🔒 Operation Modes

📼 Flight Recorder

📈 Project Status

Current Milestone: M3: Kubernetes Integration

Completed Milestones

Recent Updates (Jan 2026)

🧪 Testing

Unit Tests (No GPU Required)

Integration Tests (Requires GPU)

🏗️ Build

📦 Installation

Using npm (Recommended)

From Source

Using Go

Container Image

Helm Chart (OCI)

🤝 Contributing

Quick Contribution Guide

📚 Documentation

🔧 Technology Stack

🎯 Use Cases

1. Debugging Stuck Training Jobs

2. Thermal Management

3. Topology Validation

4. Zombie Process Hunting

🏆 Achievements

📄 License

🙏 Acknowledgments

📞 Contact

Similar Packages

Tags

Original Author

Publisher