lap — how many tokens does your API cost an LLM agent?

· MIT · Changelog · Live leaderboard

Every agent session starts by paying for tool definitions it mostly won't use — and pays again for every call and every response. lap measures that, for any OpenAPI spec, live MCP server, or your own agent config: the token cost decomposed (A menu / B call / C result), a 0–100 grade, the rule violations driving the cost, and an applicable patch for the fixable ones. Free, neutral, reproducible — a measuring stick, not a product.

Try it (60 seconds)

pip install "lap-score[mcp]"

lap stack                                   # YOUR installed MCP servers: tokens paid before you type a word
lap score  openapi.json                     # A/B/C decomposition + grade (also --mcp-url for live servers)
lap lint   openapi.json                     # the violations driving the cost (also --mcp "cmd" for servers)
lap fix    openapi.json --apply patched.json  # fixable findings as an OpenAPI Overlay patch
lap badge  openapi.json                     # shields.io grade badge for your README

$ lap stack
  server        kind   tools  menu tokens  compact
  time          stdio      2          283       31
  git           stdio     12         1418      153
  TOTAL                    14         1701      184
Your agent pays ~1,701 tokens of tool menus at session start - before you type a word.

$ lap fix api/openapi.json --apply patched.json
[written] lap-overlay.yaml  (6 action(s))
[written] patched.json  (lint findings: 15 -> 3)     # grade: B (72) -> A (91)

Everything is --json-able and CI-gateable (--max-menu-tokens, --diff --max-growth, --fail-on; a composite GitHub Action and a Spectral ruleset ship in-repo). Full CLI docs: lap/README.md.

How the measurement works

No LLM in the loop for the core numbers — a deterministic pipeline anyone can rerun:

1. List, don't execute. For an MCP server, connect over stdio/HTTP and call the standard tools/list — the exact definitions (name, description, inputSchema) it advertises. Credential-free: tool listing only, dummy env vars where a server won't boot without them. (For OpenAPI, parse the spec instead.)
2. Count the menu (bucket A). Serialize those definitions the way they enter a model's context and tokenize. Offline that's tiktoken — a fixed BPE vocabulary, an algorithm not a model, so the same input always gives the same number; set ANTHROPIC_API_KEY for Anthropic's own count_tokens. This is what a session pays before the first user message.
3. Compact what-if. Re-render the same tools as compact signatures and count that — how much is recoverable without dropping a single tool.
4. Lint the hygiene. Rules flag what burns tokens or accuracy: missing/short descriptions, undescribed params, >600-token single definitions, a whole-menu-too-heavy check.
5. Fold into a grade. menu-tokens-per-tool + hygiene findings, log-scaled → 0–100 + a letter.

Static menu costs (bucket A) are exact and reproducible; whether a deferred/tool-search server actually works and what its per-call costs are is behavioral. For that half we go beyond the static pipeline and run live A/Bs with real LLMs — actual billed calls, measuring accuracy and tokens — kept clearly separate: Anthropic's Tool Search verified at ~90%, its code-execution the one case that didn't hold up, a real server's facade that held up 8.1× at equal accuracy. Run the static check yourself: lap lint --mcp "npx -y your-server".

What the measurements show

The leaderboard — 50 real public APIs, refreshed monthly. Their naive agent menus total ~11.2M tokens; rendering the same operations compactly recovers ~82% on average (lazy tool-search: ~86%). Nobody ships the compact form.

Real tools, not just our own variants — the same accounting pointed at the ecosystem, live, with billed calls where it matters:

• 3 real OpenAPI→MCP generators all emit menus heavier than the naive baseline, 5–28× heavier than compact (GENERATORS, MCP-SERVERS).
• 20 popular published MCP servers, scored as installed — menus from 42 to 21,411 tokens per session (Notion's official server, grade F); ~64k tokens if you connect them all, before the first user message (MCP-LEADERBOARD, refreshed monthly, incl. a grade cross-check against another grader).
• Anthropic's Tool Search: verified live — ~90% billed-token cut on a real 290-op API, server-enforced (TOOL-SEARCH). Their code-execution: disputed on our workload — heavier than naive in 5/5 repeats; its saving is behavioral, not structural (CODE-EXEC).
• A third-party optimizer's self-reported % was mismeasuring — root-caused in its own source: character counts with an asymmetric formula (MCP-COMPRESSOR).
• Compression doesn't cost accuracy — a 500-run live matrix (2 models × 10 tasks × 5 forms): every compressed form matched or beat the naive menu; the cheapest correct answer turned out to be model-dependent (validation.md).
• What the replies cost, measured — a price list of 29 real read-only calls across 13 servers (1 token to 18,439 per reply), the data nobody had published. Structurally: 13 of 29 replies are free text a caller can't filter or paginate, several heavy tools accept no limit parameter at all, and MCP itself has no caller-side field/page selection — the response-side gap every menu optimization leaves untouched (RESPONSE-COSTS).

Results that don't flatter the thesis ship as prominently as the ones that do — the verified/disputed registry of the field's headline claims is docs/FIELD.md, and the standard objections are priced out in CACHE-ECONOMICS ("isn't it cached?") and TOKENIZERS ("whose tokens?").

Who it's for

• You ship an API or MCP server → lap score/lint/fix give you a number, the violations behind it, and a patch; the Action gates PRs that bloat the menu; lap badge shows the grade in your README.
• You build agents → lap stack audits what your own config burns; the leaderboard is due-diligence before wiring up an API.
• You're choosing between MCP / Tool Search / code-execution / a query DSL → this repo measured all of them on the same tasks with the same accounting (token-bench, LANDSCAPE).
• You design API conventions → the LAP profile is the rule set behind the linter; every rule cites its measurement, including the two that earned honest caveats.

Project map

• lap/ — the pip-installable toolkit (start here).
• profile/ — the LAP profile: measured conventions, L1–L4 levels, the grade formula.
• experiments/ — the benchmark (pet-zoo testbed) + every measurement script behind the docs (leaderboard, spec-#2808 simulation, cache economics, tokenizer matrix, …).
• docs/ — the receipts: leaderboard (+ its MCP-server twin), real-tool tests, FIELD claims registry, SPEC-2808 input for the MCP spec discussion.
• spectral/ — the lint rules for existing Spectral setups.
• ROADMAP.md — the full staged history and what's next.

Status & contributing

0.8.x, pre-1.0, actively maintained, MIT — no telemetry, no paid tier, no company. Issues and PRs welcome: CONTRIBUTING.md covers dev setup and the house policies (vendor neutrality, claims need receipts), and there's a "Score my API" issue template — disputes of our numbers are explicitly invited.