Mine Best Practices from PR Reviews

A pipeline that extracts generalizable best practices from GitHub PR review threads, validates them against the live codebase, and synthesizes them into a curated library with full provenance. The library can be compiled into automated review rules for tools like Cursor's Bugbot.

For detailed architecture and design rationale, see the companion blogpost.

Prerequisites

• gh CLI — authenticated with access to your repository (repo scope for PR comments).
• Claude Code — the pipeline runs as a Claude Code skill, using subagents for parallel processing.
• Python 3.10+ with pyyaml (see pyproject.toml).

Quick Start

# 1. Copy the skill into your target repo
cp -r skills/ /path/to/your-repo/.claude/skills/mine-best-practices/

# 2. Configure
cd /path/to/your-repo/.claude/skills/mine-best-practices/
cp config.example.yaml config.yaml
# Edit config.yaml: set github_repo and scopes for your codebase

Then invoke the skill from Claude Code:

/mine-best-practices --since 2024-06-01 --scope backend

The skill orchestrates the full pipeline — fetching threads, launching parallel extraction/validation/synthesis subagents, and building the library. See SKILL.md for the complete workflow and all invocation options.

Configuration

All repo-specific settings live in config.yaml. See config.example.yaml for the full reference.

Pipeline Stages

The pipeline alternates between deterministic scripts (batching, aggregation, validation) and AI agent steps (extraction, validation, topic assignment, synthesis). Each stage pauses for human review before proceeding.

1. Refresh — Fetch PR review threads from GitHub API
2. Extract — Identify generalizable patterns from review discussions (~30-40% yield)
3. Validate — Check extracted insights against the live codebase, reject stale ones
4. Topic Assignment — Group validated insights into topic categories
5. Synthesis — Merge insights into the library, consolidating duplicates with provenance
6. Build — Compile library into markdown sections and review rules

Data Locations

Cost and Time

The pipeline launches parallel subagents. Rough expectations per stage:

For a repo with 5,000 review threads: ~170 extraction calls, ~25 validation calls, ~30 synthesis calls.

Tips

• Review culture matters. The pipeline works best on repos with substantive review discussions (back-and-forth threads explaining why, not just "LGTM"). Repos with thin reviews will produce thin results.
• Start small. Mine a 1-2 weeks date range first to calibrate yield rates before processing your full history.
• Check yield rates. Typical extraction yield is 30-40%. If it's above 60% or below 20%, review the batch outputs for quality issues.
• Same-day reruns. Running the same scope twice in one day reuses the working directory. Delete or rename tmp/mining_{scope}_{date} first if you need a fresh run.
• Untrusted input. The pipeline feeds PR comment text to LLM subagents. If your repository accepts comments from untrusted contributors, review extracted insights before promoting them to your library.