agentic-qa-boilerplate

                  ░█████  ░██████ ░███████░███   ░██░████████░██ ░██████                         
                 ░██  ░██░██      ░██     ░████  ░██   ░██   ░██░██                              
                 ░███████░██  ░███░█████  ░██░██ ░██   ░██   ░██░██                              
  ██████████     ░██  ░██░██   ░██░██     ░██ ░██░██   ░██   ░██░██                              
  ██▀▀▀▀▀▀██     ░██  ░██ ░██████ ░███████░██  ░████   ░██   ░██ ░██████                         
  ██ ◉  ◉ ██     ░░   ░░  ░░░░░░  ░░░░░░░ ░░   ░░░░    ░░    ░░  ░░░░░░                          
  ██   3  ██                                                                                     
  ██████████     ░███████░███   ░██ ░██████ ░██░███   ░██░███████░███████░██████                 
   ██    ██      ░██     ░████  ░██░██      ░██░████  ░██░██     ░██     ░██  ░██                
                 ░█████  ░██░██ ░██░██  ░███░██░██░██ ░██░█████  ░█████  ░██████                 
                 ░██     ░██ ░██░██░██   ░██░██░██ ░██░██░██     ░██     ░██  ░██                
                 ░███████░██  ░████ ░██████ ░██░██  ░████░███████░███████░██  ░██                
                 ░░░░░░░ ░░   ░░░░  ░░░░░░  ░░ ░░   ░░░░ ░░░░░░░ ░░░░░░░ ░░   ░░                 
                               Quality Assurance Engineer                                        

bunx create-agentic-qa@latest <your-repo-name>

Prerequisites

Before running bunx create-agentic-qa@latest or bun install && bun run setup, install the hard blockers. The installer detects everything else and prints exact install URLs when something is missing — but front-loading these saves a fail-and-retry loop.

Hard blockers (installer exits 1 if missing)

Quasi-required (installer warns + offers install)

Per-skill CLIs (lazy-required — needed when the skill runs, not at setup)

These are not optional for the workflow — each one is required by a specific skill. They are non-blocking at setup time because the installer cannot guess which skills you will actually use. Install them up front if you plan to use the whole stack, or lazily when the skill that uses them surfaces a missing-binary error.

Convenience opt-ins (pure UX, never required)

Windows users: skip direnv. The bun run claude / bun run opencode wrappers already load .env cross-platform with zero setup. direnv on PowerShell needs version 2.37+ and is officially experimental; Git Bash works but at that point the wrapper is simpler. The installer will offer the direnv hook; just decline it.

MCP credentials (.env keys)

.mcp.json (Claude Code) and opencode.jsonc ship with ${VAR} / {env:VAR} placeholders that read from .env. Eight keys are required for the 7 canonical MCPs:

TAVILY_API_KEY
ATLASSIAN_URL · ATLASSIAN_EMAIL · ATLASSIAN_API_TOKEN
API_BASE_URL · OPENAPI_SPEC_PATH · API_TOKEN
POSTMAN_API_KEY

.env.example has the full template with per-var comments. Run bun run setup:doctor at any time to see which are still missing — it prints pending_actions[].where URLs for every credential.

When the installer tells you something is wrong

TL;DR: install Bun + Claude Code (or OpenCode) before you run setup. Everything else, the installer points you at when you hit it.

Start here — pick your path

First-timers, use the scaffolder. It handles tarball download, git scrub, rename, bun install, and the interactive installer in one shot. The manual clone is for people hacking on the boilerplate itself.

What this is

A starter for QA teams that want AI agents driving the testing workflow — not isolated test snippets, but the whole loop. Plan a sprint, document test cases in Jira/Xray, write KATA-compliant Playwright tests, run regression, sign off the release. Eight workflow skills cover the phases. A handful of slash commands handle the chores around them. The development half (project foundation, sprint dev, deploys) lives in agentic-dev-boilerplate — pair them or use one.

Scaffold a new project

create-agentic-qa is the official scaffolder (npm, source in packages/create-agentic-qa/). One command, full setup:

bunx create-agentic-qa@latest <your-repo-name>
cd <your-repo-name>

What it does:

1. Downloads upex-galaxy/agentic-qa-boilerplate (latest main) as a tarball — no git history.
2. Rewrites package.json name + .agents/project.yaml project.name.
3. Initializes a fresh git init -b main with an initial commit.
4. Runs bun install.
5. Hands off to bun run setup — gentle-ai, 14 skills, 9 community skills, 7 MCPs, .env, direnv autoload, optional gh repo create.

Useful flags (full list in packages/create-agentic-qa/README.md):

Then continue with the per-project workflow:

# Optional: open the orientation HTML (single-file tour, sidebar nav)
bun run onboarding

# Optional, Claude Code only: configure the statusline in a SEPARATE terminal
bunx -y ccstatusline@latest

# Drive the QA lifecycle inside the agent:
/agentic-qa-onboard     # first-time orientation tour
/project-discovery      # reverse-engineer the target app into .context/
/adapt-framework        # wire KATA to the target stack (auth, vars, CI, MCP) — run once after discovery
/shift-left-testing     # Stage 0: pre-sprint AC refinement on backlog batch
/sprint-testing         # in-sprint manual QA per ticket (plan + execute + report)
/test-documentation     # TMS docs + ROI scoring (Candidate / Manual / Deferred)
/test-automation        # KATA Plan -> Code -> Review
/regression-testing     # CI execution + GO / CAUTION / NO-GO

Don't chain bun run onboarding && bun run setup — the onboarding server is blocking and the chain deadlocks. Run them as separate steps.

bunx -y ccstatusline@latest is Claude Code-only and optional. Run it from a plain terminal with NO agent running — concurrent TUIs fight over stdin and the configurator silently breaks. OpenCode users skip this: the opencode-subagent-statusline plugin is already wired into opencode.jsonc.

Launching the agent

.mcp.json (Claude Code) and opencode.jsonc ship with ${VAR} / {env:VAR} placeholders — real values live in .env. Launch the agent via one of these so env vars actually load:

# Cross-platform default (uses dotenv-cli, no extra tooling required):
bun run claude        # Claude Code
bun run opencode      # OpenCode

# Optional: direnv autoload (any OS with direnv installed)
direnv allow          # one-time per repo (the installer offers to run this)
claude                # direct binary picks up .env from your shell

# Or load .env into your CURRENT shell once, then run any binary directly:
set -a; source .env; set +a   # exports every .env key into this terminal session
claude                        # now claude / opencode / acli / bun xray all see the vars

The set -a; source .env; set +a snippet is saved as the env script for reference. Run it inline (or source it) — bun run env executes in a child subshell, so the exports would not survive back to your terminal. Run it directly and every agent/CLI launched in that same shell inherits .env with no wrapper.

direnv works on macOS / Linux / Windows. On Windows install via winget install direnv — Git Bash is recommended; PowerShell support is experimental and requires direnv 2.37+. See INSTALLER.md § Launching the agent for the per-shell hook lines.

# 1. Clone the repository
git clone https://github.com/upex-galaxy/agentic-qa-boilerplate.git
cd agentic-qa-boilerplate

# 2. Install dependencies
bun install

# 3. Install Playwright browsers
bun run pw:install

# 4. Copy env template
cp .env.example .env   # then fill in the values

# 5. (Optional) Visual orientation — close tab + Ctrl-C when done.
bun run onboarding

# 6. Run the interactive setup (gentle-ai, skills, MCPs, .env, direnv)
bun run setup

# 7. Validate the install
bun cli/doctor.ts

How it works

Skills auto-trigger when your prompt matches their description frontmatter — or you force-load with a slash command (/sprint-testing). Each skill is a SKILL.md plus a references/ folder. The agent only reads what the current step needs, so context stays lean.

Project values (URLs, project key, Jira fields) live in .agents/project.yaml and get injected into prompts via a 4-syntax variable system. Skills are grouped by phase: onboarding (one-time discovery), in-sprint QA (continuous), automation (per story), regression (per release). The dev companion repo follows the same pattern.

Features

Configuration

This boilerplate has two configuration systems that serve different consumers and must not be conflated:

Both are needed. Skip neither.

(a) Runtime test config — .env

Edit .env with your project values:

# Environment selector (valid: local, staging)
TEST_ENV=local

# Test User Credentials (only the current TEST_ENV is required)
LOCAL_USER_EMAIL=
LOCAL_USER_PASSWORD=
STAGING_USER_EMAIL=your-test-user@example.com
STAGING_USER_PASSWORD=your-password

# Browser Configuration (optional — defaults shown)
HEADLESS=true
DEFAULT_TIMEOUT=30000

# TMS Integration (optional — only if AUTO_SYNC=true)
TMS_PROVIDER=xray
AUTO_SYNC=false
XRAY_CLIENT_ID=
XRAY_CLIENT_SECRET=
XRAY_PROJECT_KEY=

(b) Runtime URLs — config/variables.ts

Update envDataMap in config/variables.ts with your application URLs. The Environment type currently accepts local and staging; extend the type when you need a third environment.

const envDataMap: Record<
  Environment,
  { base: string; api: string; user: { email: string; password: string } }
> = {
  local: {
    base: 'http://localhost:3000',
    api: 'http://localhost:3000/api',
    user: userCredentialsMap.local,
  },
  staging: {
    base: 'https://staging.yourapp.com',
    api: 'https://staging.yourapp.com/api',
    user: userCredentialsMap.staging,
  },
};

(c) AI context engineering — .agents/project.yaml

The AI agents (Claude Code, Codex, Cursor, Copilot, OpenCode) resolve {{VAR}} references in skills, templates, and commands against .agents/project.yaml. Edit it manually, or run the interactive walkthrough:

bun run agents:setup

This populates project.project_name, project.project_key, issue_tracker.atlassian_url, environments.<env>.web_url, environments.<env>.api_url, environments.<env>.db_mcp, environments.<env>.api_mcp, and the rest. See .agents/README.md for the full convention.

Run Tests

# Run all tests
bun run test

# Run with UI mode (recommended for development)
bun run test:ui

# Run specific test types
bun run test:e2e           # E2E tests only
bun run test:integration   # API tests only
bun run test:smoke         # smoke / @critical tests

Project Structure

├── tests/
│   ├── components/               # KATA Components Layer
│   │   ├── TestContext.ts        # Layer 1: Base utilities + faker
│   │   ├── TestFixture.ts        # Layer 4: Unified test fixture
│   │   ├── api/                  # API components
│   │   │   ├── ApiBase.ts        # Layer 2: HTTP client base
│   │   │   └── ExampleApi.ts     # Layer 3: Domain component
│   │   ├── ui/                   # UI components
│   │   │   ├── UiBase.ts         # Layer 2: Page base
│   │   │   └── ExamplePage.ts    # Layer 3: Domain component
│   │   └── steps/                # Reusable ATC chains (preconditions)
│   │
│   ├── e2e/                      # E2E test specs
│   │   └── module-example/       # Example module
│   ├── integration/              # API integration tests
│   │   └── module-example/       # Example module
│   ├── setup/                    # Test setup files
│   │   ├── global.setup.ts       # Global setup
│   │   └── ui-auth.setup.ts      # UI authentication
│   ├── data/
│   │   ├── fixtures/             # Static test data (JSON)
│   │   ├── types.ts              # Test data types
│   │   └── DataFactory.ts        # Dynamic data generation
│   └── utils/                    # Test utilities
│       ├── decorators.ts         # @atc decorator
│       ├── jiraSync.ts           # TMS synchronization
│       └── KataReporter.ts       # Terminal reporter
│
├── config/
│   ├── variables.ts              # Runtime env vars consumed by Playwright/KATA
│   └── validateTestEnv.ts        # Test environment validation
│
├── .context/                     # AI Context Engineering (generated)
│   ├── business/                  # business-data-map / business-feature-map / business-api-map
│   ├── master-test-plan.md       # What to test and why
│   ├── PRD/                      # Product requirements
│   ├── SRS/                      # Technical specs
│   └── PBI/                      # Per-ticket backlog items
│
├── .agents/                      # Agentskills.io spec layout
│   ├── project.yaml              # AI context vars (resolved as {{VAR}} by skills)
│   ├── jira-fields.json                 # Jira custom-field catalog (synced by `bun run jira:sync-fields`)
│   ├── jira-required.yaml        # Required Jira custom-field manifest
│   └── README.md                 # Variable conventions reference
│
├── .claude/skills/               # Claude Code Skills (workflows)
│   ├── agentic-qa-core/           # Foundation: passive reference host (briefing template, dispatch patterns, orchestration doctrine)
│   ├── project-discovery/        # Onboarding + context generation
│   ├── sprint-testing/           # Planning + execution + reporting
│   ├── test-documentation/       # TMS documentation + prioritization
│   ├── test-automation/          # KATA planning + coding + review
│   ├── regression-testing/       # Regression execution + GO/NO-GO
│   ├── xray-cli/                 # Xray TMS helper
│   └── acli/                     # Atlassian CLI helper ([ISSUE_TRACKER_TOOL])
│
├── .github/workflows/            # CI/CD pipelines
│   ├── build.yml                 # PR validation
│   ├── smoke.yml                 # Daily smoke tests
│   ├── sanity.yml                # Pattern-based tests
│   └── regression.yml            # Full regression
│
├── docs/                         # Human-facing docs
│   ├── architectures/            # Architecture references
│   ├── methodology/              # QA methodology
│   ├── setup/                    # Setup guides
│   ├── testing/                  # Testing documentation
│   └── workflows/                # Workflow documentation
│
├── packages/
│   └── create-agentic-qa/        # Official npm scaffolder (bunx create-agentic-qa@latest <your-repo-name>) — own README + tests
│
├── cli/                          # install.ts, doctor.ts, update-boilerplate.ts consumed by bun scripts
│
├── playwright.config.ts          # Playwright configuration
├── INSTALLER.md                  # Contract for bun run setup — what each installer layer does
├── CLAUDE.md                     # AI memory (canonical, read by Claude Code + OpenCode)
└── package.json                  # Scripts and dependencies

KATA Architecture

This boilerplate implements KATA (Component Action Test Architecture).

Architecture Layers

TestContext (Layer 1)
    ↓ extends
UiBase / ApiBase (Layer 2) ← Helpers here
    ↓ extends
YourPage / YourApi (Layer 3) ← ATCs here
    ↓ used by
TestFixture (Layer 4) ← DI entry point
    ↓ used by
Test Files ← Orchestrate ATCs

Component Types

Example Test

import { test, expect } from '@TestFixture';

test.describe('User Dashboard', () => {
  test('@atc:UPEX-101 should display user profile', async ({ dashboardPage }) => {
    await dashboardPage.navigateToDashboard();
    await dashboardPage.openUserProfile();

    await expect(dashboardPage.profileCard).toBeVisible();
    await expect(dashboardPage.userName).toContainText('John');
  });
});

See the /test-automation skill (references/kata-architecture.md) for complete documentation.

Available Scripts

Test Execution

Reports

Code Quality

Utilities

CLI Tools

--upex flag — every jira:sync-* script accepts --upex to download the UPEX-standard reference JSON from upex-galaxy/agentic-qa-boilerplate@main instead of hitting Jira. Use when you don't have admin access, when you want a working catalog without setting up auth, or when you want the canonical UPEX standard as a reference. Examples: bun run jira:sync-fields --upex, bun run jira:sync-workflows --upex, bun run jira:sync-link-types --upex.

Keeping your project in sync with the boilerplate

The bun run up CLI keeps your project aligned with the official template by tracking which upstream commit each piece of the framework (skills/, scripts/, cli/, …) was last synced from. Instead of overwriting framework files blindly, it:

1. Reads .template/boilerplate.lock.json (committed in your repo) to find the last-synced SHA per component
2. Clones the template lazily (sparse checkout, only the dirs that get synced)
3. Computes the exact list of changed files between your synced SHA and template HEAD
4. Classifies each file: clean fast-forward, locally diverged, new upstream, deleted upstream, binary, or whitespace-only
5. In interactive mode: shows you a space-bar checkbox menu with [M/A/D] badges and +N/-M line counts. For files where you edited locally, shows you both diffs and lets you pick theirs, mine, or skip
6. In --auto / CI mode: applies safe changes, skips your edits, never deletes, prints a summary table
7. Writes the new SHA only for components where every changed file was processed (skip = no SHA advance, so a skipped file resurfaces next run)

Requirements: git ≥ 2.25 (for partial-clone with --filter=blob:none), Bun, GitHub CLI authenticated.

Run interactively:

bun run up

Run non-interactively (for CI):

bun run up --auto

Preview without writing:

bun run up --dry-run

Rollback the latest sync:

bun run up --rollback

The .template/boilerplate.lock.json file is committable — commit it so your team and CI know exactly which template version each component is on.

CI/CD Pipelines

GitHub Actions Workflows

Environment Secrets Required

Required (only the credentials matching your active TEST_ENV are validated):

# Environment selection
TEST_ENV                    # local | staging

# Test User Credentials (required for the active TEST_ENV)
LOCAL_USER_EMAIL
LOCAL_USER_PASSWORD
STAGING_USER_EMAIL
STAGING_USER_PASSWORD

Optional (only when the corresponding feature is enabled):

# Browser
HEADLESS                    # default: true
DEFAULT_TIMEOUT             # default: 30000

# TMS — set TMS_PROVIDER + AUTO_SYNC=true to push results
TMS_PROVIDER                # xray | jira
AUTO_SYNC                   # default: false

# Xray Cloud (required if TMS_PROVIDER=xray AND AUTO_SYNC=true)
XRAY_CLIENT_ID
XRAY_CLIENT_SECRET
XRAY_PROJECT_KEY

# Atlassian credentials (required if TMS_PROVIDER=jira AND AUTO_SYNC=true; also
# used by MCP atlassian, acli, xray-cli, and scripts/sync-jira-*.ts — single
# source of truth across the repo).
ATLASSIAN_URL
ATLASSIAN_EMAIL
ATLASSIAN_API_TOKEN
JIRA_TEST_STATUS_FIELD      # default: customfield_10100 — Jira-specific operational param, not a credential

# Reporting
ALLURE_RESULTS_DIR          # default: ./allure-results
SCREENSHOT_ON_FAILURE       # default: true
VIDEO_ON_FAILURE            # default: true

# CI/CD (set automatically by GitHub Actions)
CI
BUILD_ID

Skills

Workflow skills (auto-trigger)

Reusable community skills (installed by bun run setup)

These aren't committed in this repo. The installer fetches them via bunx skills add from upstream community repositories. The exact list lives in cli/install.ts — source of truth, changes faster than this README, consult the file directly.

Skill tiers (T1–T4)

Every skill belongs to one of four tiers. Each tier has different discovery and load rules. Full contract: .claude/skills/agentic-qa-core/references/skill-composition-strategy.md.

Validation: bun run skills:check checks tier coherence (orphan categories, tier mismatches, missing sections, stale doc paths).

Slash commands (utilities)

Variables system

The .agents/ directory hosts a 4-syntax variable system used by every skill and command.

See .agents/README.md for the full contract.

Validation scripts:

bun run vars:check         # Every {{VAR}} and {{jira.*}} reference resolves
bun run jira:sync-fields   # Discover Jira custom fields -> .agents/jira-fields.json
bun run jira:check         # Validate jira-required.yaml against jira-fields.json

TMS Integration (Jira/Xray)

Two TMS modalities are supported out of the box:

• Modality jira-xray: full Xray entities (Test, Test Plan, Test Execution, Test Run, Pre-Condition). Primary tooling is the /xray-cli skill plus /acli for generic Jira issues.
• Modality jira-native (no Xray): ATP/ATR live as Story custom fields + comment mirrors; TCs live as Jira Test issues. All TMS operations fall through to /acli. See .claude/skills/test-documentation/references/jira-setup.md.

For how skills resolve [ISSUE_TRACKER_TOOL] and [TMS_TOOL] tags to concrete CLIs or MCPs, see CLAUDE.md §Tool Resolution.

Configuration

1. Get Xray API credentials from Jira
2. Add to .env:

XRAY_CLIENT_ID=your-client-id
XRAY_CLIENT_SECRET=your-client-secret
XRAY_PROJECT_KEY=YOUR-PROJECT
AUTO_SYNC=true

Sync Test Results

# After test run
bun run test:sync

# Or enable auto-sync in CI
AUTO_SYNC=true bun run test

Link Tests to Test Cases

// Use @atc decorator with Jira key
test('@atc:UPEX-101 should validate login', async ({ loginPage }) => {
  // Test implementation
});

Customization Guide

1. Update Project Identity

Edit these files:

• package.json — name, description, repository
• CLAUDE.md (canonical AI memory, read by both Claude Code and OpenCode)
• .agents/project.yaml — AI context vars (or run bun run agents:setup for an interactive walkthrough)
• config/variables.ts — runtime URLs for Playwright (envDataMap)

2. Add Components

# Create a new page component
touch tests/components/ui/YourPage.ts

# Create a new API component
touch tests/components/api/YourApi.ts

Follow patterns in ExamplePage.ts and ExampleApi.ts.

3. Add Tests

# Create test directory
mkdir tests/e2e/your-module

# Create test file
touch tests/e2e/your-module/your-feature.test.ts

4. Generate Context

Load the /project-discovery skill in your AI assistant to generate project-specific context (PRD, SRS, business-data-map, business-feature-map, business-api-map, master-test-plan).

5. Adapt the Framework

Once .context/ exists, run /adapt-framework to wire the KATA architecture to your stack — auth, variables, OpenAPI facades, CI workflows, and the MCP registry. It runs a 10-phase idempotent flow (no writes before your approval) and, on re-run, reports a GENERIC / ADAPTED checklist of what is still example-project boilerplate. After it passes, you can start writing automated tests.

Companion repo

The development side lives in agentic-dev-boilerplate — project foundation, design system, sprint development, deploys. Same .agents/ variable system, same agentskills.io layout. Pair them or use one.

Cross-agent compatibility

Skills declare compatibility: [claude-code, copilot, cursor, codex, opencode] per the agentskills.io spec. Slash triggers are Claude Code specific; other agents auto-activate from the same description field. The variable system is agent-agnostic.

Future hooks

Room for per-phase model routing, an explicit skill registry, Engram-style cross-session memory, and CI-validated cross-agent portability. Notes in CLAUDE.md.

Contributing

1. Load the /test-automation skill and read its references/kata-architecture.md
2. Follow the automation standards referenced by that skill
3. Use conventional commits
4. Ensure all tests pass before PR

License

MIT — see LICENSE.