#flaker

• sampling a smaller local test run from history and changed files
• detecting flaky tests in noisy CI environments
• measuring how well local sampled runs predict CI
• embedding the same core logic in MoonBit as a library

• the full test suite is too expensive to run on every change
• CI failures are noisy because flaky tests are mixed with real regressions
• developers need a smaller local test run that still correlates well with CI

• Which tests should I run for this change?
• How much can I shrink local execution without losing too much confidence?
• Which tests are actually flaky?
• How well does local sampled execution predict CI outcomes?

Upgrading from 0.0.x / 0.1.x? See docs/how-to-use.md#config-migration for the full key rename map. Starting with 0.2.0, the CLI refuses to start on legacy configs and points to the migration guide.

Upgrading from 0.4.x? See docs/migration-0.4-to-0.5.md or docs/migration-0.4-to-0.5.ja.md. 0.5.x keeps existing profiles working, but the recommended user-facing commands are now gate-oriented.

#Install as a CLI

pnpm add -D @mizchi/flaker

pnpm dlx @mizchi/flaker --help

• Node.js 24+
• pnpm 10+

#Install as a Claude Code plugin

• flaker-setup Introduce flaker on a fresh repository. Day 0 → Week 4 onboarding flow, decision points, copy-paste commands, and pitfalls.
• flaker-management Operate flaker after setup. Advisory vs required gating, nightly triage, quarantine, flaky tag management, and staged Playwright E2E / VRT rollout.

# In Claude Code
/plugin marketplace add mizchi/flaker
/plugin install flaker@flaker

• "新しいプロジェクトに flaker をセットアップしたい"
• "flaker の advisory を required に上げる条件を決めたい"
• "E2E VRT の nightly triage を設計したい"

#Use as a MoonBit Library

• pure computation APIs
• the shared contract types they consume and return

import {
  "mizchi/flaker" @flaker,
}

test "sample from historical runs" {
  let meta = @flaker.build_sampling_meta(
    [
      @flaker.SamplingHistoryRowInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
        status: "passed",
        retry_count: 0,
        duration_ms: 1200,
        created_at: "2026-04-03T00:00:00.000Z",
      },
    ],
    [
      @flaker.SamplingListedTestInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
      },
    ],
  )

  let sampled = @flaker.sample_weighted(meta, count=1, seed=1UL)
  assert_eq(sampled.length(), 1)
}

• flaky detection: detect_flaky
• sampling: build_sampling_meta, sample_random, sample_weighted, sample_hybrid
• affected analysis: resolve_affected, build_affected_report, build_affected_report_from_input
• stable identity: create_stable_test_id, resolve_test_identity
• graph helpers: find_affected_nodes, expand_transitive, topological_sort
• report reducers: summarize_report, classify_report_diff, aggregate_report
• policy: summarize_quarantine, compute_quarantine_exit_code, run_config_check
• metrics: build_sampling_kpi

#Experimental Direct MoonBit CLI

moon run src/cmd/flaker --target js -- --help
moon run src/cmd/flaker --target native -- --version

• --help
• --version
• JS bridge exports used by the npm CLI

#Core Workflow

1. Collect or import test results from CI and local runs.
2. Build history for flaky detection and sampling.
3. Ask flaker to choose a smaller local test set.
4. Run the sampled tests and compare local outcomes with CI.
5. Evaluate whether the sampling strategy is actually trustworthy.

#Mental Model

• Gate: a decision boundary such as "can the author keep iterating?", "can this PR merge?", or "can we release?"
• Budget: the constraints that a gate must stay within
• Loop: a background routine that keeps a gate trustworthy over time
• Policy: the rules for how the system reacts when trust drops

#1. Gates

• Iteration Gate: fast local feedback for the author
• Merge Gate: the PR or mainline gate
• Release Gate: a stricter full-suite or pre-release check

#2. Budgets

• Time budget: how long humans are willing to wait
• Signal budget: how much false negative or flaky noise is acceptable
• Execution budget: CI minutes, parallelism, browser workers, compute cost
• Product performance budget: LCP, INP, bundle size, latency, or similar

• execution performance of the test system itself
• performance of the product being tested

#3. Loops

• Observation loop: scheduled full runs, history collection, holdout verification, KPI refresh
• Triage loop: quarantine review, @flaky tagging, promotion, demotion, owner assignment
• Incident loop: retry, confirm, diagnose, and fix after a real failure

#4. Policies

• when to auto-quarantine
• when to exclude @flaky tests from normal execution
• when to promote an advisory check to required
• when to demote a required check back to advisory
• how retries are interpreted

#Mapping to flaker today

• users choose gates and budgets
• operators run loops and maintain policies
• flaker chooses strategies such as affected, hybrid, or full

#Quick Start

pnpm add -D @mizchi/flaker
pnpm flaker init --adapter <playwright|vitest|jest|junit> --runner <same or actrun>
# edit [affected].resolver in flaker.toml (workspace | glob | bitflow)
pnpm flaker doctor
pnpm flaker plan          # preview what apply will do
pnpm flaker apply         # converge (safe to re-run)
pnpm flaker status        # dashboard + promotion drift

Canonical command forms (0.9.0)

The primary surface is 11 commands: init, plan, apply, status, run, doctor, debug, query, explain, import, report. The 0.7.0 legacy aliases were removed in 0.8.0. 0.9.0 re-deprecates flaker ops daily (removed in 1.0.0). ops weekly / ops incident remain Advanced first-class.

Canonical (0.9.0)	Deprecated (since → removed)
flaker apply	(replaces collect ci / local / coverage / calibrate, quarantine suggest / apply, policy quarantine, analyze flaky-tag — all removed in 0.8.0)
flaker apply --emit daily [--output <file>]	flaker ops daily (deprecated 0.9.0, remove 1.0.0)
flaker apply --emit weekly	ops weekly remains first-class (operator narrative)
flaker apply --output <file>	new in 0.9.0 — saves ApplyArtifact JSON
flaker plan --output <file>	new in 0.9.0 — saves PlanArtifact JSON
flaker status	(replaces analyze kpi, kpi — removed in 0.8.0)
flaker status --markdown	(replaces analyze eval --markdown — removed in 0.8.0)
flaker status --list flaky	(replaces analyze flaky — removed in 0.8.0)
flaker status --gate <name> --detail --json	(replaces gate review / history / explain — removed in 0.8.0)

Breaking JSON shape changes in 0.9.0: apply --json uses executed[*].status: "ok"|"failed"|"skipped" (was .ok: boolean + aborted). status --json uses drift.unmet[*].kind / .desired (was .field / .threshold). See CHANGELOG for full notes.

#Initialize

flaker init --owner your-org --name your-repo

#Collect test history

export GITHUB_TOKEN=$(gh auth token)
flaker collect --days 30

flaker import report <file> --adapter playwright --commit "$(git rev-parse HEAD)"

flaker import report <file> --adapter playwright --source ci --commit <sha>

flaker collect local --last 20

[runner]
type = "playwright"
command = "pnpm exec playwright test -c playwright.config.ts"

[runner.actrun]
workflow = ".github/workflows/ci.yml"
local = true
trust = true
# job = "e2e"

#Inspect health and operator state

flaker status
flaker gate review merge
flaker gate history merge --json
flaker quarantine suggest --json
flaker ops weekly --json

flaker analyze flaky
flaker analyze reason
flaker analyze eval --json
flaker analyze eval --markdown --window 7
flaker analyze eval --markdown --window 7 --output .artifacts/flaker-review.md

• health score
• flaky counts
• local/CI sampling correlation
• average sample ratio
• average saved minutes
• fallback rate

#Sample tests before pushing

flaker run --dry-run --strategy hybrid --count 25
flaker run --dry-run --strategy affected --changed src/foo.ts
flaker run --dry-run --gate iteration --changed src/foo.ts

#Sample and execute

flaker run --strategy hybrid --count 25
flaker run --strategy affected --changed src/foo.ts
flaker run --gate iteration --changed src/foo.ts
flaker run --runner actrun

• P(CI pass | local pass)
• P(CI fail | local fail)
• false negatives / false positives
• average saved minutes

#Execution Gates

# Auto-detects the backing profile
flaker run

# Explicit gate selection
flaker run --gate iteration
flaker run --gate merge
flaker run --gate release

# Advanced: explicit profile names still work
flaker run --profile local
flaker run --profile ci
flaker run --profile scheduled

[runner]
type = "playwright"
command = "pnpm exec playwright test -c playwright.config.ts"
flaky_tag_pattern = "@flaky"

[profile.scheduled]
strategy = "full"

[profile.ci]
strategy = "hybrid"
sample_percentage = 30
adaptive = true        # auto-adjust based on false negative rate
skip_flaky_tagged = true

[profile.local]
strategy = "affected"
max_duration_seconds = 60
fallback_strategy = "weighted"
skip_flaky_tagged = true

• release / scheduled runs all E2E tests and accumulates history
• merge / iteration exclude tests tagged with @flaky
• flaker ops weekly --json carries both quarantine and flaky-tag suggestions for operator review

1. Start with full E2E execution so flaker can accumulate baseline history.
2. When a Playwright test is known to be flaky, tag it with @flaky. flaker detects both Playwright tags and @flaky embedded in the test title.
3. Keep release / scheduled as full execution, and use merge / iteration with skip_flaky_tagged = true. For Playwright, flaker passes --grep-invert @flaky to normal runs.
4. Run weekly triage and let an AI agent update test sources based on the artifact:

flaker ops weekly --json > .artifacts/flaker-weekly.json

• suggestions.add: untagged tests that are unstable enough to move into @flaky
• suggestions.remove: currently tagged tests that have enough consecutive clean passes to return to normal execution
• suggestions.keep: currently tagged tests that should remain excluded from normal execution

flaker analyze flaky-tag --json > .artifacts/flaky-tag-triage.json

• quarantine.flaky_rate_threshold_percentage
• quarantine.min_runs

flaker analyze flaky-tag --json \
  --add-threshold 30 \
  --min-runs 10 \
  --remove-after-passes 5

flaker exec affected --changed src/foo.ts
flaker run --dry-run --gate iteration --changed src/foo.ts
flaker run --gate iteration --changed src/foo.ts

#CI Integration

#PR Comments

- name: Run flaker
  run: flaker run --gate merge

- name: Post PR comment
  if: github.event_name == 'pull_request'
  run: |
    flaker report summary --adapter vitest --input report.json --pr-comment \
      | gh pr comment ${{ github.event.pull_request.number }} --body-file -
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

#Auto-create Issues for Quarantined Tests

flaker quarantine suggest --json --output .artifacts/quarantine-plan.json
flaker quarantine apply --from .artifacts/quarantine-plan.json --create-issues

#Self-Host Rollout

• ci.yml: a self-host-advisory job runs on pull requests, executes flaker run --gate merge, snapshots kpi / eval, and updates a sticky PR comment.
• nightly-self-host.yml: a scheduled job rebuilds recent CI history with flaker collect, runs flaker run --gate release, and updates a rolling issue labeled flaker-self-host.

• matched commits >= 20 — commits where both a gated local/CI run and a release/full run exist in the same window, so that local sampling outcomes can be compared against a ground-truth full run. Increases as the nightly --gate release accumulates history.
• false negative rate <= 5% — share of commits where the merge gate passed but the full run failed (sampling missed a real regression). Measured over the same matched-commit window.
• pass correlation >= 95% — P(full run passes | merge gate passes) on matched commits. Same metric referenced elsewhere in this README as P(CI pass | local pass).
• holdout FNR <= 10% — FNR measured on the holdout slice defined by [sampling] holdout_ratio. Tests in the holdout slice are excluded from the sampled run so that their outcomes can be used to audit whether the sampler's verdict generalizes. Guards against sampler overfitting to the visible slice.
• data confidence reaches moderate or high — derived signal combining matched-commit count, history window coverage, and flaky-noise level. Rough rule of thumb: low until ~10 matched commits, moderate around 20–40 with FNR/correlation green, high beyond 40 with stable noise. Exact boundaries come from the status --gate merge --detail --json output, not from config.

#Recommended Usage Model

1. flaker run --gate release in a nightly scheduled workflow (full test + data accumulation)
2. flaker run --gate merge on PR push (selective execution, posts PR comment)
3. flaker run --gate iteration during development (fast feedback)
4. Review flaker status, flaker gate review merge, and flaker ops weekly weekly
5. Only tighten the workflow after local-to-CI correlation looks strong

• long CI times
• flaky tests
• structured reports such as Playwright JSON, JUnit XML, or Vitest JSON

#Main Commands

#Collection and import

flaker collect
flaker collect local
flaker import report <file> --adapter playwright
flaker collect coverage --format istanbul --input coverage/coverage-final.json

#Sampling and execution

flaker run --dry-run --strategy random --count 20
flaker run --dry-run --strategy weighted --count 20
flaker run --dry-run --strategy affected
flaker run --dry-run --strategy hybrid --count 50

flaker run --strategy hybrid --count 50
flaker run --strategy affected --changed src/foo.ts

#Flag precedence

Resolution order (highest to lowest):
  1. Explicit CLI flag          (--strategy, --percentage, --count)
  2. [profile.<name>] in flaker.toml   (via --profile or auto-detection)
  3. [sampling] in flaker.toml         (project default)
  4. Built-in defaults

Notes:
  --count overrides --percentage when both are given
  --changed overrides git auto-detection
  --dry-run suppresses execution, still records selection telemetry
  --explain can be combined with --dry-run or a real run

#Analysis

flaker analyze flaky
flaker analyze reason
flaker analyze eval
flaker dev train
flaker analyze query "SELECT * FROM test_results LIMIT 20"

#Confirm suspected failures

# Remote: triggers workflow_dispatch, waits for result
flaker debug confirm "tests/api.test.ts:handles timeout"
flaker debug confirm "tests/api.test.ts:handles timeout" --repeat 10

# Local: runs via test runner directly
flaker debug confirm "tests/api.test.ts:handles timeout" --runner local

• BROKEN — fails on every repeat (failures == N). Treat as a real regression.
• FLAKY — fails on some but not all repeats (0 < failures < N). Candidate for @flaky tag or quarantine.
• TRANSIENT — passes on every repeat (failures == 0). The original CI failure did not reproduce here; either CI-environment specific or a one-shot noise event.

#Retry CI failures locally

# Re-run failed tests from most recent CI failure
flaker debug retry

# From a specific workflow run
flaker debug retry --run 12345678

#Policy and ownership

flaker quarantine suggest --json --output .artifacts/quarantine-plan.json
flaker quarantine apply --from .artifacts/quarantine-plan.json --create-issues
flaker policy check
flaker exec affected --changed src/foo.ts

#Reporting

flaker report summary --adapter vitest --input report.json --markdown
flaker report summary --adapter vitest --input report.json --pr-comment
flaker report diff --base base.json --head head.json

#Minimal Configuration

[repo]
owner = "your-org"
name = "your-repo"

[storage]
path = ".flaker/data.duckdb"

[adapter]
type = "playwright"

[runner]
type = "playwright"
command = "pnpm exec playwright test"
flaky_tag_pattern = "@flaky"

[affected]
resolver = "workspace"

[flaky]
window_days = 14
detection_threshold_ratio = 0.02

[quarantine]
auto = true
flaky_rate_threshold_percentage = 30
min_runs = 10

[sampling]
strategy = "hybrid"
sample_percentage = 30
holdout_ratio = 0.1
co_failure_window_days = 90

[profile.scheduled]
strategy = "full"

[profile.ci]
strategy = "hybrid"
sample_percentage = 30
adaptive = true
skip_flaky_tagged = true

[profile.local]
strategy = "affected"
max_duration_seconds = 60
fallback_strategy = "weighted"
skip_flaky_tagged = true

#Docs

• 新規プロジェクト導入チェックリスト (ja)
• New Project Onboarding Checklist
• Usage Guide (ja)
• Usage Guide
• Operations Guide (ja)
• Operations Guide
• Detailed Command Reference
• Why flaker
• Design Partner Rollout

#Release

node_modules/@mizchi/flaker/dist/cli/main.js