#flaker

• sampling a smaller local test run from history and changed files
• detecting flaky tests in noisy CI environments
• measuring how well local sampled runs predict CI
• embedding the same core logic in MoonBit as a library

• the full test suite is too expensive to run on every change
• CI failures are noisy because flaky tests are mixed with real regressions
• developers need a smaller local test run that still correlates well with CI

• Which tests should I run for this change?
• How much can I shrink local execution without losing too much confidence?
• Which tests are actually flaky?
• How well does local sampled execution predict CI outcomes?

Upgrading from 0.0.x / 0.1.x? See docs/how-to-use.md#config-migration for the full key rename map. Starting with 0.2.0, the CLI refuses to start on legacy configs and points to the migration guide.

Upgrading from 0.4.x? See docs/migration-0.4-to-0.5.md or docs/migration-0.4-to-0.5.ja.md. 0.5.x keeps existing profiles working, but the recommended user-facing commands are now gate-oriented.

Upgrading from 0.12.x? 0.13.0 is a breaking release: --profile / [profile.*] / FLAKER_PROFILE are gone, replaced by --gate / [gate.iteration|merge|release] / FLAKER_GATE; the random, gbdt, and coverage-guided strategies, cluster_mode, model_path, [coverage], adaptive* keys, dev train, and the ops command group are removed. See docs/migration-0.12-to-0.13.md / docs/migration-0.12-to-0.13.ja.md.

#Install as a CLI

pnpm add -D @mizchi/flaker

pnpm dlx @mizchi/flaker --help

• Node.js 24+
• pnpm 10+

#Install as a Claude Code plugin

• flaker-setup Introduce flaker on a fresh repository. Day 0 → Week 4 onboarding flow, decision points, copy-paste commands, and pitfalls.
• flaker-management Operate flaker after setup. Advisory vs required gating, nightly triage, quarantine, flaky tag management, and staged Playwright E2E / VRT rollout.

# In Claude Code
/plugin marketplace add mizchi/flaker
/plugin install flaker@flaker

• "新しいプロジェクトに flaker をセットアップしたい"
• "flaker の advisory を required に上げる条件を決めたい"
• "E2E VRT の nightly triage を設計したい"

#Use as a MoonBit Library

• pure computation APIs
• the shared contract types they consume and return

import {
  "mizchi/flaker" @flaker,
}

test "sample from historical runs" {
  let meta = @flaker.build_sampling_meta(
    [
      @flaker.SamplingHistoryRowInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
        status: "passed",
        retry_count: 0,
        duration_ms: 1200,
        created_at: "2026-04-03T00:00:00.000Z",
      },
    ],
    [
      @flaker.SamplingListedTestInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
      },
    ],
  )

  let sampled = @flaker.sample_weighted(meta, count=1, seed=1UL)
  assert_eq(sampled.length(), 1)
}

• flaky detection: detect_flaky
• sampling: build_sampling_meta, sample_random, sample_weighted, sample_hybrid
• affected analysis: resolve_affected, build_affected_report, build_affected_report_from_input
• stable identity: create_stable_test_id, resolve_test_identity
• graph helpers: find_affected_nodes, expand_transitive, topological_sort
• report reducers: summarize_report, classify_report_diff, aggregate_report
• policy: summarize_quarantine, compute_quarantine_exit_code, run_config_check
• metrics: build_sampling_kpi

#Experimental Direct MoonBit CLI

moon run src/cmd/flaker --target js -- --help
moon run src/cmd/flaker --target native -- --version

• --help
• --version
• JS bridge exports used by the npm CLI

#Core Workflow

1. Collect or import test results from CI and local runs.
2. Build history for flaky detection and sampling.
3. Ask flaker to choose a smaller local test set.
4. Run the sampled tests and compare local outcomes with CI.
5. Evaluate whether the sampling strategy is actually trustworthy.

#Mental Model

• Gate: a decision boundary such as "can the author keep iterating?", "can this PR merge?", or "can we release?"
• Budget: the constraints that a gate must stay within
• Loop: a background routine that keeps a gate trustworthy over time
• Policy: the rules for how the system reacts when trust drops

#1. Gates

• Iteration Gate: fast local feedback for the author
• Merge Gate: the PR or mainline gate
• Release Gate: a stricter full-suite or pre-release check

#2. Budgets

• Time budget: how long humans are willing to wait
• Signal budget: how much false negative or flaky noise is acceptable
• Execution budget: CI minutes, parallelism, browser workers, compute cost
• Product performance budget: LCP, INP, bundle size, latency, or similar

• execution performance of the test system itself
• performance of the product being tested

#3. Loops

• Observation loop: scheduled full runs, history collection, holdout verification, KPI refresh
• Triage loop: quarantine review, @flaky tagging, promotion, demotion, owner assignment
• Incident loop: retry, confirm, diagnose, and fix after a real failure

#4. Policies

• when to auto-quarantine
• when to exclude @flaky tests from normal execution
• when to promote an advisory check to required
• when to demote a required check back to advisory
• how retries are interpreted

#Mapping to flaker today

• users choose gates and budgets
• operators run loops and maintain policies
• flaker chooses strategies such as affected, hybrid, or full

#Quick Start

pnpm add -D @mizchi/flaker
pnpm flaker init --adapter <playwright|vitest|jest|junit> --runner <same or actrun>
# edit [affected].resolver in flaker.toml (workspace | glob | bitflow)
pnpm flaker doctor
pnpm flaker plan          # preview what apply will do
pnpm flaker apply         # converge (safe to re-run)
pnpm flaker status        # dashboard + promotion drift

Canonical command forms (0.14.0)

The primary surface is: init, plan, apply, status, run, calibrate, doctor, debug, query, explain, import, report. dev is a hidden maintainer-only group.

Canonical (0.14.0)	Notes
flaker apply	reconciles the repo to flaker.toml; runs CI import + calibrate internally when needed
flaker calibrate [--window-days <n>] [--dry-run] [--json]	recommend and write [sampling]
flaker import --ci [--days <n>] [--branch-filter <branch>]	pull recent GitHub Actions results (needs GITHUB_TOKEN)
flaker status --markdown / flaker explain insights	weekly review artifacts
flaker debug retry / debug confirm / debug diagnose	incident investigation
flaker apply --output <file>	saves ApplyArtifact JSON
flaker plan --output <file>	saves PlanArtifact JSON

The full list of 0.12.x forms these replace (and why) is in docs/migration-0.12-to-0.13.md. See docs/migration-0.12-to-0.13.md for the full breaking-change list and error messages.

#Initialize

flaker init --owner your-org --name your-repo

#Collect test history

export GITHUB_TOKEN=$(gh auth token)
flaker import --ci --days 30

flaker import <file> --adapter playwright --commit "$(git rev-parse HEAD)"

flaker import <file> --adapter playwright --source ci --commit <sha>

[runner]
type = "playwright"
command = "pnpm exec playwright test -c playwright.config.ts"

[runner.actrun]
workflow = ".github/workflows/ci.yml"
local = true
trust = true
# job = "e2e"

#Inspect health and operator state

flaker status
flaker status --gate merge --detail --json
flaker status --list flaky
flaker status --list quarantined

flaker explain reason
flaker explain insights
flaker explain cluster
flaker status --markdown
flaker status --markdown > .artifacts/flaker-review.md

• health score
• flaky counts
• local/CI sampling correlation
• average sample ratio
• average saved minutes
• fallback rate

#Sample tests before pushing

flaker run --dry-run --strategy hybrid --count 25
flaker run --dry-run --strategy affected --changed src/foo.ts
flaker run --dry-run --gate iteration --changed src/foo.ts

#Sample and execute

flaker run --strategy hybrid --count 25
flaker run --strategy affected --changed src/foo.ts
flaker run --gate iteration --changed src/foo.ts
flaker run --runner actrun

• P(CI pass | local pass)
• P(CI fail | local fail)
• false negatives / false positives
• average saved minutes

#Execution Gates

# Auto-detects the gate: merge on CI, iteration otherwise
flaker run

# Explicit gate selection
flaker run --gate iteration
flaker run --gate merge
flaker run --gate release

[runner]
type = "playwright"
command = "pnpm exec playwright test -c playwright.config.ts"
flaky_tag_pattern = "@flaky"

[gate.release]
strategy = "full"

[gate.merge]
strategy = "hybrid"
sample_percentage = 30
skip_flaky_tagged = true

[gate.iteration]
strategy = "affected"
max_duration_seconds = 60
fallback_strategy = "weighted"
skip_flaky_tagged = true

• release runs all E2E tests and accumulates history
• merge / iteration exclude tests tagged with @flaky
• flaker status --list flaky surfaces quarantine and flaky-tag candidates for operator review

1. Start with full E2E execution so flaker can accumulate baseline history.
2. When a Playwright test is known to be flaky, tag it with @flaky. flaker detects both Playwright tags and @flaky embedded in the test title.
3. Keep release as full execution, and use merge / iteration with skip_flaky_tagged = true. For Playwright, flaker passes --grep-invert @flaky to normal runs.
4. Run flaker apply regularly (it auto-quarantines when [quarantine].auto = true), then review:

flaker status --list flaky
flaker status --markdown > .artifacts/flaker-weekly.md

flaker run --dry-run --gate iteration --changed src/foo.ts
flaker run --gate iteration --changed src/foo.ts

#CI Integration

#PR Comments

- name: Run flaker
  run: flaker run --gate merge

- name: Post PR comment
  if: github.event_name == 'pull_request'
  run: |
    flaker report summary --adapter vitest --input report.json --pr-comment \
      | gh pr comment ${{ github.event.pull_request.number }} --body-file -
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

#Auto-create Issues for Quarantined Tests

flaker quarantine suggest --json --output .artifacts/quarantine-plan.json
flaker quarantine apply --from .artifacts/quarantine-plan.json --create-issues

#Self-Host Rollout

• ci.yml: a self-host-advisory job runs on pull requests, executes flaker run --gate merge, snapshots kpi / eval, and updates a sticky PR comment.
• nightly-self-host.yml: a scheduled job rebuilds recent CI history with flaker import --ci, runs flaker run --gate release, and updates a rolling issue labeled flaker-self-host.

• matched commits >= 20 — commits where both a gated local/CI run and a release/full run exist in the same window, so that local sampling outcomes can be compared against a ground-truth full run. Increases as the nightly --gate release accumulates history.
• false negative rate <= 5% — share of commits where the merge gate passed but the full run failed (sampling missed a real regression). Measured over the same matched-commit window.
• pass correlation >= 95% — P(full run passes | merge gate passes) on matched commits. Same metric referenced elsewhere in this README as P(CI pass | local pass).
• holdout FNR <= 10% — FNR measured on the holdout slice defined by [sampling] holdout_ratio. Tests in the holdout slice are excluded from the sampled run so that their outcomes can be used to audit whether the sampler's verdict generalizes. Guards against sampler overfitting to the visible slice.
• data confidence reaches moderate or high — how much history the window holds, by distinct commits with results: fewer than 5 is insufficient, fewer than 30 low, fewer than 100 moderate, else high. status, plan / apply and calibrate use the same thresholds.

#Recommended Usage Model

1. flaker run --gate release in a nightly scheduled workflow (full test + data accumulation)
2. flaker run --gate merge on PR push (selective execution, posts PR comment)
3. flaker run --gate iteration during development (fast feedback)
4. Review flaker status and flaker status --gate merge --detail --json weekly
5. Only tighten the workflow after local-to-CI correlation looks strong

• long CI times
• flaky tests
• structured reports such as Playwright JSON, JUnit XML, or Vitest JSON

#Main Commands

#Collection and import

flaker import --ci --days 30
flaker import <file> --adapter playwright

#Sampling and execution

flaker run --dry-run --strategy weighted --count 20
flaker run --dry-run --strategy affected
flaker run --dry-run --strategy hybrid --count 50

flaker run --strategy hybrid --count 50
flaker run --strategy affected --changed src/foo.ts

#Flag precedence

Resolution order (highest to lowest):
  1. Explicit CLI flag          (--strategy, --percentage, --count)
  2. [gate.<name>] in flaker.toml      (via --gate or auto-detection)
  3. [sampling] in flaker.toml         (project default)
  4. Built-in defaults

Notes:
  --count overrides --percentage when both are given
  --changed overrides git auto-detection
  --dry-run suppresses execution, still records selection telemetry
  --explain can be combined with --dry-run or a real run

#Analysis

flaker status --list flaky
flaker explain reason
flaker status --markdown
flaker calibrate
flaker query "SELECT * FROM test_results LIMIT 20"

#Confirm suspected failures

# Remote: triggers workflow_dispatch, waits for result
flaker debug confirm "tests/api.test.ts:handles timeout"
flaker debug confirm "tests/api.test.ts:handles timeout" --repeat 10

# Local: runs via test runner directly
flaker debug confirm "tests/api.test.ts:handles timeout" --runner local

• BROKEN — fails on every repeat (failures == N). Treat as a real regression.
• FLAKY — fails on some but not all repeats (0 < failures < N). Candidate for @flaky tag or quarantine.
• TRANSIENT — passes on every repeat (failures == 0). The original CI failure did not reproduce here; either CI-environment specific or a one-shot noise event.

#Retry CI failures locally

# Re-run failed tests from most recent CI failure
flaker debug retry

# From a specific workflow run
flaker debug retry --run 12345678

#Policy and ownership

flaker quarantine suggest --json --output .artifacts/quarantine-plan.json
flaker quarantine apply --from .artifacts/quarantine-plan.json --create-issues
flaker policy check
flaker exec affected --changed src/foo.ts

#Reporting

flaker report summary --adapter vitest --input report.json --markdown
flaker report summary --adapter vitest --input report.json --pr-comment
flaker report diff --base base.json --head head.json

#Minimal Configuration

[repo]
owner = "your-org"
name = "your-repo"

[storage]
path = ".flaker/data.duckdb"

[adapter]
type = "playwright"

[runner]
type = "playwright"
command = "pnpm exec playwright test"
flaky_tag_pattern = "@flaky"

[affected]
resolver = "workspace"

[flaky]
window_days = 14
detection_threshold_ratio = 0.02

[quarantine]
auto = true
flaky_rate_threshold_percentage = 30
min_runs = 10

[sampling]
strategy = "hybrid"
sample_percentage = 30
holdout_ratio = 0.1
co_failure_window_days = 90

[gate.release]
strategy = "full"

[gate.merge]
strategy = "hybrid"
sample_percentage = 30
skip_flaky_tagged = true

[gate.iteration]
strategy = "affected"
max_duration_seconds = 60
fallback_strategy = "weighted"
skip_flaky_tagged = true

#Docs

• 新規プロジェクト導入チェックリスト (ja)
• New Project Onboarding Checklist
• Usage Guide (ja)
• Usage Guide
• Operations Guide (ja)
• Operations Guide
• Detailed Command Reference
• Using flaker with jev-test-filter / (ja)
• Migrating from 0.12.x to 0.13.0 (ja)
• Migrating from 0.12.x to 0.13.0
• Why flaker
• Design Partner Rollout

#Release

node_modules/@mizchi/flaker/dist/cli/main.js