#flaker

• sampling a smaller local test run from history and changed files
• detecting flaky tests in noisy CI environments
• measuring how well local sampled runs predict CI
• embedding the same core logic in MoonBit as a library

• the full test suite is too expensive to run on every change
• CI failures are noisy because flaky tests are mixed with real regressions
• developers need a smaller local test run that still correlates well with CI

• Which tests should I run for this change?
• How much can I shrink local execution without losing too much confidence?
• Which tests are actually flaky?
• How well does local sampled execution predict CI outcomes?

#Install as a CLI

pnpm add -D @mizchi/flaker

pnpm dlx @mizchi/flaker --help

• Node.js 24+
• pnpm 10+

#Use as a MoonBit Library

• pure computation APIs
• the shared contract types they consume and return

import {
  "mizchi/flaker" @flaker,
}

test "sample from historical runs" {
  let meta = @flaker.build_sampling_meta(
    [
      @flaker.SamplingHistoryRowInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
        status: "passed",
        retry_count: 0,
        duration_ms: 1200,
        created_at: "2026-04-03T00:00:00.000Z",
      },
    ],
    [
      @flaker.SamplingListedTestInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
      },
    ],
  )

  let sampled = @flaker.sample_weighted(meta, count=1, seed=1UL)
  assert_eq(sampled.length(), 1)
}

• flaky detection: detect_flaky
• sampling: build_sampling_meta, sample_random, sample_weighted, sample_hybrid
• affected analysis: resolve_affected, build_affected_report, build_affected_report_from_input
• stable identity: create_stable_test_id, resolve_test_identity
• graph helpers: find_affected_nodes, expand_transitive, topological_sort
• report reducers: summarize_report, classify_report_diff, aggregate_report
• policy: summarize_quarantine, compute_quarantine_exit_code, run_config_check
• metrics: build_sampling_kpi

#Experimental Direct MoonBit CLI

moon run src/cmd/flaker --target js -- --help
moon run src/cmd/flaker --target native -- --version

• --help
• --version
• JS bridge exports used by the npm CLI

#Core Workflow

1. Collect or import test results from CI and local runs.
2. Build history for flaky detection and sampling.
3. Ask flaker to choose a smaller local test set.
4. Run the sampled tests and compare local outcomes with CI.
5. Evaluate whether the sampling strategy is actually trustworthy.

#Quick Start

#1. Initialize

flaker init --owner your-org --name your-repo

#2. Collect test history

export GITHUB_TOKEN=$(gh auth token)
flaker collect --last 30

flaker import report.json --adapter playwright --commit "$(git rev-parse HEAD)"

flaker collect-local --last 20

#3. Inspect flakiness

flaker flaky
flaker reason
flaker eval

flaker eval --json
flaker eval --markdown --window 7

• health score
• flaky counts
• local/CI sampling correlation
• average sample ratio
• average saved minutes
• fallback rate

#4. Sample tests before pushing

flaker sample --strategy hybrid --count 25
flaker sample --strategy affected --changed src/foo.ts

#5. Sample and execute

flaker run --strategy hybrid --count 25
flaker run --strategy affected --changed src/foo.ts

• P(CI pass | local pass)
• P(CI fail | local fail)
• false negatives / false positives
• average saved minutes

#Recommended Usage Model

• use flaker run --strategy hybrid --count N locally
• keep full CI as the source of truth
• collect local and CI results into the same database
• review flaker eval weekly
• only tighten the workflow after local-to-CI correlation looks strong

• long CI times
• flaky tests
• structured reports such as Playwright JSON, JUnit XML, or Vitest JSON

#Main Commands

#Collection and import

flaker collect
flaker collect-local
flaker import report.json --adapter playwright

#Sampling and execution

flaker sample --strategy random --count 20
flaker sample --strategy weighted --count 20
flaker sample --strategy affected
flaker sample --strategy hybrid --count 50

flaker run --strategy hybrid --count 50
flaker run --strategy affected --changed src/foo.ts

#Analysis

flaker flaky
flaker reason
flaker eval
flaker query "SELECT * FROM test_results LIMIT 20"

#Policy and ownership

flaker quarantine
flaker check
flaker affected --changed src/foo.ts

#Minimal Configuration

[repo]
owner = "your-org"
name = "your-repo"

[storage]
path = ".flaker/data.duckdb"

[adapter]
type = "playwright"

[runner]
type = "vitest"
command = "pnpm vitest"

[affected]
resolver = "workspace"

[flaky]
window_days = 14
detection_threshold = 2.0

[quarantine]
auto = true
flaky_rate_threshold = 30.0
min_runs = 10

#Docs

• Usage Guide
• Why flaker
• Design Partner Rollout

#Release

node_modules/@mizchi/flaker/dist/cli/main.js