moonhermetic

    A static hermeticity auditor for MoonBit tests.

    testing
    lint
    deterministic
    hermeticity
    flaky-tests
    ci
    moonbit
    Download zip
    Author
    Version
    0.16.1
    License
    Apache-2.0
    Last updated
    3 hours ago
    Downloads
    2

    #MoonHermetic

    MoonHermetic is a static hermeticity auditor for MoonBit tests. It finds tests that silently depend on the machine, clock, network, filesystem, random state, or timing before those tests become flaky in CI.

    Its default analysis is test-oriented rather than file-wide: a MoonBit-aware call graph links named tests to local helpers and reports the effect path that makes a test depend on ambient host state.

    It is intentionally narrower than a general static-analysis framework and different from runtime flake classifiers: MoonHermetic does not rerun failed tests or infer a root cause from historical results. It makes likely non-hermetic code visible during review and fails a release gate early.

    #Why this project

    MoonBit's structured concurrency and cross-target toolchain make it practical to run the same tests in many environments. Tests that read @env.now(), an ambient environment variable, a live endpoint, or an unseeded random source defeat that portability.

    The initial Mooncakes keyword audit found adjacent runtime flake and test tooling, including mizchi/flaker, but no direct static "test hermeticity" scanner. MoonHermetic complements those tools by reporting the risky source location and a concrete remediation before the test is executed.

    #Features

    • Seven rules: wall clock, randomness, environment, network, subprocess, filesystem, and sleep/timing dependencies.
    • Test-reachability semantics that trace effects through helpers across files in the same MoonBit package and ignore unreachable helpers.
    • Comment- and literal-aware scanning with precise file, line, and column locations.
    • Rule-specific inline suppression with owners, expiry dates, and // hermetic:ignore-next.
    • Suppression governance reports for active, expiring, expired, legacy, and invalid directives.
    • A reusable GitHub Action with cross-platform smoke coverage.
    • Checksum-verified native binaries for Linux, macOS, and Windows releases.
    • Validated Mooncakes package assets and a guarded publication workflow.
    • GitHub job summaries and an optional managed pull-request comment.
    • Saved baseline mode for adopting MoonHermetic in existing projects.
    • A one-command init flow that creates configuration and a safe first baseline.
    • A doctor command for validating setup, policy, buffers, and CI readiness.
    • Project configuration for rule disabling, severity overrides, exclusions, and the default CI failure threshold.
    • Unified-diff mode that reports only findings introduced on changed lines.
    • A pinned ten-repository precision baseline for detecting heuristic drift.
    • A 19-case reviewed sample for precision, recall, and specificity checks.
    • Deterministic stratified sampling with a 20-case reviewed precision sample.
    • Text, JSON, Markdown, and SARIF 2.1.0 output.
    • CI-friendly exit codes and deterministic file traversal.
    • No opinionated test-replacement framework: it points to the boundary that should be injected or faked.

    #Quick start

    Requirements: a current MoonBit toolchain and MoonBit 0.1.20260920 or newer.

    git clone https://github.com/kobe108/moonhermetic cd moonhermetic moon check moon test moon run cmd/main -- . --fail-on warning

    Published Mooncakes releases can be added without cloning the repository:

    moon add kobe108/moonhermetic

    Initialize an existing project with the default configuration and baseline:

    moon run cmd/main -- init

    This creates .moonhermetic.json and .moonhermetic-baseline.json. Existing files are never overwritten unless --force is supplied. Use --no-baseline when only configuration is wanted, or provide a target and custom paths:

    moon run cmd/main -- init subproject \ --config config/moonhermetic.json \ --baseline config/moonhermetic-baseline.json

    Example output:

    examples/risky/checkout_test.mbt:20:18: error [HERM-NET] External network dependency evidence: let response = fetch(endpoint) fix: Use a fake transport or a deterministic local fixture instead of a live service. MoonHermetic scanned 2 test file(s): 3 finding(s), 2 error(s), 1 warning(s).

    #CLI

    moonhermetic [options] [file-or-directory ...] moonhermetic init [path] [options] moonhermetic doctor [path] [options] -f, --format <text|json|markdown|sarif> --fail-on <never|warning|error> --config <path> --no-config --diff <path> --audit-suppressions --fail-on-suppression --baseline <path> --write-baseline <path> --summary-file <path> --no-reachability --exclude <text> --rules -h, --help -V, --version

    Run moonhermetic init --help for initialization options.

    Semantic mode is the default. MoonHermetic parses test and fn bodies, builds a package-local multi-file call graph, and reports only effects reachable from a named test. Findings include the concrete call path:

    test: request reads an ambient token path: request reads an ambient token -> prepare_request -> read_ambient_token

    Use --no-reachability to scan every discovered .mbt source line with the legacy name-based behavior. See docs/SEMANTIC_ANALYSIS.md for the concrete case, algorithm, and scope.

    Run project diagnostics with:

    moon run cmd/main -- doctor moon run cmd/main -- doctor --format json --strict

    Without a path, MoonHermetic scans the current directory. It only selects files ending in _test.mbt or _wbtest.mbt, and skips _build, .git, .mooncakes, .moonagent, target, and node_modules.

    #Configuration

    MoonHermetic reads .moonhermetic.json from the current working directory when it exists. Use --config <path> to select another file or --no-config to disable discovery entirely.

    { "version": 1, "failOn": "error", "exclude": ["vendor"], "rules": { "HERM-NET": "warning", "HERM-FS": "off", "HERM-SLEEP": "error" } }

    Each rule accepts off, warning, or error. CLI --fail-on and --exclude values override or extend the corresponding config values for one run. Unknown rule IDs and misspelled top-level keys are rejected so a broken policy cannot silently pass CI.

    A complete example is available at examples/moonhermetic.example.json.

    #Pull-request diff mode

    Generate a unified diff in CI and pass it with --diff. MoonHermetic still scans the complete test files so it can resolve findings correctly, but only reports findings whose source line appears as an added line in the patch.

    git diff --unified=0 "$BASE_SHA...HEAD" -- '*.mbt' > moonhermetic.diff moon run cmd/main -- . --diff moonhermetic.diff --fail-on warning

    Findings outside the patch are counted in summary.filteredOut and omitted from text, Markdown, and SARIF output. Deleted lines do not count as changed lines. A demo patch is available at examples/changes.diff.

    #Baseline mode

    Freeze the current findings when first adopting MoonHermetic:

    moon run cmd/main -- . --write-baseline .moonhermetic-baseline.json

    Later scans accept those recorded findings and fail only for new risks:

    moon run cmd/main -- . \ --baseline .moonhermetic-baseline.json \ --fail-on warning

    Baseline fingerprints use the rule, repository-relative path, and normalized evidence. Moving a known finding to another line or changing whitespace keeps it accepted. Repeated occurrences are counted, so adding a new duplicate still creates a finding. Reports expose both accepted and stale counts; regenerate the baseline to remove stale entries. See docs/BASELINE.md for the schema and review policy.

    #Rules

    RuleDefault severityDetects
    HERM-CLOCKwarningWall-clock or local-time reads
    HERM-RANDwarningRandomness without an explicit seed
    HERM-ENVwarningAmbient process environment reads
    HERM-NETerrorLive network or socket dependencies
    HERM-PROCwarningChild processes and shell execution
    HERM-FSwarningFiles outside a managed fixture
    HERM-SLEEPerrorSleep- or elapsed-time-based synchronization

    Run moon run cmd/main -- --rules for the current catalog.

    #Suppression

    The legacy blanket directive remains available:

    sleep(20) // hermetic:ignore: timing service must expire the lease

    For reviewed exceptions, name only the affected rules and attach an owner and UTC expiry date:

    fetch("https://example.test") // hermetic:ignore HERM-NET owner=kobe108 expires=2026-12-31
    sleep(10) // hermetic:ignore HERM-SLEEP,HERM-CLOCK owner=kobe108 expires=2026-12-31

    Use // hermetic:ignore-next to target the following line instead. A scoped directive must include owner= and expires=YYYY-MM-DD. Once the UTC date is past the expiry date, or when the directive is malformed, it stops suppressing and the original finding reappears.

    Inline suppression stays local. Rule disabling is explicit in the versioned config file, so policy changes remain reviewable.

    #Suppression governance

    Use --audit-suppressions to review every directive without changing the normal finding behavior:

    moon run cmd/main -- . --audit-suppressions moon run cmd/main -- . --audit-suppressions --format json

    The report groups directives as:

    • active: expires more than 30 days after the scan date.
    • expiring-soon: expires within the next 30 days, including today.
    • expired: the expiry date is already in the past.
    • legacy: a blanket hermetic:ignore: ... directive without governance metadata.
    • invalid: a scoped directive with a missing, unknown, or malformed field.

    An ordinary audit exits successfully even when it finds expired or invalid directives. For a CI gate, use:

    moon run cmd/main -- . --fail-on-suppression

    This enables the audit and exits with code 1 when any scoped directive is expired or invalid. Legacy directives remain visible but do not fail this optional gate. Strings, block comments, and snapshot content are ignored when collecting directives.

    #GitHub Actions

    The repository is also a composite Action. On release tags it downloads the checksum-verified binary for the runner platform; local or branch checkouts fall back to installing MoonBit. It scans the caller's workspace, respects .moonhermetic.json, and exposes the generated report path:

    permissions: contents: read security-events: write steps: - uses: actions/checkout@v5 - name: MoonHermetic id: hermetic uses: kobe108/moonhermetic@v0.16.1 with: path: . format: sarif fail-on: error - name: Upload SARIF if: always() uses: github/codeql-action/upload-sarif@v4 with: sarif_file: ${{ steps.hermetic.outputs.report-path }}

    The Action defaults to SARIF for scans and JSON for suppression audits. It generates a conventional report filename when output-file is omitted. See docs/ACTION.md for all inputs, PR diff examples, the governance gate, and artifact uploads.

    For a non-Action CI runner, the same compact Markdown can be generated with:

    moon run cmd/main -- . --summary-file moonhermetic-summary.md

    Direct native binaries and their checksums are available from GitHub Releases. Release asset names include the version, platform, and architecture, for example moonhermetic-v0.16.1-linux-x86_64. Each release also includes a validated Mooncakes ZIP and checksum.

    The equivalent manual workflow is:

    - name: Hermeticity audit env: BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} run: | git diff --unified=0 "$BASE_SHA" "$GITHUB_SHA" -- '*.mbt' > moonhermetic.diff moon run cmd/main -- . --diff moonhermetic.diff \ --format sarif \ --fail-on error > moonhermetic.sarif - name: Upload SARIF if: always() uses: github/codeql-action/upload-sarif@v4 with: sarif_file: moonhermetic.sarif - name: Suppression governance run: moon run cmd/main -- . --exclude examples --fail-on-suppression

    The repository's own workflow runs moon check, moon test, the scanner, and a packaging smoke test.

    #Library API

    The analyzer is also usable without the CLI:

    let findings = @moonhermetic.scan_source_with_options_and_date(
    "billing_test.mbt",
    source,
    @moonhermetic.default_scan_options(),
    "2026-11-05",
    )
    let report = @moonhermetic.make_report(1, findings)
    println(@moonhermetic.render_markdown(report))

    let audits = @moonhermetic.audit_suppressions(
    "billing_test.mbt",
    source,
    "2026-11-05",
    )
    let governance = @moonhermetic.make_suppression_audit_report(
    1,
    "2026-11-05",
    audits,
    )
    println(@moonhermetic.render_suppression_audit_json(governance))

    #Roadmap

    • Add precision fixtures from real MoonBit projects.
    • Add code signing and notarization for native binaries.

    #Development

    moon fmt moon check moon test moon info node --test tests/action-runner.test.mjs

    Run the network-backed real-project benchmark with:

    pwsh scripts/run-precision.ps1

    See docs/PROJECT_PLAN.md for the one-month delivery plan, docs/RULES.md for rule semantics and false-positive guidance, docs/PRECISION.md for the pinned corpus, and docs/RELEASES.md for native binary publishing, docs/BASELINE.md for incremental adoption, docs/SEMANTIC_ANALYSIS.md for test reachability, docs/DOCTOR.md for setup diagnostics, docs/MOONCAKES.md for package publication, and docs/ACCEPTANCE.md for the acceptance checklist, and docs/API.md for the public API overview, docs/DEMO.zh.md for the Chinese demo script, and docs/HACKATHON_PITCH.zh.md for the Chinese proposal and demo script.

    The repository uses the commit identity kobe108 <3360944005@qq.com>. When Git is installed, scripts/init-git.ps1 -RemoteUrl <url> initializes the local repository with that identity and configures the remote.

    #License

    Apache-2.0.

    BaselineError

    pub(all) suberror BaselineError {
    BaselineError(String)
    }

    DiffParseError

    pub(all) suberror DiffParseError {
    DiffParseError(String)
    }

    Baseline

    pub(all) struct Baseline {
    created_on : String
    created_with : String
    entries : Array[BaselineEntry]
    }

    BaselineEntry

    pub(all) struct BaselineEntry {
    fingerprint : String
    rule_id : String
    path : String
    evidence : String
    count : Int
    }

    BaselineFilterResult

    pub(all) struct BaselineFilterResult {
    findings : Array[Finding]
    suppressed_count : Int
    stale_count : Int
    }

    ChangedFile

    pub(all) struct ChangedFile {
    path : String
    lines : Array[Int]
    }

    DiffFilter

    pub(all) struct DiffFilter {
    files : Array[ChangedFile]
    }

    Added line numbers indexed by new-file path.

    DiffFilter::includes

    fn DiffFilter::includes(self : DiffFilter, path : String, line : Int) -> Bool

    Check whether an added line is present in the diff.

    Finding

    pub(all) struct Finding {
    path : String
    line : Int
    column : Int
    rule : RuleKind
    severity : Severity
    evidence : String
    test_name : String?
    call_path : Array[String]
    }

    One source location and its explanation.

    Finding::to_text

    fn Finding::to_text(self : Finding) -> String

    RuleKind

    pub(all) enum RuleKind {
    Clock
    Random
    Environment
    Network
    Process
    Filesystem
    Sleep
    }

    The category of implicit dependency found in a test.
    impl Eq for RuleKind

    RuleKind::code

    fn RuleKind::code(self : RuleKind) -> String

    RuleKind::equal

    fn RuleKind::equal(self : RuleKind, other : RuleKind) -> Bool

    RuleKind::from_code

    fn RuleKind::from_code(value : String) -> RuleKind?

    Parse a stable rule ID. Matching is ASCII case-insensitive.

    RuleKind::message

    fn RuleKind::message(self : RuleKind) -> String

    RuleKind::not_equal

    fn RuleKind::not_equal(x : RuleKind, y : RuleKind) -> Bool

    RuleKind::remediation

    fn RuleKind::remediation(self : RuleKind) -> String

    RuleKind::title

    fn RuleKind::title(self : RuleKind) -> String

    RuleOverride

    pub(all) struct RuleOverride {
    rule : RuleKind
    severity : Severity?
    }

    Per-rule policy. None disables the rule; Some overrides its severity.

    ScanOptions

    pub(all) struct ScanOptions {
    overrides : Array[RuleOverride]
    }

    Options applied while scanning source.

    ScanReport

    pub(all) struct ScanReport {
    scanned_files : Int
    findings : Array[Finding]
    filtered_out : Int
    baseline_suppressed : Int
    baseline_stale : Int
    baseline_path : String?
    }

    Findings produced by one scan invocation.

    Severity

    pub(all) enum Severity {
    Warning
    Error
    }

    Severity assigned to a finding.
    impl Eq for Severity

    Severity::equal

    fn Severity::equal(self : Severity, other : Severity) -> Bool

    Severity::not_equal

    fn Severity::not_equal(x : Severity, y : Severity) -> Bool

    Severity::to_string

    fn Severity::to_string(self : Severity) -> String

    SuppressionAudit

    pub(all) struct SuppressionAudit {
    path : String
    line : Int
    column : Int
    directive : String
    rules : Array[String]
    owner : String?
    expires : String?
    reason : String?
    status : SuppressionStatus
    message : String
    }

    SuppressionAuditReport

    pub(all) struct SuppressionAuditReport {
    scanned_files : Int
    today : String
    audits : Array[SuppressionAudit]
    }

    SuppressionStatus

    pub(all) enum SuppressionStatus {
    Active
    ExpiringSoon
    Expired
    Legacy
    Invalid
    }

    SuppressionStatus::equal

    fn SuppressionStatus::equal(self : SuppressionStatus, other : SuppressionStatus) -> Bool

    SuppressionStatus::not_equal

    fn SuppressionStatus::not_equal(x : SuppressionStatus, y : SuppressionStatus) -> Bool

    SuppressionStatus::to_string

    fn SuppressionStatus::to_string(self : SuppressionStatus) -> String

    audit_suppressions

    fn audit_suppressions(path : String, source : String, today : String) -> Array[SuppressionAudit]

    count_by_severity

    fn count_by_severity(report : ScanReport, severity : Severity) -> Int

    count_suppression_status

    fn count_suppression_status(report : SuppressionAuditReport, status : SuppressionStatus) -> Int

    create_baseline

    fn create_baseline(findings : Array[Finding], root : String, created_on : String, created_with : String) -> Baseline

    default_scan_options

    fn default_scan_options() -> ScanOptions

    default_severity

    fn default_severity(rule : RuleKind) -> Severity

    filter_findings

    fn filter_findings(findings : Array[Finding], filter : DiffFilter) -> Array[Finding]

    Keep only findings located on lines added by the diff.

    filter_with_baseline

    fn filter_with_baseline(findings : Array[Finding], baseline : Baseline, root : String) -> BaselineFilterResult

    finding_fingerprint

    fn finding_fingerprint(root : String, finding : Finding) -> String

    Compute a SHA-256 identity that survives line and column movement.

    The identity includes the rule, repository-relative path, and whitespace- normalized evidence. Formatting-only changes therefore retain the same fingerprint while changed operations become new findings.

    has_suppression_errors

    fn has_suppression_errors(report : SuppressionAuditReport) -> Bool

    Expired and malformed directives fail the governance gate.

    make_baseline_report

    fn make_baseline_report(scanned_files : Int, findings : Array[Finding], filtered_out : Int, baseline_suppressed : Int, baseline_stale : Int, baseline_path : String) -> ScanReport

    Build a report after applying both a saved baseline and optional diff filter.

    make_filtered_report

    fn make_filtered_report(scanned_files : Int, findings : Array[Finding], filtered_out : Int) -> ScanReport

    Build a report after diff-aware filtering.

    make_report

    fn make_report(scanned_files : Int, findings : Array[Finding]) -> ScanReport

    Build an aggregate report from findings collected by the CLI.

    make_suppression_audit_report

    fn make_suppression_audit_report(scanned_files : Int, today : String, audits : Array[SuppressionAudit]) -> SuppressionAuditReport

    parse_baseline

    fn parse_baseline(text : String) -> Baseline raise BaselineError

    parse_unified_diff

    fn parse_unified_diff(patch : String) -> DiffFilter raise DiffParseError

    Parse a unified diff and collect added line numbers in the new files.

    render_baseline_json

    fn render_baseline_json(baseline : Baseline) -> String

    render_ci_summary

    fn render_ci_summary(report : ScanReport) -> String

    Render a compact Markdown report for GitHub job summaries and PR comments.

    At most 20 findings are shown to keep CI reports readable and within API limits. The complete report remains available through the selected format.

    render_json

    fn render_json(report : ScanReport) -> String

    Render machine-readable JSON.

    render_markdown

    fn render_markdown(report : ScanReport) -> String

    Render a Markdown report intended for PR comments and project dashboards.

    render_sarif

    fn render_sarif(report : ScanReport) -> String

    Render SARIF 2.1.0 for GitHub code scanning and other CI consumers.

    render_suppression_audit_json

    fn render_suppression_audit_json(report : SuppressionAuditReport) -> String

    render_suppression_audit_markdown

    fn render_suppression_audit_markdown(report : SuppressionAuditReport) -> String

    Render a compact suppression governance summary for GitHub CI surfaces.

    render_suppression_audit_text

    fn render_suppression_audit_text(report : SuppressionAuditReport) -> String

    Render a line-oriented governance report for terminals and CI logs.

    render_text

    fn render_text(report : ScanReport) -> String

    Render a stable, line-oriented report suitable for terminals and CI logs.

    resolve_severity

    fn resolve_severity(rule : RuleKind, options : ScanOptions) -> Severity?

    Resolve the effective severity for a rule after applying overrides.

    rule_catalog

    fn rule_catalog() -> Array[RuleKind]

    rule_catalog_markdown

    fn rule_catalog_markdown() -> String

    Markdown catalog for documentation and --rules output.

    scan_source

    fn scan_source(path : String, source : String) -> Array[Finding]

    Scan one MoonBit source file with the default rule policy.

    scan_source_semantically

    fn scan_source_semantically(path : String, source : String) -> Array[Finding]

    Scan with test reachability semantics and the default rule policy.

    scan_source_semantically_with_options_and_date

    fn scan_source_semantically_with_options_and_date(path : String, source : String, options : ScanOptions, today : String) -> Array[Finding]

    Scan with test reachability, rule overrides, and a UTC reference date.

    scan_source_with_options

    fn scan_source_with_options(path : String, source : String, options : ScanOptions) -> Array[Finding]

    Scan one MoonBit source file with explicit rule overrides.

    Expiring, scoped suppressions require the date-aware overload.

    scan_source_with_options_and_date

    fn scan_source_with_options_and_date(path : String, source : String, options : ScanOptions, today : String) -> Array[Finding]

    Scan with rule overrides and an ISO-8601 UTC date for expiry evaluation.

    scan_sources_semantically

    fn scan_sources_semantically(sources : Array[(String, String)]) -> Array[Finding]

    Scan a MoonBit package across files while tracing only test-reachable code.

    scan_sources_semantically_with_options_and_date

    fn scan_sources_semantically_with_options_and_date(sources : Array[(String, String)], options : ScanOptions, today : String) -> Array[Finding]

    Cross-file semantic scan with rule overrides and a UTC reference date.

    should_fail

    fn should_fail(report : ScanReport, fail_on : Severity) -> Bool