🥮 mooncakes.io
    Sign in

    moon_scrub

    Offline sensitive-data detection, redaction, JSON and streaming pipelines for MoonBit applications.

    security
    redaction
    secrets
    pii
    logs
    streaming
    json
    detection
    Download zip
    Version
    0.5.2
    License
    Apache-2.0
    Repository
    Last updated
    16 days ago
    Downloads
    50

    #Moon Scrub

    Offline sensitive-data detection and redaction for MoonBit applications. Findings contain category, confidence, and UTF-16 offsets, but never retain the matched value.

    Version 0.1.0 is published on MoonCakes. Install it with moon add 123123213weqw/moon_scrub.

    #Public API

    Scanning and redaction:

    • scan(text, policy~) -> Array[Finding]
    • scan_with_config(text, config) -> Array[Finding]
    • scan_with_rules(text, rules, policy~) -> Array[Finding]
    • scan_full(text, rules, config) -> Array[Finding]
    • redact(text, style~, policy~) -> RedactionResult
    • redact_with_config(text, config, style~) -> RedactionResult
    • redact_full(text, rules, config, style~) -> RedactionResult
    • redact_with_rules(text, rules, style~, policy~) -> RedactionResult
    • verify_clean(text, policy~) -> Bool
    • verify_clean_with_config(text, config) -> Bool
    • redact_batch(lines, style~, policy~) -> BatchResult
    • redact_batch_with_config(lines, config, style~) -> BatchResult
    • redact_fields(fields, style~, policy~) -> StructuredResult
    • redact_fields_with_config(fields, config, style~) -> StructuredResult
    • redact_json(text, config~, style~) -> JsonRedactionResult
    • redact_json_with_rules(text, rules, config~, style~) -> JsonRedactionResult

    Streaming:

    • ChunkScanner::new(config~, overlap~)
    • ChunkScanner::push(chunk) -> Array[Finding]
    • ChunkScanner::push_lines(lines) -> Array[Finding]
    • ChunkScanner::finish() -> Array[Finding]

    Reporting and exceptions:

    • findings_summary(text, config~) -> Array[KindCount]
    • explain(text, config~) -> Array[String]
    • scan_except(text, keep, config~) -> Array[Finding]
    • redact_except(text, keep, style~, config~) -> RedactionResult

    Configuration:

    • ScanPolicy::secrets_only() and ScanPolicy::standard()
    • ScanConfig::secrets_only() and ScanConfig::standard(): per-family switches, false-positive suppressor toggles, and extra TokenPrefixRule prefixed-token rules
    • CustomRule exact-value rules

    Detectors cover structured access tokens across GitHub, GitLab, Slack, AWS, Google, Stripe, Anthropic, OpenAI-style, SendGrid, Doppler and npm prefixes, bearer and Basic credentials, JWTs, sensitive assignments (including JSON-style quoted keys), webhook URLs, email addresses, IPv4 and IPv6, Luhn-valid payment cards with major-network BIN prefixes (Amex spacing included), Chinese resident identity numbers with checksum validation, MAC addresses, UUIDs, SSH public keys and certificate blocks, wallet addresses, and opt-in E.164 phone numbers. This is a deterministic helper, not a complete DLP or compliance system. See the repository README and security model for limitations.

    BatchResult

    source
    pub(all) struct BatchResult {
      lines : Array[String]
      summary : BatchSummary
      changed_indices : Array[Int]
    } derive(Eq, 
    Debug    )

    BatchResult::equal

    source
    fn BatchResult::equal(BatchResult, BatchResult) -> Bool

    BatchResult::not_equal

    source
    fn BatchResult::not_equal(x : BatchResult, y : BatchResult) -> Bool

    BatchResult::to_repr

    source

    BatchSummary

    source
    pub(all) struct BatchSummary {
      lines : Int
      changed_lines : Int
      findings : Int
      access_tokens : Int
      bearer_tokens : Int
      jwts : Int
      credentials : Int
      emails : Int
      ipv4s : Int
      payment_cards : Int
      private_keys : Int
      url_credentials : Int
      custom_secrets : Int
      resident_ids : Int
      mac_addresses : Int
      uuids : Int
      ipv6s : Int
      phones : Int
      public_keys : Int
      wallet_addresses : Int
      webhook_urls : Int
      high_entropy_secrets : Int
    } derive(Eq, 
    Debug    )

    BatchSummary::equal

    source

    BatchSummary::not_equal

    source
    fn BatchSummary::not_equal(x : BatchSummary, y : BatchSummary) -> Bool

    BatchSummary::to_repr

    source

    CharClass

    source
    pub(all) enum CharClass {
      AlnumUndash
      HexLower
      HexAny
      Base64Url
      Digits
    } derive(Eq, 
    Debug    )

    Character vocabulary a custom pattern rule matches after its prefix.

    CharClass::equal

    source
    fn CharClass::equal(CharClass, CharClass) -> Bool

    CharClass::not_equal

    source
    fn CharClass::not_equal(x : CharClass, y : CharClass) -> Bool

    CharClass::to_repr

    source

    ChunkScanner

    source
    type ChunkScanner

    Streaming scanner for input that arrives in chunks: log tailing, socket reads, or prompt fragments. Findings carry absolute offsets into the whole stream, never into the current chunk.

    The scanner keeps an overlap-sized tail of unseen data so secrets that straddle a chunk boundary are still detected. Correctness holds for any secret whose full span is no longer than overlap; longer spans can be split and missed, so pick overlap above the largest expected secret (PEM private-key blocks are the usual ceiling; the default fits 8 KB keys).

    ChunkScanner::finish

    source
    fn ChunkScanner::finish(self : ChunkScanner) -> Array[Finding]

    test {
      let scanner = @moon_scrub.ChunkScanner::new(overlap=64)
      ignore(scanner.push("password=hunter2 padding padding padding padding"))
      inspect(scanner.finish().length(), content="1")
    }

    ChunkScanner::new

    source
    fn ChunkScanner::new(config? : ScanConfig, overlap? : Int) -> ChunkScanner

    Build a chunked scanner. overlap is clamped to at least 64 code units.

    ChunkScanner::push

    source
    fn ChunkScanner::push(self : ChunkScanner, chunk : String) -> Array[Finding]

    test {
      let scanner = @moon_scrub.ChunkScanner::new(overlap=64)
      inspect(
        scanner
        .push(
          "token=ghp_1234567890abcdefghij " +
          "padding padding padding padding padding padding padding padding",
        )
        .length(),
        content="1",
      )
    }

    ChunkScanner::push_lines

    source
    fn ChunkScanner::push_lines(self : ChunkScanner, lines : Array[String]) -> Array[Finding]

    Push complete log lines with their newline included and collect settled findings. Equivalent to one push per line, but a single buffer append.

    ChunkScanner::summary

    source
    fn ChunkScanner::summary(self : ChunkScanner) -> BatchSummary

    Running aggregate over everything emitted so far. lines counts non-empty pushes; category counters match BatchSummary. Returns a copy — BatchSummary is plain integers.

    Confidence

    source
    pub(all) enum Confidence {
      High
      Medium
    } derive(Eq, 
    Debug    )

    Confidence::equal

    source
    fn Confidence::equal(Confidence, Confidence) -> Bool

    Confidence::not_equal

    source
    fn Confidence::not_equal(x : Confidence, y : Confidence) -> Bool

    Confidence::to_repr

    source

    Confidence::to_string

    source
    fn Confidence::to_string(self : Confidence) -> String

    CustomRule

    source
    pub(all) struct CustomRule {
      label : String
      value : String
    } derive(Eq, 
    Debug    )

    A caller-supplied exact value that should never leave the process. Rules shorter than four UTF-16 code units are ignored to limit accidental broad matches. The label is metadata only and never appears in output markers.

    CustomRule::equal

    source
    fn CustomRule::equal(CustomRule, CustomRule) -> Bool

    CustomRule::not_equal

    source
    fn CustomRule::not_equal(x : CustomRule, y : CustomRule) -> Bool

    CustomRule::to_repr

    source

    Finding

    source
    pub(all) struct Finding {
      kind : SensitiveKind
      start : Int
      end : Int
      confidence : Confidence
    } derive(Eq, 
    Debug    )

    Half-open UTF-16 offsets. A finding never stores or returns the matched value, making reports safer to log than the original input.

    Finding::equal

    source
    fn Finding::equal(Finding, Finding) -> Bool

    Finding::not_equal

    source
    fn Finding::not_equal(x : Finding, y : Finding) -> Bool

    Finding::to_repr

    source

    JsonFinding

    source
    pub(all) struct JsonFinding {
      path : String
      finding : Finding
    } derive(Eq, 
    Debug    )

    A finding located inside a JSON document, addressed by a JSON-pointer-like path such as /user/0/email. Array positions appear as numeric segments. The field name is metadata; the matched value itself is never stored.

    JsonFinding::equal

    source
    fn JsonFinding::equal(JsonFinding, JsonFinding) -> Bool

    JsonFinding::not_equal

    source
    fn JsonFinding::not_equal(x : JsonFinding, y : JsonFinding) -> Bool

    JsonFinding::to_repr

    source

    JsonRedactionResult

    source
    pub(all) struct JsonRedactionResult {
      text : String
      changed_fields : Int
      findings : Array[JsonFinding]
      parsed : Bool
    } derive(Eq, 
    Debug    )

    JsonRedactionResult::equal

    source

    JsonRedactionResult::not_equal

    source

    JsonRedactionResult::paths

    source
    fn JsonRedactionResult::paths(self : JsonRedactionResult) -> Array[String]

    Distinct JSON-pointer-like paths that produced findings, sorted, so callers can compare document coverage across runs without values.

    JsonRedactionResult::to_repr

    source

    KindCount

    source
    pub(all) struct KindCount {
      kind : SensitiveKind
      count : Int
    } derive(Eq, 
    Debug    )

    One kind-to-count pair; summaries are ordered by kind name so identical inputs always produce identical reports.

    KindCount::equal

    source
    fn KindCount::equal(KindCount, KindCount) -> Bool

    KindCount::not_equal

    source
    fn KindCount::not_equal(x : KindCount, y : KindCount) -> Bool

    KindCount::to_repr

    source

    PatternRule

    source
    pub(all) struct PatternRule {
      prefix : String
      chars : CharClass
      min_length : Int
      max_length : Int
    } derive(Eq, 
    Debug    )

    A caller-defined shape rule: a prefix followed by min_length to max_length total code units drawn from a character class. More expressive than TokenPrefixRule when secrets carry a constrained alphabet, e.g. lowercase-hex transaction ids or numeric tenant keys.

    PatternRule::equal

    source
    fn PatternRule::equal(PatternRule, PatternRule) -> Bool

    PatternRule::not_equal

    source
    fn PatternRule::not_equal(x : PatternRule, y : PatternRule) -> Bool

    PatternRule::to_repr

    source

    RedactionResult

    source
    pub(all) struct RedactionResult {
      text : String
      findings : Array[Finding]
      changed : Bool
    } derive(Eq, 
    Debug    )

    RedactionResult::equal

    source

    RedactionResult::not_equal

    source
    fn RedactionResult::not_equal(x : RedactionResult, y : RedactionResult) -> Bool

    RedactionResult::to_repr

    source

    RedactionStyle

    source
    pub(all) enum RedactionStyle {
      Marker
      Typed
      PreserveLast4
    } derive(Eq, 
    Debug    )

    RedactionStyle::equal

    source

    RedactionStyle::not_equal

    source
    fn RedactionStyle::not_equal(x : RedactionStyle, y : RedactionStyle) -> Bool

    RedactionStyle::to_repr

    source

    ScanConfig

    source
    pub(all) struct ScanConfig {
      detect_access_token : Bool
      detect_bearer : Bool
      detect_jwt : Bool
      detect_private_key : Bool
      detect_url_credential : Bool
      detect_assignment : Bool
      detect_email : Bool
      detect_ipv4 : Bool
      detect_payment_card : Bool
      detect_resident_id : Bool
      detect_webhook : Bool
      detect_wallet : Bool
      detect_public_key : Bool
      detect_phone : Bool
      detect_high_entropy : Bool
      detect_ipv6 : Bool
      detect_uuid : Bool
      detect_mac : Bool
      suppress_version_ipv4 : Bool
      require_known_card_prefix : Bool
      suppress_example_domains : Bool
      extra_prefixes : Array[TokenPrefixRule]
      extra_patterns : Array[PatternRule]
    } derive(Eq, 
    Debug    )

    Full detector configuration. Every detector family can be switched off independently, and the false-positive suppressors can be relaxed by callers who prefer recall over precision. Turning credential detectors off creates surprising leak paths, so the preset constructors keep them on.

    ScanConfig::equal

    source
    fn ScanConfig::equal(ScanConfig, ScanConfig) -> Bool

    ScanConfig::not_equal

    source
    fn ScanConfig::not_equal(x : ScanConfig, y : ScanConfig) -> Bool

    ScanConfig::pii_only

    source
    fn ScanConfig::pii_only() -> ScanConfig

    Only the personal-identifier families run; every credential detector is off. Useful for telemetry scrubbing where secrets are handled elsewhere.

    ScanConfig::secrets_only

    source
    fn ScanConfig::secrets_only() -> ScanConfig

    Only credential families run; emails, IPv4 addresses and payment cards are left untouched.

    ScanConfig::standard

    source
    fn ScanConfig::standard() -> ScanConfig

    All detectors on with false-positive suppression enabled.

    ScanConfig::to_repr

    source

    ScanPolicy

    source
    pub(all) struct ScanPolicy {
      detect_email : Bool
      detect_ipv4 : Bool
      detect_payment_card : Bool
    } derive(Eq, 
    Debug    )

    A policy controls which detector families run. Known credentials are always enabled because suppressing them creates surprising leak paths.

    ScanPolicy::equal

    source
    fn ScanPolicy::equal(ScanPolicy, ScanPolicy) -> Bool

    ScanPolicy::not_equal

    source
    fn ScanPolicy::not_equal(x : ScanPolicy, y : ScanPolicy) -> Bool

    ScanPolicy::secrets_only

    source
    fn ScanPolicy::secrets_only() -> ScanPolicy

    ScanPolicy::standard

    source
    fn ScanPolicy::standard() -> ScanPolicy

    ScanPolicy::to_repr

    source

    SensitiveKind

    source
    pub(all) enum SensitiveKind {
      AccessToken
      BearerToken
      Jwt
      CredentialAssignment
      Email
      Ipv4
      PaymentCard
      PrivateKey
      UrlCredential
      CustomSecret
      ResidentId
      MacAddress
      Uuid
      Ipv6
      Phone
      PublicKey
      WalletAddress
      WebhookUrl
      HighEntropySecret
    } derive(Eq, 
    Debug    )

    Detection categories intentionally stay small and auditable. This library detects structured secrets and common identifiers; it is not an NLP system.

    SensitiveKind::equal

    source

    SensitiveKind::not_equal

    source
    fn SensitiveKind::not_equal(x : SensitiveKind, y : SensitiveKind) -> Bool

    SensitiveKind::to_repr

    source

    SensitiveKind::to_string

    source
    fn SensitiveKind::to_string(self : SensitiveKind) -> String

    StructuredField

    source
    pub(all) struct StructuredField {
      path : String
      value : String
    } derive(Eq, 
    Debug    )

    Domain-neutral structured field adapter. path may be a JSON pointer, dotted property path, form field name, or database column name.

    StructuredField::equal

    source

    StructuredField::not_equal

    source
    fn StructuredField::not_equal(x : StructuredField, y : StructuredField) -> Bool

    StructuredField::to_repr

    source

    StructuredFieldResult

    source
    pub(all) struct StructuredFieldResult {
      path : String
      value : String
      findings : Array[Finding]
      changed : Bool
      sensitive_path : Bool
    } derive(Eq, 
    Debug    )

    StructuredFieldResult::equal

    source

    StructuredFieldResult::not_equal

    source

    StructuredFieldResult::to_repr

    source

    StructuredResult

    source
    pub(all) struct StructuredResult {
      fields : Array[StructuredFieldResult]
      changed_fields : Int
      findings : Int
    } derive(Eq, 
    Debug    )

    StructuredResult::equal

    source

    StructuredResult::not_equal

    source
    fn StructuredResult::not_equal(x : StructuredResult, y : StructuredResult) -> Bool

    StructuredResult::to_repr

    source

    TokenPrefixRule

    source
    pub(all) struct TokenPrefixRule {
      prefix : String
      min_length : Int
    } derive(Eq, 
    Debug    )

    A caller-defined prefixed-token rule. min_length is the minimum total match length including the prefix. Prefixes shorter than two code units or minimums below the prefix length are ignored to limit accidental broad matches.

    TokenPrefixRule::equal

    source

    TokenPrefixRule::not_equal

    source
    fn TokenPrefixRule::not_equal(x : TokenPrefixRule, y : TokenPrefixRule) -> Bool

    TokenPrefixRule::to_repr

    source

    explain

    source
    fn explain(text : String, config? : ScanConfig) -> Array[String]

    test {
      let lines = @moon_scrub.explain("mail a@b.co")
      inspect(lines.length(), content="1")
      inspect(lines[0], content="EMAIL 5-11 HIGH")
    }

    explain_json

    source
    fn explain_json(text : String, config? : ScanConfig) -> String

    explain as a JSON array: one object per finding with kind, start, end, and confidence — no matched text anywhere. Fits report pipelines that already speak JSON.

    findings_summary

    source
    fn findings_summary(text : String, config? : ScanConfig) -> Array[KindCount]

    test {
      let rows = @moon_scrub.findings_summary("a@b.co x@y.org")
      inspect(rows.length(), content="1")
      inspect(rows[0].kind.to_string(), content="EMAIL")
      inspect(rows[0].count, content="2")
    }

    redact

    source
    fn redact(text : String, style? : RedactionStyle, policy? : ScanPolicy) -> RedactionResult

    test {
      let result = @moon_scrub.redact("user=demo@example.com password=x")
      inspect(
        result.text,
        content="user=[REDACTED:EMAIL] password=[REDACTED:CREDENTIAL]",
      )
    }

    redact_batch

    source
    fn redact_batch(lines : Array[String], style? : RedactionStyle, policy? : ScanPolicy) -> BatchResult

    test {
      let result = @moon_scrub.redact_batch(["password=x", "plain"])
      inspect(result.summary.changed_lines, content="1")
    }

    redact_batch_with_config

    source
    fn redact_batch_with_config(lines : Array[String], config : ScanConfig, style? : RedactionStyle) -> BatchResult

    Batch redaction under a full configuration, enabling post-0.1.0 families such as resident ids and MAC addresses in log pipelines.

    redact_except

    source
    fn redact_except(text : String, keep : Array[String], style? : RedactionStyle, config? : ScanConfig) -> RedactionResult

    Redact while sparing caller-certified literals, such as a load balancer IP that is safe to publish. Everything else runs under the given config.

    redact_fields

    source
    fn redact_fields(fields : Array[StructuredField], style? : RedactionStyle, policy? : ScanPolicy) -> StructuredResult

    Redact already-parsed structured fields. Sensitive paths are fail-closed: their complete value is replaced even when its shape matches no detector. Other fields use content scanning. Paths are retained for caller-side joins.

    redact_fields_with_config

    source
    fn redact_fields_with_config(fields : Array[StructuredField], config : ScanConfig, style? : RedactionStyle) -> StructuredResult

    Redact structured fields under a full configuration, enabling post-0.1.0 families for callers that normalize records themselves.

    redact_full

    source
    fn redact_full(text : String, rules : Array[CustomRule], config : ScanConfig, style? : RedactionStyle) -> RedactionResult

    Redact with both built-in detectors under full configuration and caller-supplied exact-value rules.

    redact_json

    source
    fn redact_json(text : String, config? : ScanConfig, style? : RedactionStyle) -> JsonRedactionResult

    test {
      let doc = @moon_scrub.redact_json("{\"k\": \"password=hunter2\"}")
      inspect(doc.text, content="{\"k\":\"password=[REDACTED:CREDENTIAL]\"}")
    }

    redact_json_with_rules

    source
    fn redact_json_with_rules(text : String, rules : Array[CustomRule], config? : ScanConfig, style? : RedactionStyle) -> JsonRedactionResult

    redact_with_config

    source
    fn redact_with_config(text : String, config : ScanConfig, style? : RedactionStyle) -> RedactionResult

    Redact with full detector configuration, suppressor toggles and extra prefixed-token rules.

    redact_with_rules

    source
    fn redact_with_rules(text : String, rules : Array[CustomRule], style? : RedactionStyle, policy? : ScanPolicy) -> RedactionResult

    Redact text with built-in and caller-supplied exact-value rules.

    scan

    source
    fn scan(text : String, policy? : ScanPolicy) -> Array[Finding]

    Scan text without retaining sensitive values in the returned findings.

    scan_except

    source
    fn scan_except(text : String, keep : Array[String], config? : ScanConfig) -> Array[Finding]

    test {
      let kept = @moon_scrub.scan_except("ip 10.0.0.1 ip 10.0.0.2", ["10.0.0.1"])
      inspect(kept.length(), content="1")
    }

    scan_full

    source
    fn scan_full(text : String, rules : Array[CustomRule], config : ScanConfig) -> Array[Finding]

    scan_with_config

    source
    fn scan_with_config(text : String, config : ScanConfig) -> Array[Finding]

    test {
      let config = @moon_scrub.ScanConfig::secrets_only()
      let findings = @moon_scrub.scan_with_config("ip 10.0.0.8", config)
      inspect(findings.length(), content="0")
    }

    scan_with_rules

    source
    fn scan_with_rules(text : String, rules : Array[CustomRule], policy? : ScanPolicy) -> Array[Finding]

    Add caller-known exact secrets to the built-in detector set. The returned findings do not carry rule labels or values. Empty and very short values are deliberately ignored.

    verify_clean

    source
    fn verify_clean(text : String, policy? : ScanPolicy) -> Bool

    test {
      let clean = @moon_scrub.redact("password=hunter2").text
      inspect(@moon_scrub.verify_clean(clean), content="true")
    }

    verify_clean_with_config

    source
    fn verify_clean_with_config(text : String, config : ScanConfig) -> Bool

    verify_clean_with_rules

    source
    fn verify_clean_with_rules(text : String, rules : Array[CustomRule], config? : ScanConfig) -> Bool

    Return true only when neither built-in detectors under the configuration nor caller-supplied exact-value rules find anything.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io