🥮 mooncakes.io

    lunasieve

    Streaming sensitive-data detection and redaction engine for MoonBit

    security
    redaction
    logging
    streaming
    secret-scanner
    Download zip
    Version
    0.1.0
    License
    Apache-2.0
    Repository
    Last updated
    5 hours ago
    Downloads
    2

    #LunaSieve

    LunaSieve 是一个使用 MoonBit 实现的流式敏感信息检测与脱敏引擎。它可以在日志、配置、HTTP 报文和数据导出内容中发现访问令牌、密码、连接串、个人信息与高熵 Secret，并在不泄露原文的前提下生成净化文本和审计报告。

    CI

    #特性

    • 31 类常见令牌前缀和 113 条敏感配置键规则。
    • 邮箱、中国大陆手机号、身份证号、银行卡号、IPv4、JWT、PEM 和连接串检测。
    • 高熵令牌发现，并通过语义优先级避免覆盖更明确的检测结果。
    • 分块流式扫描，能够发现跨 chunk 的敏感信息。
    • 可配置的掩码、替换、删除和稳定哈希脱敏策略。
    • 自定义字面量规则、边界模式、大小写模式和显式白名单。
    • 多文档扫描、稳定指纹、基线过滤、文本报告和 SARIF 2.1.0 输出。
    • 默认不在 Finding 或报告中保存原始 Secret。
    • 229 项自动化测试；CI 执行格式、检查、测试、Release 构建和打包。

    #安装

    moon add sujy123456/lunasieve

    在 moon.pkg 中导入：

    ///|
    import {
      "sujy123456/lunasieve",
    }

    #快速开始

    ///|
    fn main {
      let input = "user=alice@example.com token=ghp_abcdefghijklmnopqrstuvwxyz0123456789"
      let result = @lunasieve.scan(input)
      println("findings: \{result.findings.length()}")
      println(result.redacted)
    }

    运行仓库示例：

    moon run cmd/main

    #自定义策略

    ///|
    let policy = @lunasieve.Policy::new([
      @lunasieve.Rule::new("email", Mask(fill='*', keep_start=2, keep_end=3)),
      @lunasieve.Rule::new("github-pat", Replace("[GITHUB_TOKEN]")),
      @lunasieve.Rule::new("cn-identity", Drop),
    ])

    ///|
    let result = @lunasieve.Scanner::new(policy~).scan(input)

    #流式扫描

    StreamScanner 保留有限的重叠窗口，因此令牌跨越两个输入块时仍能被识别：

    ///|
    let stream = @lunasieve.StreamScanner::new(overlap=128)

    ///|
    let first = stream.push("contact alice@")

    ///|
    let second = stream.push("example.com next")

    ///|
    let remaining = stream.finish()

    #自定义规则与白名单

    ///|
    let scanner = @lunasieve.LiteralScanner::new([
      @lunasieve.LiteralRule::new(
        "internal-key",
        "ACME-SECRET",
        boundary=Word,
        case_sensitive=false,
      ),
    ])

    ///|
    let allowlist = [
      @lunasieve.AllowRule::new(
        detector_id="email",
        exact_value="example@example.com",
        reason="documentation fixture",
      ),
    ]

    建议白名单只记录公开测试值，不要把真实 Secret 写入仓库。

    #批量扫描与 SARIF

    let result = @lunasieve.scan_documents([
      @lunasieve.Document::new("service.env", env_text),
      @lunasieve.Document::new("application.log", log_text),
    ])
    println(@lunasieve.batch_to_text(result))
    let sarif = @lunasieve.batch_to_sarif(result)

    每条结果都包含稳定指纹，filter_new_findings 可用来过滤历史基线，适合在 CI 中只阻止新增问题。

    #安全边界

    • LunaSieve 是检测与脱敏组件，不是凭据保险库。
    • Finding 默认只保存位置、类型、置信度及隐藏预览。
    • 高熵检测属于启发式规则，应结合语义规则与白名单使用。
    • Hash 动作用于稳定关联，不提供密码学不可逆性。
    • 核心库处理内存文本；文件遍历、权限和符号链接策略由调用方负责。

    #开发

    moon fmt --check
moon check
moon test
moon build --release
moon package

    重新生成规则目录测试：

    ./tools/generate_catalog_tests.ps1

    #项目结构

    • prefix_detector.mbt：供应商令牌前缀。
    • contextual_detector.mbt：配置键、PEM 与连接串。
    • structured_detectors.mbt：邮箱、手机号、证件、银行卡、IP 和 JWT。
    • entropy_detector.mbt：高熵令牌。
    • policy.mbt：重叠消解和脱敏策略。
    • streaming.mbt：分块流式扫描。
    • custom_rules.mbt：自定义规则、白名单和位置。
    • batch.mbt：多文档、基线、文本和 SARIF 报告。
    • docs/ARCHITECTURE.md：设计与边界。

    #许可证

    Apache-2.0。项目为原创实现；使用 Luhn、FNV-1a 等公开算法思想，不复制第三方项目源代码。

    AllowRule

    source
    pub(all) struct AllowRule {
      detector_id : String?
      exact_value : String?
      line_contains : String?
      reason : String
    } derive(Eq, 
    Debug    )

    AllowRule::new

    source
    fn AllowRule::new(detector_id? : String, exact_value? : String, line_contains? : String, reason? : String) -> AllowRule

    BatchResult

    source
    pub(all) struct BatchResult {
      findings : Array[DocumentFinding]
      summary : BatchSummary
    } derive(Eq, 
    Debug    )

    BatchSummary

    source
    pub(all) struct BatchSummary {
      documents : Int
      bytes : Int
      findings : Int
      critical : Int
      high : Int
      medium : Int
      low : Int
      info : Int
      by_detector : Array[DetectorCount]
    } derive(Eq, 
    Debug    )

    BoundaryMode

    source
    pub(all) enum BoundaryMode {
      Anywhere
      Word
      Line
    } derive(Eq, 
    Debug    )

    DetectorCount

    source
    pub(all) struct DetectorCount {
      detector_id : String
      count : Int
    } derive(Eq, 
    Debug    )

    DetectorKind

    source
    pub(all) enum DetectorKind {
      Prefix
      Email
      Phone
      Identity
      BankCard
      IPv4
      Jwt
      HighEntropy
      Custom
    } derive(Eq, 
    Debug    )

    DetectorKind::name

    source
    fn DetectorKind::name(self : DetectorKind) -> String

    Document

    source
    pub(all) struct Document {
      path : String
      content : String
    } derive(Eq, 
    Debug    )

    Document::new

    source
    fn Document::new(path : String, content : String) -> Document

    DocumentFinding

    source
    pub(all) struct DocumentFinding {
      path : String
      finding : Finding
      start : SourceLocation
      end : SourceLocation
      fingerprint : String
    } derive(Eq, 
    Debug    )

    Finding

    source
    pub(all) struct Finding {
      detector_id : String
      kind : DetectorKind
      severity : Severity
      span : Span
      confidence : Int
      preview : String
      message : String
    } derive(Eq, 
    Debug    )

    Finding::new

    source
    fn Finding::new(detector_id : String, kind : DetectorKind, severity : Severity, start : Int, end : Int, confidence : Int, preview : String, message : String) -> Finding

    FindingSummary

    source
    pub(all) struct FindingSummary {
      total : Int
      critical : Int
      high : Int
      medium : Int
      low : Int
      info : Int
    } derive(Eq, 
    Debug    )

    KeyPattern

    source
    pub(all) struct KeyPattern {
      id : String
      key : String
      severity : Severity
      minimum_length : Int
      message : String
    } derive(Eq, 
    Debug    )

    KeyPattern::new

    source
    fn KeyPattern::new(id : String, key : String, severity : Severity, minimum_length? : Int, message? : String) -> KeyPattern

    LiteralRule

    source
    pub(all) struct LiteralRule {
      id : String
      literal : String
      case_sensitive : Bool
      boundary : BoundaryMode
      severity : Severity
      message : String
    } derive(Eq, 
    Debug    )

    LiteralRule::new

    source
    fn LiteralRule::new(id : String, literal : String, severity? : Severity, case_sensitive? : Bool, boundary? : BoundaryMode, message? : String) -> LiteralRule

    LiteralScanner

    source
    pub(all) struct LiteralScanner {
      rules : Array[LiteralRule]
      include_preview : Bool
    } derive(Eq, 
    Debug    )

    LiteralScanner::find

    source
    fn LiteralScanner::find(self : LiteralScanner, text : String) -> Array[Finding]

    LiteralScanner::new

    source
    fn LiteralScanner::new(rules : Array[LiteralRule], include_preview? : Bool) -> LiteralScanner

    LiteralScanner::redact

    source
    fn LiteralScanner::redact(self : LiteralScanner, text : String, policy? : Policy) -> String

    LocatedFinding

    source
    pub(all) struct LocatedFinding {
      finding : Finding
      start : SourceLocation
      end : SourceLocation
    } derive(Eq, 
    Debug    )

    Policy

    source
    pub(all) struct Policy {
      rules : Array[Rule]
      fallback : RedactionAction
    } derive(Eq, 
    Debug    )

    Policy::new

    source
    fn Policy::new(rules : Array[Rule], fallback? : RedactionAction) -> Policy

    Policy::secure_default

    source
    fn Policy::secure_default() -> Policy

    PrefixPattern

    source
    pub(all) struct PrefixPattern {
      id : String
      prefix : String
      minimum_tail : Int
      maximum_tail : Int
      severity : Severity
      message : String
    } derive(Eq, 
    Debug    )

    PrefixPattern::new

    source
    fn PrefixPattern::new(id : String, prefix : String, minimum_tail : Int, maximum_tail : Int, severity : Severity, message : String) -> PrefixPattern

    RedactionAction

    source
    pub(all) enum RedactionAction {
      Mask(fill~ : Char, keep_start~ : Int, keep_end~ : Int)
      Replace(String)
      Drop
      Hash
    } derive(Eq, 
    Debug    )

    Rule

    source
    pub(all) struct Rule {
      detector_id : String
      minimum_severity : Severity
      action : RedactionAction
      enabled : Bool
    } derive(Eq, 
    Debug    )

    Rule::new

    source
    fn Rule::new(detector_id : String, action : RedactionAction, minimum_severity? : Severity, enabled? : Bool) -> Rule

    ScanOptions

    source
    pub(all) struct ScanOptions {
      detect_entropy : Bool
      minimum_entropy_length : Int
      entropy_threshold_milli : Int
      include_preview : Bool
      merge_overlaps : Bool
    } derive(Eq, 
    Debug    )

    ScanOptions::default

    source
    fn ScanOptions::default() -> ScanOptions

    ScanResult

    source
    pub(all) struct ScanResult {
      input_length : Int
      findings : Array[Finding]
      redacted : String
    } derive(Eq, 
    Debug    )

    Scanner

    source
    pub(all) struct Scanner {
      options : ScanOptions
      policy : Policy
    }

    Scanner::find

    source
    fn Scanner::find(self : Scanner, text : String) -> Array[Finding]

    Scanner::new

    source
    fn Scanner::new(options? : ScanOptions, policy? : Policy) -> Scanner

    Scanner::scan

    source
    fn Scanner::scan(self : Scanner, text : String) -> ScanResult

    Severity

    source
    pub(all) enum Severity {
      Info
      Low
      Medium
      High
      Critical
    } derive(Eq, 
    Debug    )

    Severity::name

    source
    fn Severity::name(self : Severity) -> String

    Severity::rank

    source
    fn Severity::rank(self : Severity) -> Int

    SourceLocation

    source
    pub(all) struct SourceLocation {
      offset : Int
      line : Int
      column : Int
    } derive(Eq, 
    Debug    )

    Span

    source
    pub(all) struct Span {
      start : Int
      end : Int
    } derive(Eq, 
    Debug    )

    Span::length

    source
    fn Span::length(self : Span) -> Int

    Span::new

    source
    fn Span::new(start : Int, end : Int) -> Span raise

    Span::overlaps

    source
    fn Span::overlaps(self : Span, other : Span) -> Bool

    StreamScanner

    source
    pub(all) struct StreamScanner {
      scanner : Scanner
      overlap : Int
      pending : String
      absolute_offset : Int
      finished : Bool
    }

    StreamScanner::buffered_bytes

    source
    fn StreamScanner::buffered_bytes(self : StreamScanner) -> Int

    StreamScanner::finish

    source
    fn StreamScanner::finish(self : StreamScanner) -> Array[Finding] raise

    StreamScanner::new

    source
    fn StreamScanner::new(scanner? : Scanner, overlap? : Int) -> StreamScanner

    StreamScanner::push

    source
    fn StreamScanner::push(self : StreamScanner, chunk : String) -> Array[Finding] raise

    apply_allowlist

    source
    fn apply_allowlist(text : String, findings : Array[Finding], rules : Array[AllowRule]) -> Array[Finding]

    batch_to_sarif

    source
    fn batch_to_sarif(result : BatchResult) -> String

    batch_to_text

    source
    fn batch_to_text(result : BatchResult) -> String

    builtin_key_patterns

    source
    fn builtin_key_patterns() -> Array[KeyPattern]

    builtin_prefix_patterns

    source
    fn builtin_prefix_patterns() -> Array[PrefixPattern]

    filter_new_findings

    source
    fn filter_new_findings(findings : Array[DocumentFinding], known_fingerprints : Array[String]) -> Array[DocumentFinding]

    find

    source
    fn find(text : String) -> Array[Finding]

    findings_to_json

    source
    fn findings_to_json(findings : Array[Finding]) -> String

    locate

    source
    fn locate(text : String, offset : Int) -> SourceLocation

    locate_findings

    source
    fn locate_findings(text : String, findings : Array[Finding]) -> Array[LocatedFinding]

    located_findings_to_text

    source
    fn located_findings_to_text(items : Array[LocatedFinding]) -> String

    redact

    source
    fn redact(text : String, findings : Array[Finding], policy? : Policy) -> String

    redact_text

    source
    fn redact_text(text : String) -> String

    scan

    source
    fn scan(text : String) -> ScanResult

    scan_documents

    source
    fn scan_documents(documents : Array[Document], scanner? : Scanner, allowlist? : Array[AllowRule]) -> BatchResult

    summarize

    source
    fn summarize(findings : Array[Finding]) -> FindingSummary

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io