🥮 mooncakes.io

    robots-gate

    审计 robots.txt 策略、解释访问决策、校验站点地图并规划合规抓取任务

    robots
    robots-txt
    crawler
    sitemap
    audit
    moonbit
    Download zip
    Version
    0.2.2
    License
    Apache-2.0
    Repository
    Last updated
    2 months ago
    Downloads
    48

    No dependencies

    #MoonRobots Gate

    MoonRobots Gate 是一套使用 MoonBit 实现的 Robots Exclusion Protocol 策略工程工具。它可以解析并规范化 robots.txt、解释访问判断、审计策略质量、比较策略版本、校验站点地图交互，并在明确的预算约束下生成确定性抓取计划。

    本项目面向爬虫开发者、文档索引器、链接检查器、静态站点工具、CI 流程和 AI 智能体，使它们无须引入完整爬虫框架，也能获得可复现的策略判断。

    #当前状态

    • MoonBit 可复用核心代码 4,310 行，CLI 与示例 260 行。
    • MoonBit 测试代码 1,276 行，合计 5,846 行。
    • 共 107 项测试，在 wasm、wasm-gc、JavaScript 和 native 四个目标全部通过，覆盖解析、匹配、URL 处理、语义检查、渲染、策略差异、访问矩阵、站点地图、抓取规划和审计报告。
    • 除 moonbitlang/core 外没有其他运行时依赖。
    • moon check、moon build、moon test 和 moon package --list 均已通过。
    • GitHub 与 Mooncakes 命名空间均已独立核对为 2111950632。

    #主要能力

    #协议解析

    • 解析 User-agent、Allow 和 Disallow 指令。
    • 保存 Sitemap、Crawl-delay、Host、Clean-param 和未知扩展，同时避免扩展指令破坏标准分组。
    • 处理注释、CRLF、UTF-8 BOM、空记录、重复值、格式错误和防御式输入限制。
    • 生成带稳定代码、严重程度、行号、问题说明和修复建议的源位置诊断。

    #访问判断

    • 选择匹配度最高的产品令牌。
    • 合并所有同等匹配度的用户代理组。
    • 使用最长路径匹配规则，并在同等长度下由 Allow 胜出。
    • 支持 *、末尾 $、区分大小写的路径，以及对未保留字符百分号编码的 RFC 风格归一化。
    • 对 /robots.txt 提供隐式允许。
    • 返回完整决策轨迹，而不只是布尔值。

    #策略工程

    • 语义检查器可发现全站阻断、缺少兜底组、重复和冲突规则、无效百分号转义、错误的 $ 位置、冗余通配符、异常抓取延迟、查询字符串敏感规则和非标准指令。
    • 提供规范化、紧凑化和带源位置注释的渲染结果。
    • 支持用户代理组、规则、站点地图、主机和抓取延迟的结构差异比较。
    • 支持基于调用方探针的行为差异比较。
    • 支持 Markdown 和 CSV 访问矩阵。

    #站点地图与抓取规划

    • 解析 XML urlset、XML sitemapindex 和纯文本站点地图。
    • 校验绝对 URL、重复地址、日期前缀、优先级、更新频率和跨域条目。
    • 报告被指定爬虫策略阻止的站点地图 URL。
    • 将站点地图条目转换为带优先级的抓取任务。
    • 按请求数、字节数、深度、单主机配额、Robots 规则、重复 URL 和抓取延迟约束规划任务。
    • 输出顺序稳定的已接受任务和被拒绝任务报告。

    #审计报告

    • 在一次操作中完成解析、语义检查和可配置访问矩阵。
    • 生成有上限的风险分数和等级。
    • 为人工审阅输出 Markdown，为 CI 或智能体输出稳定 JSON。

    #安装

    moon add 2111950632/robots-gate

    包命名空间已通过 moon whoami 核对为 2111950632。

    #库使用示例

    let policy = @robots_gate.parse(
      #|User-agent: *
      #|Disallow: /private/
      #|Allow: /private/public/
      #|Crawl-delay: 2
    )

    let decision = @robots_gate.decide(
      policy,
      "ExampleBot/1.0",
      "/private/public/index.html",
    )

    println(decision.summary())
    println(@robots_gate.render_decision_trace(decision))

    运行完整审计：

    let audit = @robots_gate.audit(
      robots_text,
      ["Googlebot", "Bingbot", "GPTBot"],
      ["/", "/admin/", "/api/", "/private/", "/public/"],
    )

    println(@robots_gate.render_audit_markdown(audit))

    校验站点地图并生成抓取计划：

    let sitemap = @robots_gate.parse_sitemap(sitemap_text)
    let tasks = @robots_gate.tasks_from_sitemap(sitemap, 32768)
    let budget = @robots_gate.crawl_budget(100, 10000000, 8, 25, 1000)
    let plan = @robots_gate.plan_crawl(policy, "ExampleBot", tasks, budget)

    println(@robots_gate.render_crawl_plan(plan))

    #命令行工具

    CLI 默认使用内置策略和站点地图。可以通过 MOONROBOTS_POLICY 或 MOONROBOTS_SITEMAP 提供自定义内容，无须增加文件系统或网络依赖。

    moon run cmd/main -- decide ExampleBot /private/report
moon run cmd/main -- lint
moon run cmd/main -- normalize
moon run cmd/main -- matrix
moon run cmd/main -- sitemap
moon run cmd/main -- plan
moon run cmd/main -- diff
moon run cmd/main -- audit
moon run cmd/main -- audit json
moon run cmd/main -- stats

    使用自定义输入的 PowerShell 示例：

    $env:MOONROBOTS_POLICY = @"
User-agent: *
Disallow: /internal/
Allow: /internal/public/
Sitemap: https://example.com/sitemap.xml
"@

moon run cmd/main -- audit

    #可运行示例

    moon run examples/basic
moon run examples/advanced
moon run examples/sitemap
moon run examples/planner

    #架构

    模块职责
    model.mbt公共协议类型、诊断类型、决策类型和统计类型
    parser.mbt防御式逐行解析和源位置诊断
    matcher.mbt用户代理选择、通配符匹配和决策轨迹
    url.mbtHTTP URL 解析及来源、路径辅助函数
    lint.mbt语义和可维护性诊断
    render.mbt策略、诊断、轨迹和摘要输出
    matrix.mbt批量探针和行为比较
    diff.mbt结构变化和探针驱动的行为变化
    sitemap.mbt站点地图解析、校验和策略过滤
    frontier.mbt带预算约束的确定性抓取规划
    audit.mbt聚合评分以及 Markdown、JSON 报告

    模块边界和数据流详见架构文档。

    #标准定位

    匹配模型遵循 RFC 9309 的核心规则，包括产品令牌选择、最长路径匹配、同等匹配下的 Allow 优先、区分大小写的路径、百分号编码处理，以及对 /robots.txt 的隐式访问。

    Sitemap、Crawl-delay、Host 和 Clean-param 属于扩展指令，而不是 RFC 9309 的规范记录。MoonRobots Gate 会保存并报告这些指令，同时让扩展解析与标准分组边界保持独立。

    #范围与非目标

    0.2.0 版本是协议与规划核心，明确不负责以下工作：

    • 获取远程资源或跟随重定向。
    • 执行 DNS、TLS、HTTP 缓存或内容解码。
    • 实际执行抓取或持久化分布式任务队列。
    • 实现通用 XML 解析器。
    • 宣称与所有搜索引擎扩展逐字节兼容。

    调用方负责网络传输和持久化。这样的边界使核心库可以移植到不同 MoonBit 目标，并保证策略行为在测试、CI 和 WebAssembly 应用中保持确定。

    #生态查重

    选题检查范围包括：

    • 较早一次 MoonBit 公开活动展示的 30 个项目，该页面仅作为历史生态查重来源。
    • Mooncakes 中与 robots、robots.txt、爬虫、站点地图和 SEO 有关的搜索。
    • GitHub 中将 MoonBit 与 robots.txt、User-agent、Crawl-delay、爬虫和站点地图组合的搜索。

    公开结果中未发现专门面向 Robots 协议策略工程和抓取治理的 MoonBit 包或展示项目。相邻项目 MoonSEO 用于生成和审计 SEO 落地页，并不解析 robots.txt、判断爬虫访问权限、校验站点地图与策略冲突，也不规划受策略约束的抓取任务。

    详细且带日期的搜索记录见生态查重报告。

    #开发

    moon info
moon fmt
moon check --target all --deny-warn
moon build --target all
moon test --target all --deny-warn
moon package --list

    CI 会在推送和拉取请求中运行同一套验证流程。

    #文档

    #许可证

    Apache-2.0。

    AccessProbe

    source
    pub struct AccessProbe {
      name : String
      agent : String
      path : String
    } derive(Eq, 
    Debug    )

    A named user-agent and path probe.

    AccessProbe::agent

    source
    fn AccessProbe::agent(self : AccessProbe) -> String

    AccessProbe::name

    source
    fn AccessProbe::name(self : AccessProbe) -> String

    AccessProbe::path

    source
    fn AccessProbe::path(self : AccessProbe) -> String

    BehaviorChange

    source
    pub(all) enum BehaviorChange {
      UnchangedAllow
      UnchangedDeny
      BecameAllowed
      BecameDenied
      MissingBaseline
    } derive(Eq, 
    Debug    )

    Behavior classification used when comparing two matrices.

    BehaviorChange::label

    source
    fn BehaviorChange::label(self : BehaviorChange) -> String

    ChangeKind

    source
    pub(all) enum ChangeKind {
      GroupAdded
      GroupRemoved
      RuleAdded
      RuleRemoved
      SitemapAdded
      SitemapRemoved
      HostAdded
      HostRemoved
      DelayChanged
      AccessBecameAllowed
      AccessBecameDenied
    } derive(Eq, 
    Debug    )

    Structural or behavioral policy change.

    ChangeKind::is_behavioral

    source
    fn ChangeKind::is_behavioral(self : ChangeKind) -> Bool

    ChangeKind::label

    source
    fn ChangeKind::label(self : ChangeKind) -> String

    CrawlBudget

    source
    pub struct CrawlBudget {
      max_requests : Int
      max_bytes : Int
      max_depth : Int
      max_requests_per_host : Int
      fallback_delay_millis : Int
    } derive(Eq, 
    Debug    )

    Resource and safety limits for a crawl planning run.

    CrawlBudget::fallback_delay_millis

    source
    fn CrawlBudget::fallback_delay_millis(self : CrawlBudget) -> Int

    CrawlBudget::max_bytes

    source
    fn CrawlBudget::max_bytes(self : CrawlBudget) -> Int

    CrawlBudget::max_depth

    source
    fn CrawlBudget::max_depth(self : CrawlBudget) -> Int

    CrawlBudget::max_requests

    source
    fn CrawlBudget::max_requests(self : CrawlBudget) -> Int

    CrawlBudget::max_requests_per_host

    source
    fn CrawlBudget::max_requests_per_host(self : CrawlBudget) -> Int

    CrawlPlan

    source
    pub struct CrawlPlan {
      accepted : Array[PlannedTask]
      rejected : Array[RejectedTask]
      total_estimated_bytes : Int
      estimated_duration_millis : Int
    } derive(Eq, 
    Debug    )

    Output of a deterministic, policy-aware crawl planning pass.

    CrawlPlan::accepted

    source
    fn CrawlPlan::accepted(self : CrawlPlan) -> Array[PlannedTask]

    CrawlPlan::accepted_count

    source
    fn CrawlPlan::accepted_count(self : CrawlPlan) -> Int

    CrawlPlan::estimated_duration_millis

    source
    fn CrawlPlan::estimated_duration_millis(self : CrawlPlan) -> Int

    CrawlPlan::rejected

    source
    fn CrawlPlan::rejected(self : CrawlPlan) -> Array[RejectedTask]

    CrawlPlan::rejected_count

    source
    fn CrawlPlan::rejected_count(self : CrawlPlan) -> Int

    CrawlPlan::total_estimated_bytes

    source
    fn CrawlPlan::total_estimated_bytes(self : CrawlPlan) -> Int

    CrawlTask

    source
    pub struct CrawlTask {
      id : String
      url : String
      priority : Int
      depth : Int
      estimated_bytes : Int
      discovered_from : String
    } derive(Eq, 
    Debug    )

    Candidate URL submitted to the deterministic crawl planner.

    CrawlTask::depth

    source
    fn CrawlTask::depth(self : CrawlTask) -> Int

    CrawlTask::discovered_from

    source
    fn CrawlTask::discovered_from(self : CrawlTask) -> String

    CrawlTask::estimated_bytes

    source
    fn CrawlTask::estimated_bytes(self : CrawlTask) -> Int

    CrawlTask::id

    source
    fn CrawlTask::id(self : CrawlTask) -> String

    CrawlTask::priority

    source
    fn CrawlTask::priority(self : CrawlTask) -> Int

    CrawlTask::url

    source
    fn CrawlTask::url(self : CrawlTask) -> String

    Decision

    source
    pub struct Decision {
      allowed : Bool
      agent : String
      path : String
      normalized_path : String
      rule_kind : RuleKind
      rule_pattern : String
      line : Int
      reason : String
      agent_specificity : Int
      rule_specificity : Int
      matched_groups : Int
      trace : Array[DecisionStep]
    } derive(Eq, 
    Debug    )

    Decision returned for a user-agent and path.

    Decision::agent

    source
    fn Decision::agent(self : Decision) -> String

    Decision::agent_specificity

    source
    fn Decision::agent_specificity(self : Decision) -> Int

    Decision::allowed

    source
    fn Decision::allowed(self : Decision) -> Bool

    Decision::line

    source
    fn Decision::line(self : Decision) -> Int

    Decision::matched_groups

    source
    fn Decision::matched_groups(self : Decision) -> Int

    Decision::normalized_path

    source
    fn Decision::normalized_path(self : Decision) -> String

    Decision::path

    source
    fn Decision::path(self : Decision) -> String

    Decision::reason

    source
    fn Decision::reason(self : Decision) -> String

    Decision::rule_kind

    source
    fn Decision::rule_kind(self : Decision) -> RuleKind

    Decision::rule_pattern

    source
    fn Decision::rule_pattern(self : Decision) -> String

    Decision::rule_specificity

    source
    fn Decision::rule_specificity(self : Decision) -> Int

    Decision::summary

    source
    fn Decision::summary(self : Decision) -> String

    Decision::trace

    source
    fn Decision::trace(self : Decision) -> Array[DecisionStep]

    Decision::verdict

    source
    fn Decision::verdict(self : Decision) -> String

    DecisionMatrix

    source
    pub struct DecisionMatrix {
      outcomes : Array[ProbeOutcome]
      allowed_count : Int
      denied_count : Int
      default_count : Int
    } derive(Eq, 
    Debug    )

    Deterministic batch of access decisions.

    DecisionMatrix::all_allowed

    source
    fn DecisionMatrix::all_allowed(self : DecisionMatrix) -> Bool

    DecisionMatrix::allowed_count

    source
    fn DecisionMatrix::allowed_count(self : DecisionMatrix) -> Int

    DecisionMatrix::default_count

    source
    fn DecisionMatrix::default_count(self : DecisionMatrix) -> Int

    DecisionMatrix::denied_count

    source
    fn DecisionMatrix::denied_count(self : DecisionMatrix) -> Int

    DecisionMatrix::outcomes

    source

    DecisionMatrix::total

    source
    fn DecisionMatrix::total(self : DecisionMatrix) -> Int

    DecisionStep

    source
    pub struct DecisionStep {
      stage : String
      detail : String
    } derive(Eq, 
    Debug    )

    One step in an explainable access decision.

    DecisionStep::detail

    source
    fn DecisionStep::detail(self : DecisionStep) -> String

    DecisionStep::stage

    source
    fn DecisionStep::stage(self : DecisionStep) -> String

    DecisionStep::summary

    source
    fn DecisionStep::summary(self : DecisionStep) -> String

    Diagnostic

    source
    pub struct Diagnostic {
      severity : Severity
      code : String
      message : String
      span : SourceSpan
      hint : String
    } derive(Eq, 
    Debug    )

    A source-aware parser or policy diagnostic.

    Diagnostic::code

    source
    fn Diagnostic::code(self : Diagnostic) -> String

    Diagnostic::hint

    source
    fn Diagnostic::hint(self : Diagnostic) -> String

    Diagnostic::line

    source
    fn Diagnostic::line(self : Diagnostic) -> Int

    Diagnostic::message

    source
    fn Diagnostic::message(self : Diagnostic) -> String

    Diagnostic::severity

    source
    fn Diagnostic::severity(self : Diagnostic) -> Severity

    Diagnostic::span

    source
    fn Diagnostic::span(self : Diagnostic) -> SourceSpan

    Diagnostic::summary

    source
    fn Diagnostic::summary(self : Diagnostic) -> String

    Directive

    source
    pub struct Directive {
      kind : DirectiveKind
      key : String
      value : String
      raw : String
      span : SourceSpan
    } derive(Eq, 
    Debug    )

    A preserved source directive, including extensions and unknown fields.

    Directive::is_standard

    source
    fn Directive::is_standard(self : Directive) -> Bool

    Directive::key

    source
    fn Directive::key(self : Directive) -> String

    Directive::kind

    source
    fn Directive::kind(self : Directive) -> DirectiveKind

    Directive::line

    source
    fn Directive::line(self : Directive) -> Int

    Directive::raw

    source
    fn Directive::raw(self : Directive) -> String

    Directive::span

    source
    fn Directive::span(self : Directive) -> SourceSpan

    Directive::value

    source
    fn Directive::value(self : Directive) -> String

    DirectiveKind

    source
    pub(all) enum DirectiveKind {
      UserAgentDirective
      AllowDirective
      DisallowDirective
      SitemapDirective
      CrawlDelayDirective
      HostDirective
      CleanParamDirective
      UnknownDirective
    } derive(Eq, 
    Debug    )

    Classification of a parsed robots.txt record.

    DirectiveKind::label

    source
    fn DirectiveKind::label(self : DirectiveKind) -> String

    Group

    source
    pub struct Group {
      agents : Array[String]
      rules : Array[Rule]
      crawl_delay_millis : Int
      start_line : Int
      end_line : Int
    } derive(Eq, 
    Debug    )

    A user-agent group in robots.txt.

    Group::agent_count

    source
    fn Group::agent_count(self : Group) -> Int

    Group::agents

    source
    fn Group::agents(self : Group) -> Array[String]

    Group::crawl_delay_millis

    source
    fn Group::crawl_delay_millis(self : Group) -> Int

    Group::end_line

    source
    fn Group::end_line(self : Group) -> Int

    Group::has_crawl_delay

    source
    fn Group::has_crawl_delay(self : Group) -> Bool

    Group::rule_count

    source
    fn Group::rule_count(self : Group) -> Int

    Group::rules

    source
    fn Group::rules(self : Group) -> Array[Rule]

    Group::start_line

    source
    fn Group::start_line(self : Group) -> Int

    LintOptions

    source
    pub struct LintOptions {
      require_wildcard_group : Bool
      require_sitemap : Bool
      warn_unknown_directives : Bool
      max_crawl_delay_millis : Int
      max_rules_per_group : Int
    } derive(Eq, 
    Debug    )

    Configuration for policy linting.

    LintOptions::max_crawl_delay_millis

    source
    fn LintOptions::max_crawl_delay_millis(self : LintOptions) -> Int

    LintOptions::max_rules_per_group

    source
    fn LintOptions::max_rules_per_group(self : LintOptions) -> Int

    LintOptions::require_sitemap

    source
    fn LintOptions::require_sitemap(self : LintOptions) -> Bool

    LintOptions::require_wildcard_group

    source
    fn LintOptions::require_wildcard_group(self : LintOptions) -> Bool

    LintOptions::warn_unknown_directives

    source
    fn LintOptions::warn_unknown_directives(self : LintOptions) -> Bool

    MatrixComparison

    source
    pub struct MatrixComparison {
      name : String
      before : ProbeOutcome?
      after : ProbeOutcome
      change : BehaviorChange
    } derive(Eq, 
    Debug    )

    MatrixComparison::after

    source

    MatrixComparison::before

    source

    MatrixComparison::change

    source

    MatrixComparison::name

    source
    fn MatrixComparison::name(self : MatrixComparison) -> String

    ParseOptions

    source
    pub struct ParseOptions {
      max_input_chars : Int
      max_lines : Int
      max_line_chars : Int
      max_groups : Int
      max_rules : Int
      preserve_unknown : Bool
    } derive(Eq, 
    Debug    )

    Limits used by the defensive parser.

    ParseOptions::max_groups

    source
    fn ParseOptions::max_groups(self : ParseOptions) -> Int

    ParseOptions::max_input_chars

    source
    fn ParseOptions::max_input_chars(self : ParseOptions) -> Int

    ParseOptions::max_line_chars

    source
    fn ParseOptions::max_line_chars(self : ParseOptions) -> Int

    ParseOptions::max_lines

    source
    fn ParseOptions::max_lines(self : ParseOptions) -> Int

    ParseOptions::max_rules

    source
    fn ParseOptions::max_rules(self : ParseOptions) -> Int

    ParseOptions::preserve_unknown

    source
    fn ParseOptions::preserve_unknown(self : ParseOptions) -> Bool

    ParsedUrl

    source
    pub struct ParsedUrl {
      scheme : String
      authority : String
      host : String
      port : Int
      path_query : String
      fragment : String
      valid : Bool
      error : String
    } derive(Eq, 
    Debug    )

    Parsed HTTP or HTTPS URL used by sitemap and audit helpers.

    ParsedUrl::authority

    source
    fn ParsedUrl::authority(self : ParsedUrl) -> String

    ParsedUrl::error

    source
    fn ParsedUrl::error(self : ParsedUrl) -> String

    ParsedUrl::fragment

    source
    fn ParsedUrl::fragment(self : ParsedUrl) -> String

    ParsedUrl::host

    source
    fn ParsedUrl::host(self : ParsedUrl) -> String

    ParsedUrl::origin

    source
    fn ParsedUrl::origin(self : ParsedUrl) -> String

    ParsedUrl::path_query

    source
    fn ParsedUrl::path_query(self : ParsedUrl) -> String

    ParsedUrl::port

    source
    fn ParsedUrl::port(self : ParsedUrl) -> Int

    ParsedUrl::robots_url

    source
    fn ParsedUrl::robots_url(self : ParsedUrl) -> String

    ParsedUrl::scheme

    source
    fn ParsedUrl::scheme(self : ParsedUrl) -> String

    ParsedUrl::valid

    source
    fn ParsedUrl::valid(self : ParsedUrl) -> Bool

    ParsedUrl::without_fragment

    source
    fn ParsedUrl::without_fragment(self : ParsedUrl) -> String

    PlannedTask

    source
    pub struct PlannedTask {
      task : CrawlTask
      canonical_url : String
      host : String
      path : String
      sequence : Int
      not_before_millis : Int
      decision : Decision
    } derive(Eq, 
    Debug    )

    A task accepted into the crawl plan.

    PlannedTask::canonical_url

    source
    fn PlannedTask::canonical_url(self : PlannedTask) -> String

    PlannedTask::decision

    source
    fn PlannedTask::decision(self : PlannedTask) -> Decision

    PlannedTask::host

    source
    fn PlannedTask::host(self : PlannedTask) -> String

    PlannedTask::not_before_millis

    source
    fn PlannedTask::not_before_millis(self : PlannedTask) -> Int

    PlannedTask::path

    source
    fn PlannedTask::path(self : PlannedTask) -> String

    PlannedTask::sequence

    source
    fn PlannedTask::sequence(self : PlannedTask) -> Int

    PlannedTask::task

    source
    fn PlannedTask::task(self : PlannedTask) -> CrawlTask

    Policy

    source
    pub struct Policy {
      groups : Array[Group]
      sitemaps : Array[String]
      hosts : Array[String]
      directives : Array[Directive]
      diagnostics : Array[Diagnostic]
      source_lines : Int
      truncated : Bool
    } derive(Eq, 
    Debug    )

    Parsed robots.txt policy with source metadata and diagnostics.

    Policy::diagnostic_count

    source
    fn Policy::diagnostic_count(self : Policy) -> Int

    Policy::diagnostics

    source
    fn Policy::diagnostics(self : Policy) -> Array[Diagnostic]

    Policy::directives

    source
    fn Policy::directives(self : Policy) -> Array[Directive]

    Policy::error_count

    source
    fn Policy::error_count(self : Policy) -> Int

    Policy::group_count

    source
    fn Policy::group_count(self : Policy) -> Int

    Policy::groups

    source
    fn Policy::groups(self : Policy) -> Array[Group]

    Policy::hosts

    source
    fn Policy::hosts(self : Policy) -> Array[String]

    Policy::is_valid

    source
    fn Policy::is_valid(self : Policy) -> Bool

    Policy::rule_count

    source
    fn Policy::rule_count(self : Policy) -> Int

    Policy::sitemaps

    source
    fn Policy::sitemaps(self : Policy) -> Array[String]

    Policy::source_lines

    source
    fn Policy::source_lines(self : Policy) -> Int

    Policy::stats

    source
    fn Policy::stats(self : Policy) -> PolicyStats

    Policy::truncated

    source
    fn Policy::truncated(self : Policy) -> Bool

    Policy::warning_count

    source
    fn Policy::warning_count(self : Policy) -> Int

    PolicyAudit

    source
    pub struct PolicyAudit {
      policy : Policy
      diagnostics : Array[Diagnostic]
      matrix : DecisionMatrix
      risk_score : Int
      grade : String
      explicit_decisions : Int
      default_decisions : Int
    } derive(Eq, 
    Debug    )

    Aggregated, deterministic policy audit suitable for CI output.

    PolicyAudit::default_decisions

    source
    fn PolicyAudit::default_decisions(self : PolicyAudit) -> Int

    PolicyAudit::diagnostics

    source
    fn PolicyAudit::diagnostics(self : PolicyAudit) -> Array[Diagnostic]

    PolicyAudit::explicit_decisions

    source
    fn PolicyAudit::explicit_decisions(self : PolicyAudit) -> Int

    PolicyAudit::grade

    source
    fn PolicyAudit::grade(self : PolicyAudit) -> String

    PolicyAudit::matrix

    source
    fn PolicyAudit::matrix(self : PolicyAudit) -> DecisionMatrix

    PolicyAudit::passed

    source
    fn PolicyAudit::passed(self : PolicyAudit) -> Bool

    PolicyAudit::policy

    source
    fn PolicyAudit::policy(self : PolicyAudit) -> Policy

    PolicyAudit::risk_score

    source
    fn PolicyAudit::risk_score(self : PolicyAudit) -> Int

    PolicyChange

    source
    pub struct PolicyChange {
      kind : ChangeKind
      subject : String
      before : String
      after : String
      line_before : Int
      line_after : Int
    } derive(Eq, 
    Debug    )

    One source-aware difference between two policies.

    PolicyChange::after

    source
    fn PolicyChange::after(self : PolicyChange) -> String

    PolicyChange::before

    source
    fn PolicyChange::before(self : PolicyChange) -> String

    PolicyChange::kind

    source

    PolicyChange::line_after

    source
    fn PolicyChange::line_after(self : PolicyChange) -> Int

    PolicyChange::line_before

    source
    fn PolicyChange::line_before(self : PolicyChange) -> Int

    PolicyChange::subject

    source
    fn PolicyChange::subject(self : PolicyChange) -> String

    PolicyChange::summary

    source
    fn PolicyChange::summary(self : PolicyChange) -> String

    PolicyDiff

    source
    pub struct PolicyDiff {
      changes : Array[PolicyChange]
      structural_count : Int
      behavioral_count : Int
    } derive(Eq, 
    Debug    )

    Complete structural and optional behavioral difference.

    PolicyDiff::behavioral_count

    source
    fn PolicyDiff::behavioral_count(self : PolicyDiff) -> Int

    PolicyDiff::changes

    source
    fn PolicyDiff::changes(self : PolicyDiff) -> Array[PolicyChange]

    PolicyDiff::is_empty

    source
    fn PolicyDiff::is_empty(self : PolicyDiff) -> Bool

    PolicyDiff::structural_count

    source
    fn PolicyDiff::structural_count(self : PolicyDiff) -> Int

    PolicyStats

    source
    pub struct PolicyStats {
      groups : Int
      agents : Int
      rules : Int
      allows : Int
      disallows : Int
      sitemaps : Int
      diagnostics : Int
    } derive(Eq, 
    Debug    )

    Compact aggregate statistics for a policy.

    PolicyStats::agents

    source
    fn PolicyStats::agents(self : PolicyStats) -> Int

    PolicyStats::allows

    source
    fn PolicyStats::allows(self : PolicyStats) -> Int

    PolicyStats::diagnostics

    source
    fn PolicyStats::diagnostics(self : PolicyStats) -> Int

    PolicyStats::disallows

    source
    fn PolicyStats::disallows(self : PolicyStats) -> Int

    PolicyStats::groups

    source
    fn PolicyStats::groups(self : PolicyStats) -> Int

    PolicyStats::rules

    source
    fn PolicyStats::rules(self : PolicyStats) -> Int

    PolicyStats::sitemaps

    source
    fn PolicyStats::sitemaps(self : PolicyStats) -> Int

    ProbeOutcome

    source
    pub struct ProbeOutcome {
      probe : AccessProbe
      decision : Decision
    } derive(Eq, 
    Debug    )

    Result of running one access probe.

    ProbeOutcome::allowed

    source
    fn ProbeOutcome::allowed(self : ProbeOutcome) -> Bool

    ProbeOutcome::decision

    source
    fn ProbeOutcome::decision(self : ProbeOutcome) -> Decision

    ProbeOutcome::probe

    source

    ProbeOutcome::summary

    source
    fn ProbeOutcome::summary(self : ProbeOutcome) -> String

    RejectedTask

    source
    pub struct RejectedTask {
      task : CrawlTask
      reason : RejectionReason
      detail : String
    } derive(Eq, 
    Debug    )

    A task rejected by one deterministic planning constraint.

    RejectedTask::detail

    source
    fn RejectedTask::detail(self : RejectedTask) -> String

    RejectedTask::reason

    source

    RejectedTask::task

    source
    fn RejectedTask::task(self : RejectedTask) -> CrawlTask

    RejectionReason

    source
    pub(all) enum RejectionReason {
      InvalidUrl
      RobotsDenied
      DepthExceeded
      RequestBudgetExceeded
      ByteBudgetExceeded
      HostBudgetExceeded
      DuplicateUrl
    } derive(Eq, 
    Debug    )

    Why a task was rejected by the planner.

    RejectionReason::label

    source
    fn RejectionReason::label(self : RejectionReason) -> String

    RenderStyle

    source
    pub(all) enum RenderStyle {
      Canonical
      Compact
      Annotated
    } derive(Eq, 
    Debug    )

    Formatting style for normalized robots.txt output.
    impl Show for RenderStyle

    RenderStyle::label

    source
    fn RenderStyle::label(self : RenderStyle) -> String

    Rule

    source
    pub struct Rule {
      kind : RuleKind
      pattern : String
      normalized_pattern : String
      line : Int
    } derive(Eq, 
    Debug    )

    One parsed Allow or Disallow rule.

    Rule::is_empty

    source
    fn Rule::is_empty(self : Rule) -> Bool

    Rule::kind

    source
    fn Rule::kind(self : Rule) -> RuleKind

    Rule::line

    source
    fn Rule::line(self : Rule) -> Int

    Rule::normalized_pattern

    source
    fn Rule::normalized_pattern(self : Rule) -> String

    Rule::pattern

    source
    fn Rule::pattern(self : Rule) -> String

    Rule::specificity

    source
    fn Rule::specificity(self : Rule) -> Int

    RuleKind

    source
    pub(all) enum RuleKind {
      Allow
      Disallow
    } derive(Eq, 
    Debug    )

    Whether a matching path is allowed or disallowed.

    RuleKind::directive_name

    source
    fn RuleKind::directive_name(self : RuleKind) -> String

    RuleKind::label

    source
    fn RuleKind::label(self : RuleKind) -> String

    RuleKind::tie_priority

    source
    fn RuleKind::tie_priority(self : RuleKind) -> Int

    Severity

    source
    pub(all) enum Severity {
      Info
      Warning
      Error
    } derive(Eq, 
    Debug    )

    Severity of a parser or policy diagnostic.

    Severity::label

    source
    fn Severity::label(self : Severity) -> String

    Severity::rank

    source
    fn Severity::rank(self : Severity) -> Int

    SitemapDocument

    source
    pub struct SitemapDocument {
      kind : SitemapKind
      entries : Array[SitemapEntry]
      child_sitemaps : Array[String]
      diagnostics : Array[Diagnostic]
      source_items : Int
    } derive(Eq, 
    Debug    )

    Parsed sitemap entries, child indexes, and diagnostics.

    SitemapDocument::child_count

    source
    fn SitemapDocument::child_count(self : SitemapDocument) -> Int

    SitemapDocument::child_sitemaps

    source
    fn SitemapDocument::child_sitemaps(self : SitemapDocument) -> Array[String]

    SitemapDocument::diagnostics

    source
    fn SitemapDocument::diagnostics(self : SitemapDocument) -> Array[Diagnostic]

    SitemapDocument::entries

    source

    SitemapDocument::entry_count

    source
    fn SitemapDocument::entry_count(self : SitemapDocument) -> Int

    SitemapDocument::is_valid

    source
    fn SitemapDocument::is_valid(self : SitemapDocument) -> Bool

    SitemapDocument::kind

    source

    SitemapDocument::source_items

    source
    fn SitemapDocument::source_items(self : SitemapDocument) -> Int

    SitemapEntry

    source
    pub struct SitemapEntry {
      location : String
      last_modified : String
      change_frequency : String
      priority_millis : Int
      ordinal : Int
    } derive(Eq, 
    Debug    )

    One URL from a sitemap document.

    SitemapEntry::change_frequency

    source
    fn SitemapEntry::change_frequency(self : SitemapEntry) -> String

    SitemapEntry::has_last_modified

    source
    fn SitemapEntry::has_last_modified(self : SitemapEntry) -> Bool

    SitemapEntry::has_priority

    source
    fn SitemapEntry::has_priority(self : SitemapEntry) -> Bool

    SitemapEntry::last_modified

    source
    fn SitemapEntry::last_modified(self : SitemapEntry) -> String

    SitemapEntry::location

    source
    fn SitemapEntry::location(self : SitemapEntry) -> String

    SitemapEntry::ordinal

    source
    fn SitemapEntry::ordinal(self : SitemapEntry) -> Int

    SitemapEntry::path

    source
    fn SitemapEntry::path(self : SitemapEntry) -> String

    SitemapEntry::priority_millis

    source
    fn SitemapEntry::priority_millis(self : SitemapEntry) -> Int

    SitemapKind

    source
    pub(all) enum SitemapKind {
      UrlSet
      SitemapIndex
      TextSitemap
      UnknownSitemap
    } derive(Eq, 
    Debug    )

    Detected sitemap representation.

    SitemapKind::label

    source
    fn SitemapKind::label(self : SitemapKind) -> String

    SourceSpan

    source
    pub struct SourceSpan {
      line : Int
      column : Int
      length : Int
    } derive(Eq, 
    Debug    )

    Source location for a line-oriented robots.txt record.

    SourceSpan::column

    source
    fn SourceSpan::column(self : SourceSpan) -> Int

    SourceSpan::length

    source
    fn SourceSpan::length(self : SourceSpan) -> Int

    SourceSpan::line

    source
    fn SourceSpan::line(self : SourceSpan) -> Int

    SourceSpan::summary

    source
    fn SourceSpan::summary(self : SourceSpan) -> String

    access_probe

    source
    fn access_probe(name : String, agent : String, path : String) -> AccessProbe

    all_diagnostics

    source
    fn all_diagnostics(policy : Policy) -> Array[Diagnostic]

    all_diagnostics_with_options

    source
    fn all_diagnostics_with_options(policy : Policy, options : LintOptions) -> Array[Diagnostic]

    allowed_sitemap_entries

    source
    fn allowed_sitemap_entries(policy : Policy, agent : String, document : SitemapDocument) -> Array[SitemapEntry]

    annotated_policy

    source
    fn annotated_policy(policy : Policy) -> String

    audit

    source
    fn audit(input : String, agents : Array[String], paths : Array[String]) -> PolicyAudit

    Parses, lints, and probes a policy in one deterministic operation.

    audit_default

    source
    fn audit_default(input : String) -> PolicyAudit

    audit_exit_code

    source
    fn audit_exit_code(audit : PolicyAudit) -> Int

    behavior_changes

    source
    fn behavior_changes(diff : PolicyDiff) -> Array[PolicyChange]

    behavior_flip_count

    source
    fn behavior_flip_count(comparisons : Array[MatrixComparison]) -> Int

    blocked_sitemap_entries

    source
    fn blocked_sitemap_entries(policy : Policy, agent : String, document : SitemapDocument) -> Array[SitemapEntry]

    build_probe_grid

    source
    fn build_probe_grid(agents : Array[String], paths : Array[String]) -> Array[AccessProbe]

    can_fetch

    source
    fn can_fetch(policy : Policy, agent : String, path : String) -> Bool

    Convenience helper that returns only the allow/deny boolean.

    canonical_url

    source
    fn canonical_url(input : String) -> String

    canonicalize

    source
    fn canonicalize(input : String) -> String

    compact_policy

    source
    fn compact_policy(policy : Policy) -> String

    compare_matrices

    source
    fn compare_matrices(before : DecisionMatrix, after : DecisionMatrix) -> Array[MatrixComparison]

    crawl_budget

    source
    fn crawl_budget(max_requests : Int, max_bytes : Int, max_depth : Int, max_requests_per_host : Int, fallback_delay_millis : Int) -> CrawlBudget

    crawl_delay

    source
    fn crawl_delay(policy : Policy, agent : String) -> Int

    Returns the applicable crawl delay in milliseconds, or -1 when absent.

    crawl_task

    source
    fn crawl_task(id : String, url : String, priority : Int, depth : Int, estimated_bytes : Int) -> CrawlTask

    cross_origin_entries

    source
    fn cross_origin_entries(document : SitemapDocument, expected_origin : String) -> Array[SitemapEntry]

    decide

    source
    fn decide(policy : Policy, agent : String, path : String) -> Decision

    Returns the full matching decision for a user-agent and URL path.

    default_crawl_budget

    source
    fn default_crawl_budget() -> CrawlBudget

    default_lint_options

    source
    fn default_lint_options() -> LintOptions

    default_outcomes

    source
    fn default_outcomes(matrix : DecisionMatrix) -> Array[ProbeOutcome]

    default_parse_options

    source
    fn default_parse_options() -> ParseOptions

    denied_outcomes

    source
    fn denied_outcomes(matrix : DecisionMatrix) -> Array[ProbeOutcome]

    diagnostics_at_least

    source
    fn diagnostics_at_least(diagnostics : Array[Diagnostic], severity : Severity) -> Array[Diagnostic]

    diff_policies

    source
    fn diff_policies(before : Policy, after : Policy, probes : Array[AccessProbe]) -> PolicyDiff

    Compares policy structure and access behavior on caller-provided probes.

    discovered_crawl_task

    source
    fn discovered_crawl_task(id : String, url : String, priority : Int, depth : Int, estimated_bytes : Int, discovered_from : String) -> CrawlTask

    effective_rules

    source
    fn effective_rules(policy : Policy, agent : String) -> Array[Rule]

    Returns every effective rule from equally specific matching groups.

    find_outcome

    source
    fn find_outcome(matrix : DecisionMatrix, name : String) -> ProbeOutcome?

    has_diagnostic_code

    source
    fn has_diagnostic_code(diagnostics : Array[Diagnostic], code : String) -> Bool

    lint

    source
    fn lint(policy : Policy) -> Array[Diagnostic]

    lint_with_options

    source
    fn lint_with_options(policy : Policy, options : LintOptions) -> Array[Diagnostic]

    Performs semantic and maintainability checks beyond parsing.

    normalize_path

    source
    fn normalize_path(path : String) -> String

    Normalizes a URL path for RFC 9309 comparison.

    normalize_percent_encoding

    source
    fn normalize_percent_encoding(input : String) -> String

    Decodes percent-encoded unreserved ASCII while preserving reserved octets.

    parse

    source
    fn parse(input : String) -> Policy

    Parses robots.txt with safe defaults.

    parse_options

    source
    fn parse_options(max_input_chars : Int, max_lines : Int, max_line_chars : Int, max_groups : Int, max_rules : Int, preserve_unknown : Bool) -> ParseOptions

    parse_render_style

    source
    fn parse_render_style(input : String) -> Result[RenderStyle, String]

    parse_sitemap

    source
    fn parse_sitemap(input : String) -> SitemapDocument

    Parses XML urlsets, XML sitemap indexes, or line-oriented text sitemaps.

    parse_url

    source
    fn parse_url(input : String) -> ParsedUrl

    Parses an absolute HTTP or HTTPS URL without network access.

    parse_with_options

    source
    fn parse_with_options(input : String, options : ParseOptions) -> Policy

    Parses robots.txt using defensive limits and preserves diagnostics.

    plan_crawl

    source
    fn plan_crawl(policy : Policy, agent : String, tasks : Array[CrawlTask], budget : CrawlBudget) -> CrawlPlan

    Plans crawl order by priority while enforcing protocol and budget limits.

    rejected_for_reason

    source
    fn rejected_for_reason(plan : CrawlPlan, reason : RejectionReason) -> Array[RejectedTask]

    remove_url_fragment

    source
    fn remove_url_fragment(input : String) -> String

    render_audit_json

    source
    fn render_audit_json(audit : PolicyAudit) -> String

    render_audit_markdown

    source
    fn render_audit_markdown(audit : PolicyAudit) -> String

    render_blocked_sitemap_report

    source
    fn render_blocked_sitemap_report(policy : Policy, agent : String, document : SitemapDocument) -> String

    render_crawl_plan

    source
    fn render_crawl_plan(plan : CrawlPlan) -> String

    render_decision_trace

    source
    fn render_decision_trace(decision : Decision) -> String

    render_diagnostics

    source
    fn render_diagnostics(diagnostics : Array[Diagnostic]) -> String

    render_diff

    source
    fn render_diff(diff : PolicyDiff) -> String

    render_matrix_comparison

    source
    fn render_matrix_comparison(comparisons : Array[MatrixComparison]) -> String

    render_matrix_csv

    source
    fn render_matrix_csv(matrix : DecisionMatrix) -> String

    render_matrix_markdown

    source
    fn render_matrix_markdown(matrix : DecisionMatrix) -> String

    render_policy

    source
    fn render_policy(policy : Policy, style : RenderStyle) -> String

    Serializes the semantic policy into deterministic robots.txt text.

    render_policy_summary

    source
    fn render_policy_summary(policy : Policy) -> String

    render_sitemap_summary

    source
    fn render_sitemap_summary(document : SitemapDocument) -> String

    robots_url_for

    source
    fn robots_url_for(input : String) -> String

    run_matrix

    source
    fn run_matrix(policy : Policy, probes : Array[AccessProbe]) -> DecisionMatrix

    Evaluates a list of named probes against one policy.

    same_origin

    source
    fn same_origin(left : String, right : String) -> Bool

    strict_lint_options

    source
    fn strict_lint_options() -> LintOptions

    strict_parse_options

    source
    fn strict_parse_options() -> ParseOptions

    structural_diff

    source
    fn structural_diff(before : Policy, after : Policy) -> PolicyDiff

    tasks_from_sitemap

    source
    fn tasks_from_sitemap(document : SitemapDocument, default_bytes : Int) -> Array[CrawlTask]

    url_depth

    source
    fn url_depth(input : String) -> Int

    url_has_query

    source
    fn url_has_query(input : String) -> Bool

    url_path

    source
    fn url_path(input : String) -> String

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io