Sign in

    robopolicy

    MoonBit AI crawler intent gate with robots evidence, remediation gaps, and release capsules

    ai-crawler
    intent
    compliance
    evidence
    release
    robots
    audit
    Download zip
    Author
    Version
    0.2.0
    License
    MIT
    Last updated
    2 months ago
    Downloads
    30

    No dependencies

    #RoboPolicy Intent Gate

    RoboPolicy Intent Gate 是一个纯 MoonBit 实现的 AI 抓取意图合规验收库。它不再把“robots.txt 解析器”作为核心卖点,而是把 robots 规则、目标路径、AI/爬虫用途、数据敏感级别、授权方式、留存期限和证据编号合并成一份可复现的发布前验收报告,并进一步生成带稳定指纹、缺口修复清单和评审说明的 Evidence Capsule。

    一句话定位:给 MoonBit/Wasm 里的 AI 数据抓取、搜索索引、监控、归档和安全扫描流程做“能不能抓、为什么、缺什么证据、是否需要人工复核”的确定性检查。

    #与已有 robots 项目的关系

    本项目已针对初审反馈重新定位。Mooncakes 上已有可用 robots 项目:

    • cauchyQ/moonbit-robots:小型 robots.txt 解析和访问决策库。
    • 2111950632/robots-gate:robots.txt 策略审计、站点地图校验与合规抓取规划工具。

    RoboPolicy Intent Gate 不与它们竞争“解析 robots.txt”这一基础能力。当前项目的新增边界是 AI/数据抓取意图验收:

    • 记录抓取目的:搜索索引、监控、研究、归档、AI 训练、安全扫描、价格比较等。
    • 标记数据敏感级别:公开页面、账号数据、个人数据、凭据、管理后台、付费内容、训练语料。
    • 区分授权方式:无授权、仅 robots、站点所有者批准、合同授权、内部测试。
    • 对每个 URL 输出 verdict:permitted、needs-review、denied。
    • 输出缺失证据、阻断原因、留存要求、人工复核义务和 Markdown/CSV/JSON 证据行。
    • 生成 Evidence Capsule:稳定发布指纹、验收 readiness、差异化说明、修复清单和 Markdown/JSON 评审包。

    robots 解析和访问决策在这里是底层信号;项目核心是把“能不能访问”提升为“这个 AI/爬虫意图能不能进入发布流程”。

    #安装方式

    moon add JQ-ai-nb/robopolicy

    #最小使用示例

    fn main {
    let policy = @robopolicy.parse(@robopolicy.sample_robots())
    let intents = [
    @robopolicy.safe_public_ai_intent(
    "docs-monitor",
    "ResearchBot",
    "/private/summary.html",
    "owner-ticket-42",
    ),
    @robopolicy.training_intent(
    "training-review",
    "ResearchBot",
    "/private/summary.html",
    @robopolicy.ConsentRobotsOnly,
    "dataset-plan-7",
    ),
    ]
    let report = @robopolicy.review_intents(policy, intents)
    println(report.to_markdown())
    println(report.to_evidence_capsule("RoboPolicy Intent Gate", "0.2.0").to_markdown())
    }

    #主要功能

    • review_intent(policy, intent):审查单个 AI/爬虫抓取意图。
    • review_intents(policy, intents):生成发布前意图验收报告。
    • CrawlIntent:记录 user-agent、目标路径、抓取目的、敏感级别、授权方式、留存天数和证据编号。
    • IntentReview:输出 robots 结果、风险等级、阻断原因、义务清单、证据行和 verdict。
    • IntentGateReport::to_markdown() / to_csv() / to_json():用于 CI、验收材料和自动化发布报告。
    • build_evidence_capsule(report, project, version):把报告封装为可复核的发布证据胶囊。
    • EvidenceCapsule::to_markdown() / to_json():输出稳定指纹、发布 readiness、差异化说明和修复清单。
    • 保留底层能力:robots.txt 解析、访问决策、路径规范化、规则画像、sitemap 校验、批量目标审计和访问矩阵。

    #本地运行

    moon fmt --check moon info moon check --target all --deny-warn moon build moon test --target wasm --deny-warn moon test --target wasm-gc --deny-warn moon run cmd/main

    CI 还会检查示例输出中包含 RoboPolicy Evidence Capsule,确保发布证据胶囊能力随主流程一起运行。

    #功能边界

    RoboPolicy Intent Gate 不是网络爬虫,不下载网页,不抓取 sitemap 内容,不调度任务,不替用户绕过 robots.txt。它只做离线、确定性、可测试的发布前验收:给定 robots 文本和意图清单,输出是否允许进入抓取/训练/发布流程的证据报告,并把报告封装成可随 CI、发布说明或验收材料提交的 Evidence Capsule。

    #验收资料

    • 申报书:SUBMISSION.md
    • 差异与扩展关系:docs/overlap-response.md
    • 设计说明:docs/design.md
    • 调研记录:docs/research.md
    • 测试记录:docs/test-record.md
    • 验收清单:docs/acceptance-checklist.md
    • 发布说明:docs/release.md
    • 更新日志:CHANGELOG.md

    #Mooncakes

    • 包名:JQ-ai-nb/robopolicy
    • 文档地址:https://mooncakes.io/docs/JQ-ai-nb/robopolicy
    • Manifest:https://mooncakes.io/api/v0/manifest/JQ-ai-nb/robopolicy

    #开源许可证和第三方说明

    本项目使用 MIT 许可证。核心代码为原创 MoonBit 实现,不移植第三方源码,不包含来源不明素材、图片或私有代码。robots.txt 语义参考通用 Robots Exclusion Protocol 约定;差异化功能为 AI/爬虫意图验收、授权证据、敏感级别、缺口修复清单和发布证据胶囊。

    AccessDecision

    pub(all) struct AccessDecision {
    allowed : Bool
    user_agent : String
    path : String
    matched_group : String
    matched_rule : Rule?
    reason : String
    } derive(Eq,
    Debug
    )

    AccessDecision::to_json

    fn AccessDecision::to_json(self : AccessDecision) -> String

    AccessDecision::to_line

    fn AccessDecision::to_line(self : AccessDecision) -> String

    AgentGroup

    pub(all) struct AgentGroup {
    agents : Array[String]
    rules : Array[Rule]
    crawl_delay : Int?
    line : Int
    } derive(Eq,
    Debug
    )

    AgentGroup::profiles

    fn AgentGroup::profiles(self : AgentGroup) -> Array[RuleProfile]

    AgentGroup::rule_pairs

    fn AgentGroup::rule_pairs(self : AgentGroup) -> Array[RulePairAnalysis]

    AgentGroup::summary

    fn AgentGroup::summary(self : AgentGroup, has_sitemap_hint : Bool) -> GroupRuleSummary

    AgentGroup::to_json

    fn AgentGroup::to_json(self : AgentGroup) -> String

    AgentMatrix

    pub(all) struct AgentMatrix {
    rows : Array[AgentMatrixRow]
    agent_count : Int
    path_count : Int
    allowed_count : Int
    blocked_count : Int
    } derive(Eq,
    Debug
    )

    AgentMatrix::to_json

    fn AgentMatrix::to_json(self : AgentMatrix) -> String

    AgentMatrix::to_markdown

    fn AgentMatrix::to_markdown(self : AgentMatrix) -> String

    AgentMatrixRow

    pub(all) struct AgentMatrixRow {
    user_agent : String
    path : String
    allowed : Bool
    reason : String
    matched_group : String
    matched_pattern : String
    } derive(Eq,
    Debug
    )

    AgentMatrixRow::to_json

    fn AgentMatrixRow::to_json(self : AgentMatrixRow) -> String

    AgentMatrixRow::to_markdown_row

    fn AgentMatrixRow::to_markdown_row(self : AgentMatrixRow) -> String

    AuditReport

    pub(all) struct AuditReport {
    policy : RobotsPolicy
    findings : Array[Finding]
    } derive(Eq,
    Debug
    )

    AuditReport::has_finding

    fn AuditReport::has_finding(self : AuditReport, code : StringView) -> Bool

    AuditReport::to_json

    fn AuditReport::to_json(self : AuditReport) -> String

    AuditReport::to_markdown

    fn AuditReport::to_markdown(self : AuditReport) -> String

    AuditTarget

    pub(all) struct AuditTarget {
    label : String
    user_agent : String
    url_or_path : String
    expected_allowed : Bool?
    importance : TargetImportance
    note : String
    } derive(Eq,
    Debug
    )

    AuditTarget::normalized_path

    fn AuditTarget::normalized_path(self : AuditTarget) -> String

    AuditTarget::to_json

    fn AuditTarget::to_json(self : AuditTarget) -> String

    AuditTarget::to_line

    fn AuditTarget::to_line(self : AuditTarget) -> String

    AuditTarget::with_expected

    fn AuditTarget::with_expected(self : AuditTarget, expected_allowed : Bool) -> AuditTarget

    AuditTarget::with_importance

    fn AuditTarget::with_importance(self : AuditTarget, importance : TargetImportance) -> AuditTarget

    AuditTarget::with_note

    fn AuditTarget::with_note(self : AuditTarget, note : StringView) -> AuditTarget

    CapsuleReadiness

    pub(all) enum CapsuleReadiness {
    CapsuleReady
    CapsuleNeedsReview
    CapsuleBlocked
    } derive(Eq,
    Debug
    )

    Evidence Capsule converts an Intent Gate report into a deterministic release-review artifact. It gives reviewers a stable fingerprint, a remediation checklist, and a concise non-overlap statement that sits above ordinary robots.txt parsing or crawl planning.

    CapsuleReadiness::label

    fn CapsuleReadiness::label(self : CapsuleReadiness) -> String

    ConsentMode

    pub(all) enum ConsentMode {
    ConsentNone
    ConsentRobotsOnly
    ConsentOwnerApproved
    ConsentContractual
    ConsentInternalTest
    } derive(Eq,
    Debug
    )

    ConsentMode::label

    fn ConsentMode::label(self : ConsentMode) -> String

    CoverageBand

    pub(all) enum CoverageBand {
    CoverageEmpty
    CoverageWeak
    CoverageFair
    CoverageStrong
    } derive(Eq,
    Debug
    )

    CoverageBand::label

    fn CoverageBand::label(self : CoverageBand) -> String

    CrawlIntent

    pub(all) struct CrawlIntent {
    label : String
    user_agent : String
    url_or_path : String
    purpose : IntentPurpose
    sensitivity : IntentSensitivity
    consent : ConsentMode
    expected_allowed : Bool?
    retention_days : Int
    evidence_id : String
    note : String
    } derive(Eq,
    Debug
    )

    CrawlIntent::has_evidence

    fn CrawlIntent::has_evidence(self : CrawlIntent) -> Bool

    CrawlIntent::has_explicit_authorization

    fn CrawlIntent::has_explicit_authorization(self : CrawlIntent) -> Bool

    CrawlIntent::importance

    fn CrawlIntent::importance(self : CrawlIntent) -> TargetImportance

    CrawlIntent::needs_explicit_authorization

    fn CrawlIntent::needs_explicit_authorization(self : CrawlIntent) -> Bool

    CrawlIntent::purpose_summary

    fn CrawlIntent::purpose_summary(self : CrawlIntent) -> String

    CrawlIntent::to_target

    fn CrawlIntent::to_target(self : CrawlIntent) -> AuditTarget

    CrawlIntent::with_evidence

    fn CrawlIntent::with_evidence(self : CrawlIntent, evidence_id : StringView) -> CrawlIntent

    CrawlIntent::with_expected

    fn CrawlIntent::with_expected(self : CrawlIntent, expected_allowed : Bool) -> CrawlIntent

    CrawlIntent::with_note

    fn CrawlIntent::with_note(self : CrawlIntent, note : StringView) -> CrawlIntent

    CrawlIntent::with_retention

    fn CrawlIntent::with_retention(self : CrawlIntent, retention_days : Int) -> CrawlIntent

    DirectiveKind

    pub(all) enum DirectiveKind {
    UserAgent
    Allow
    Disallow
    Sitemap
    CrawlDelay
    Host
    Unknown
    } derive(Eq,
    Debug
    )

    DirectiveKind::label

    fn DirectiveKind::label(self : DirectiveKind) -> String

    EvidenceCapsule

    pub(all) struct EvidenceCapsule {
    project_name : String
    version : String
    fingerprint : String
    readiness : CapsuleReadiness
    source_summary : String
    novelty_statement : String
    reviewer_note : String
    checklist : Array[String]
    remediation_items : Array[RemediationItem]
    report : IntentGateReport
    } derive(Eq,
    Debug
    )

    EvidenceCapsule::has_remediation

    fn EvidenceCapsule::has_remediation(self : EvidenceCapsule, kind : RemediationKind) -> Bool

    EvidenceCapsule::is_release_ready

    fn EvidenceCapsule::is_release_ready(self : EvidenceCapsule) -> Bool

    EvidenceCapsule::to_json

    fn EvidenceCapsule::to_json(self : EvidenceCapsule) -> String

    EvidenceCapsule::to_line

    fn EvidenceCapsule::to_line(self : EvidenceCapsule) -> String

    EvidenceCapsule::to_markdown

    fn EvidenceCapsule::to_markdown(self : EvidenceCapsule) -> String

    ExpectationStatus

    pub(all) enum ExpectationStatus {
    ExpectationNotSet
    ExpectationMatched
    ExpectationMismatch
    } derive(Eq,
    Debug
    )

    ExpectationStatus::label

    fn ExpectationStatus::label(self : ExpectationStatus) -> String

    ExportDetailLevel

    pub(all) enum ExportDetailLevel {
    ExportCompact
    ExportNormal
    ExportVerbose
    } derive(Eq,
    Debug
    )

    ExportDetailLevel::label

    fn ExportDetailLevel::label(self : ExportDetailLevel) -> String

    ExportOptions

    pub(all) struct ExportOptions {
    detail : ExportDetailLevel
    include_parse_issues : Bool
    include_rule_profiles : Bool
    include_sitemaps : Bool
    include_targets : Bool
    include_notes : Bool
    } derive(Eq,
    Debug
    )

    Finding

    pub(all) struct Finding {
    code : String
    severity : FindingSeverity
    message : String
    detail : String
    } derive(Eq,
    Debug
    )

    Finding::to_json

    fn Finding::to_json(self : Finding) -> String

    FindingSeverity

    pub(all) enum FindingSeverity {
    Error
    Warning
    Info
    } derive(Eq,
    Debug
    )

    FindingSeverity::label

    fn FindingSeverity::label(self : FindingSeverity) -> String

    GroupRuleSummary

    pub(all) struct GroupRuleSummary {
    agents : Array[String]
    group_line : Int
    rule_count : Int
    allow_count : Int
    disallow_count : Int
    wildcard_rule_count : Int
    anchored_rule_count : Int
    root_block_count : Int
    empty_rule_count : Int
    max_depth : Int
    max_literal_weight : Int
    crawl_delay : Int?
    has_sitemap_hint : Bool
    notes : Array[String]
    } derive(Eq,
    Debug
    )

    GroupRuleSummary::to_json

    fn GroupRuleSummary::to_json(self : GroupRuleSummary) -> String

    GroupRuleSummary::to_line

    fn GroupRuleSummary::to_line(self : GroupRuleSummary) -> String

    GroupRuleSummary::to_markdown

    fn GroupRuleSummary::to_markdown(self : GroupRuleSummary) -> String

    IntentGateReport

    pub(all) struct IntentGateReport {
    policy : RobotsPolicy
    intents : Array[CrawlIntent]
    reviews : Array[IntentReview]
    summary : IntentGateSummary
    target_summary : SiteAuditSummary
    } derive(Eq,
    Debug
    )

    IntentGateReport::to_csv

    fn IntentGateReport::to_csv(self : IntentGateReport) -> String

    IntentGateReport::to_evidence_capsule

    fn IntentGateReport::to_evidence_capsule(self : IntentGateReport, project_name : StringView, version : StringView) -> EvidenceCapsule

    IntentGateReport::to_json

    fn IntentGateReport::to_json(self : IntentGateReport) -> String

    IntentGateReport::to_markdown

    fn IntentGateReport::to_markdown(self : IntentGateReport) -> String

    IntentGateSummary

    pub(all) struct IntentGateSummary {
    total_intents : Int
    permitted_intents : Int
    denied_intents : Int
    review_intents : Int
    robots_allowed : Int
    robots_blocked : Int
    high_risk_intents : Int
    missing_evidence : Int
    explicit_authorization : Int
    average_risk : Int
    score : Int
    notes : Array[String]
    } derive(Eq,
    Debug
    )

    IntentGateSummary::is_release_ready

    fn IntentGateSummary::is_release_ready(self : IntentGateSummary) -> Bool

    IntentGateSummary::pass_rate_percent

    fn IntentGateSummary::pass_rate_percent(self : IntentGateSummary) -> Int

    IntentGateSummary::to_json

    fn IntentGateSummary::to_json(self : IntentGateSummary) -> String

    IntentGateSummary::to_line

    fn IntentGateSummary::to_line(self : IntentGateSummary) -> String

    IntentGateSummary::to_markdown

    fn IntentGateSummary::to_markdown(self : IntentGateSummary) -> String

    IntentPurpose

    pub(all) enum IntentPurpose {
    PurposeSearchIndex
    PurposeMonitoring
    PurposeResearch
    PurposeArchive
    PurposeAiTraining
    PurposeSecurityScan
    PurposePriceComparison
    PurposeUnknown
    } derive(Eq,
    Debug
    )

    Intent Gate turns raw robots decisions into reviewable AI-crawler acceptance evidence. It is the higher-level layer that checks why an agent wants a URL, what data class is involved, and whether enough authorization evidence exists before a crawl or model-training job ships.

    IntentPurpose::label

    fn IntentPurpose::label(self : IntentPurpose) -> String

    IntentReview

    pub(all) struct IntentReview {
    intent : CrawlIntent
    target_result : TargetAuditResult
    verdict : IntentVerdict
    risk_band : IntentRiskBand
    risk_points : Int
    robots_allowed : Bool
    requires_human_review : Bool
    blockers : Array[String]
    obligations : Array[String]
    evidence_line : String
    tags : Array[String]
    } derive(Eq,
    Debug
    )

    IntentReview::has_blocker

    fn IntentReview::has_blocker(self : IntentReview, blocker : StringView) -> Bool

    IntentReview::to_csv_row

    fn IntentReview::to_csv_row(self : IntentReview) -> String

    IntentReview::to_json

    fn IntentReview::to_json(self : IntentReview) -> String

    IntentReview::to_line

    fn IntentReview::to_line(self : IntentReview) -> String

    IntentReview::to_markdown_row

    fn IntentReview::to_markdown_row(self : IntentReview) -> String

    IntentRiskBand

    pub(all) enum IntentRiskBand {
    IntentRiskGreen
    IntentRiskYellow
    IntentRiskOrange
    IntentRiskRed
    } derive(Eq,
    Debug
    )

    IntentRiskBand::label

    fn IntentRiskBand::label(self : IntentRiskBand) -> String

    IntentSensitivity

    pub(all) enum IntentSensitivity {
    SensitivityPublic
    SensitivityAccount
    SensitivityPersonalData
    SensitivityCredential
    SensitivityAdmin
    SensitivityPaidContent
    SensitivityTrainingCorpus
    } derive(Eq,
    Debug
    )

    IntentSensitivity::label

    fn IntentSensitivity::label(self : IntentSensitivity) -> String

    IntentVerdict

    pub(all) enum IntentVerdict {
    IntentPermitted
    IntentNeedsReview
    IntentDenied
    } derive(Eq,
    Debug
    )

    IntentVerdict::label

    fn IntentVerdict::label(self : IntentVerdict) -> String

    IssueKind

    pub(all) enum IssueKind {
    MissingColon
    EmptyUserAgent
    EmptyDirective
    InvalidCrawlDelay
    RuleBeforeUserAgent
    UnknownDirective
    } derive(Eq,
    Debug
    )

    RoboPolicy parses robots.txt documents and answers crawler access decisions. It is designed for deterministic CI checks, site audits, and Wasm tools.

    IssueKind::label

    fn IssueKind::label(self : IssueKind) -> String

    NormalizedPathBody

    pub(all) struct NormalizedPathBody {
    normalized : String
    segments : Array[String]
    collapsed_slashes : Bool
    removed_dot_segments : Bool
    root_fallback : Bool
    } derive(Eq,
    Debug
    )

    ParseIssue

    pub(all) struct ParseIssue {
    kind : IssueKind
    line : Int
    directive : String
    value : String
    message : String
    } derive(Eq,
    Debug
    )

    PathReferenceParts

    pub(all) struct PathReferenceParts {
    source : PathSourceKind
    path : String
    query : String?
    fragment : String?
    had_query : Bool
    had_fragment : Bool
    } derive(Eq,
    Debug
    )

    PathSnapshot

    pub(all) struct PathSnapshot {
    original : String
    normalized : String
    source : PathSourceKind
    query : String?
    fragment : String?
    had_query : Bool
    had_fragment : Bool
    collapsed_slashes : Bool
    removed_dot_segments : Bool
    root_fallback : Bool
    } derive(Eq,
    Debug
    )

    PathSnapshot::as_path

    fn PathSnapshot::as_path(self : PathSnapshot) -> String

    PathSnapshot::debug_flags

    fn PathSnapshot::debug_flags(self : PathSnapshot) -> String

    PathSnapshot::has_fragment

    fn PathSnapshot::has_fragment(self : PathSnapshot) -> Bool

    PathSnapshot::has_query

    fn PathSnapshot::has_query(self : PathSnapshot) -> Bool

    PathSnapshot::is_root

    fn PathSnapshot::is_root(self : PathSnapshot) -> Bool

    PathSnapshot::to_json

    fn PathSnapshot::to_json(self : PathSnapshot) -> String

    PathSnapshot::to_line

    fn PathSnapshot::to_line(self : PathSnapshot) -> String

    PathSnapshot::to_markdown

    fn PathSnapshot::to_markdown(self : PathSnapshot) -> String

    PathSourceKind

    pub(all) enum PathSourceKind {
    Empty
    BarePath
    AbsolutePath
    AbsoluteUrl
    } derive(Eq,
    Debug
    )

    PathSourceKind::label

    fn PathSourceKind::label(self : PathSourceKind) -> String

    PolicyProfile

    pub(all) struct PolicyProfile {
    group_summaries : Array[GroupRuleSummary]
    total_rules : Int
    total_allows : Int
    total_disallows : Int
    total_wildcards : Int
    total_anchors : Int
    total_root_blocks : Int
    highest_depth : Int
    highest_weight : Int
    has_wildcard_group : Bool
    has_specific_group : Bool
    has_sitemap : Bool
    issue_count : Int
    notes : Array[String]
    } derive(Eq,
    Debug
    )

    PolicyProfile::to_json

    fn PolicyProfile::to_json(self : PolicyProfile) -> String

    PolicyProfile::to_line

    fn PolicyProfile::to_line(self : PolicyProfile) -> String

    PolicyProfile::to_markdown

    fn PolicyProfile::to_markdown(self : PolicyProfile) -> String

    QueryFragmentParts

    type QueryFragmentParts derive(Eq,
    Debug
    )

    RemediationItem

    pub(all) struct RemediationItem {
    id : String
    intent_label : String
    kind : RemediationKind
    severity : RemediationSeverity
    verdict : IntentVerdict
    action : String
    evidence_hint : String
    } derive(Eq,
    Debug
    )

    RemediationItem::to_json

    fn RemediationItem::to_json(self : RemediationItem) -> String

    RemediationItem::to_markdown_row

    fn RemediationItem::to_markdown_row(self : RemediationItem) -> String

    RemediationKind

    pub(all) enum RemediationKind {
    RemediationNoIntent
    RemediationRobotsBlock
    RemediationExplicitAuthorization
    RemediationMissingEvidence
    RemediationHumanReview
    RemediationHighRisk
    RemediationRetention
    } derive(Eq,
    Debug
    )

    RemediationKind::label

    fn RemediationKind::label(self : RemediationKind) -> String

    RemediationSeverity

    pub(all) enum RemediationSeverity {
    RemediationInfo
    RemediationWarning
    RemediationCritical
    } derive(Eq,
    Debug
    )

    RemediationSeverity::label

    fn RemediationSeverity::label(self : RemediationSeverity) -> String

    RobotsPolicy

    pub(all) struct RobotsPolicy {
    groups : Array[AgentGroup]
    sitemaps : Array[String]
    host : String?
    issues : Array[ParseIssue]
    } derive(Eq,
    Debug
    )

    RobotsPolicy::audit_targets

    fn RobotsPolicy::audit_targets(self : RobotsPolicy, targets : Array[AuditTarget]) -> SiteAuditReport

    RobotsPolicy::crawl_delay_for

    fn RobotsPolicy::crawl_delay_for(self : RobotsPolicy, user_agent : StringView) -> Int?

    RobotsPolicy::decide

    fn RobotsPolicy::decide(self : RobotsPolicy, user_agent : StringView, url_or_path : StringView) -> AccessDecision

    RobotsPolicy::is_allowed

    fn RobotsPolicy::is_allowed(self : RobotsPolicy, user_agent : StringView, url_or_path : StringView) -> Bool

    RobotsPolicy::profile

    fn RobotsPolicy::profile(self : RobotsPolicy) -> PolicyProfile

    RobotsPolicy::review_intents

    fn RobotsPolicy::review_intents(self : RobotsPolicy, intents : Array[CrawlIntent]) -> IntentGateReport

    RobotsPolicy::sitemap_entries

    fn RobotsPolicy::sitemap_entries(self : RobotsPolicy) -> Array[SitemapEntry]

    RobotsPolicy::sitemap_summary

    fn RobotsPolicy::sitemap_summary(self : RobotsPolicy) -> SitemapSummary

    RobotsPolicy::to_compact_json

    fn RobotsPolicy::to_compact_json(self : RobotsPolicy) -> String

    RobotsPolicy::to_json

    fn RobotsPolicy::to_json(self : RobotsPolicy) -> String

    RobotsPolicy::to_json_with_options

    fn RobotsPolicy::to_json_with_options(self : RobotsPolicy, options : ExportOptions) -> String

    Rule

    pub(all) struct Rule {
    allow : Bool
    pattern : String
    line : Int
    } derive(Eq,
    Debug
    )

    Rule::to_json

    fn Rule::to_json(self : Rule) -> String

    RuleImpact

    pub(all) enum RuleImpact {
    ImpactAllowEmpty
    ImpactAllowNarrow
    ImpactAllowWide
    ImpactBlockRoot
    ImpactBlockDirectory
    ImpactBlockFile
    ImpactBlockWildcard
    ImpactNeutral
    } derive(Eq,
    Debug
    )

    RuleImpact::label

    fn RuleImpact::label(self : RuleImpact) -> String

    RulePairAnalysis

    pub(all) struct RulePairAnalysis {
    first : RuleProfile
    second : RuleProfile
    relation : RuleRelation
    message : String
    same_action : Bool
    same_pattern : Bool
    first_matches_second_prefix : Bool
    second_matches_first_prefix : Bool
    } derive(Eq,
    Debug
    )

    RulePairAnalysis::to_line

    fn RulePairAnalysis::to_line(self : RulePairAnalysis) -> String

    RulePatternKind

    pub(all) enum RulePatternKind {
    PatternEmpty
    PatternRoot
    PatternExactPath
    PatternDirectoryPrefix
    PatternFilePath
    PatternExtensionGlob
    PatternWildcardPrefix
    PatternWildcardSuffix
    PatternWildcardMiddle
    PatternAnchored
    PatternCatchAll
    } derive(Eq,
    Debug
    )

    RulePatternKind::label

    fn RulePatternKind::label(self : RulePatternKind) -> String

    RuleProfile

    pub(all) struct RuleProfile {
    rule : Rule
    kind : RulePatternKind
    impact : RuleImpact
    risk : RuleRiskBand
    normalized_pattern : String
    literal_weight : Int
    depth : Int
    wildcard_count : Int
    has_anchor : Bool
    has_query_like_text : Bool
    has_fragment_like_text : Bool
    is_root_rule : Bool
    is_empty_pattern : Bool
    is_directory_rule : Bool
    is_file_rule : Bool
    is_wildcard_rule : Bool
    prefix : String
    extension : String?
    explanation : String
    notes : Array[String]
    } derive(Eq,
    Debug
    )

    RuleProfile::action_label

    fn RuleProfile::action_label(self : RuleProfile) -> String

    RuleProfile::is_broad

    fn RuleProfile::is_broad(self : RuleProfile) -> Bool

    RuleProfile::is_narrow

    fn RuleProfile::is_narrow(self : RuleProfile) -> Bool

    RuleProfile::is_precise_allow

    fn RuleProfile::is_precise_allow(self : RuleProfile) -> Bool

    RuleProfile::is_risky_block

    fn RuleProfile::is_risky_block(self : RuleProfile) -> Bool

    RuleProfile::matches_path

    fn RuleProfile::matches_path(self : RuleProfile, path : StringView) -> Bool

    RuleProfile::notes_text

    fn RuleProfile::notes_text(self : RuleProfile) -> String

    RuleProfile::to_json

    fn RuleProfile::to_json(self : RuleProfile) -> String

    RuleProfile::to_line

    fn RuleProfile::to_line(self : RuleProfile) -> String

    RuleProfile::to_markdown_row

    fn RuleProfile::to_markdown_row(self : RuleProfile) -> String

    RuleRelation

    pub(all) enum RuleRelation {
    RelationDuplicate
    RelationBroader
    RelationNarrower
    RelationConflict
    RelationIndependent
    } derive(Eq,
    Debug
    )

    RuleRelation::label

    fn RuleRelation::label(self : RuleRelation) -> String

    RuleRiskBand

    pub(all) enum RuleRiskBand {
    RuleRiskLow
    RuleRiskMedium
    RuleRiskHigh
    RuleRiskCritical
    } derive(Eq,
    Debug
    )

    RuleRiskBand::label

    fn RuleRiskBand::label(self : RuleRiskBand) -> String

    SiteAuditReport

    pub(all) struct SiteAuditReport {
    policy : RobotsPolicy
    targets : Array[AuditTarget]
    results : Array[TargetAuditResult]
    summary : SiteAuditSummary
    rule_profile : PolicyProfile
    } derive(Eq,
    Debug
    )

    SiteAuditReport::allowed_results

    fn SiteAuditReport::allowed_results(self : SiteAuditReport) -> Array[TargetAuditResult]

    SiteAuditReport::blocked_results

    fn SiteAuditReport::blocked_results(self : SiteAuditReport) -> Array[TargetAuditResult]

    SiteAuditReport::critical_results

    fn SiteAuditReport::critical_results(self : SiteAuditReport) -> Array[TargetAuditResult]

    SiteAuditReport::failed_results

    SiteAuditReport::is_clean

    fn SiteAuditReport::is_clean(self : SiteAuditReport) -> Bool

    SiteAuditReport::to_csv

    fn SiteAuditReport::to_csv(self : SiteAuditReport) -> String

    SiteAuditReport::to_json

    fn SiteAuditReport::to_json(self : SiteAuditReport) -> String

    SiteAuditReport::to_markdown

    fn SiteAuditReport::to_markdown(self : SiteAuditReport) -> String

    SiteAuditSummary

    pub(all) struct SiteAuditSummary {
    total_targets : Int
    allowed_targets : Int
    blocked_targets : Int
    expected_targets : Int
    matched_expectations : Int
    mismatched_expectations : Int
    critical_targets : Int
    critical_mismatches : Int
    high_risk_targets : Int
    normalized_targets : Int
    query_targets : Int
    fragment_targets : Int
    unique_agents : Int
    coverage_band : CoverageBand
    score : Int
    notes : Array[String]
    } derive(Eq,
    Debug
    )

    SiteAuditSummary::is_clean

    fn SiteAuditSummary::is_clean(self : SiteAuditSummary) -> Bool

    SiteAuditSummary::pass_rate_percent

    fn SiteAuditSummary::pass_rate_percent(self : SiteAuditSummary) -> Int

    SiteAuditSummary::to_json

    fn SiteAuditSummary::to_json(self : SiteAuditSummary) -> String

    SiteAuditSummary::to_line

    fn SiteAuditSummary::to_line(self : SiteAuditSummary) -> String

    SiteAuditSummary::to_markdown

    fn SiteAuditSummary::to_markdown(self : SiteAuditSummary) -> String

    SitemapEntry

    pub(all) struct SitemapEntry {
    raw : String
    scheme : SitemapScheme
    host : String
    path : String
    normalized_path : String
    format : SitemapFormat
    secure : Bool
    absolute : Bool
    query : String?
    fragment : String?
    depth : Int
    issue_count : Int
    issues : Array[SitemapIssue]
    } derive(Eq,
    Debug
    )

    SitemapEntry::has_issue

    fn SitemapEntry::has_issue(self : SitemapEntry, kind : SitemapIssueKind) -> Bool

    SitemapEntry::is_cross_host

    fn SitemapEntry::is_cross_host(self : SitemapEntry, host : StringView) -> Bool

    SitemapEntry::is_xml_like

    fn SitemapEntry::is_xml_like(self : SitemapEntry) -> Bool

    SitemapEntry::issues_text

    fn SitemapEntry::issues_text(self : SitemapEntry) -> String

    SitemapEntry::to_json

    fn SitemapEntry::to_json(self : SitemapEntry) -> String

    SitemapEntry::to_line

    fn SitemapEntry::to_line(self : SitemapEntry) -> String

    SitemapEntry::to_markdown_row

    fn SitemapEntry::to_markdown_row(self : SitemapEntry) -> String

    SitemapFormat

    pub(all) enum SitemapFormat {
    SitemapXml
    SitemapXmlGzip
    SitemapIndex
    SitemapText
    SitemapUnknownFormat
    } derive(Eq,
    Debug
    )

    SitemapFormat::label

    fn SitemapFormat::label(self : SitemapFormat) -> String

    SitemapIssue

    pub(all) struct SitemapIssue {
    kind : SitemapIssueKind
    sitemap : String
    message : String
    } derive(Eq,
    Debug
    )

    SitemapIssue::to_json

    fn SitemapIssue::to_json(self : SitemapIssue) -> String

    SitemapIssue::to_line

    fn SitemapIssue::to_line(self : SitemapIssue) -> String

    SitemapIssueKind

    pub(all) enum SitemapIssueKind {
    SitemapMissingHost
    SitemapInsecureHttp
    SitemapUnsupportedScheme
    SitemapMissingPath
    SitemapUnexpectedExtension
    SitemapDuplicate
    SitemapContainsQuery
    SitemapContainsFragment
    } derive(Eq,
    Debug
    )

    SitemapIssueKind::label

    fn SitemapIssueKind::label(self : SitemapIssueKind) -> String

    SitemapReferenceParts

    type SitemapReferenceParts derive(Eq,
    Debug
    )

    SitemapScheme

    pub(all) enum SitemapScheme {
    SitemapHttp
    SitemapHttps
    SitemapRelative
    SitemapUnknown
    } derive(Eq,
    Debug
    )

    SitemapScheme::label

    fn SitemapScheme::label(self : SitemapScheme) -> String

    SitemapSummary

    pub(all) struct SitemapSummary {
    entries : Array[SitemapEntry]
    total : Int
    https_count : Int
    http_count : Int
    relative_count : Int
    xml_count : Int
    gzip_count : Int
    index_count : Int
    issue_count : Int
    duplicate_count : Int
    unique_hosts : Array[String]
    notes : Array[String]
    } derive(Eq,
    Debug
    )

    SitemapSummary::has_cross_host_entries

    fn SitemapSummary::has_cross_host_entries(self : SitemapSummary, expected_host : StringView) -> Bool

    SitemapSummary::has_only_secure_absolute_urls

    fn SitemapSummary::has_only_secure_absolute_urls(self : SitemapSummary) -> Bool

    SitemapSummary::issue_lines

    fn SitemapSummary::issue_lines(self : SitemapSummary) -> Array[String]

    SitemapSummary::to_json

    fn SitemapSummary::to_json(self : SitemapSummary) -> String

    SitemapSummary::to_line

    fn SitemapSummary::to_line(self : SitemapSummary) -> String

    SitemapSummary::to_markdown

    fn SitemapSummary::to_markdown(self : SitemapSummary) -> String

    TargetAuditResult

    pub(all) struct TargetAuditResult {
    target : AuditTarget
    decision : AccessDecision
    snapshot : PathSnapshot
    expectation_status : ExpectationStatus
    matched_expected : Bool
    importance_score : Int
    risk_points : Int
    tags : Array[String]
    } derive(Eq,
    Debug
    )

    TargetAuditResult::expectation_label

    fn TargetAuditResult::expectation_label(self : TargetAuditResult) -> String

    TargetAuditResult::status_label

    fn TargetAuditResult::status_label(self : TargetAuditResult) -> String

    TargetAuditResult::to_csv_row

    fn TargetAuditResult::to_csv_row(self : TargetAuditResult) -> String

    TargetAuditResult::to_json

    fn TargetAuditResult::to_json(self : TargetAuditResult) -> String

    TargetAuditResult::to_line

    fn TargetAuditResult::to_line(self : TargetAuditResult) -> String

    TargetAuditResult::to_markdown_row

    fn TargetAuditResult::to_markdown_row(self : TargetAuditResult) -> String

    TargetImportance

    pub(all) enum TargetImportance {
    ImportanceLow
    ImportanceNormal
    ImportanceHigh
    ImportanceCritical
    } derive(Eq,
    Debug
    )

    TargetImportance::label

    fn TargetImportance::label(self : TargetImportance) -> String

    TargetImportance::score

    fn TargetImportance::score(self : TargetImportance) -> Int

    analyze_rule

    fn analyze_rule(rule : Rule) -> RuleProfile

    analyze_rule_pairs

    fn analyze_rule_pairs(rules : Array[Rule]) -> Array[RulePairAnalysis]

    analyze_rules

    fn analyze_rules(rules : Array[Rule]) -> Array[RuleProfile]

    audit

    fn audit(policy : RobotsPolicy) -> AuditReport

    audit_target

    fn audit_target(policy : RobotsPolicy, target : AuditTarget) -> TargetAuditResult

    audit_targets

    fn audit_targets(policy : RobotsPolicy, targets : Array[AuditTarget]) -> SiteAuditReport

    blocked_private_intent

    fn blocked_private_intent(label : StringView, user_agent : StringView, url_or_path : StringView) -> CrawlIntent

    build_agent_matrix

    fn build_agent_matrix(policy : RobotsPolicy, user_agents : Array[String], paths : Array[String]) -> AgentMatrix

    build_evidence_capsule

    fn build_evidence_capsule(report : IntentGateReport, project_name : StringView, version : StringView) -> EvidenceCapsule

    compact_export_options

    fn compact_export_options() -> ExportOptions

    compare_rule_profiles

    fn compare_rule_profiles(first : RuleProfile, second : RuleProfile) -> RulePairAnalysis

    compare_rules

    fn compare_rules(first : Rule, second : Rule) -> RulePairAnalysis

    crawl_intent

    fn crawl_intent(label : StringView, user_agent : StringView, url_or_path : StringView, purpose : IntentPurpose, sensitivity : IntentSensitivity, consent : ConsentMode) -> CrawlIntent

    critical_private_target

    fn critical_private_target(label : StringView, user_agent : StringView, url_or_path : StringView) -> AuditTarget

    decision_trace_text

    fn decision_trace_text(policy : RobotsPolicy, user_agent : StringView, url_or_path : StringView) -> String

    default_export_options

    fn default_export_options() -> ExportOptions

    expected_target

    fn expected_target(label : StringView, user_agent : StringView, url_or_path : StringView, expected_allowed : Bool, importance : TargetImportance, note : StringView) -> AuditTarget

    extract_path

    fn extract_path(input : StringView) -> String

    find_broad_rules

    fn find_broad_rules(rules : Array[Rule]) -> Array[RuleProfile]

    find_conflicting_rules

    fn find_conflicting_rules(rules : Array[Rule]) -> Array[RulePairAnalysis]

    find_duplicate_rules

    fn find_duplicate_rules(rules : Array[Rule]) -> Array[RulePairAnalysis]

    find_precise_allows

    fn find_precise_allows(rules : Array[Rule]) -> Array[RuleProfile]

    find_risky_blocks

    fn find_risky_blocks(rules : Array[Rule]) -> Array[RuleProfile]

    group_to_json

    fn group_to_json(group : AgentGroup, options : ExportOptions) -> String

    normalize_target

    fn normalize_target(input : StringView) -> PathSnapshot

    parse

    fn parse(raw : StringView) -> RobotsPolicy

    parse_sitemap_url

    fn parse_sitemap_url(raw : StringView) -> SitemapEntry

    parse_sitemaps

    fn parse_sitemaps(raw_sitemaps : Array[String]) -> Array[SitemapEntry]

    path_contains_segment

    fn path_contains_segment(path : StringView, segment : StringView) -> Bool

    path_depth

    fn path_depth(path : StringView) -> Int

    path_extension

    fn path_extension(path : StringView) -> String?

    path_has_trailing_slash

    fn path_has_trailing_slash(path : StringView) -> Bool

    path_join

    fn path_join(segments : Array[String]) -> String

    path_leaf

    fn path_leaf(path : StringView) -> String

    path_parent

    fn path_parent(path : StringView) -> String

    path_segments

    fn path_segments(path : StringView) -> Array[String]

    path_starts_with_segment

    fn path_starts_with_segment(path : StringView, segment : StringView) -> Bool

    path_without_trailing_slash

    fn path_without_trailing_slash(path : StringView) -> String

    policy_summary_text

    fn policy_summary_text(policy : RobotsPolicy) -> String

    policy_to_json

    fn policy_to_json(policy : RobotsPolicy, options : ExportOptions) -> String

    private_target

    fn private_target(label : StringView, url_or_path : StringView) -> AuditTarget

    profile_policy

    fn profile_policy(policy : RobotsPolicy) -> PolicyProfile

    public_target

    fn public_target(label : StringView, url_or_path : StringView) -> AuditTarget

    review_intent

    fn review_intent(policy : RobotsPolicy, intent : CrawlIntent) -> IntentReview

    review_intents

    fn review_intents(policy : RobotsPolicy, intents : Array[CrawlIntent]) -> IntentGateReport

    rules_to_markdown

    fn rules_to_markdown(rules : Array[Rule]) -> String

    safe_public_ai_intent

    fn safe_public_ai_intent(label : StringView, user_agent : StringView, url_or_path : StringView, evidence_id : StringView) -> CrawlIntent

    sample_audit_markdown

    fn sample_audit_markdown() -> String

    sample_evidence_capsule_markdown

    fn sample_evidence_capsule_markdown() -> String

    sample_intent_gate_markdown

    fn sample_intent_gate_markdown() -> String

    sample_intents

    fn sample_intents() -> Array[CrawlIntent]

    sample_robots

    fn sample_robots() -> String

    sample_site_audit_markdown

    fn sample_site_audit_markdown() -> String

    sample_targets

    fn sample_targets() -> Array[AuditTarget]

    sitemap_host_report

    fn sitemap_host_report(entries : Array[SitemapEntry]) -> String

    sitemap_issue_report

    fn sitemap_issue_report(entries : Array[SitemapEntry]) -> String

    summarize_group

    fn summarize_group(group : AgentGroup, has_sitemap_hint : Bool) -> GroupRuleSummary

    summarize_intent_reviews

    fn summarize_intent_reviews(reviews : Array[IntentReview]) -> IntentGateSummary

    summarize_site_results

    fn summarize_site_results(results : Array[TargetAuditResult]) -> SiteAuditSummary

    summarize_sitemaps

    fn summarize_sitemaps(raw_sitemaps : Array[String]) -> SitemapSummary

    target

    fn target(label : StringView, user_agent : StringView, url_or_path : StringView) -> AuditTarget

    training_intent

    fn training_intent(label : StringView, user_agent : StringView, url_or_path : StringView, consent : ConsentMode, evidence_id : StringView) -> CrawlIntent

    verbose_export_options

    fn verbose_export_options() -> ExportOptions