🥮 mooncakes.io

robopolicy

Pure MoonBit robots.txt parser, crawler access decision, and policy audit library

robots
crawler
policy
sitemap
wasm
audit
moon add JQ-ai-nb/robopolicy@0.1.0
Download zip
Author
Version
0.1.0
License
MIT
Repository
Last updated
5 hours ago
Downloads
3
README

#RoboPolicy

RoboPolicy 是一个纯 MoonBit 实现的 robots.txt 解析、爬虫访问决策与站点抓取策略审计库。它面向 MoonBit/Wasm 工具、爬虫配置校验、站点发布检查和教学场景，帮助开发者在不依赖外部语言库的情况下判断某个 user-agent 是否可以抓取指定 URL。

#解决的问题

robots.txt 看起来简单，但实际包含分组、最长规则匹配、Allow/Disallow 冲突、通配符、结尾锚点、sitemap、crawl-delay 等细节。很多抓取工具只把 robots.txt 当普通文本处理，容易出现误抓、漏抓或无法解释的策略问题。RoboPolicy 将这些规则转成 MoonBit 数据结构，提供可测试、可审计、可导出的访问决策。

#适用场景

  • MoonBit 或 Wasm 爬虫工具的 robots.txt 访问判断。
  • CI 中检查站点发布前的 robots.txt 是否格式正确。
  • 搜索、归档、监控、教学项目中的抓取策略解释器。
  • 需要离线、确定性、无网络依赖的 robots.txt 解析库。

#安装方式

发布到 Mooncakes 后：

moon add JQ-ai-nb/robopolicy

#最小使用示例

fn main {
  let text = "User-agent: *\nDisallow: /admin/\nAllow: /admin/help"
  let policy = @robopolicy.parse(text)
  println(policy.is_allowed("ExampleBot", "/admin/panel"))
  println(policy.is_allowed("ExampleBot", "/admin/help"))
}

输出：

false
true

#本地运行

moon fmt --check
moon info
moon check --target all --deny-warn
moon build
moon test --target wasm --deny-warn
moon test --target wasm-gc --deny-warn
moon run cmd/main

发布前检查：

moon publish --dry-run

#核心功能

  • parse(text)：解析 robots.txt 文本。
  • RobotsPolicy::decide(user_agent, url_or_path)：返回可解释访问决策。
  • RobotsPolicy::is_allowed(user_agent, url_or_path)：快速判断是否允许抓取。
  • RobotsPolicy::crawl_delay_for(user_agent)：读取选中分组的 crawl-delay。
  • audit(policy)：生成格式、策略和可维护性审计报告。
  • AuditReport::to_markdown()：导出 Markdown 报告。
  • normalize_target(url_or_path)：规范化绝对 URL、裸路径、查询串、片段、重复斜杠和 ./.. 路径段。
  • analyze_rule(rule) / RobotsPolicy::profile()：生成规则画像，识别根路径阻断、目录规则、文件规则、通配符、锚点、风险等级和组级摘要。
  • parse_sitemap_url(url) / RobotsPolicy::sitemap_summary()：解析和校验 sitemap 指令，识别 HTTPS、HTTP、相对路径、重复 sitemap、格式和扩展名问题。
  • audit_targets(policy, targets)：对关键 URL 清单做批量 allow/block 验证，输出期望匹配率、关键路径风险和 Markdown/CSV/JSON 报告。
  • build_agent_matrix(policy, agents, paths)：生成多 user-agent、多路径的访问矩阵，方便展示和回归测试。
  • to_json()、policy_summary_text()、decision_trace_text()：用于 CI、示例程序和自动化报告的结构化导出。
  • 支持 sitemap、host、未知指令记录、错误行记录。

#支持范围

  • User-agent、Allow、Disallow、Sitemap、Crawl-delay、Host。
  • 多 user-agent 分组。
  • user-agent 最长 token 匹配。
  • 规则最长匹配优先。
  • Allow 与 Disallow 等长冲突时优先允许。
  • * 通配符和 $ 结尾锚点。
  • 绝对 URL、相对路径、裸路径的路径提取。

#暂不支持范围

  • 在线抓取 robots.txt。
  • URL 百分号编码规范化。
  • IDN 域名规范化。
  • 根据 sitemap 发起网络请求。
  • 分布式爬虫调度、限速执行器或持久化队列。

这些属于后续扩展方向，当前版本聚焦可复现的解析、访问判断和审计。

#测试与验收命令

moon fmt --check
moon info
moon check --target all --deny-warn
moon build
moon test --target wasm --deny-warn
moon test --target wasm-gc --deny-warn
moon run cmd/main
moon publish --dry-run

当前测试覆盖正常输入、错误输入、边界匹配、数据结构转换、核心匹配算法、路径规范化、规则画像、Sitemap 解析与校验、批量目标审计、访问矩阵、Markdown/CSV/JSON 导出、示例 smoke test 和主要错误路径。当前本地测试数为 66 个，已在 wasm 和 wasm-gc 目标通过。

#参赛验收资料

  • 申报书：SUBMISSION.md
  • 设计说明：docs/design.md
  • 测试记录：docs/test-record.md
  • 维护说明：docs/maintenance.md
  • 验收清单：docs/acceptance-checklist.md
  • 发布说明：docs/release.md
  • 更新日志：CHANGELOG.md
  • 贡献说明：CONTRIBUTING.md

#Mooncakes 包名

  • 包名：JQ-ai-nb/robopolicy
  • 文档地址：https://mooncakes.io/docs/JQ-ai-nb/robopolicy
  • Manifest：https://mooncakes.io/api/v0/manifest/JQ-ai-nb/robopolicy

#开源许可证和第三方说明

本项目使用 MIT 许可证。核心代码为原创 MoonBit 实现，不移植第三方源码，不包含来源不明素材或私有代码。robots.txt 语义参考通用 Robots Exclusion Protocol 约定，当前版本只实现离线解析和策略判断，不包含第三方抓取器代码。

#
AccessDecision

source
pub(all) struct AccessDecision {
  allowed : Bool
  user_agent : String
  path : String
  matched_group : String
  matched_rule : Rule?
  reason : String
} derive(Eq, 
Debug)

#
AccessDecision::to_json

source
fn AccessDecision::to_json(self : AccessDecision) -> String

#
AccessDecision::to_line

source
fn AccessDecision::to_line(self : AccessDecision) -> String

#
AgentGroup

source
pub(all) struct AgentGroup {
  agents : Array[String]
  rules : Array[Rule]
  crawl_delay : Int?
  line : Int
} derive(Eq, 
Debug)

#
AgentGroup::profiles

source
fn AgentGroup::profiles(self : AgentGroup) -> Array[RuleProfile]

#
AgentGroup::rule_pairs

source
fn AgentGroup::rule_pairs(self : AgentGroup) -> Array[RulePairAnalysis]

#
AgentGroup::summary

source
fn AgentGroup::summary(self : AgentGroup, has_sitemap_hint : Bool) -> GroupRuleSummary

#
AgentGroup::to_json

source
fn AgentGroup::to_json(self : AgentGroup) -> String

#
AgentMatrix

source
pub(all) struct AgentMatrix {
  rows : Array[AgentMatrixRow]
  agent_count : Int
  path_count : Int
  allowed_count : Int
  blocked_count : Int
} derive(Eq, 
Debug)

#
AgentMatrix::to_json

source
fn AgentMatrix::to_json(self : AgentMatrix) -> String

#
AgentMatrix::to_markdown

source
fn AgentMatrix::to_markdown(self : AgentMatrix) -> String

#
AgentMatrixRow

source
pub(all) struct AgentMatrixRow {
  user_agent : String
  path : String
  allowed : Bool
  reason : String
  matched_group : String
  matched_pattern : String
} derive(Eq, 
Debug)

#
AgentMatrixRow::to_json

source
fn AgentMatrixRow::to_json(self : AgentMatrixRow) -> String

#
AgentMatrixRow::to_markdown_row

source
fn AgentMatrixRow::to_markdown_row(self : AgentMatrixRow) -> String

#
AuditReport

source
pub(all) struct AuditReport {
  policy : RobotsPolicy
  findings : Array[Finding]
} derive(Eq, 
Debug)

#
AuditReport::has_finding

source
fn AuditReport::has_finding(self : AuditReport, code : StringView) -> Bool

#
AuditReport::to_json

source
fn AuditReport::to_json(self : AuditReport) -> String

#
AuditReport::to_markdown

source
fn AuditReport::to_markdown(self : AuditReport) -> String

#
AuditTarget

source
pub(all) struct AuditTarget {
  label : String
  user_agent : String
  url_or_path : String
  expected_allowed : Bool?
  importance : TargetImportance
  note : String
} derive(Eq, 
Debug)

#
AuditTarget::normalized_path

source
fn AuditTarget::normalized_path(self : AuditTarget) -> String

#
AuditTarget::to_json

source
fn AuditTarget::to_json(self : AuditTarget) -> String

#
AuditTarget::to_line

source
fn AuditTarget::to_line(self : AuditTarget) -> String

#
AuditTarget::with_expected

source
fn AuditTarget::with_expected(self : AuditTarget, expected_allowed : Bool) -> AuditTarget

#
AuditTarget::with_importance

source
fn AuditTarget::with_importance(self : AuditTarget, importance : TargetImportance) -> AuditTarget

#
AuditTarget::with_note

source
fn AuditTarget::with_note(self : AuditTarget, note : StringView) -> AuditTarget

#
CoverageBand

source
pub(all) enum CoverageBand {
  CoverageEmpty
  CoverageWeak
  CoverageFair
  CoverageStrong
} derive(Eq, 
Debug)

#
CoverageBand::label

source
fn CoverageBand::label(self : CoverageBand) -> String

#
DirectiveKind

source
pub(all) enum DirectiveKind {
  UserAgent
  Allow
  Disallow
  Sitemap
  CrawlDelay
  Host
  Unknown
} derive(Eq, 
Debug)

#
DirectiveKind::label

source
fn DirectiveKind::label(self : DirectiveKind) -> String

#
ExpectationStatus

source
pub(all) enum ExpectationStatus {
  ExpectationNotSet
  ExpectationMatched
  ExpectationMismatch
} derive(Eq, 
Debug)

#
ExpectationStatus::label

source
fn ExpectationStatus::label(self : ExpectationStatus) -> String

#
ExportDetailLevel

source
pub(all) enum ExportDetailLevel {
  ExportCompact
  ExportNormal
  ExportVerbose
} derive(Eq, 
Debug)

#
ExportDetailLevel::label

source
fn ExportDetailLevel::label(self : ExportDetailLevel) -> String

#
ExportOptions

source
pub(all) struct ExportOptions {
  detail : ExportDetailLevel
  include_parse_issues : Bool
  include_rule_profiles : Bool
  include_sitemaps : Bool
  include_targets : Bool
  include_notes : Bool
} derive(Eq, 
Debug)

#
Finding

source
pub(all) struct Finding {
  code : String
  severity : FindingSeverity
  message : String
  detail : String
} derive(Eq, 
Debug)

#
Finding::to_json

source
fn Finding::to_json(self : Finding) -> String

#
FindingSeverity

source
pub(all) enum FindingSeverity {
  Error
  Warning
  Info
} derive(Eq, 
Debug)

#
FindingSeverity::label

source
fn FindingSeverity::label(self : FindingSeverity) -> String

#
GroupRuleSummary

source
pub(all) struct GroupRuleSummary {
  agents : Array[String]
  group_line : Int
  rule_count : Int
  allow_count : Int
  disallow_count : Int
  wildcard_rule_count : Int
  anchored_rule_count : Int
  root_block_count : Int
  empty_rule_count : Int
  max_depth : Int
  max_literal_weight : Int
  crawl_delay : Int?
  has_sitemap_hint : Bool
  notes : Array[String]
} derive(Eq, 
Debug)

#
GroupRuleSummary::to_json

source
fn GroupRuleSummary::to_json(self : GroupRuleSummary) -> String

#
GroupRuleSummary::to_line

source
fn GroupRuleSummary::to_line(self : GroupRuleSummary) -> String

#
GroupRuleSummary::to_markdown

source
fn GroupRuleSummary::to_markdown(self : GroupRuleSummary) -> String

#
IssueKind

source
pub(all) enum IssueKind {
  MissingColon
  EmptyUserAgent
  EmptyDirective
  InvalidCrawlDelay
  RuleBeforeUserAgent
  UnknownDirective
} derive(Eq, 
Debug)

RoboPolicy parses robots.txt documents and answers crawler access decisions. It is designed for deterministic CI checks, site audits, and Wasm tools.

#
IssueKind::label

source
fn IssueKind::label(self : IssueKind) -> String

#
NormalizedPathBody

source
pub(all) struct NormalizedPathBody {
  normalized : String
  segments : Array[String]
  collapsed_slashes : Bool
  removed_dot_segments : Bool
  root_fallback : Bool
} derive(Eq, 
Debug)

#
ParseIssue

source
pub(all) struct ParseIssue {
  kind : IssueKind
  line : Int
  directive : String
  value : String
  message : String
} derive(Eq, 
Debug)

#
PathReferenceParts

source
pub(all) struct PathReferenceParts {
  source : PathSourceKind
  path : String
  query : String?
  fragment : String?
  had_query : Bool
  had_fragment : Bool
} derive(Eq, 
Debug)

#
PathSnapshot

source
pub(all) struct PathSnapshot {
  original : String
  normalized : String
  source : PathSourceKind
  query : String?
  fragment : String?
  had_query : Bool
  had_fragment : Bool
  collapsed_slashes : Bool
  removed_dot_segments : Bool
  root_fallback : Bool
} derive(Eq, 
Debug)

#
PathSnapshot::as_path

source
fn PathSnapshot::as_path(self : PathSnapshot) -> String

#
PathSnapshot::debug_flags

source
fn PathSnapshot::debug_flags(self : PathSnapshot) -> String

#
PathSnapshot::has_fragment

source
fn PathSnapshot::has_fragment(self : PathSnapshot) -> Bool

#
PathSnapshot::has_query

source
fn PathSnapshot::has_query(self : PathSnapshot) -> Bool

#
PathSnapshot::is_root

source
fn PathSnapshot::is_root(self : PathSnapshot) -> Bool

#
PathSnapshot::to_json

source
fn PathSnapshot::to_json(self : PathSnapshot) -> String

#
PathSnapshot::to_line

source
fn PathSnapshot::to_line(self : PathSnapshot) -> String

#
PathSnapshot::to_markdown

source
fn PathSnapshot::to_markdown(self : PathSnapshot) -> String

#
PathSourceKind

source
pub(all) enum PathSourceKind {
  Empty
  BarePath
  AbsolutePath
  AbsoluteUrl
} derive(Eq, 
Debug)

#
PathSourceKind::label

source
fn PathSourceKind::label(self : PathSourceKind) -> String

#
PolicyProfile

source
pub(all) struct PolicyProfile {
  group_summaries : Array[GroupRuleSummary]
  total_rules : Int
  total_allows : Int
  total_disallows : Int
  total_wildcards : Int
  total_anchors : Int
  total_root_blocks : Int
  highest_depth : Int
  highest_weight : Int
  has_wildcard_group : Bool
  has_specific_group : Bool
  has_sitemap : Bool
  issue_count : Int
  notes : Array[String]
} derive(Eq, 
Debug)

#
PolicyProfile::to_json

source
fn PolicyProfile::to_json(self : PolicyProfile) -> String

#
PolicyProfile::to_line

source
fn PolicyProfile::to_line(self : PolicyProfile) -> String

#
PolicyProfile::to_markdown

source
fn PolicyProfile::to_markdown(self : PolicyProfile) -> String

#
QueryFragmentParts

source
type QueryFragmentParts derive(Eq, 
Debug)

#
RobotsPolicy

source
pub(all) struct RobotsPolicy {
  groups : Array[AgentGroup]
  sitemaps : Array[String]
  host : String?
  issues : Array[ParseIssue]
} derive(Eq, 
Debug)

#
RobotsPolicy::audit_targets

source
fn RobotsPolicy::audit_targets(self : RobotsPolicy, targets : Array[AuditTarget]) -> SiteAuditReport

#
RobotsPolicy::crawl_delay_for

source
fn RobotsPolicy::crawl_delay_for(self : RobotsPolicy, user_agent : StringView) -> Int?

#
RobotsPolicy::decide

source
fn RobotsPolicy::decide(self : RobotsPolicy, user_agent : StringView, url_or_path : StringView) -> AccessDecision

#
RobotsPolicy::is_allowed

source
fn RobotsPolicy::is_allowed(self : RobotsPolicy, user_agent : StringView, url_or_path : StringView) -> Bool

#
RobotsPolicy::profile

source
fn RobotsPolicy::profile(self : RobotsPolicy) -> PolicyProfile

#
RobotsPolicy::sitemap_entries

source
fn RobotsPolicy::sitemap_entries(self : RobotsPolicy) -> Array[SitemapEntry]

#
RobotsPolicy::sitemap_summary

source
fn RobotsPolicy::sitemap_summary(self : RobotsPolicy) -> SitemapSummary

#
RobotsPolicy::to_compact_json

source
fn RobotsPolicy::to_compact_json(self : RobotsPolicy) -> String

#
RobotsPolicy::to_json

source
fn RobotsPolicy::to_json(self : RobotsPolicy) -> String

#
RobotsPolicy::to_json_with_options

source
fn RobotsPolicy::to_json_with_options(self : RobotsPolicy, options : ExportOptions) -> String

#
Rule

source
pub(all) struct Rule {
  allow : Bool
  pattern : String
  line : Int
} derive(Eq, 
Debug)

#
Rule::to_json

source
fn Rule::to_json(self : Rule) -> String

#
RuleImpact

source
pub(all) enum RuleImpact {
  ImpactAllowEmpty
  ImpactAllowNarrow
  ImpactAllowWide
  ImpactBlockRoot
  ImpactBlockDirectory
  ImpactBlockFile
  ImpactBlockWildcard
  ImpactNeutral
} derive(Eq, 
Debug)

#
RuleImpact::label

source
fn RuleImpact::label(self : RuleImpact) -> String

#
RulePairAnalysis

source
pub(all) struct RulePairAnalysis {
  first : RuleProfile
  second : RuleProfile
  relation : RuleRelation
  message : String
  same_action : Bool
  same_pattern : Bool
  first_matches_second_prefix : Bool
  second_matches_first_prefix : Bool
} derive(Eq, 
Debug)

#
RulePairAnalysis::to_line

source
fn RulePairAnalysis::to_line(self : RulePairAnalysis) -> String

#
RulePatternKind

source
pub(all) enum RulePatternKind {
  PatternEmpty
  PatternRoot
  PatternExactPath
  PatternDirectoryPrefix
  PatternFilePath
  PatternExtensionGlob
  PatternWildcardPrefix
  PatternWildcardSuffix
  PatternWildcardMiddle
  PatternAnchored
  PatternCatchAll
} derive(Eq, 
Debug)

#
RulePatternKind::label

source
fn RulePatternKind::label(self : RulePatternKind) -> String

#
RuleProfile

source
pub(all) struct RuleProfile {
  rule : Rule
  kind : RulePatternKind
  impact : RuleImpact
  risk : RuleRiskBand
  normalized_pattern : String
  literal_weight : Int
  depth : Int
  wildcard_count : Int
  has_anchor : Bool
  has_query_like_text : Bool
  has_fragment_like_text : Bool
  is_root_rule : Bool
  is_empty_pattern : Bool
  is_directory_rule : Bool
  is_file_rule : Bool
  is_wildcard_rule : Bool
  prefix : String
  extension : String?
  explanation : String
  notes : Array[String]
} derive(Eq, 
Debug)

#
RuleProfile::action_label

source
fn RuleProfile::action_label(self : RuleProfile) -> String

#
RuleProfile::is_broad

source
fn RuleProfile::is_broad(self : RuleProfile) -> Bool

#
RuleProfile::is_narrow

source
fn RuleProfile::is_narrow(self : RuleProfile) -> Bool

#
RuleProfile::is_precise_allow

source
fn RuleProfile::is_precise_allow(self : RuleProfile) -> Bool

#
RuleProfile::is_risky_block

source
fn RuleProfile::is_risky_block(self : RuleProfile) -> Bool

#
RuleProfile::matches_path

source
fn RuleProfile::matches_path(self : RuleProfile, path : StringView) -> Bool

#
RuleProfile::notes_text

source
fn RuleProfile::notes_text(self : RuleProfile) -> String

#
RuleProfile::to_json

source
fn RuleProfile::to_json(self : RuleProfile) -> String

#
RuleProfile::to_line

source
fn RuleProfile::to_line(self : RuleProfile) -> String

#
RuleProfile::to_markdown_row

source
fn RuleProfile::to_markdown_row(self : RuleProfile) -> String

#
RuleRelation

source
pub(all) enum RuleRelation {
  RelationDuplicate
  RelationBroader
  RelationNarrower
  RelationConflict
  RelationIndependent
} derive(Eq, 
Debug)

#
RuleRelation::label

source
fn RuleRelation::label(self : RuleRelation) -> String

#
RuleRiskBand

source
pub(all) enum RuleRiskBand {
  RuleRiskLow
  RuleRiskMedium
  RuleRiskHigh
  RuleRiskCritical
} derive(Eq, 
Debug)

#
RuleRiskBand::label

source
fn RuleRiskBand::label(self : RuleRiskBand) -> String

#
SiteAuditReport

source
pub(all) struct SiteAuditReport {
  policy : RobotsPolicy
  targets : Array[AuditTarget]
  results : Array[TargetAuditResult]
  summary : SiteAuditSummary
  rule_profile : PolicyProfile
} derive(Eq, 
Debug)

#
SiteAuditReport::allowed_results

source
fn SiteAuditReport::allowed_results(self : SiteAuditReport) -> Array[TargetAuditResult]

#
SiteAuditReport::blocked_results

source
fn SiteAuditReport::blocked_results(self : SiteAuditReport) -> Array[TargetAuditResult]

#
SiteAuditReport::critical_results

source
fn SiteAuditReport::critical_results(self : SiteAuditReport) -> Array[TargetAuditResult]

#
SiteAuditReport::failed_results

source

#
SiteAuditReport::is_clean

source
fn SiteAuditReport::is_clean(self : SiteAuditReport) -> Bool

#
SiteAuditReport::to_csv

source
fn SiteAuditReport::to_csv(self : SiteAuditReport) -> String

#
SiteAuditReport::to_json

source
fn SiteAuditReport::to_json(self : SiteAuditReport) -> String

#
SiteAuditReport::to_markdown

source
fn SiteAuditReport::to_markdown(self : SiteAuditReport) -> String

#
SiteAuditSummary

source
pub(all) struct SiteAuditSummary {
  total_targets : Int
  allowed_targets : Int
  blocked_targets : Int
  expected_targets : Int
  matched_expectations : Int
  mismatched_expectations : Int
  critical_targets : Int
  critical_mismatches : Int
  high_risk_targets : Int
  normalized_targets : Int
  query_targets : Int
  fragment_targets : Int
  unique_agents : Int
  coverage_band : CoverageBand
  score : Int
  notes : Array[String]
} derive(Eq, 
Debug)

#
SiteAuditSummary::is_clean

source
fn SiteAuditSummary::is_clean(self : SiteAuditSummary) -> Bool

#
SiteAuditSummary::pass_rate_percent

source
fn SiteAuditSummary::pass_rate_percent(self : SiteAuditSummary) -> Int

#
SiteAuditSummary::to_json

source
fn SiteAuditSummary::to_json(self : SiteAuditSummary) -> String

#
SiteAuditSummary::to_line

source
fn SiteAuditSummary::to_line(self : SiteAuditSummary) -> String

#
SiteAuditSummary::to_markdown

source
fn SiteAuditSummary::to_markdown(self : SiteAuditSummary) -> String

#
SitemapEntry

source
pub(all) struct SitemapEntry {
  raw : String
  scheme : SitemapScheme
  host : String
  path : String
  normalized_path : String
  format : SitemapFormat
  secure : Bool
  absolute : Bool
  query : String?
  fragment : String?
  depth : Int
  issue_count : Int
  issues : Array[SitemapIssue]
} derive(Eq, 
Debug)

#
SitemapEntry::has_issue

source
fn SitemapEntry::has_issue(self : SitemapEntry, kind : SitemapIssueKind) -> Bool

#
SitemapEntry::is_cross_host

source
fn SitemapEntry::is_cross_host(self : SitemapEntry, host : StringView) -> Bool

#
SitemapEntry::is_xml_like

source
fn SitemapEntry::is_xml_like(self : SitemapEntry) -> Bool

#
SitemapEntry::issues_text

source
fn SitemapEntry::issues_text(self : SitemapEntry) -> String

#
SitemapEntry::to_json

source
fn SitemapEntry::to_json(self : SitemapEntry) -> String

#
SitemapEntry::to_line

source
fn SitemapEntry::to_line(self : SitemapEntry) -> String

#
SitemapEntry::to_markdown_row

source
fn SitemapEntry::to_markdown_row(self : SitemapEntry) -> String

#
SitemapFormat

source
pub(all) enum SitemapFormat {
  SitemapXml
  SitemapXmlGzip
  SitemapIndex
  SitemapText
  SitemapUnknownFormat
} derive(Eq, 
Debug)

#
SitemapFormat::label

source
fn SitemapFormat::label(self : SitemapFormat) -> String

#
SitemapIssue

source
pub(all) struct SitemapIssue {
  kind : SitemapIssueKind
  sitemap : String
  message : String
} derive(Eq, 
Debug)

#
SitemapIssue::to_json

source
fn SitemapIssue::to_json(self : SitemapIssue) -> String

#
SitemapIssue::to_line

source
fn SitemapIssue::to_line(self : SitemapIssue) -> String

#
SitemapIssueKind

source
pub(all) enum SitemapIssueKind {
  SitemapMissingHost
  SitemapInsecureHttp
  SitemapUnsupportedScheme
  SitemapMissingPath
  SitemapUnexpectedExtension
  SitemapDuplicate
  SitemapContainsQuery
  SitemapContainsFragment
} derive(Eq, 
Debug)

#
SitemapIssueKind::label

source
fn SitemapIssueKind::label(self : SitemapIssueKind) -> String

#
SitemapReferenceParts

source
type SitemapReferenceParts derive(Eq, 
Debug)

#
SitemapScheme

source
pub(all) enum SitemapScheme {
  SitemapHttp
  SitemapHttps
  SitemapRelative
  SitemapUnknown
} derive(Eq, 
Debug)

#
SitemapScheme::label

source
fn SitemapScheme::label(self : SitemapScheme) -> String

#
SitemapSummary

source
pub(all) struct SitemapSummary {
  entries : Array[SitemapEntry]
  total : Int
  https_count : Int
  http_count : Int
  relative_count : Int
  xml_count : Int
  gzip_count : Int
  index_count : Int
  issue_count : Int
  duplicate_count : Int
  unique_hosts : Array[String]
  notes : Array[String]
} derive(Eq, 
Debug)

#
SitemapSummary::has_cross_host_entries

source
fn SitemapSummary::has_cross_host_entries(self : SitemapSummary, expected_host : StringView) -> Bool

#
SitemapSummary::has_only_secure_absolute_urls

source
fn SitemapSummary::has_only_secure_absolute_urls(self : SitemapSummary) -> Bool

#
SitemapSummary::issue_lines

source
fn SitemapSummary::issue_lines(self : SitemapSummary) -> Array[String]

#
SitemapSummary::to_json

source
fn SitemapSummary::to_json(self : SitemapSummary) -> String

#
SitemapSummary::to_line

source
fn SitemapSummary::to_line(self : SitemapSummary) -> String

#
SitemapSummary::to_markdown

source
fn SitemapSummary::to_markdown(self : SitemapSummary) -> String

#
TargetAuditResult

source
pub(all) struct TargetAuditResult {
  target : AuditTarget
  decision : AccessDecision
  snapshot : PathSnapshot
  expectation_status : ExpectationStatus
  matched_expected : Bool
  importance_score : Int
  risk_points : Int
  tags : Array[String]
} derive(Eq, 
Debug)

#
TargetAuditResult::expectation_label

source
fn TargetAuditResult::expectation_label(self : TargetAuditResult) -> String

#
TargetAuditResult::status_label

source
fn TargetAuditResult::status_label(self : TargetAuditResult) -> String

#
TargetAuditResult::to_csv_row

source
fn TargetAuditResult::to_csv_row(self : TargetAuditResult) -> String

#
TargetAuditResult::to_json

source
fn TargetAuditResult::to_json(self : TargetAuditResult) -> String

#
TargetAuditResult::to_line

source
fn TargetAuditResult::to_line(self : TargetAuditResult) -> String

#
TargetAuditResult::to_markdown_row

source
fn TargetAuditResult::to_markdown_row(self : TargetAuditResult) -> String

#
TargetImportance

source
pub(all) enum TargetImportance {
  ImportanceLow
  ImportanceNormal
  ImportanceHigh
  ImportanceCritical
} derive(Eq, 
Debug)

#
TargetImportance::label

source
fn TargetImportance::label(self : TargetImportance) -> String

#
TargetImportance::score

source
fn TargetImportance::score(self : TargetImportance) -> Int

#
analyze_rule

source
fn analyze_rule(rule : Rule) -> RuleProfile

#
analyze_rule_pairs

source
fn analyze_rule_pairs(rules : Array[Rule]) -> Array[RulePairAnalysis]

#
analyze_rules

source
fn analyze_rules(rules : Array[Rule]) -> Array[RuleProfile]

#
audit

source
fn audit(policy : RobotsPolicy) -> AuditReport

#
audit_target

source
fn audit_target(policy : RobotsPolicy, target : AuditTarget) -> TargetAuditResult

#
audit_targets

source
fn audit_targets(policy : RobotsPolicy, targets : Array[AuditTarget]) -> SiteAuditReport

#
build_agent_matrix

source
fn build_agent_matrix(policy : RobotsPolicy, user_agents : Array[String], paths : Array[String]) -> AgentMatrix

#
compact_export_options

source
fn compact_export_options() -> ExportOptions

#
compare_rule_profiles

source
fn compare_rule_profiles(first : RuleProfile, second : RuleProfile) -> RulePairAnalysis

#
compare_rules

source
fn compare_rules(first : Rule, second : Rule) -> RulePairAnalysis

#
critical_private_target

source
fn critical_private_target(label : StringView, user_agent : StringView, url_or_path : StringView) -> AuditTarget

#
decision_trace_text

source
fn decision_trace_text(policy : RobotsPolicy, user_agent : StringView, url_or_path : StringView) -> String

#
default_export_options

source
fn default_export_options() -> ExportOptions

#
expected_target

source
fn expected_target(label : StringView, user_agent : StringView, url_or_path : StringView, expected_allowed : Bool, importance : TargetImportance, note : StringView) -> AuditTarget

#
extract_path

source
fn extract_path(input : StringView) -> String

#
find_broad_rules

source
fn find_broad_rules(rules : Array[Rule]) -> Array[RuleProfile]

#
find_conflicting_rules

source
fn find_conflicting_rules(rules : Array[Rule]) -> Array[RulePairAnalysis]

#
find_duplicate_rules

source
fn find_duplicate_rules(rules : Array[Rule]) -> Array[RulePairAnalysis]

#
find_precise_allows

source
fn find_precise_allows(rules : Array[Rule]) -> Array[RuleProfile]

#
find_risky_blocks

source
fn find_risky_blocks(rules : Array[Rule]) -> Array[RuleProfile]

#
group_to_json

source
fn group_to_json(group : AgentGroup, options : ExportOptions) -> String

#
normalize_target

source
fn normalize_target(input : StringView) -> PathSnapshot

#
parse

source
fn parse(raw : StringView) -> RobotsPolicy

#
parse_sitemap_url

source
fn parse_sitemap_url(raw : StringView) -> SitemapEntry

#
parse_sitemaps

source
fn parse_sitemaps(raw_sitemaps : Array[String]) -> Array[SitemapEntry]

#
path_contains_segment

source
fn path_contains_segment(path : StringView, segment : StringView) -> Bool

#
path_depth

source
fn path_depth(path : StringView) -> Int

#
path_extension

source
fn path_extension(path : StringView) -> String?

#
path_has_trailing_slash

source
fn path_has_trailing_slash(path : StringView) -> Bool

#
path_join

source
fn path_join(segments : Array[String]) -> String

#
path_leaf

source
fn path_leaf(path : StringView) -> String

#
path_parent

source
fn path_parent(path : StringView) -> String

#
path_segments

source
fn path_segments(path : StringView) -> Array[String]

#
path_starts_with_segment

source
fn path_starts_with_segment(path : StringView, segment : StringView) -> Bool

#
path_without_trailing_slash

source
fn path_without_trailing_slash(path : StringView) -> String

#
policy_summary_text

source
fn policy_summary_text(policy : RobotsPolicy) -> String

#
policy_to_json

source
fn policy_to_json(policy : RobotsPolicy, options : ExportOptions) -> String

#
private_target

source
fn private_target(label : StringView, url_or_path : StringView) -> AuditTarget

#
profile_policy

source
fn profile_policy(policy : RobotsPolicy) -> PolicyProfile

#
public_target

source
fn public_target(label : StringView, url_or_path : StringView) -> AuditTarget

#
rules_to_markdown

source
fn rules_to_markdown(rules : Array[Rule]) -> String

#
sample_audit_markdown

source
fn sample_audit_markdown() -> String

#
sample_robots

source
fn sample_robots() -> String

#
sample_site_audit_markdown

source
fn sample_site_audit_markdown() -> String

#
sample_targets

source
fn sample_targets() -> Array[AuditTarget]

#
sitemap_host_report

source
fn sitemap_host_report(entries : Array[SitemapEntry]) -> String

#
sitemap_issue_report

source
fn sitemap_issue_report(entries : Array[SitemapEntry]) -> String

#
summarize_group

source
fn summarize_group(group : AgentGroup, has_sitemap_hint : Bool) -> GroupRuleSummary

#
summarize_site_results

source
fn summarize_site_results(results : Array[TargetAuditResult]) -> SiteAuditSummary

#
summarize_sitemaps

source
fn summarize_sitemaps(raw_sitemaps : Array[String]) -> SitemapSummary

#
target

source
fn target(label : StringView, user_agent : StringView, url_or_path : StringView) -> AuditTarget

#
verbose_export_options

source
fn verbose_export_options() -> ExportOptions

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io