🥮 mooncakes.io

    moon-robots

    RFC 9309 robots.txt parser, matcher, serializer and audit toolkit for MoonBit.

    robots
    robots-txt
    rfc9309
    parser
    matcher
    audit
    moonbit
    Download zip
    Version
    0.1.1
    License
    Apache-2.0
    Repository
    Last updated
    2 months ago
    Downloads
    34

    No dependencies

    #moon-robots

    • Module: liuhang0001/moon-robots
    • Version: 0.1.1
    • Status: first public release
    • Repository: https://github.com/liuhang0001/moon-robots
    • Maintainer: liuhang0001

    #Overview

    moon-robots is a strict, local-only RFC 9309 robots.txt parser, matcher, serializer and audit toolkit for MoonBit.

    It turns existing robots.txt text into a semantic model, selects the applicable User-agent group for a crawler product token, evaluates URI paths with RFC-style pattern matching, emits deterministic robots.txt text, and reports lightweight advisory audit findings.

    #Why

    Robots exclusion logic is easy to under-specify. This project keeps the MVP focused on the core interpretation path:

    robots.txt -> parse -> groups/rules -> product-token selection -> path matching -> access decision

    #RFC 9309 Scope

    Implemented:

    • User-agent, Allow, Disallow
    • comments, inline comments and blank lines
    • CR, LF and CRLF inputs
    • UTF-8 text
    • multiple matching groups merged by equal specificity
    • wildcard *
    • end marker $
    • percent-encoding normalization for unreserved octets
    • longest matching rule, with Allow winning equal-length ties
    • implicit allow for /robots.txt
    • deterministic serializer
    • lightweight audit
    • defensive resource limits

    Recognized but non-normative extension:

    • Sitemap

    Unknown records are preserved as OtherRecord; they do not alter group parsing or matching.

    #Quick Start

    let text = "User-agent: *\nDisallow: /private/\nAllow: /private/public/\n"
    match parse_robots(text) {
      Ok(file) => {
        let decision = evaluate(file, "ExampleBot", "/private/public/page").unwrap()
        println(decision.summary())
      }
      Err(e) => println(e.to_display())
    }

    #Parsing

    Use parse_robots(input) for default limits or parse_robots_with_limits(input, limits) for explicit limits.

    Every public parser returns Result[RobotsFile, RobotsError]; external input must not panic.

    #Checking Access

    Use evaluate(file, product_token, uri_path) for full decision metadata, or can_fetch(file, product_token, uri_path) for a boolean convenience API.

    #Pattern Matching

    Matching is case-sensitive for paths. Product tokens use case-insensitive exact matching, with * used only as a fallback. Within path rules, * matches zero or more characters and $ anchors the pattern to the end of the input path. Raw non-ASCII text is converted to percent-encoded UTF-8 octets; percent-encoded unreserved ASCII is decoded for comparison, while reserved octets remain encoded.

    #Serialization

    serialize_robots(file) emits deterministic LF-terminated robots.txt text using canonical field names:

    • User-agent
    • Allow
    • Disallow
    • Sitemap

    Whitespace round-tripping is intentionally out of scope.

    #Audit

    audit_robots(file) reports advisory issues such as duplicate rules, duplicate agents, empty rules, broad Disallow: /, suspicious patterns and unknown records. Audit findings do not make otherwise valid robots.txt input invalid.

    #CLI

    The portable CLI is robots-tool.

    Commands:

    • parse
    • validate
    • check
    • format
    • audit
    • stats

    Target-portable input uses --text:

    robots-tool check --agent ExampleBot --path /private/page --text "User-agent: *\nDisallow: /private/\n"

    CLI output is deterministic JSON for command results and errors.

    #Examples

    Four executable examples are included:

    • examples/parse
    • examples/check_access
    • examples/multi_bot
    • examples/audit

    #Testing

    The test suite covers parser behavior, comments, newlines, group selection, wildcard fallback, merged groups, allow/disallow decisions, longest match, equal-length ties, *, $, percent encoding, UTF-8, unknown records, serializer, audit, limits, property tests and truncation safety.

    #Security

    robots.txt is not an access-control system. Disallow does not protect private resources. This library only interprets a policy document supplied by the caller. It does not fetch URLs, crawl the web, parse HTML, resolve DNS, perform TLS, or enforce authorization.

    #Limitations

    This MVP does not implement a web crawler, HTTP client, sitemap XML parser, full RFC 3986 URL library, database, cache, scheduler, search engine, indexing layer, DNS or TLS.

    #License

    Apache-2.0.

    #Release Status

    Version 0.1.1 fixes RFC 9309 product-token selection and Unicode path normalization.

    RobotsError

    source
    pub(all) suberror RobotsError {
      RobotsError(RobotsErrorStage, RobotsErrorKind, Int, Int, String)
    }

    RobotsError::byte_offset

    source
    fn RobotsError::byte_offset(self : RobotsError) -> Int

    RobotsError::context

    source
    fn RobotsError::context(self : RobotsError) -> String

    RobotsError::kind

    source

    RobotsError::line

    source
    fn RobotsError::line(self : RobotsError) -> Int

    RobotsError::stage

    source

    RobotsError::to_display

    source
    fn RobotsError::to_display(self : RobotsError) -> String

    AccessDecision

    source
    pub struct AccessDecision {
      allowed : Bool
      product_token : String
      uri_path : String
      normalized_path : String
      matched_rule : RobotsRule?
      matched_length : Int
      matched_groups : Int
      reason : String
    } derive(Eq, 
    Debug    )

    AccessDecision::allowed

    source
    fn AccessDecision::allowed(self : AccessDecision) -> Bool

    AccessDecision::matched_groups

    source
    fn AccessDecision::matched_groups(self : AccessDecision) -> Int

    AccessDecision::matched_length

    source
    fn AccessDecision::matched_length(self : AccessDecision) -> Int

    AccessDecision::matched_rule

    source
    fn AccessDecision::matched_rule(self : AccessDecision) -> RobotsRule?

    AccessDecision::normalized_path

    source
    fn AccessDecision::normalized_path(self : AccessDecision) -> String

    AccessDecision::product_token

    source
    fn AccessDecision::product_token(self : AccessDecision) -> String

    AccessDecision::reason

    source
    fn AccessDecision::reason(self : AccessDecision) -> String

    AccessDecision::summary

    source
    fn AccessDecision::summary(self : AccessDecision) -> String

    AccessDecision::uri_path

    source
    fn AccessDecision::uri_path(self : AccessDecision) -> String

    AuditIssue

    source
    pub struct AuditIssue {
      severity : AuditSeverity
      kind : AuditKind
      line : Int
      message : String
    } derive(Eq, 
    Debug    )

    AuditIssue::kind

    source
    fn AuditIssue::kind(self : AuditIssue) -> AuditKind

    AuditIssue::line

    source
    fn AuditIssue::line(self : AuditIssue) -> Int

    AuditIssue::message

    source
    fn AuditIssue::message(self : AuditIssue) -> String

    AuditIssue::severity

    source
    fn AuditIssue::severity(self : AuditIssue) -> AuditSeverity

    AuditKind

    source
    pub(all) enum AuditKind {
      EmptyUserAgent
      RuleBeforeFirstGroup
      EmptyRule
      DuplicateRule
      DuplicateUserAgent
      UnknownRecord
      VeryBroadDisallow
      UnusedGroup
      SuspiciousPattern
    } derive(Eq, 
    Debug    )

    AuditKind::label

    source
    fn AuditKind::label(self : AuditKind) -> String

    AuditSeverity

    source
    pub(all) enum AuditSeverity {
      Info
      Warning
    } derive(Eq, 
    Debug    )

    AuditSeverity::label

    source
    fn AuditSeverity::label(self : AuditSeverity) -> String

    Limits

    source
    pub struct Limits {
      max_input_bytes : Int
      max_lines : Int
      max_groups : Int
      max_user_agents_per_group : Int
      max_rules_per_group : Int
      max_pattern_bytes : Int
      max_record_bytes : Int
    } derive(Eq, 
    Debug    )

    Limits::custom

    source
    fn Limits::custom(max_input_bytes : Int, max_lines : Int, max_groups : Int, max_user_agents_per_group : Int, max_rules_per_group : Int, max_pattern_bytes : Int, max_record_bytes : Int) -> Limits

    Limits::default

    source
    fn Limits::default() -> Limits

    Limits::max_groups

    source
    fn Limits::max_groups(self : Limits) -> Int

    Limits::max_input_bytes

    source
    fn Limits::max_input_bytes(self : Limits) -> Int

    Limits::max_lines

    source
    fn Limits::max_lines(self : Limits) -> Int

    Limits::max_pattern_bytes

    source
    fn Limits::max_pattern_bytes(self : Limits) -> Int

    Limits::max_record_bytes

    source
    fn Limits::max_record_bytes(self : Limits) -> Int

    Limits::max_rules_per_group

    source
    fn Limits::max_rules_per_group(self : Limits) -> Int

    Limits::max_user_agents_per_group

    source
    fn Limits::max_user_agents_per_group(self : Limits) -> Int

    Limits::permissive

    source
    fn Limits::permissive() -> Limits

    Limits::strict

    source
    fn Limits::strict() -> Limits

    OtherRecord

    source
    pub struct OtherRecord {
      name : String
      value : String
      line : Int
      byte_offset : Int
    } derive(Eq, 
    Debug    )

    OtherRecord::byte_offset

    source
    fn OtherRecord::byte_offset(self : OtherRecord) -> Int

    OtherRecord::canonical_name

    source
    fn OtherRecord::canonical_name(self : OtherRecord) -> String

    OtherRecord::line

    source
    fn OtherRecord::line(self : OtherRecord) -> Int

    OtherRecord::name

    source
    fn OtherRecord::name(self : OtherRecord) -> String

    OtherRecord::value

    source
    fn OtherRecord::value(self : OtherRecord) -> String

    RecordKind

    source
    pub(all) enum RecordKind {
      UserAgentRecord
      AllowRecord
      DisallowRecord
      SitemapRecord
      OtherRecordKind
    } derive(Eq, 
    Debug    )

    RobotsErrorKind

    source
    pub(all) enum RobotsErrorKind {
      EmptyInput
      MissingColon
      EmptyUserAgent
      InvalidUserAgent
      InvalidRecordName
      InvalidPattern
      InvalidPercentEncoding
      InvalidUtf8
      LimitExceeded
      RuleBeforeFirstGroup
      ControlCharacter
    } derive(Eq)

    RobotsErrorKind::to_string

    source
    fn RobotsErrorKind::to_string(self : RobotsErrorKind) -> String

    RobotsErrorStage

    source
    pub(all) enum RobotsErrorStage {
      Input
      Line
      UserAgent
      Rule
      Pattern
      Utf8
      Limit
    } derive(Eq)

    RobotsErrorStage::to_string

    source
    fn RobotsErrorStage::to_string(self : RobotsErrorStage) -> String

    RobotsFile

    source
    pub struct RobotsFile {
      groups : Array[RobotsGroup]
      other_records : Array[OtherRecord]
      comments : Int
      blank_lines : Int
      source_lines : Int
    } derive(Eq, 
    Debug    )

    RobotsFile::blank_lines

    source
    fn RobotsFile::blank_lines(self : RobotsFile) -> Int

    RobotsFile::comments

    source
    fn RobotsFile::comments(self : RobotsFile) -> Int

    RobotsFile::group_count

    source
    fn RobotsFile::group_count(self : RobotsFile) -> Int

    RobotsFile::groups

    source
    fn RobotsFile::groups(self : RobotsFile) -> Array[RobotsGroup]

    RobotsFile::other_record_count

    source
    fn RobotsFile::other_record_count(self : RobotsFile) -> Int

    RobotsFile::other_records

    source
    fn RobotsFile::other_records(self : RobotsFile) -> Array[OtherRecord]

    RobotsFile::rule_count

    source
    fn RobotsFile::rule_count(self : RobotsFile) -> Int

    RobotsFile::source_lines

    source
    fn RobotsFile::source_lines(self : RobotsFile) -> Int

    RobotsFile::stats

    source
    fn RobotsFile::stats(self : RobotsFile) -> RobotsStats

    RobotsFile::user_agent_count

    source
    fn RobotsFile::user_agent_count(self : RobotsFile) -> Int

    RobotsGroup

    source
    pub struct RobotsGroup {
      user_agents : Array[String]
      rules : Array[RobotsRule]
      start_line : Int
      end_line : Int
    } derive(Eq, 
    Debug    )

    RobotsGroup::agent_count

    source
    fn RobotsGroup::agent_count(self : RobotsGroup) -> Int

    RobotsGroup::end_line

    source
    fn RobotsGroup::end_line(self : RobotsGroup) -> Int

    RobotsGroup::rule_count

    source
    fn RobotsGroup::rule_count(self : RobotsGroup) -> Int

    RobotsGroup::rules

    source
    fn RobotsGroup::rules(self : RobotsGroup) -> Array[RobotsRule]

    RobotsGroup::start_line

    source
    fn RobotsGroup::start_line(self : RobotsGroup) -> Int

    RobotsGroup::user_agents

    source
    fn RobotsGroup::user_agents(self : RobotsGroup) -> Array[String]

    RobotsRule

    source
    pub struct RobotsRule {
      kind : RuleKind
      pattern : String
      line : Int
      byte_offset : Int
    } derive(Eq, 
    Debug    )

    RobotsRule::byte_offset

    source
    fn RobotsRule::byte_offset(self : RobotsRule) -> Int

    RobotsRule::is_empty

    source
    fn RobotsRule::is_empty(self : RobotsRule) -> Bool

    RobotsRule::kind

    source
    fn RobotsRule::kind(self : RobotsRule) -> RuleKind

    RobotsRule::line

    source
    fn RobotsRule::line(self : RobotsRule) -> Int

    RobotsRule::pattern

    source
    fn RobotsRule::pattern(self : RobotsRule) -> String

    RobotsStats

    source
    pub struct RobotsStats {
      groups : Int
      user_agents : Int
      rules : Int
      allows : Int
      disallows : Int
      other_records : Int
      comments : Int
      blank_lines : Int
      source_lines : Int
    } derive(Eq, 
    Debug    )

    RobotsStats::allows

    source
    fn RobotsStats::allows(self : RobotsStats) -> Int

    RobotsStats::blank_lines

    source
    fn RobotsStats::blank_lines(self : RobotsStats) -> Int

    RobotsStats::comments

    source
    fn RobotsStats::comments(self : RobotsStats) -> Int

    RobotsStats::disallows

    source
    fn RobotsStats::disallows(self : RobotsStats) -> Int

    RobotsStats::groups

    source
    fn RobotsStats::groups(self : RobotsStats) -> Int

    RobotsStats::other_records

    source
    fn RobotsStats::other_records(self : RobotsStats) -> Int

    RobotsStats::rules

    source
    fn RobotsStats::rules(self : RobotsStats) -> Int

    RobotsStats::source_lines

    source
    fn RobotsStats::source_lines(self : RobotsStats) -> Int

    RobotsStats::user_agents

    source
    fn RobotsStats::user_agents(self : RobotsStats) -> Int

    RuleKind

    source
    pub(all) enum RuleKind {
      Allow
      Disallow
    } derive(Eq, 
    Debug    )

    RuleKind::directive_name

    source
    fn RuleKind::directive_name(self : RuleKind) -> String

    RuleKind::json_name

    source
    fn RuleKind::json_name(self : RuleKind) -> String

    RuleKind::label

    source
    fn RuleKind::label(self : RuleKind) -> String

    access_decision

    source
    fn access_decision(allowed : Bool, product_token : String, uri_path : String, normalized_path : String, matched_rule : RobotsRule?, matched_length : Int, matched_groups : Int, reason : String) -> AccessDecision

    audit_robots

    source
    fn audit_robots(file : RobotsFile) -> Array[AuditIssue]

    audit_to_text

    source
    fn audit_to_text(issues : Array[AuditIssue]) -> String

    can_fetch

    source
    fn can_fetch(file : RobotsFile, product_token : String, uri_path : String) -> Bool

    empty_robots_file

    source
    fn empty_robots_file() -> RobotsFile

    evaluate

    source
    fn evaluate(file : RobotsFile, product_token : String, uri_path : String) -> Result[AccessDecision, RobotsError]

    is_allowed

    source
    fn is_allowed(file : RobotsFile, product_token : String, uri_path : String) -> Result[AccessDecision, RobotsError]

    is_sitemap

    source
    fn is_sitemap(record : OtherRecord) -> Bool

    json_string

    source
    fn json_string(input : String) -> String

    library_version

    source
    fn library_version() -> String

    max_context_bytes

    source
    fn max_context_bytes() -> Int

    normalize_path_for_match

    source
    fn normalize_path_for_match(path : String) -> Result[String, RobotsError]

    other_record

    source
    fn other_record(name : String, value : String, line : Int, byte_offset : Int) -> OtherRecord

    parse_robots

    source
    fn parse_robots(input : String) -> Result[RobotsFile, RobotsError]

    parse_robots_with_limits

    source
    fn parse_robots_with_limits(input : String, limits : Limits) -> Result[RobotsFile, RobotsError]

    pattern_matches

    source
    fn pattern_matches(pattern : String, path : String) -> Bool

    pattern_specificity

    source
    fn pattern_specificity(pattern : String) -> Int

    robots_error

    source
    fn robots_error(stage : RobotsErrorStage, kind : RobotsErrorKind, line : Int, byte_offset : Int, context : String) -> RobotsError

    robots_file

    source
    fn robots_file(groups : Array[RobotsGroup], other_records : Array[OtherRecord], comments : Int, blank_lines : Int, source_lines : Int) -> RobotsFile

    robots_group

    source
    fn robots_group(user_agents : Array[String], rules : Array[RobotsRule], start_line : Int, end_line : Int) -> RobotsGroup

    robots_rule

    source
    fn robots_rule(kind : RuleKind, pattern : String, line : Int, byte_offset : Int) -> RobotsRule

    rules_for

    source
    fn rules_for(file : RobotsFile, product_token : String) -> Array[RobotsRule]

    serialize_robots

    source
    fn serialize_robots(file : RobotsFile) -> String

    stats_to_text

    source
    fn stats_to_text(file : RobotsFile) -> String

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io