🥮 mooncakes.io

    moonlexicon

    High-performance dictionary compilation and multi-pattern matching library for MoonBit.

    trie
    aho-corasick
    dictionary
    nlp
    moonbit
    Download zip
    Author
    Version
    0.3.0
    License
    Apache-2.0
    Repository
    Last updated
    2 months ago
    Downloads
    35

    No dependencies

    #moonlexicon

    A reusable MoonBit library for compiling dictionaries and scanning text with prefix tries and Aho–Corasick automata. It supports search, rule-based entity recognition, log auditing, tokenization, and deterministic multi-pattern matching.

    #Core capabilities

    • Trie compilation with exact lookup, prefix expansion, longest-at-position, and greedy segmentation.
    • Aho–Corasick compilation with overlapping matches, leftmost-longest selection, label counts, and candidate aggregation.
    • Bounded Levenshtein fuzzy lookup and fuzzy match filtering.
    • Dictionary-aware tokenization, chunk-oriented scanning, policies, statistics, and TSV import/export.
    • Reversible MLX1 dictionary snapshots, validation, merge/diff operations, and normalization-aware compilation.
    • Query filters, context windows, redaction/highlighting, explainable ranking, multi-dictionary catalogs, and match analysis reports.
    • Incremental ASCII byte scanning and deterministic workload evaluation APIs.
    • Runnable examples and a deterministic benchmark workload.

    #Quick start

    ///|
    test {
      let entries = [
        @moonlexicon.Entry::{ keyword: "MoonBit", label: "language", weight: 10 },
        { keyword: "lexicon", label: "library", weight: 8 },
      ]
      let lexicon = @moonlexicon.compile(entries)
      assert_true(lexicon.contains("MoonBit"))
      let automaton = @moonlexicon.compile_automaton(entries)
      let hits = automaton.find_leftmost_longest("MoonBit lexicon")
      debug_inspect(
        [
          for hit in hits => hit.keyword
        ],
        content=(
          #|["MoonBit", "lexicon"]
        ),
      )
    }

    Run the example CLI:

    moon run cmd/main

    #CLI

    The example command prints matches as TSV:

    MoonBit	language	0	7	10
lexicon	library	8	15	8

    The benchmark command uses a deterministic 1,000-entry dictionary and a 397,970-character input:

    moon run --target native cmd/bench

    #Dictionary format

    parse_tsv accepts one entry per line:

    keyword<TAB>label<TAB>weight
MoonBit<TAB>language<TAB>10
lexicon<TAB>library<TAB>8

    Blank lines and lines beginning with # are ignored. Weight is optional and defaults to 1.

    #Architecture

    Lexicon stores prefix-shared trie nodes. Automaton adds failure links and suffix output propagation for one-pass multi-pattern scanning. Filtering and policies are separate composition layers for deterministic application output. See docs/ARCHITECTURE.md for invariants and complexity notes.

    #Benchmarks

    Five consecutive native runs on the development machine measured 183.1 ms, 159.5 ms, 161.8 ms, 185.5 ms, and 203.4 ms wall-clock time, including process startup and compilation-cache lookup. The workload reported 1,000 entries, 1,017 automaton states, 397,970 input characters, 57,800 matches, and 1,000 distinct surfaces. These are reproducible workload measurements, not universal speed claims. See docs/BENCHMARKS.md for the environment and command.

    1..5 | ForEach-Object {
  (Measure-Command { moon run --target native cmd/bench | Out-Null }).TotalMilliseconds
}

    #Testing

    The suite covers trie behavior, overlapping outputs, filtering, tokenization, fuzzy matching, scanner boundaries, TSV parsing, statistics, policies, and deterministic boundary fixtures.

    moon fmt --check
moon check --target all --deny-warn
moon test --target all --deny-warn
moon run --target native cmd/bench

    The test suite includes focused black-box, white-box, documentation, and boundary cases. Generated build output is excluded from version control.

    #CI

    GitHub Actions runs on Ubuntu, macOS, and Windows. Jobs install the current stable MoonBit CLI, require moonc >= v0.10.9, update dependencies, check formatting, type check all stable targets, regenerate public interfaces, verify a clean interface diff, and run all tests. See .github/workflows/test.yml.

    #License

    Apache-2.0. See LICENSE.

    AcNode

    source

    AnalysisReport

    source
    pub(all) struct AnalysisReport {
      text_length : Int
      match_count : Int
      distinct_label_count : Int
      distinct_keyword_count : Int
      total_weight : Int
      covered_characters : Int
      unmatched_characters : Int
      coverage_percent : Int
      overlap_count : Int
      longest_span : Int
      matches : Array[LexiconMatch]
    } derive(Eq, ToJson, 
    Debug    )

    Aggregate analysis of one scanned document.

    Automaton

    source
    pub struct Automaton {
      states : Array[AcNode]
      entries : Array[Entry]
    } derive(
    Debug    )

    Compiled Aho-Corasick automaton for streaming multi-pattern search.

    Automaton::ascii_entry_count

    source
    fn Automaton::ascii_entry_count(self : Automaton) -> Int

    Return the number of ASCII-compatible entries in an automaton.

    Automaton::compile

    source
    fn Automaton::compile(entries : ArrayView[Entry]) -> Automaton

    Build an Aho-Corasick automaton from entries.

    Automaton::contains_any

    source
    fn Automaton::contains_any(self : Automaton, text : String) -> Bool

    True if at least one dictionary item occurs in the text.

    Automaton::count_labels

    source
    fn Automaton::count_labels(self : Automaton, text : String) -> Map[String, Int]

    Count emitted matches grouped by entry label.

    Automaton::find_all

    source
    fn Automaton::find_all(self : Automaton, text : String) -> Array[LexiconMatch]

    Find all matches in left-to-right order.

    Automaton::find_ascii_bytes

    source
    fn Automaton::find_ascii_bytes(self : Automaton, bytes : Bytes) -> Array[LexiconMatch]

    Scan a complete ASCII byte payload in one pass.

    Automaton::find_bytes

    source
    fn Automaton::find_bytes(self : Automaton, bytes : Bytes) -> Array[LexiconMatch]

    Alias suitable for code that supports both text and byte backends.

    Automaton::find_leftmost_longest

    source
    fn Automaton::find_leftmost_longest(self : Automaton, text : String) -> Array[LexiconMatch]

    Emit non-overlapping matches by preferring longer spans, then heavier entries.

    Automaton::fuzzy_matches

    source
    fn Automaton::fuzzy_matches(self : Automaton, text : String, query : String, max_distance? : Int) -> Array[LexiconMatch]

    Return automaton matches whose surface form is close to a query.

    Automaton::recall_candidates

    source
    fn Automaton::recall_candidates(self : Automaton, text : String) -> Array[Candidate]

    Recall entity candidates by label and surface form.

    Automaton::state_count

    source
    fn Automaton::state_count(self : Automaton) -> Int

    Number of automaton states.

    Automaton::supports_ascii_backend

    source
    fn Automaton::supports_ascii_backend(self : Automaton) -> Bool

    Return true when all dictionary keywords can be represented by the byte API.

    BenchmarkResult

    source
    pub(all) struct BenchmarkResult {
      entry_count : Int
      state_count : Int
      input_chars : Int
      repeat_count : Int
      match_count : Int
      distinct_surface_count : Int
      total_weight : Int
      coverage_percent : Int
    } derive(Eq, ToJson, 
    Debug    )

    Structural result for a deterministic dictionary workload.

    Builder

    source
    pub struct Builder {
      entries : Array[Entry]
    } derive(
    Debug    )

    Small fluent builder for application dictionaries.

    Builder::add

    source
    fn Builder::add(self : Builder, keyword : String, label? : String, weight? : Int) -> Builder

    Builder::add_all

    source
    fn Builder::add_all(self : Builder, entries : ArrayView[Entry]) -> Builder

    Builder::new

    source
    fn Builder::new() -> Builder

    Builder::to_automaton

    source
    fn Builder::to_automaton(self : Builder) -> Automaton

    Builder::to_lexicon

    source
    fn Builder::to_lexicon(self : Builder) -> Lexicon

    ByteScanStats

    source
    pub(all) struct ByteScanStats {
      byte_count : Int
      ascii : Bool
      non_ascii_count : Int
      match_count : Int
      distinct_surface_count : Int
      covered_bytes : Int
    } derive(Eq, ToJson, 
    Debug    )

    Summary of a byte scan.

    ByteScanner

    source
    pub struct ByteScanner {
      automaton : Automaton
      state : Int
      position : Int
    } derive(
    Debug    )

    An incremental scanner for ASCII dictionary workloads stored as bytes.

    ByteScanner::finish

    source
    fn ByteScanner::finish(self : ByteScanner) -> Array[LexiconMatch]

    Reset a byte scanner at an end-of-stream boundary.

    ByteScanner::new

    source
    fn ByteScanner::new(automaton : Automaton) -> ByteScanner

    ByteScanner::push

    source
    fn ByteScanner::push(self : ByteScanner, bytes : Bytes) -> Array[LexiconMatch]

    Scan one byte chunk and emit matches with byte offsets.

    Candidate

    source
    pub(all) struct Candidate {
      text : String
      label : String
      score : Int
      span_count : Int
    } derive(Eq, ToJson, 
    Debug    )

    A candidate produced for entity recall.

    CandidateExplanation

    source
    pub(all) struct CandidateExplanation {
      occurrences : Int
      score : Int
      average_weight : Int
      first_position : Int
      last_position : Int
      label : String
    } derive(Eq, ToJson, 
    Debug    )

    Explain how a candidate score was accumulated.

    CatalogHit

    source
    pub(all) struct CatalogHit {
      source : String
      hit : LexiconMatch
    } derive(Eq, ToJson, 
    Debug    )

    A match annotated with the dictionary that produced it.

    CatalogStats

    source
    pub(all) struct CatalogStats {
      dictionary_count : Int
      entry_count : Int
      node_count : Int
      match_count : Int
      distinct_sources : Int
    } derive(Eq, ToJson, 
    Debug    )

    Summary counters for a dictionary catalog.

    ContextWindow

    source
    pub(all) struct ContextWindow {
      source : String
      before : String
      matched : String
      after : String
      label : String
      start : Int
      end : Int
      weight : Int
    } derive(Eq, ToJson, 
    Debug    )

    A match together with bounded source context.

    DictionaryCatalog

    source
    pub struct DictionaryCatalog {
      dictionaries : Map[String, Lexicon]
    } derive(
    Debug    )

    A named collection of dictionaries for applications with multiple domains.

    DictionaryCatalog::contains

    source
    fn DictionaryCatalog::contains(self : DictionaryCatalog, keyword : String) -> Bool

    Return whether any named dictionary contains a keyword.

    DictionaryCatalog::dictionary_count

    source
    fn DictionaryCatalog::dictionary_count(self : DictionaryCatalog) -> Int

    DictionaryCatalog::entries

    source
    fn DictionaryCatalog::entries(self : DictionaryCatalog) -> Array[(String, Entry)]

    Return all entries in source-name order.

    DictionaryCatalog::lookup

    source
    fn DictionaryCatalog::lookup(self : DictionaryCatalog, name : String, keyword : String) -> Array[Entry]

    Look up an exact keyword in one named dictionary.

    DictionaryCatalog::merge

    source
    fn DictionaryCatalog::merge(self : DictionaryCatalog, other : DictionaryCatalog) -> Unit

    Merge another catalog into this catalog, replacing equal names.

    DictionaryCatalog::names

    source
    fn DictionaryCatalog::names(self : DictionaryCatalog) -> Array[String]

    Return registered names in lexical order.

    DictionaryCatalog::new

    source

    DictionaryCatalog::register

    source
    fn DictionaryCatalog::register(self : DictionaryCatalog, name : String, lexicon : Lexicon) -> Unit

    Register or replace a named compiled dictionary.

    DictionaryCatalog::register_entries

    source
    fn DictionaryCatalog::register_entries(self : DictionaryCatalog, name : String, entries : ArrayView[Entry]) -> Unit

    Compile and register entries under one name.

    DictionaryCatalog::remove

    source
    fn DictionaryCatalog::remove(self : DictionaryCatalog, name : String) -> Bool

    Remove a dictionary and report whether it existed.

    DictionaryCatalog::scan

    source
    fn DictionaryCatalog::scan(self : DictionaryCatalog, text : String) -> Array[CatalogHit]

    Scan all named dictionaries and retain source ownership.

    DictionaryCatalog::scan_query

    source
    fn DictionaryCatalog::scan_query(self : DictionaryCatalog, text : String, query : MatchQuery) -> Array[CatalogHit]

    Scan the catalog with one shared query policy.

    DictionaryCatalog::stats

    source
    fn DictionaryCatalog::stats(self : DictionaryCatalog, text : String) -> CatalogStats

    Return aggregate catalog counters for a text.

    DictionaryCatalog::to_tsv

    source
    fn DictionaryCatalog::to_tsv(self : DictionaryCatalog) -> String

    Export catalog entries with a source column.

    DictionaryDelta

    source
    pub(all) struct DictionaryDelta {
      added : Array[Entry]
      removed : Array[Entry]
      updated : Array[Entry]
    } derive(Eq, ToJson, 
    Debug    )

    A change set that can be applied to a base dictionary.

    DictionaryDelta::change_count

    source
    fn DictionaryDelta::change_count(self : DictionaryDelta) -> Int

    Count all operations represented by a change set.

    DictionaryDelta::empty

    source

    Return an empty change set.

    DictionaryDelta::is_empty

    source
    fn DictionaryDelta::is_empty(self : DictionaryDelta) -> Bool

    True if applying the change set would leave the dictionary unchanged.

    DictionaryQuality

    source
    pub(all) struct DictionaryQuality {
      entry_count : Int
      valid_count : Int
      invalid_count : Int
      duplicate_count : Int
      label_count : Int
      minimum_keyword_length : Int
      maximum_keyword_length : Int
      average_keyword_length : Int
    } derive(Eq, ToJson, 
    Debug    )

    Quality counters for dictionary ingestion pipelines.

    DictionaryStats

    source
    pub(all) struct DictionaryStats {
      entry_count : Int
      node_count : Int
      label_count : Int
      min_keyword_length : Int
      max_keyword_length : Int
      total_keyword_length : Int
    } derive(Eq, ToJson, 
    Debug    )

    Summary statistics for a compiled dictionary.

    DictionaryStats::sharing_percent

    source
    fn DictionaryStats::sharing_percent(self : DictionaryStats) -> Int

    Estimate prefix-sharing efficiency as a percentage.

    Entry

    source
    pub(all) struct Entry {
      keyword : String
      label : String
      weight : Int
    } derive(Eq, ToJson, 
    Debug    )

    A dictionary item stored in a compiled lexicon.

    Entry::new

    source
    fn Entry::new(keyword : String, label? : String, weight? : Int) -> Entry

    Create a dictionary entry.

    EntryNormalizationReport

    source
    pub(all) struct EntryNormalizationReport {
      original : Entry
      normalized : Entry
      changed : Bool
      issue_count : Int
    } derive(Eq, ToJson, 
    Debug    )

    Return a report for every entry, retaining source order.

    KeywordSummary

    source
    pub(all) struct KeywordSummary {
      keyword : String
      label : String
      match_count : Int
      total_weight : Int
      first_position : Int
      last_position : Int
    } derive(Eq, ToJson, 
    Debug    )

    A keyword-level aggregate for identifying noisy or high-value rules.

    LabelSummary

    source
    pub(all) struct LabelSummary {
      label : String
      match_count : Int
      total_weight : Int
      maximum_weight : Int
      covered_characters : Int
      average_span : Int
    } derive(Eq, ToJson, 
    Debug    )

    A label-level aggregate for dashboards and audit reports.

    Lexicon

    source
    pub struct Lexicon {
      nodes : Array[TrieNode]
      entries : Array[Entry]
    } derive(
    Debug    )

    Mutable builder for lexicon entries and trie nodes.

    Lexicon::add

    source
    fn Lexicon::add(self : Lexicon, entry : Entry) -> Unit

    Add one entry to the lexicon. Empty keywords are ignored.

    Lexicon::contains

    source
    fn Lexicon::contains(self : Lexicon, keyword : String) -> Bool

    Check whether the exact keyword exists.

    Lexicon::entries

    source
    fn Lexicon::entries(self : Lexicon) -> Array[Entry]

    Return a stable copy of entries stored in the lexicon.

    Lexicon::entry_count

    source
    fn Lexicon::entry_count(self : Lexicon) -> Int

    Number of dictionary entries.

    Lexicon::fuzzy_lookup

    source
    fn Lexicon::fuzzy_lookup(self : Lexicon, query : String, max_distance? : Int) -> Array[Entry]

    Return dictionary entries within an edit-distance threshold.

    Lexicon::longest_at

    source
    fn Lexicon::longest_at(self : Lexicon, text : String, offset : Int) -> LexiconMatch?

    Find the longest dictionary item that starts at offset.

    Lexicon::lookup

    source
    fn Lexicon::lookup(self : Lexicon, keyword : String) -> Array[Entry]

    Return entries whose keyword exactly equals keyword.

    Lexicon::new

    source
    fn Lexicon::new() -> Lexicon

    Create an empty lexicon builder.

    Lexicon::node_count

    source
    fn Lexicon::node_count(self : Lexicon) -> Int

    Number of trie nodes after prefix sharing.
    fn Lexicon::prefix_search(self : Lexicon, prefix : String) -> Array[Entry]

    Return every entry whose keyword starts with prefix.

    Lexicon::prefixes_at

    source
    fn Lexicon::prefixes_at(self : Lexicon, text : String, offset : Int) -> Array[LexiconMatch]

    Return all dictionary prefixes that start at offset.

    Lexicon::segment_longest

    source
    fn Lexicon::segment_longest(self : Lexicon, text : String) -> Array[LexiconMatch]

    Greedy longest-match segmentation over the text.

    Lexicon::stats

    source
    fn Lexicon::stats(self : Lexicon) -> DictionaryStats

    Lexicon::tokenize

    source
    fn Lexicon::tokenize(self : Lexicon, text : String) -> Array[Token]

    Tokenize text into dictionary hits and single-character gaps.

    LexiconMatch

    source
    pub(all) struct LexiconMatch {
      keyword : String
      label : String
      start : Int
      end : Int
      weight : Int
    } derive(Eq, ToJson, 
    Debug    )

    A match emitted by exact, prefix, longest or Aho-Corasick search.

    MatchPolicy

    source
    pub(all) struct MatchPolicy {
      min_weight : Int
      max_matches : Int
      require_label : String
      prefer_longest : Bool
    } derive(Eq, ToJson, 
    Debug    )

    Match selection policy used by applications that need deterministic output.

    MatchPolicy::accepts

    source
    fn MatchPolicy::accepts(self : MatchPolicy, hit : LexiconMatch) -> Bool

    MatchPolicy::default

    source
    fn MatchPolicy::default() -> MatchPolicy

    MatchQuery

    source
    pub(all) struct MatchQuery {
      min_weight : Int
      require_label : String
      prefer_longest : Bool
      include_overlaps : Bool
      max_matches : Int
    } derive(Eq, ToJson, 
    Debug    )

    A composable query for extracting matches from an automaton.

    MatchQuery::accepts

    source
    fn MatchQuery::accepts(self : MatchQuery, hit : LexiconMatch) -> Bool

    Decide whether a match satisfies a query's scalar filters.

    MatchQuery::default

    source
    fn MatchQuery::default() -> MatchQuery

    Return a query with permissive defaults.

    MatchQuery::for_label

    source
    fn MatchQuery::for_label(label : String) -> MatchQuery

    Return a query specialized for one label.

    MatchQuery::with_limit

    source
    fn MatchQuery::with_limit(self : MatchQuery, limit : Int) -> MatchQuery

    Return a query with an explicit output bound.

    MatchQuery::with_min_weight

    source
    fn MatchQuery::with_min_weight(self : MatchQuery, minimum : Int) -> MatchQuery

    Return a query that excludes low-confidence dictionary entries.

    MatchStats

    source
    pub(all) struct MatchStats {
      match_count : Int
      distinct_surface_count : Int
      total_weight : Int
      longest_span : Int
    } derive(Eq, ToJson, 
    Debug    )

    Return aggregate counts for matches and distinct surfaces.

    NormalizationOptions

    source
    pub(all) struct NormalizationOptions {
      lowercase : Bool
      trim : Bool
      collapse_whitespace : Bool
      map_separators : Bool
      drop_controls : Bool
      max_length : Int
    } derive(Eq, ToJson, 
    Debug    )

    Text normalization controls for dictionaries and search keys.

    NormalizationOptions::default

    source

    Defaults for case-insensitive human-language lookup.

    NormalizationReport

    source
    pub(all) struct NormalizationReport {
      text : String
      input_length : Int
      output_length : Int
      changed_count : Int
      whitespace_count : Int
      dropped_count : Int
      truncated : Bool
    } derive(Eq, ToJson, 
    Debug    )

    Detailed information about one normalization pass.

    Normalizer

    source
    pub struct Normalizer {
      options : NormalizationOptions
    } derive(
    Debug    )

    A reusable normalizer keeps policy configuration separate from data.

    Normalizer::apply

    source
    fn Normalizer::apply(self : Normalizer, text : String) -> String

    Normalizer::apply_entries

    source
    fn Normalizer::apply_entries(self : Normalizer, entries : ArrayView[Entry]) -> Array[Entry]

    Normalizer::apply_entry

    source
    fn Normalizer::apply_entry(self : Normalizer, item : Entry) -> Entry

    Normalizer::new

    source

    RankedCandidate

    source
    pub(all) struct RankedCandidate {
      candidate : Candidate
      explanation : CandidateExplanation
    } derive(Eq, ToJson, 
    Debug    )

    A ranked candidate paired with its evidence.

    Scanner

    source
    pub struct Scanner {
      automaton : Automaton
      state : Int
      position : Int
    } derive(
    Debug    )

    A streaming scanner keeps the automaton state between input chunks.

    Scanner::finish

    source
    fn Scanner::finish(self : Scanner) -> Array[LexiconMatch]

    Finish a scanner. Complete matches are emitted by push, so this returns an empty tail and makes the end-of-stream step explicit.

    Scanner::new

    source
    fn Scanner::new(automaton : Automaton) -> Scanner

    Scanner::push

    source
    fn Scanner::push(self : Scanner, chunk : String) -> Array[LexiconMatch]

    Append a chunk and emit matches without retaining the source text.

    SegmentationOptions

    source
    pub(all) struct SegmentationOptions {
      combine_unmatched : Bool
      prefer_longest : Bool
      include_empty : Bool
    } derive(Eq, ToJson, 
    Debug    )

    Options controlling dictionary-aware segmentation.

    SegmentationOptions::default

    source

    Conservative segmentation defaults suitable for user-facing text tools.

    SerializationStats

    source
    pub(all) struct SerializationStats {
      version : String
      entry_count : Int
      byte_length : Int
      checksum : Int
      valid_header : Bool
    } derive(Eq, ToJson, 
    Debug    )

    Statistics about a serialized dictionary payload.

    Token

    source
    pub(all) struct Token {
      text : String
      start : Int
      end : Int
      matched : Bool
      label : String
    } derive(Eq, ToJson, 
    Debug    )

    A token emitted by the lightweight dictionary-aware tokenizer.

    TokenSequence

    source
    pub(all) struct TokenSequence {
      tokens : Array[Token]
      source_length : Int
    } derive(Eq, ToJson, 
    Debug    )

    A token sequence retains source order and original character offsets.

    TokenSequence::for_label

    source
    fn TokenSequence::for_label(self : TokenSequence, label : String) -> Array[Token]

    TokenSequence::has_contiguous_offsets

    source
    fn TokenSequence::has_contiguous_offsets(self : TokenSequence) -> Bool

    TokenSequence::matched_count

    source
    fn TokenSequence::matched_count(self : TokenSequence) -> Int

    TokenSequence::to_text

    source
    fn TokenSequence::to_text(self : TokenSequence) -> String

    TokenSequence::unmatched_count

    source
    fn TokenSequence::unmatched_count(self : TokenSequence) -> Int

    TokenSequence::unmatched_text

    source
    fn TokenSequence::unmatched_text(self : TokenSequence) -> String

    TokenStats

    source
    pub(all) struct TokenStats {
      token_count : Int
      matched_count : Int
      unmatched_count : Int
      total_length : Int
      matched_length : Int
      unmatched_length : Int
      distinct_label_count : Int
    } derive(Eq, ToJson, 
    Debug    )

    TrieNode

    source

    DICTIONARY_FORMAT_VERSION

    source
    let DICTIONARY_FORMAT_VERSION : String

    Current version marker for the portable dictionary text format.

    analyze_matches

    source
    fn analyze_matches(text_length : Int, matches : ArrayView[LexiconMatch]) -> AnalysisReport

    Summarize an existing match array when scanning was performed elsewhere.

    analyze_text

    source
    fn analyze_text(automaton : Automaton, text : String) -> AnalysisReport

    Summarize all matches emitted for a document.

    apply_delta

    source
    fn apply_delta(base : ArrayView[Entry], delta : DictionaryDelta) -> Array[Entry]

    Apply a dictionary change set and deduplicate the result.

    apply_policy

    source
    fn apply_policy(matches : ArrayView[LexiconMatch], policy : MatchPolicy) -> Array[LexiconMatch]

    Filter, order, and cap matches according to a policy.

    ascii_bytes_to_string

    source
    fn ascii_bytes_to_string(bytes : Bytes) -> String

    Convert ASCII bytes to text, returning an empty string for non-ASCII input.

    assess_dictionary

    source
    fn assess_dictionary(entries : ArrayView[Entry]) -> DictionaryQuality

    Inspect entries before compilation and return deterministic quality data.

    Importers can use this report to reject malformed input without losing the rest of the batch. Duplicate identities are counted after validation so a malformed duplicate is still reported as malformed rather than hidden.

    average_span

    source
    fn average_span(matches : ArrayView[LexiconMatch]) -> Int

    Calculate the arithmetic mean span length.

    benchmark_checksum

    source
    fn benchmark_checksum(results : ArrayView[BenchmarkResult]) -> Int

    Calculate a simple checksum for a benchmark result collection.

    benchmark_coverage_ratio

    source
    fn benchmark_coverage_ratio(result : BenchmarkResult) -> Int

    Return the ratio of matched characters to total input characters.

    benchmark_description

    source
    fn benchmark_description() -> String

    Return a short description of the standard workload.

    benchmark_entries

    source
    fn benchmark_entries(size : Int) -> Array[Entry]

    Generate numbered entries for repeatable local and CI workloads.

    benchmark_expected_surface_count

    source
    fn benchmark_expected_surface_count(size : Int) -> Int

    Return the expected number of dictionary surfaces in a standard workload.

    benchmark_expected_weight

    source
    fn benchmark_expected_weight(size : Int) -> Int

    Return the total weight of generated benchmark entries.

    benchmark_match_density

    source
    fn benchmark_match_density(result : BenchmarkResult) -> Int

    Return matches per one hundred input characters for a benchmark result.

    benchmark_matches_expected

    source
    fn benchmark_matches_expected(result : BenchmarkResult, dictionary_size : Int, repeats : Int) -> Bool

    Verify the counters that are independent of compiler backend.

    benchmark_matches_per_repeat

    source
    fn benchmark_matches_per_repeat(result : BenchmarkResult) -> Int

    Return the average number of matches per repeated dictionary pass.

    benchmark_report_tsv

    source
    fn benchmark_report_tsv(results : ArrayView[BenchmarkResult]) -> String

    Return a stable text report for benchmark artifacts.

    benchmark_result_is_consistent

    source
    fn benchmark_result_is_consistent(result : BenchmarkResult) -> Bool

    Check that a result is internally consistent with a workload.

    benchmark_result_summary

    source
    fn benchmark_result_summary(result : BenchmarkResult) -> String

    Return an operational summary suitable for a one-line CI log.

    benchmark_results_equal

    source
    fn benchmark_results_equal(left : BenchmarkResult, right : BenchmarkResult) -> Bool

    Compare structural counters from two runs.

    benchmark_scale

    source
    fn benchmark_scale(sizes : ArrayView[Int], repeats : Int) -> Array[BenchmarkResult]

    Run several independent dictionary sizes using the same corpus recipe.

    benchmark_sharing_margin

    source
    fn benchmark_sharing_margin(result : BenchmarkResult) -> Int

    Return the number of states beyond the dictionary entry count.

    benchmark_signature

    source
    fn benchmark_signature(result : BenchmarkResult) -> String

    Return a compact workload identifier suitable for cache keys.

    benchmark_sizes_are_ordered

    source
    fn benchmark_sizes_are_ordered(results : ArrayView[BenchmarkResult]) -> Bool

    Return true when all results have non-decreasing dictionary sizes.

    benchmark_surface_coverage

    source
    fn benchmark_surface_coverage(result : BenchmarkResult) -> Int

    Return matched surfaces as a percentage of emitted matches.

    benchmark_text

    source
    fn benchmark_text(entries : ArrayView[Entry], repeats : Int) -> String

    Build a deterministic text corpus from a dictionary.

    benchmark_text_checksum

    source
    fn benchmark_text_checksum(text : String) -> Int

    Compute a small deterministic rolling checksum for text fixtures.

    benchmark_text_for_size

    source
    fn benchmark_text_for_size(size : Int, repeats : Int) -> String

    Generate a text corpus without first materializing entries for callers that only need an input fixture.

    benchmark_unique_ratio

    source
    fn benchmark_unique_ratio(result : BenchmarkResult) -> Int

    Return distinct matched surfaces as a percentage of the dictionary.

    benchmark_workload_checksum

    source
    fn benchmark_workload_checksum(size : Int, repeats : Int) -> Int

    Return a deterministic checksum over a generated workload.

    best_per_position

    source
    fn best_per_position(matches : ArrayView[LexiconMatch]) -> Array[LexiconMatch]

    Return the highest-confidence match at each source position.

    byte_backend_description

    source
    fn byte_backend_description() -> String

    Return a stable diagnostic for the byte backend.

    byte_slice

    source
    fn byte_slice(bytes : Bytes, start : Int, end : Int) -> Bytes

    Return bytes in a source span, clamped to valid byte boundaries.

    bytes_are_ascii

    source
    fn bytes_are_ascii(bytes : Bytes) -> Bool

    True when every byte is a seven-bit ASCII value.

    bytes_hex_preview

    source
    fn bytes_hex_preview(bytes : Bytes, limit : Int) -> String

    Return a hexadecimal preview for diagnostics.

    candidate_contribution

    source
    fn candidate_contribution(hit : LexiconMatch) -> Candidate

    Return the score contribution of one match.

    candidate_report_tsv

    source
    fn candidate_report_tsv(candidates : ArrayView[RankedCandidate]) -> String

    Return a stable TSV report suitable for audit artifacts.

    candidates_at_least

    source
    fn candidates_at_least(candidates : ArrayView[RankedCandidate], minimum_score : Int) -> Array[RankedCandidate]

    Return candidates with a minimum accumulated score.

    candidates_with_occurrences

    source
    fn candidates_with_occurrences(candidates : ArrayView[RankedCandidate], minimum_occurrences : Int) -> Array[RankedCandidate]

    Return candidates whose evidence contains at least this many occurrences.

    catalog_count_by_source

    source
    fn catalog_count_by_source(hits : ArrayView[CatalogHit]) -> Map[String, Int]

    Count hits by source name.

    catalog_hit_signature

    source
    fn catalog_hit_signature(hit : CatalogHit) -> String

    Return a stable catalog hit signature.

    catalog_hits_for_source

    source
    fn catalog_hits_for_source(hits : ArrayView[CatalogHit], source : String) -> Array[CatalogHit]

    Return hits from one source only.

    catalog_matches

    source
    fn catalog_matches(hits : ArrayView[CatalogHit]) -> Array[LexiconMatch]

    Flatten catalog hits into normal match values.

    clamp_entry_weights

    source
    fn clamp_entry_weights(entries : ArrayView[Entry], minimum : Int, maximum : Int) -> Array[Entry]

    Return a copy with weights clamped to the supplied inclusive range.

    clamp_match

    source
    fn clamp_match(hit : LexiconMatch, text_length : Int) -> LexiconMatch

    Clamp a match to a source string, preserving its metadata.

    combine_unmatched_tokens

    source
    fn combine_unmatched_tokens(tokens : ArrayView[Token]) -> Array[Token]

    compile

    source
    fn compile(entries : ArrayView[Entry]) -> Lexicon

    Compile entries into a lexicon in one pass.

    compile_automaton

    source
    fn compile_automaton(entries : ArrayView[Entry]) -> Automaton

    Convenience wrapper around Automaton::compile.

    compile_normalized

    source
    fn compile_normalized(entries : ArrayView[Entry], options : NormalizationOptions) -> Lexicon

    Compile a dictionary after applying the same normalization policy to every keyword. Callers should normalize queries with the same options.

    context_windows

    source
    fn context_windows(text : String, matches : ArrayView[LexiconMatch], radius : Int) -> Array[ContextWindow]

    Extract context windows around each match.

    count_by_label

    source
    fn count_by_label(matches : ArrayView[LexiconMatch]) -> Map[String, Int]

    Count matches by label.

    count_entries_by_label

    source
    fn count_entries_by_label(entries : ArrayView[Entry]) -> Map[String, Int]

    Count entries by label without compiling a trie.

    count_normalization_whitespace

    source
    fn count_normalization_whitespace(text : String) -> Int

    Count whitespace characters using the same classifier as the normalizer.

    coverage_ranges

    source
    fn coverage_ranges(text_length : Int, matches : ArrayView[LexiconMatch]) -> Array[(Int, Int)]

    Return source intervals that contain at least one match.

    covered_byte_count

    source
    fn covered_byte_count(byte_length : Int, matches : ArrayView[LexiconMatch]) -> Int

    Count covered byte positions using merged match intervals.

    covered_characters

    source
    fn covered_characters(text_length : Int, matches : ArrayView[LexiconMatch]) -> Int

    Count covered source characters using a union of match intervals.

    decode_entries

    source
    fn decode_entries(payload : String) -> Array[Entry]

    Decode a MLX1 payload. Malformed rows are skipped and never become empty keywords, which makes it safe to inspect partially written cache files.

    decode_entries_checked

    source
    fn decode_entries_checked(payload : String) -> Array[Entry]

    Decode a checked payload only when its checksum agrees with the rows.

    deduplicate_entries

    source
    fn deduplicate_entries(entries : ArrayView[Entry]) -> Array[Entry]

    Keep one entry for each keyword/label pair, retaining the highest weight.

    dictionary_checksum

    source
    fn dictionary_checksum(entries : ArrayView[Entry]) -> Int

    Compute a stable non-cryptographic checksum for dictionary entries.

    dictionary_format_description

    source
    fn dictionary_format_description() -> String

    Return a short format description for CLI help and diagnostics.

    dictionary_quality_is_usable

    source
    fn dictionary_quality_is_usable(report : DictionaryQuality) -> Bool

    Return whether a dictionary is safe to compile without data loss.

    dictionary_quality_tsv

    source
    fn dictionary_quality_tsv(report : DictionaryQuality) -> String

    Render a stable one-row report for import logs and CI artifacts.

    diff_entries

    source
    fn diff_entries(before : ArrayView[Entry], after : ArrayView[Entry]) -> DictionaryDelta

    Compare dictionaries by keyword/label identity and weight.

    distinct_entry_count

    source
    fn distinct_entry_count(entries : ArrayView[Entry]) -> Int

    Return the number of distinct keyword/label identities.

    distinct_label_count

    source
    fn distinct_label_count(matches : ArrayView[LexiconMatch]) -> Int

    Return the count of distinct labels in a match array.

    duplicate_entry_count

    source
    fn duplicate_entry_count(entries : ArrayView[Entry]) -> Int

    Return the number of repeated keyword/label identities in a batch.

    encode_entries

    source
    fn encode_entries(entries : ArrayView[Entry]) -> String

    Return a deterministic, line-oriented representation of dictionary entries.

    encode_entries_checked

    source
    fn encode_entries_checked(entries : ArrayView[Entry]) -> String

    Serialize entries and append a checksum trailer for cache validation.

    encoded_size

    source
    fn encoded_size(entries : ArrayView[Entry]) -> Int

    Estimate the encoded size of entries without allocating the payload.

    encoded_size_matches

    source
    fn encoded_size_matches(entries : ArrayView[Entry]) -> Bool

    Compare the encoded size estimate with the actual payload length.

    entries_equal_identity

    source
    fn entries_equal_identity(left : Entry, right : Entry) -> Bool

    True when two entries represent the same keyword and label.

    entries_for_label

    source
    fn entries_for_label(entries : ArrayView[Entry], label : String) -> Array[Entry]

    Select entries that belong to one label.

    entries_with_min_weight

    source
    fn entries_with_min_weight(entries : ArrayView[Entry], minimum : Int) -> Array[Entry]

    Select entries whose weight reaches minimum.

    entry

    source
    fn entry(keyword : String, label? : String, weight? : Int) -> Entry

    Free-function shorthand for Entry::new.

    entry_display_key

    source
    fn entry_display_key(entry : Entry) -> String

    Return a human-readable identity that is safe to use in reports.

    entry_key

    source
    fn entry_key(entry : Entry) -> String

    A stable identity for an entry. The label is part of the identity so a keyword may intentionally participate in more than one domain.

    escape_dictionary_field

    source
    fn escape_dictionary_field(value : String) -> String

    Escape a dictionary field for the MLX1 line format.

    escaped_character_count

    source
    fn escaped_character_count(value : String) -> Int

    Return the number of escaped control characters in a field.

    explain_candidate

    source
    fn explain_candidate(matches : ArrayView[LexiconMatch], keyword : String, label : String) -> CandidateExplanation?

    Return the evidence for one candidate identity.

    first_byte_match

    source
    fn first_byte_match(automaton : Automaton, bytes : Bytes) -> LexiconMatch?

    Return the first byte position containing a match, if any.

    first_matched_token

    source
    fn first_matched_token(tokens : ArrayView[Token]) -> Token?

    group_entries_by_label

    source
    fn group_entries_by_label(entries : ArrayView[Entry]) -> Map[String, Array[Entry]]

    Return entries grouped by label, retaining source order inside each group.

    highlight_html

    source
    fn highlight_html(text : String, matches : ArrayView[LexiconMatch]) -> String

    Render match text as HTML-safe highlighted output.

    highlight_matches

    source
    fn highlight_matches(text : String, matches : ArrayView[LexiconMatch], prefix : String, suffix : String) -> String

    Add a prefix and suffix around every non-overlapping match.

    integer_width

    source
    fn integer_width(value : Int) -> Int

    Return the integer width used by the text serializer.

    invert_delta

    source
    fn invert_delta(before : ArrayView[Entry], after : ArrayView[Entry], delta : DictionaryDelta) -> DictionaryDelta

    Invert a change set so it can be used for rollback.

    is_dictionary_payload

    source
    fn is_dictionary_payload(payload : String) -> Bool

    True when a payload has the MLX1 header.

    keyword_length_histogram

    source
    fn keyword_length_histogram(entries : ArrayView[Entry]) -> Map[Int, Int]

    Return the frequency of each keyword length.

    keyword_summaries

    source
    fn keyword_summaries(matches : ArrayView[LexiconMatch]) -> Array[KeywordSummary]

    Summarize repeated keyword occurrences.

    label_report_tsv

    source
    fn label_report_tsv(matches : ArrayView[LexiconMatch]) -> String

    Render label summaries as a stable TSV report.

    label_summaries

    source
    fn label_summaries(matches : ArrayView[LexiconMatch]) -> Array[LabelSummary]

    Group matches into deterministic label summaries.

    largest_benchmark

    source
    fn largest_benchmark(results : ArrayView[BenchmarkResult]) -> BenchmarkResult?

    Return the result with the largest input, if one exists.

    last_matched_token

    source
    fn last_matched_token(tokens : ArrayView[Token]) -> Token?

    levenshtein

    source
    fn levenshtein(a : String, b : String, max_distance? : Int) -> Int

    A bounded Levenshtein distance implementation for dictionary suggestions.

    limit_matches

    source
    fn limit_matches(matches : ArrayView[LexiconMatch], limit : Int) -> Array[LexiconMatch]

    Limit output while preserving the order produced by the query.

    longest_keyword

    source
    fn longest_keyword(entries : ArrayView[Entry]) -> String

    Return the longest keyword in a dictionary, or an empty string.

    longest_match

    source
    fn longest_match(matches : ArrayView[LexiconMatch]) -> LexiconMatch?

    Return the match with the largest span, if any.

    match_contains

    source
    fn match_contains(outer : LexiconMatch, inner : LexiconMatch) -> Bool

    Return true when one match fully contains another.

    match_density

    source
    fn match_density(matches : ArrayView[LexiconMatch], text_length : Int) -> Int

    Calculate matches per one hundred source characters.

    match_signature

    source
    fn match_signature(hit : LexiconMatch) -> String

    Return a compact, deterministic signature for a match.

    match_span_length

    source
    fn match_span_length(hit : LexiconMatch) -> Int

    Return the length of a match span.

    match_stats

    source
    fn match_stats(matches : ArrayView[LexiconMatch]) -> MatchStats

    matched_tokens

    source
    fn matched_tokens(tokens : ArrayView[Token]) -> Array[Token]

    Return only the matched tokens from a token stream.

    matched_tokens_at_least

    source
    fn matched_tokens_at_least(tokens : ArrayView[Token], minimum_length : Int) -> Array[Token]

    matches_for_label

    source
    fn matches_for_label(matches : ArrayView[LexiconMatch], label : String) -> Array[LexiconMatch]

    Keep matches for one label.

    matches_in_range

    source
    fn matches_in_range(matches : ArrayView[LexiconMatch], start : Int, end : Int) -> Array[LexiconMatch]

    Keep matches fully contained in a source interval.

    matches_overlap

    source
    fn matches_overlap(left : LexiconMatch, right : LexiconMatch) -> Bool

    Return true when two matches share at least one source character.

    matches_to_tsv

    source
    fn matches_to_tsv(matches : ArrayView[LexiconMatch]) -> String

    Render matches as tab-separated rows.

    median_span

    source
    fn median_span(matches : ArrayView[LexiconMatch]) -> Int

    Return the median span length, using lower-middle selection for even input.

    merge_entries

    source
    fn merge_entries(left : ArrayView[Entry], right : ArrayView[Entry]) -> Array[Entry]

    Merge two dictionaries while preserving first-seen order.

    merge_overlapping

    source
    fn merge_overlapping(matches : ArrayView[LexiconMatch]) -> Array[LexiconMatch]

    Collapse overlaps using leftmost-longest selection.

    most_frequent_keyword

    source
    fn most_frequent_keyword(matches : ArrayView[LexiconMatch]) -> String?

    Return the most frequently emitted keyword, if any.

    needs_normalization

    source
    fn needs_normalization(text : String, options : NormalizationOptions) -> Bool

    Return true when normalization changes a string.

    non_ascii_byte_count

    source
    fn non_ascii_byte_count(bytes : Bytes) -> Int

    Count bytes outside the ASCII range.

    normalization_description

    source
    fn normalization_description(options : NormalizationOptions) -> String

    Return a readable policy description.

    normalization_report_tsv

    source
    fn normalization_report_tsv(entries : ArrayView[Entry], options : NormalizationOptions) -> String

    Render normalization diagnostics as TSV.

    normalization_reports

    source
    fn normalization_reports(entries : ArrayView[Entry], options : NormalizationOptions) -> Array[EntryNormalizationReport]

    normalize_entries

    source
    fn normalize_entries(entries : ArrayView[Entry], options : NormalizationOptions) -> Array[Entry]

    Normalize and remove entries whose keyword becomes empty.

    normalize_entry

    source
    fn normalize_entry(item : Entry, options : NormalizationOptions) -> Entry

    Normalize one entry without changing its label or weight.

    normalize_for_lookup

    source
    fn normalize_for_lookup(query : String, options : NormalizationOptions) -> String

    Produce a normalized query key.

    normalize_serialized_entries

    source
    fn normalize_serialized_entries(entries : ArrayView[Entry]) -> Array[Entry]

    Return a copy of entries after a serialize/decode cycle.

    normalize_with_report

    source
    fn normalize_with_report(text : String, options : NormalizationOptions) -> NormalizationReport

    Normalize text while retaining counters useful for data-quality reports.

    normalized_keyword_length

    source
    fn normalized_keyword_length(item : Entry, options : NormalizationOptions) -> Int

    Return the normalized keyword length of an entry.

    normalized_lookup

    source
    fn normalized_lookup(lexicon : Lexicon, query : String, options : NormalizationOptions) -> Array[Entry]

    Look up an exact normalized key in a normalized lexicon.

    overlap_count

    source
    fn overlap_count(matches : ArrayView[LexiconMatch]) -> Int

    Count pairwise overlapping match spans.

    overlapping_matches

    source
    fn overlapping_matches(matches : ArrayView[LexiconMatch], target : LexiconMatch) -> Array[LexiconMatch]

    Return all matches that overlap a target span.

    parse_tsv

    source
    fn parse_tsv(text : String) -> Array[Entry]

    Parse tab-separated dictionary lines: keyword, label, optional weight.

    percentage

    source
    fn percentage(numerator : Int, denominator : Int) -> Int

    Convert a numerator into a rounded percentage in the range 0..100.

    query_lexicon

    source
    fn query_lexicon(lexicon : Lexicon, text : String, query : MatchQuery) -> Array[LexiconMatch]

    Query a lexicon by compiling its current entries into an automaton.

    query_matches

    source
    fn query_matches(automaton : Automaton, text : String, query : MatchQuery) -> Array[LexiconMatch]

    Extract matches according to query filters and output policy.

    rank_candidates

    source
    fn rank_candidates(matches : ArrayView[LexiconMatch]) -> Array[RankedCandidate]

    Aggregate matches into ranked, explainable candidates.

    rank_candidates_top

    source
    fn rank_candidates_top(matches : ArrayView[LexiconMatch], limit : Int) -> Array[RankedCandidate]

    Keep the highest-ranked candidates up to a caller-supplied limit.

    ranked_candidates_are_ordered

    source
    fn ranked_candidates_are_ordered(candidates : ArrayView[RankedCandidate]) -> Bool

    Return true when ranking order is deterministic and non-increasing.

    redact_labels

    source
    fn redact_labels(text : String, matches : ArrayView[LexiconMatch], labels : ArrayView[String], replacement : String) -> String

    Replace matches only when their label belongs to a supplied set.

    redact_matches

    source
    fn redact_matches(text : String, matches : ArrayView[LexiconMatch], replacement : String) -> String

    Replace every non-overlapping match with replacement.

    remove_empty_entries

    source
    fn remove_empty_entries(entries : ArrayView[Entry]) -> Array[Entry]

    Remove entries with empty keywords while retaining all other fields.

    replace_keyword

    source
    fn replace_keyword(entry : Entry, keyword : String) -> Entry

    Replace one keyword while preserving label and weight.

    replace_label

    source
    fn replace_label(entry : Entry, label : String) -> Entry

    Return a copy with a new label.

    replace_weight

    source
    fn replace_weight(entry : Entry, weight : Int) -> Entry

    Return a copy with a new weight.

    run_benchmark

    source
    fn run_benchmark(entries : ArrayView[Entry], repeats : Int) -> BenchmarkResult

    Run the standard scan and return backend-independent counters.

    scan_byte_chunks

    source
    fn scan_byte_chunks(automaton : Automaton, chunks : ArrayView[Bytes]) -> Array[LexiconMatch]

    Scan multiple byte chunks with one retained automaton state.

    scan_byte_stats

    source
    fn scan_byte_stats(automaton : Automaton, bytes : Bytes) -> ByteScanStats

    Scan bytes and calculate operational counters.

    scan_bytes_for_label

    source
    fn scan_bytes_for_label(automaton : Automaton, bytes : Bytes, label : String) -> Array[LexiconMatch]

    Filter byte matches by a label without recompiling the automaton.

    segment_text

    source
    fn segment_text(lexicon : Lexicon, text : String) -> TokenSequence

    Segment with the standard longest-match policy.

    segment_with_options

    source
    fn segment_with_options(lexicon : Lexicon, text : String, options : SegmentationOptions) -> TokenSequence

    Segment text with explicit matching and gap policies.

    select_leftmost_longest

    source
    fn select_leftmost_longest(matches : ArrayView[LexiconMatch]) -> Array[LexiconMatch]

    Keep non-overlapping matches using leftmost-longest, then highest weight.

    serialization_stats

    source
    fn serialization_stats(payload : String) -> SerializationStats

    Compute payload metadata without decoding the dictionary twice.

    serialize_lexicon

    source
    fn serialize_lexicon(lexicon : Lexicon) -> String

    Serialize all entries currently stored in a lexicon.

    shortest_keyword

    source
    fn shortest_keyword(entries : ArrayView[Entry]) -> String

    Return the shortest keyword in a dictionary, or an empty string.

    similarity_score

    source
    fn similarity_score(a : String, b : String) -> Int

    Compute a normalized similarity score in the inclusive range 0..100.

    sort_entries

    source
    fn sort_entries(entries : ArrayView[Entry]) -> Array[Entry]

    Sort entries by keyword, label, then descending weight.

    sort_entries_by_weight

    source
    fn sort_entries_by_weight(entries : ArrayView[Entry]) -> Array[Entry]

    Sort entries by descending weight, then longest keyword first.

    sort_matches_by_position

    source
    fn sort_matches_by_position(matches : ArrayView[LexiconMatch]) -> Array[LexiconMatch]

    Sort matches by start, then by span length and weight.

    span_histogram

    source
    fn span_histogram(matches : ArrayView[LexiconMatch]) -> Map[Int, Int]

    Return a histogram of match span lengths.

    spans_valid

    source
    fn spans_valid(text : String, matches : ArrayView[LexiconMatch]) -> Bool

    Check that all match spans are valid for a source string.

    string_to_ascii_bytes

    source
    fn string_to_ascii_bytes(text : String) -> Bytes?

    Build an ASCII byte payload from a string, if every character fits.

    token_span

    source
    fn token_span(tokens : ArrayView[Token]) -> (Int, Int)

    token_stats

    source
    fn token_stats(tokens : ArrayView[Token]) -> TokenStats

    tokens_containing

    source
    fn tokens_containing(tokens : ArrayView[Token], needle : String) -> Array[Token]

    tokens_have_valid_offsets

    source
    fn tokens_have_valid_offsets(tokens : ArrayView[Token]) -> Bool

    tokens_to_tsv

    source
    fn tokens_to_tsv(tokens : ArrayView[Token]) -> String

    top_candidates

    source
    fn top_candidates(candidates : ArrayView[Candidate], limit? : Int) -> Array[Candidate]

    Keep at most limit highest-scoring candidates.

    total_entry_weight

    source
    fn total_entry_weight(entries : ArrayView[Entry]) -> Int

    Return the total weight of all entries.

    uncovered_ranges

    source
    fn uncovered_ranges(text_length : Int, matches : ArrayView[LexiconMatch]) -> Array[(Int, Int)]

    Return gaps between covered intervals.

    unescape_dictionary_field

    source
    fn unescape_dictionary_field(value : String) -> String

    Decode one escaped MLX1 field. Unknown escapes retain their character.

    unique_keywords

    source
    fn unique_keywords(entries : ArrayView[Entry]) -> Array[Entry]

    Return only the first occurrence of each keyword, regardless of label.

    unmatched_text

    source
    fn unmatched_text(tokens : ArrayView[Token]) -> String

    Return the unmatched text while preserving source order.

    valid_entry_count

    source
    fn valid_entry_count(entries : ArrayView[Entry]) -> Int

    Return the number of valid entries in a batch.

    validate_entries

    source
    fn validate_entries(entries : ArrayView[Entry]) -> Array[String]

    Validate a dictionary and include the zero-based entry index in each issue.

    validate_entry

    source
    fn validate_entry(entry : Entry) -> Array[String]

    Validate one entry and return stable diagnostic codes.

    The validator is deliberately non-raising so import pipelines can collect every problem in one pass instead of failing on the first malformed row.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io