🥮 mooncakes.io
    Sign in

    moon_textstat

    Readability metrics for MoonBit — text counts plus the eight textstat formulas, with all data embedded (no runtime I/O).

    readability
    textstat
    flesch
    syllable
    hyphenation
    moonbit
    Download zip
    Version
    0.1.0
    License
    MIT AND MPL-1.1 AND LGPL-2.1-or-later
    Repository
    Last updated
    3 hours ago
    Downloads
    2

    No dependencies

    #vicoproplus/moon_textstat

    A MoonBit port of the readability metrics of Python textstat 0.7.13: one call turns a text into counts, eight readability scores and a reading-time estimate. The library is pure computation — it reads no file, opens no socket, and consults no environment variable, so the hyphenation dictionaries and word lists it needs are embedded in the package.

    #Metrics

    One call returns the 19 fields of TextStats (PRD-A 10.3, unchanged names and units):

    • Counters (10) — char_count, letter_count, lexicon_count, sentence_count, syllable_count, polysyllabcount, monosyllabcount, difficult_words, difficult_words_list, long_word_count.
    • Formulas (8) — flesch_reading_ease, flesch_kincaid_grade, gunning_fog, smog_index, automated_readability_index, coleman_liau_index, linsear_write_formula, dale_chall_readability_score.
    • Reading time (1) — reading_time, in seconds.

    Every metric is also a standalone entry with the same name (@moon_textstat.flesch_reading_ease(text), @moon_textstat.char_count(text)…) whose optional arguments carry the same defaults as analyze.

    #Installation

    Not yet published to the registry, so moon add vicoproplus/moon_textstat 404s today. Use a path dependency instead: a moon.work beside your module with members = ["demo", "../moon_textstat/lib"], and in demo/moon.mod the versioned import { "vicoproplus/moon_textstat@0.1.0" }. The moon add form is the future install path, once published.

    Then import the module from a package's moon.pkg:

    import {
  "vicoproplus/moon_textstat",
}

    #Example

    ///|
    fn main {
      let stats = try! @moon_textstat.analyze("The cat sat on the mat.")
      println(stats.flesch_reading_ease) // 116.14500000000001
      println(stats.flesch_kincaid_grade) // -1.4499999999999993
      println(stats.gunning_fog) // 2.4000000000000004
      println(stats.smog_index) // 3.1291
      println(stats.automated_readability_index) // -4.300000000000001
      println(stats.coleman_liau_index) // -4.299999999999997
      println(stats.linsear_write_formula) // 2
      println(stats.dale_chall_readability_score) // 0.2976
    }

    analyze raises TextstatError; use try! where a failure is impossible (English input, no unsupported language) and try … catch elsewhere.

    #The lang parameter

    analyze(text, lang="en_US") — the default is en_US. The code selects the hyphenation dictionary and the easy-word list (an en or es list, otherwise the English one); the readability constants are keyed by the nine roots de, en, es, fr, hu, it, nl, pl, ru, with per-key fallback to the English row. Ten dictionary codes are embedded: en_US, en_GB, de_DE, es, fr, it_IT, nl_NL, pl_PL, ru_RU, hu_HU, plus the short aliases en (→ en_GB, upstream's short-name winner), es, fr, it, nl, pl, ru, hu. Codes are case-insensitive and - equals _ (EN-us works). de and pt raise TextstatError::MetricFailure, because upstream maps them to de_AT/pt_BR, which are not embedded. The low-level TextstatError::UnsupportedLanguage carries the raw code; the public entries wrap it into MetricFailure.

    #Fidelity

    Values are bit-for-bit aligned with textstat 0.7.13 on Python 3.12.10, pyphen 0.17.2 and nltk 3.10.3: 366 frozen cells (288 sample-side, 78 language-side) are compared as exact integers or IEEE-754 bit patterns, not as rounded floats. Run the contract check from the module root:

    moon run cmd/gate --target native

    Exit code 0 means every cell matches, 1 means a mismatch, 2 means the environment or the build failed.

    #Scope

    The Chinese-specific metrics of textstat (chinese_*, the zh language support) are out of scope for this module version: TextStats intentionally carries no Chinese-extension field, and no function or data table covers them.

    TextstatError

    source
    pub suberror TextstatError {
      UnsupportedLanguage(String)
      MetricFailure(String)
    }

    Failure of a public entry point.

    UnsupportedLanguage(requested) reports that no embedded hyphenation dictionary can serve requested, the caller's raw spelling before case and separator normalization. It is the low-level variant: only the package-private hyphenation and syllable functions raise it, so white-box tests can still observe the resolved code directly.

    MetricFailure(message) is the product-facing variant raised at every public boundary that can fail. Its text deliberately carries the three items of spec §4.5.4 / A-5.4 — metric name, language code and failure reason — which upstream's bare KeyError: None does not. That enrichment is a product contract, not an alignment item: the gate compares the failure semantics (raised vs. a value), never this text (A-5.2 vs. A-5.4).

    TextStats

    source
    pub(all) struct TextStats {
      char_count : Int
      letter_count : Int
      lexicon_count : Int
      sentence_count : Int
      syllable_count : Int
      polysyllabcount : Int
      monosyllabcount : Int
      difficult_words : Int
      difficult_words_list : Array[String]
      long_word_count : Int
      reading_time : Double
      flesch_reading_ease : Double
      flesch_kincaid_grade : Double
      gunning_fog : Double
      smog_index : Double
      automated_readability_index : Double
      coleman_liau_index : Double
      linsear_write_formula : Double
      dale_chall_readability_score : Double
    }

    The value object returned by one analysis call.

    The 19 fields, their order, types and units are a read-only reference to PRD-A 10.3 (AD-4): names are not renamed, added or dropped. Counting fields are 32-bit integers, while formula scores and reading_time are double-precision floats; difficult_words_list is a word sequence whose de-duplicated mode is compared as a set (order is not contractual).

    Units: char_count/letter_count characters, lexicon_count/ syllable_count/polysyllabcount/monosyllabcount/difficult_words/ long_word_count words, sentence_count sentences, reading_time seconds, formula fields in their respective scale or grade. This value object intentionally carries no Chinese-extension field (spec §4.2 constraint 1).

    analyze

    source
    fn analyze(text : String, lang? : String) -> TextStats raise TextstatError

    Analyze text once and return the 19-field value object of spec §4.2 (PRD-A FR-01..FR-23; spec §1.3.2; design D-5).

    The fields are computed by the 19 public single-metric entries, each under the default shape frozen in the oracle's results keys (spec §4.2 constraint 3):

    • char_count with ignore_spaces=true;
    • lexicon_count with removepunct=true, split_contractions=false and split_hyphens=false;
    • syllable_count, polysyllabcount, monosyllabcount, flesch_reading_ease, flesch_kincaid_grade, gunning_fog, smog_index and dale_chall_readability_score with the requested lang;
    • difficult_words and difficult_words_list with syllable_threshold=2 and unique=true;
    • long_word_count with threshold=6;
    • reading_time with 14.69 ms per non-whitespace character;
    • linsear_write_formula with strict_upper=true and strict_lower=false;
    • automated_readability_index and coleman_liau_index with no language parameter.

    The record literal lists the 19 fields in the spec §4.2 order (the order of TextStats), so the returned value object has a stable shape. Nothing is reimplemented here: the entry only sequences the existing public entries, and no second tokenization path exists (spec §2.1 constraint 2).

    Failure channel. syllable_count is the first syllable-based metric in field order, so an unsupported language on non-empty text raises there and the call stops before any English substitution (spec §4.5.4). The error is the inner metric's MetricFailure, whose text still carries the three required items — metric name, language code and failure reason (A-5.4) — and it propagates unchanged rather than being rewrapped under analyze. The empty string keeps that entry's documented early return (syllable_count("") is 0 for every language), and the other entries then return their zero shapes, exactly as the single-metric contracts describe.

    The function is pure: no I/O. This entry exposes only lang; the variant switches (difficult-word syllable threshold and de-duplication, long-word threshold, reading-time pace, Linsear strictness) remain reachable through the corresponding single-metric entry, which is their contract surface.

    automated_readability_index

    source
    fn automated_readability_index(text : String) -> Double

    Automated Readability Index (spec §4.4 table row 5; PRD-A FR-20; upstream _automated_readability_index.py:10-38).

    The expression is the upstream one, verbatim:

    a = chars_per_word(text)
b = words_per_sentence(text)
if a == 0 or b == 0: return 0.0
return (4.71 * a) + (0.5 * b) - 21.43

    Asymmetric denominators (spec §4.4 constraint 4). a is the space-free character count over the punctuation-preserving word count; b is the punctuation-stripped word count over the sentence count. Both averages come from the package-private helpers in averages.mbt, so this entry neither tokenizes nor re-counts anything (spec §2.1 constraint 2). On the punctuation_only sample (" .? ") the two word counts differ — 1 versus 0 — so a is 2.0 and b is 0.0; the oracle still pins 0.0 for that row.

    Evaluation order. (4.71 * a) + (0.5 * b) - 21.43 parses as ((4.71 * a) + (0.5 * b)) - 21.43: * binds tighter than +/-, which associate to the left; moon fmt drops the redundant parentheses but not the tree. Spec §4.4 constraint 1 forbids rewriting it; measured on the eleven frozen samples the reassociated 4.71 * a + (0.5 * b - 21.43) happens to agree bit for bit, so the tree is kept as upstream wrote it rather than because a sample forces it.

    Zero branch. Upstream computes both averages and returns 0.0 if either is 0. The empty string has no characters, words or sentences (0 / 0 → 0.0 for both), and whitespace-only text also has a zero character count and no words, so both return 0.0. Punctuation-only text is the asymmetric case: a is 2.0 but b is 0.0 (one sentence, no words), so the guard still returns 0.0 (spec §4.3.3, A-5.1).

    No language parameter. Row 5 declares no language dependency: ARI resolves no constants, easy-word lists, pronunciation tables or hyphenation dictionaries, so the function cannot raise. Pure: no I/O.

    char_count

    source
    fn char_count(text : String, ignore_spaces? : Bool) -> Int

    Count the characters of text as Unicode code points (spec §4.3.1 row 1; upstream _count_chars.py:25-27).

    With the default ignore_spaces=true only characters that are not whitespace are counted: the loop skips every code point matched by Python's re.sub(r"\s", "", text) (\n, \t, NBSP, and also U+001C..U+001F). With ignore_spaces=false every code point is counted unchanged, matching upstream's plain len(text).

    Length is measured in code points, not UTF-8 bytes and not UTF-16 code units: String::char_length() is used because String::length() returns the UTF-16 code-unit count (an astral character takes two units there). This is exactly Python len(str) semantics. The empty string yields 0 and whitespace-only input yields 0 when spaces are ignored (spec §4.3.3). The function is pure: no I/O, no exceptions.

    coleman_liau_index

    source
    fn coleman_liau_index(text : String) -> Double

    Coleman-Liau Index (spec §4.4 table row 6; PRD-A FR-21; upstream _coleman_liau_index.py:9-37).

    The expression is the upstream one, verbatim:

    letters = letters_per_word(text) * 100
sentences = sentences_per_word(text) * 100
if letters == 0 or sentences == 0: return 0.0
return (0.058 * letters) - (0.296 * sentences) - 15.8

    Hundred-word scaling before the guard. letters_per_word and sentences_per_word are per-word averages; upstream multiplies each by 100 to obtain letters and sentences per hundred words, and only then tests the scaled values for zero. This port keeps that order: the averages come from the package-private helpers in averages.mbt (count layer only, so this entry neither tokenizes nor re-counts anything, spec §2.1 constraint 2), the two multiplications run before the branch, and the branch tests the scaled locals. Multiplication by 100 cannot turn a non-zero average into 0.0, so the guard fires exactly when an average is 0.0.

    Evaluation order. (0.058 * letters) - (0.296 * sentences) - 15.8 parses as ((0.058 * letters) - (0.296 * sentences)) - 15.8: * binds tighter than -, which associates to the left; moon fmt drops the redundant parentheses but not the tree. Spec §4.4 constraint 1 forbids rewriting it. The reassociated 0.058 * letters - (0.296 * sentences + 15.8) moves four of the eleven frozen rows by 1 to 64 ulp (long_multi_sentence_paragraph 2, abbreviations_and_initials 64 — cancellation around 0.17 —, hyphenated_words 1, very_short_under_5_words 1), so the left-associative form is observable inside the sample set, not merely stylistic.

    Zero branch. The empty string has no letters, words or sentences (0 / 0 → 0.0 for both averages) and whitespace-only text also tokenizes to no words, so both return 0.0. Punctuation-only text (" .? ") has one sentence but no letters and no words, so letters_per_word hits its zero denominator and returns 0.0; the letters == 0.0 half of the guard fires and the result is 0.0 (spec §4.3.3, A-5.1). The frozen oracle pins 0.0 for all three shapes.

    No language parameter. Row 6 declares no language dependency: this entry resolves no constants, easy-word lists, pronunciation tables or hyphenation dictionaries, so the function cannot raise and its return type carries no raise. Pure: no I/O.

    dale_chall_readability_score

    source
    fn dale_chall_readability_score(text : String, lang? : String) -> Double raise TextstatError

    Dale-Chall readability score (spec §4.4 table row 8; PRD-A FR-23; upstream _dale_chall_readability_score.py:9-56).

    The expression is the upstream one, verbatim:

    word_count = count_words(text)
hard_count = count_difficult_words(text, lang, syllable_threshold=0)
if word_count == 0: return 0.0
per_difficult_words = 100 * hard_count / word_count
score = (0.1579 * per_difficult_words) + (0.0496 * words_per_sentence(text))
if per_difficult_words > 5: score += 3.6365
return score

    Difficult words are non-unique at threshold 0. Row 8 of spec §4.4 names the word list and the syllable layer as dependencies, and §4.3.4 fixes the two call-site defaults this formula must not inherit: Dale-Chall forces syllable_threshold=0, and the counter runs with unique=false, the backend default. The public difficult_words field defaults to threshold 2 and unique=true (spec §4.3.4, R-09/R-32/R-43), so it must not be used here. Both settings are observable in the frozen samples: at threshold 0 long_multi_sentence_paragraph has 36 difficult occurrences out of 88 words (33 distinct spellings), abbreviations_and_initials 17 out of 31 (6 at threshold 2) and very_short_under_5_words 1 out of 2 (0 at threshold 2). The unique count would give 9.6136.../13.6159...-style values instead of the frozen 10.969005454545455 / 12.551798924731184.

    Evaluation order. 100 * hard_count is the integer product, converted to Double only for the division, matching Python's int multiply followed by true division. The score is the add-before-anything shape (0.1579 * per_difficult_words) + (0.0496 * words_per_sentence(text)); the bonus is a separate += 3.6365 after it, applied only when the percentage is strictly greater than 5. Exactly 5.0 percent therefore receives no bonus. Spec §4.4 constraint 1 forbids rewriting either expression.

    Zero branch. If word_count is 0 the function returns 0.0, exactly as upstream's except ZeroDivisionError. The empty string, whitespace-only text and punctuation-only text all tokenize to no words and land here. The difficult counter runs first, as upstream evaluates it before dividing, but zero-word texts collect no tokens and never resolve a hyphenation dictionary: they return 0.0 even for an unsupported lang (spec §4.3.3).

    Failure channel. A non-easy word reaches the syllable layer inside the difficult counter, so a hard non-empty text under ar/xx_YY raises the product-facing MetricFailure naming this metric, the code and the missing dictionary before the zero guard (upstream raises KeyError: None; spec §4.5.4, A-5.4). Text made only of easy words — e.g. "the cat sat on the mat" — stays on the easy-list branch, which falls back to the English list, and scores 0.2976 for every language. This port copies that behavior and never substitutes a dictionary.

    The default is en_US. The function is pure: no I/O.

    difficult_words

    source
    fn difficult_words(text : String, lang? : String, syllable_threshold? : Int, unique? : Bool) -> Int raise TextstatError

    Number of difficult words in text (spec §4.3.4; upstream public difficult_words, textstat.py:723-749).

    Defaults are lang="en_US", syllable_threshold=2 and unique=true. The de-duplicated default is the public-field contract of spec §4.2 constraint 3; it is not the mode Gunning Fog uses. Gunning Fog calls the internal count_difficult_words with the language-root threshold and unique=false, so the two callers must not be conflated (R-09/R-32/R-43).

    unique=true counts distinct spellings exactly, so "Extraordinary" and "extraordinary" are two members; unique=false counts every occurrence in order. See count_difficult_words for the tokenization evidence behind preserving case.

    Failure follows the per-word predicate. Empty, whitespace-only and punctuation-only text tokenizes to no words and returns 0 even for an unsupported lang; easy words return false before any dictionary is resolved. Verified against the vendored upstream (0.7.13) under ar: "", " ", " .? ", "the", "cat sat" and "hello world" all return 0, while "xyzzy" and the language-probe sentence raise KeyError: None. This port raises the product-facing MetricFailure naming this metric, the caller's raw spelling and the missing dictionary (spec §4.5.4, A-5.4) and never substitutes a supported dictionary.

    The count is 32-bit (Int), the type of TextStats.difficult_words (PRD-A 10.3). The function is pure: no I/O.

    difficult_words_list

    source
    fn difficult_words_list(text : String, lang? : String, syllable_threshold? : Int, unique? : Bool) -> Array[String] raise TextstatError

    The difficult words of text (spec §4.3.4; upstream public difficult_words_list, textstat.py:751-776).

    Defaults are lang="en_US", syllable_threshold=2 and unique=true. Selection is list_difficult_words — case-preserving list_words tokens filtered by is_difficult_word — the same implementation behind the difficult_words count, so membership and spelling agree by construction.

    unique=true returns distinct spellings by exact String equality, mirroring upstream's set(...) membership: Readability and readability are two entries. Upstream then passes the set through list(...), whose iteration order varies with PYTHONHASHSEED (D-16); order is not contractual (spec §4.3.4, C-4), and this port returns a deterministic first-occurrence order instead. unique=false returns every occurrence in token order, including repetitions, matching upstream's list path (FR-13).

    Failure follows the per-word predicate, exactly like difficult_words: zero-word or easy-only text never resolves a dictionary and returns [] even for an unsupported lang; a non-easy candidate raises the product-facing MetricFailure naming this metric, the raw code and the missing dictionary (spec §4.5.4, A-5.4) where upstream raises KeyError: None.

    The function is pure: no I/O.

    flesch_kincaid_grade

    source
    fn flesch_kincaid_grade(text : String, lang? : String) -> Double raise TextstatError

    Flesch-Kincaid Grade (spec §4.4 table row 2; PRD-A FR-17; upstream _flesch_kincaid_grade.py:9-39).

    The expression is the upstream one, verbatim:

    0.39 * sl + 11.8 * spw - 15.59

    with sl = words_per_sentence(text) and spw = syllables_per_word(text,lang). * binds tighter than + and -, and +/- associate to the left, so the parse tree is exactly upstream's ((0.39 * sl) + (11.8 * spw)) - 15.59; moon fmt drops the redundant parentheses but not the tree. Spec §4.4 constraint 1 forbids rewriting it into an algebraically equivalent form, because reassociation changes the floating-point result. Measured on the eleven frozen samples with the upstream counts, 0.39 * sl + (11.8 * spw - 15.59) moves abbreviations_and_initials, hyphenated_words, multiple_paragraphs_newlines and very_short_under_5_words by 1-2 ulp, so the left-associative form above is the only one that reproduces the oracle bit for bit.

    Fixed coefficients (C-2 correction). Unlike Flesch Reading Ease there is no per-language constant row: 0.39, 11.8 and 15.59 are literals here, and the language affects the result only through syllables_per_word (spec §4.4 row 2, "语种依赖: 仅音节（系数固定）"). lang is forwarded unchanged to the syllable layer, which is also the only layer that resolves or rejects it.

    Zero branch. If either average is 0.0 the function returns 0.0 before any formula arithmetic, matching upstream's if sentence_length == 0 or syllables == 0: return 0.0. That covers the empty string and every text that tokenizes to no words (whitespace-only, punctuation-only) under a supported language (spec §4.3.3).

    Failure channel. syllables_per_word calls syllable_count before the zero guard, exactly as upstream evaluates count_syllables / count_words, so an unsupported language on non-empty text raises instead of returning 0.0 — including whitespace-only and punctuation-only text. The empty string still returns 0.0 for every language, because syllable_count("") returns 0 before resolving a dictionary (verified against the vendored upstream: flesch_kincaid_grade("", "ar") is 0.0, ("Hi there!", "ar") raises). The failure is raised as MetricFailure naming this metric, the language code and the missing hyphenation dictionary (spec §4.5.4, A-5.4); no fallback to English is attempted.

    The default is en_US.

    flesch_reading_ease

    source
    fn flesch_reading_ease(text : String, lang? : String) -> Double raise TextstatError

    Flesch Reading Ease (spec §4.4 table row 1; PRD-A FR-16; upstream _flesch_reading_ease.py:11-37).

    The expression is the upstream one, verbatim:

    fre_base - fre_sentence_length * sl - fre_syll_per_word * spw

    with sl = words_per_sentence(text), spw = syllables_per_word(text, lang) and the three constants resolved through lang_config(lang). Python parses the two subtractions left to right, so the value is (fre_base - a * sl) - b * spw; spec §4.4 constraint 1 forbids rewriting it into an algebraically equivalent form, because reassociation changes the floating-point result.

    Zero branch. If either average is 0.0 the function returns 0.0 without touching the constant table, matching upstream's if sentence_length == 0 or syllables == 0: return 0.0. That covers the empty string and every text that tokenizes to no words (whitespace-only, punctuation-only) under a supported language (spec §4.3.3).

    Failure channel. syllables_per_word calls syllable_count before the zero guard, exactly as upstream evaluates count_syllables / count_words, so an unsupported language on non-empty text raises instead of returning 0.0 — including text that tokenizes to nothing (" ", " .? "). The empty string still returns 0.0 for every language, because syllable_count("") returns 0 before resolving a dictionary (verified: upstream flesch_reading_ease("", "ar") is 0.0, ("hello", "ar") raises). The failure is raised as MetricFailure naming this metric, the language code and the missing hyphenation dictionary (spec §4.5.4, A-5.4); no fallback to English is attempted.

    Language constants. lang_config keys on the raw root (en, de, es, fr, it, nl, pl, ru, hu) and silently falls back to the English row, so pl_PL inherits the English FRE constants and pt uses English constants with a Portuguese hyphenation dictionary (spec §4.5.1/§4.5.3). The default is en_US.

    gunning_fog

    source
    fn gunning_fog(text : String, lang? : String) -> Double raise TextstatError

    Gunning Fog Index (spec §4.4 table row 3; PRD-A FR-18; upstream _gunning_fog.py:12-36).

    The expression is the upstream one, verbatim:

    syllable_threshold = int(get_lang_cfg(lang, "syllable_threshold"))
diff_words = count_difficult_words(text, lang, syllable_threshold)
tot_words = count_words(text)
if tot_words == 0: return 0.0
per_diff_words = 100 * diff_words / tot_words
return 0.4 * (words_per_sentence(text) + per_diff_words)

    Difficult words are non-unique at the language-root threshold. The threshold comes from lang_config(lang).syllable_threshold — root-resolved, so en/en_US/en_GB and every root without an override use 3, pl_PL uses 4 and hu_HU uses 5 (spec §4.3.4, §4.5.1). The counter is the package-private count_difficult_words(..., unique=false), not the public difficult_words field, which defaults to threshold 2 and unique=true. E-28/A-3.3 pin the difference on the 11-word fixture: at threshold 3 the non-unique count is 9, the unique count is 4, and 0.4 * (11 + 100 * 9 / 11) is 37.127272727272725; passing the unique count would yield 18.94545454545455.

    Evaluation order. 100 * diff_words is the integer product, converted to Double only for the division, matching Python's int multiply followed by true division; (diff_words / tot_words) * 100 moves the hyphenated_words and numbers_and_symbols samples by 1 ulp. The final 0.4 * (wps + per_diff_words) adds before multiplying; the reassociated 0.4 * wps + 0.4 * per_diff_words moves long_multi_sentence_paragraph and numbers_and_symbols by 1 ulp. Spec §4.4 constraint 1 forbids both rewrites, so the tree is kept as upstream wrote it.

    Zero branch. If tot_words is 0 the function returns 0.0 after the difficult count, exactly as upstream's except ZeroDivisionError. The empty string, whitespace-only text and punctuation-only text all tokenize to no words and land here. Because the difficult counter runs first and finds no tokens, it never resolves a hyphenation dictionary: these inputs return 0.0 even for an unsupported lang.

    Failure channel. A non-easy word reaches the syllable layer inside the difficult counter, so a hard non-empty text under ar/xx_YY raises the product-facing MetricFailure naming this metric, the code and the missing dictionary before the zero guard can hide it (upstream raises KeyError: None; spec §4.5.4, A-5.4). Text made only of easy words — e.g. hello — stays on the easy-list branch and does not fail; upstream returns 0.4 there and this port matches.

    The default is en_US.

    letter_count

    source
    fn letter_count(text : String) -> Int

    Count the letters of text (spec §4.3.1 row 2; upstream _count_letters.py:25-26).

    Upstream deletes every Python \s character (re.sub(r"\s", "", text)), then deletes every character matched by [^\w\s] (remove_punctuation with rm_apostrophe=True) and returns len of the result. Whitespace is already gone at that point, so the length is exactly the number of \w characters: one pass over the code points is enough. Digits and _ count, punctuation (-, ., ', …) does not, and empty / whitespace-only / punctuation-only input yields 0.

    Python's \w is Unicode-aware (alphanumerics plus underscore), while MoonBit has no Unicode letter classification. This port therefore accepts ASCII alphanumerics, _, and every non-ASCII code point up to U+FFFF, after rejecting whitespace (which upstream removes in either pass). The frozen oracle corpus contains exactly one non-ASCII character, ó (U+00F3, a letter), so this approximation reproduces every frozen cell exactly. Known limits, none present in the frozen cells or the supported languages' common text: BMP punctuation or symbols (e.g. «) would be over-counted, and astral letters would be under-counted. Astral non-letters such as emoji are correctly excluded — Python's "🤣".isalnum() is False — because their code points are above U+FFFF. The function is pure: no I/O, no exceptions.

    lexicon_count

    source
    fn lexicon_count(text : String, removepunct? : Bool, split_contractions? : Bool, split_hyphens? : Bool) -> Int

    Count the words of text (spec §4.3.1 row 3; upstream _count_words.py:39-46).

    Literally len(list_words(text, rm_punctuation, split_contractions,split_hyphens)): list_words (list_words.mbt) already replicates the five upstream tokenization steps in order, so this entry only forwards the three documented switches and takes the token count. Nothing is filtered, stemmed or re-tokenized afterwards; upstream returns len(...) unconditionally.

    Defaults reproduce the frozen oracle's default column: hyphens fuse (well-known is one word), contraction apostrophes survive (aren't is one word), and punctuation is deleted, so punctuation-only text counts 0. removepunct=false stops the deletion, so punctuation survives inside tokens and " .? " counts as one word. split_hyphens=true runs before punctuation removal and turns each ASCII - into a separator, so well-known counts as two words. split_contractions=true replaces the apostrophe of a contraction with a space after punctuation removal, so aren't counts as aren + t; the split is case-sensitive and is a no-op when no apostrophe survived (spec §4.3.2).

    Token-level detail (the \w/\s predicates and their documented approximations) is stated once in list_words.mbt; this function adds no rule of its own. The count is 32-bit (Int), the counting type of TextStats.lexicon_count (PRD-A 10.3). The function is pure: no I/O, no exceptions.

    linsear_write_formula

    source
    fn linsear_write_formula(text : String, lang? : String, strict_lower? : Bool, strict_upper? : Bool) -> Double raise TextstatError

    Linsear Write (spec §4.4 table row 7; PRD-A FR-22; upstream _linsear_write_formula.py:12-83).

    The body is the upstream one, verbatim:

    text_list = list_words(text, rm_punctuation=False)
if strict_upper and len(text_list) > 100:
    words_list = []; i_text = 0
    while i_text < len(text_list) and len(words_list) < 100:
        word = remove_punctuation(text_list[i_text], rm_apostrophe=False)
        i_text += 1
        if len(word) > 0: words_list.append(word)
else:
    words_list = list_words(text, rm_punctuation=True)
    i_text = len(text_list)
if strict_lower and len(words_list) < 100: return 0.0
easy = 0; difficult = 0
for word in words_list:
    n = count_syllables(word, lang)
    if n >= 3: difficult += 1
    elif n > 0: easy += 1
text_trunc = " ".join(text_list[:i_text])
if count_sentences(text_trunc) == 0: return 0.0
number = float((easy * 1 + difficult * 3) / count_sentences(text_trunc))
if number <= 20: number -= 2
return number / 2

    Two tokenizations, two roles. text_list is the punctuation-preserving whitespace split (rm_punctuation=false); it only ever feeds the sentence denominator. words_list is the punctuation-free split (rm_punctuation=true) and feeds the syllable classification. They differ on punctuation-only tokens, which advance i_text but never reach words_list.

    Truncation (spec §4.4 constraint 3). The prefix is taken only when strict_upper and text_list holds more than 100 tokens. The loop stops after 100 words, not tokens: a punctuation-only token such as ".?" is consumed (i_text += 1) and dropped, so i_text can exceed 100 and is the count of consumed tokens. The denominator is count_sentences of the space-joined consumed prefix — not of text and not of words_list — so a cut that lands mid-sentence counts that partial sentence, and the re-join collapses newlines and tabs inside the prefix to single spaces exactly like Python's " ".join. With strict_upper=false (or at most 100 tokens) the prefix is the whole text_list and words_list is the untruncated split.

    strict_lower order. The guard runs after the truncation decision and tests the resulting words_list. Under the default strict_upper=true, a text longer than 100 tokens truncates to exactly 100 words and therefore passes the guard; a short text keeps its own word count and returns 0.0. With strict_upper=false the guard sees the full word count.

    Classification and arithmetic. A word with three or more syllables is difficult, one or two syllables easy, and a syllable count of 0 contributes to neither branch (unreachable from the public surface: every entry of words_list is non-empty, and syllables_for_word always returns at least 1). The numerator is easy + 3 * difficult in integer arithmetic, then divided by the sentence count as a Double — Python's int / int true division, not integer division. If the quotient is at most 20 it is reduced by 2, and the result is finally halved. Both steps stay in Double, so number / 2 is number / 2.0.

    Zero branch. count_sentences is 0 only for the empty prefix, which is reachable only through the empty string: every non-empty prefix yields at least 1, and the truncation branch requires more than 100 tokens. A wordless non-empty text is not zero: punctuation_only has one sentence, no words and no syllables, so 0 / 1 is 0.0, the <= 20 branch subtracts 2 and the result is exactly -1.0 (spec §4.3.3).

    Failure channel. syllables_for_word resolves the languages per word, so any text whose words_list is non-empty raises the product-facing MetricFailure naming this metric, the language code and the missing dictionary for ar/xx_YY before the sentence denominator is computed (spec §4.5.4, A-5.4). Zero-word texts — the empty string, whitespace-only and punctuation-only input — never resolve a dictionary and keep 0.0 / 0.0 / -1.0 for every language, including unsupported codes. Verified against the vendored upstream (0.7.13): linsear_write_formula("", "ar") is 0.0, (" \n\t ", "ar") is 0.0, (" .? ", "ar") is -1.0 and ("Hi there!","ar") raises KeyError: None.

    The entries of words_list from the truncation branch keep their original case, whereas upstream's count_syllables lowercases internally; the pronunciation table and the hyphenation engine lowercase ASCII themselves, so the two agree for ASCII input (spec §4.3.2's approximation). The function is pure: no I/O. The defaults are en_US, strict_lower=false and strict_upper=true (spec §4.2 constraint 3).

    Registered exception to global constraint 2 ("公式层只吃计数层"). This formula calls the counting-layer helpers list_words and remove_punctuation inside its own body (lines below: the two list_words calls and the remove_punctuation call in the truncation loop) instead of consuming precomputed counts. That is upstream-mandated and not a design choice of this port: textstat/textstat/backend/metrics/_linsear_write_formula.py:12-83 does exactly this, and the formula's contract needs which words were seen — the syllable classification of each of the first 100 words plus the consumed-token count that becomes the sentence denominator — which no aggregate count exposes. Reproducing the upstream numbers (spec §4.4 constraint 3) therefore requires re-tokenizing here. Registered in the deviation register of cmd/gate/main.mbt; every other formula layer consumes counts only.

    long_word_count

    source
    fn long_word_count(text : String, threshold? : Int) -> Int

    Count the words longer than threshold characters (spec §4.3.1 row 10; upstream _count_long_words.py:23).

    Literally len([w for w in list_words(text, rm_apostrophe=True) iflen(w) > threshold]). The tokenizer is called with only rm_apostrophe=true; the other four switches keep their defaults (rm_punctuation=true, lowercase=false, split_contractions=false, split_hyphens=false), so this entry forwards exactly one flag and adds no rule of its own. Every behavior below was verified against the vendored upstream 0.7.13:

    • Punctuation is deleted before anything else and apostrophes go with it, so don't becomes dont (4 characters): it counts above threshold 3 but not above 4. Keeping the apostrophe would make it 5 and change the threshold-4 answer, which is exactly what rm_apostrophe=true rules out.
    • No lowercasing runs (lowercase=false). The length is case-insensitive, so this only matters for the token text, not the count.
    • Hyphens are not split (split_hyphens=false), so well-known is a single fused token wellknown (9 characters) and counts above 6, whereas splitting it would yield well (4) and known (5) and count nothing.
    • Digits are word characters and count toward the length: $1,299.99 collapses to the single token 129999 (6 characters), and 1234567 is 7 characters. Non-ASCII BMP letters also survive (fórmulas is 8).
    • The comparison is strict >: a token of exactly threshold characters is not counted. With the default threshold=6 only 7+ character tokens count, so twelve (6) is 0 while seventeen (9) is 1.

    Length is Python's len(str) over code points, hence String::char_length() rather than the UTF-16 String::length() (the same distinction documented in char_count.mbt). Astral code points never reach this comparison because the tokenizer's \w approximation excludes them (see is_python_word_char in letter_count.mbt), so the two lengths agree on every token this function can see.

    The metric is language-independent: there is no lang parameter and no dictionary or hyphenation lookup, so no TextstatError can arise (unlike the syllable-based counts). Empty, whitespace-only and punctuation-only input yields 0 because the token list is empty (spec §4.3.3).

    The count is 32-bit (Int), the counting type of TextStats.long_word_count (PRD-A 10.3). The function is pure: no I/O, no exceptions.

    monosyllabcount

    source
    fn monosyllabcount(text : String, lang? : String) -> Int raise TextstatError

    Count the words with exactly one syllable (spec §4.3.1 row 7; upstream _count_monosyllable_words.py:24).

    Upstream's whole implementation is len([w for w in list_words(text) if count_syllables(w, lang) == 1]). The tokenizer keeps its defaults (lowercase=false, punctuation removed, hyphens and contractions fused), yet the syllable lookup is effectively lowercase: count_syllables treats its argument as text and re-tokenizes it with lowercase=True. syllables_for_word applies the same lowercase inside the pronunciation table and the hyphenation engine, so the caller's word is passed through unchanged and no case handling is added here.

    The comparison is == 1, not <= 1: a word of zero syllables would not be counted, which matches upstream's equality test exactly.

    The language is resolved per word, inside syllables_for_word, where upstream's count_syllables builds its dictionaries. Text with no words therefore never resolves the language: "", whitespace-only and punctuation-only input return 0 even for an unsupported code, because upstream's comprehension is empty and count_syllables is never called. Verified against the vendored upstream (0.7.13): under ar, monosyllabcount returns 0 for "", " " and " .? ", while any text with a word raises there and now surfaces the product-facing MetricFailure naming this metric, the code and the missing dictionary (spec §4.5.4, A-5.4). This differs from syllable_count, whose entry resolves the language before its loop.

    The result is 32-bit (Int), the counting type of TextStats.monosyllabcount (PRD-A 10.3). The function is pure: no I/O.

    polysyllabcount

    source
    fn polysyllabcount(text : String, lang? : String) -> Int raise TextstatError

    Count the words with three or more syllables (spec §4.3.1 row 6; upstream _count_polysyllable_words.py:26).

    Upstream's whole implementation is len([w for w in list_words(text) if count_syllables(w, lang) >= 3]). The tokenizer keeps its defaults (lowercase=false, punctuation removed, hyphens and contractions fused), yet the syllable lookup is effectively lowercase: count_syllables treats its argument as text and re-tokenizes it with lowercase=True. syllables_for_word applies the same lowercase inside the pronunciation table and the hyphenation engine, so the caller's word is passed through unchanged and no case handling is added here.

    The language is resolved per word, inside syllables_for_word, where upstream's count_syllables builds its dictionaries. Text with no words therefore never resolves the language: "", whitespace-only and punctuation-only input return 0 even for an unsupported code, because upstream's comprehension is empty and count_syllables is never called. Verified against the vendored upstream (0.7.13): under ar, polysyllabcount returns 0 for "", " " and " .? ", while any text with a word raises there and now surfaces the product-facing MetricFailure naming this metric, the code and the missing dictionary (spec §4.5.4, A-5.4). This differs from syllable_count, whose entry resolves the language before its loop.

    The result is 32-bit (Int), the counting type of TextStats.polysyllabcount (PRD-A 10.3). The function is pure: no I/O.

    reading_time

    source
    fn reading_time(text : String, ms_per_char? : Double) -> Double

    Reading time in seconds (spec §4.2 unit: seconds).

    Literally ms_per_char * count_chars(text, ignore_spaces=True) / 1000 (upstream backend/metrics/_reading_time.py). Python evaluates the multiplication before the division, so the expression keeps that order: ms_per_char * chars / 1000, never ms_per_char * (chars / 1000).

    char_count already skips Python whitespace, so empty, whitespace-only and punctuation-only inputs follow spec §4.3.3 (0.0 / 0.0 / 0.02938 with the default 14.69 ms per character). The metric is language-independent: no lang parameter, no dictionary lookup, no failure channel.

    sentence_count

    source
    fn sentence_count(text : String) -> Int

    Count the sentences of text (spec §4.3.1 row 4; upstream _count_sentences.py:24-32).

    Upstream returns 0 for the empty string and otherwise iterates re.findall(r"\b[^.!?]+[.!?]*", text, re.UNICODE), discarding candidates whose default word count is at most two and returning max(1, candidates - discarded). This port reproduces that scan without a regex library, left to right and non-overlapping exactly like findall: at each position p it tries to match the pattern starting there; on success it records the candidate and resumes at the match end, otherwise it advances p by one code point.

    The three parts of the pattern are reproduced literally:

    1. \b is a word boundary at p: exactly one of the code points before and at p is a word character, with the start and end of the text counted as non-word. is_python_word_char (letter_count.mbt) supplies the word-character approximation; its doc comment states the limits.
    2. [^.!?]+ consumes one or more code points that are not ., ! or ?. Only these three ASCII terminators split a sentence — newlines, ;, : and … (U+2026) are ordinary content (spec §4.3.2).
    3. [.!?]* consumes the whole run of terminators that follows.

    A boundary in front of a terminator (a word character followed by ., ! or ?) starts no match, because [^.!?]+ cannot consume the terminator itself; the scan advances one position instead.

    Each candidate is word-counted with the package-private list_words under its documented defaults (removepunct=true, no lowercasing, no contraction or hyphen splitting), which is the same call upstream's count_words(sentence) makes. Candidates with at most two words are discarded, and the result is max(1, candidates - discarded), so every non-empty text yields at least 1: whitespace-only and punctuation-only input both yield 1 while the empty string yields 0 (spec §4.3.3). The function is pure: no I/O, no exceptions.

    smog_index

    source
    fn smog_index(text : String, lang? : String) -> Double raise TextstatError

    SMOG Index (spec §4.4 table row 4; PRD-A FR-19; upstream _smog_index.py:9-40).

    The expression is the upstream one, verbatim:

    sentences = count_sentences(text)
poly_syllab = count_polysyllable_words(text, lang)
return (1.043 * (30 * (poly_syllab / sentences)) ** 0.5) + 3.1291

    Evaluation order. poly_syllab / sentences is a true division of the two counts, then multiplied by 30, raised to the power 0.5, multiplied by 1.043 and finally increased by 3.1291. Spec §4.4 constraints 1 and 2 forbid rewriting the tree: the division is not integer division, and the power keeps the x ** 0.5 semantics, written @math.pow(30.0 * ratio, 0.5). The builtin Double::pow used in the plan is deprecated in toolchain 0.1.20261012 in favour of @math.pow — both are the same fdlibm power routine, only the spelling changed, and the package imports moonbitlang/core/math to keep the build warning-free. sqrt happens to produce the same bits for all eleven frozen samples (the domains are 0, 10, 36, 50, 105 and 144), but spec §4.4 constraint 2 forbids substituting it; the reopening condition is a bit difference, and then the oracle bits win. Multiplying 30 by the ratio rather than by the numerator is also kept as upstream wrote it.

    Zero branch. Upstream computes both counts first and only the division sits inside try: ... except ZeroDivisionError: return 0.0, so the polysyllable count runs before the guard. The empty string is the only input whose sentence count is 0 (every non-empty text yields at least 1), and it tokenizes to no words, so the guard returns 0.0 for every language.

    Degenerate inputs (spec §4.3.3 / A-5.1). Whitespace-only and punctuation-only text is non-empty: the sentence count is 1, there are no words, hence no polysyllables, the ratio is 0.0, 0.0 ** 0.5 is 0.0 and the result is exactly 3.1291. Only the empty string returns 0.0.

    Failure channel. The polysyllable counter resolves the language per word, so a text with at least one word under ar/xx_YY raises the product-facing MetricFailure naming this metric, the code and the missing dictionary from the count before the guard can run (spec §4.5.4, A-5.4); zero-word text never resolves a dictionary and keeps the values above even for an unsupported code. Verified against the vendored upstream (0.7.13): smog_index("", "ar") is 0.0, smog_index(" ", "ar") is 3.1291, smog_index(" .? ", "ar") is 3.1291 and ("Hi there!", "ar") raises KeyError: None.

    The counts are 32-bit (Int), the counting type of TextStats.polysyllabcount (PRD-A 10.3), and are widened to Double only for the division, the sole place where the ratio is formed. The function is pure: no I/O. The default is en_US.

    syllable_count

    source
    fn syllable_count(text : String, lang? : String) -> Int raise TextstatError

    Count the syllables of text (spec §4.3.1 row 5; upstream _count_syllables.py:23-34).

    The empty string returns 0 before any language lookup, exactly upstream's if not text: return 0, which precedes both get_cmudict and get_pyphen. Verified against the vendored upstream (0.7.13): syllable_count("", "ar") and syllable_count("", "xx_YY") both return 0, so the empty string never reaches the failure channel even for an unsupported language.

    Any other text resolves the language first, the position where upstream constructs Pyphen(lang=...) before its word loop. This is observable on non-empty input that tokenizes to nothing: upstream still raises for an unsupported language on " .? " or " ", so a plain sum over zero words must not turn the failure into 0.

    The tokens come from list_words(text, lowercase=true) under its documented defaults: punctuation is removed (so punctuation-only and whitespace-only text sum over no words and give 0) and hyphens stay fused, because upstream hyphenates the compound as one word. The per-word count is syllables_for_word (T-16): pronunciation table first for English roots, hyphenation positions + 1 otherwise, raising UnsupportedLanguage(requested) from the resolved dictionary.

    The language is a formal parameter, not upstream's deprecated no-effect argument, and there is no silent fallback (spec §4.5.1): an unsupported code fails with the product-facing MetricFailure, whose text names this metric, the language code and the missing hyphenation dictionary (spec §4.5.4, A-5.4). The low-level syllable_count_low keeps the raw UnsupportedLanguage for internal callers such as syllables_per_word. The count is 32-bit (Int), the counting type of TextStats.syllable_count (PRD-A 10.3). The function is pure: no I/O.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io