moongettext

    Pure MoonBit GNU gettext catalog parser, compiler, and runtime lookup library.

    gettext
    i18n
    po
    pot
    mo
    localization
    Download zip
    Version
    0.1.1
    License
    Apache-2.0
    Last updated
    2 months ago
    Downloads
    70

    #moongettext

    Pure MoonBit tooling for GNU gettext catalogs.

    moongettext parses and writes PO/POT text, compiles and reads MO revision 0 binaries, evaluates Plural-Forms, validates catalogs, merges templates, and provides context-aware runtime lookup. The library code is backend-neutral and is checked on MoonBit's wasm, wasm-gc, js, and native stable targets.

    The file-oriented CLI uses moonbitlang/x/fs, so that package is intentionally native-only.

    #Why this package?

    Applications often need a translation format that translators and existing localization platforms already understand. PO is reviewable text, MO is a compact runtime format, and the plural-expression language covers grammatical rules that cannot be represented by a simple singular/plural Boolean.

    moongettext keeps those concerns in one typed model:

    • five PO comment classes: translator, extracted, reference, flag, previous;
    • contexts, singular and plural entries, multiline strings, standard C escapes;
    • obsolete #~ entries and fuzzy filtering;
    • deterministic semantic PO/POT serialization;
    • GNU MO revision 0 read/write in both byte orders;
    • a C-like Plural-Forms parser with short-circuit and ternary evaluation;
    • runtime gettext, pgettext, ngettext, and npgettext operations;
    • explicit catalog fallback;
    • validation, statistics, source-reference queries, and POT/PO merging.

    #Install

    Install the package with Moon:

    moon add YanFangNie/moongettext

    For this source checkout, no global installation is required:

    moon check --target all --deny-warn --warn-list +73 moon test --target all --deny-warn --warn-list +73 moon run examples/basic

    The module pins moonbitlang/x for the native file CLI. The root library itself uses only MoonBit core APIs.

    #Quick start: parse and normalize PO

    parse_po raises a structured GettextError for malformed input. write_po normalizes layout but preserves the represented messages and comments.

    ///|
    test "parse and normalize a PO entry" {
    let source =
    #|#. Shown on the launch screen
    #|#: src/main.mbt:12
    #|msgctxt "button"
    #|msgid "Open"
    #|msgstr "Ouvrir"
    #|
    let document = @moongettext.parse_po(source)
    assert_eq(document.entries.length(), 1)
    assert_true(
    document.find_entry("Open", context="button") is Some(entry) &&
    entry.translation(0) == Some("Ouvrir"),
    )
    let reparsed = @moongettext.parse_po(@moongettext.write_po(document))
    assert_true(reparsed == document)
    }

    PO serialization is a semantic round trip, not a byte-for-byte formatter: indentation, blank lines, quoting, and multiline wrapping are canonicalized.

    #Runtime lookup

    Catalog construction reads Plural-Forms from the empty-msgid metadata header. Missing translations return source text. Fuzzy and obsolete entries are excluded from runtime catalogs by default.

    ///|
    test "context and plural-aware lookup" {
    let document = @moongettext.PoFile::new([
    @moongettext.PoEntry::singular(
    "",
    translation=(
    #|Content-Type: text/plain; charset=UTF-8
    #|Language: fr
    #|Plural-Forms: nplurals=2; plural=(n > 1);
    #|
    ),
    ),
    @moongettext.PoEntry::singular(
    "Open",
    translation="Ouvrir",
    context="button",
    ),
    @moongettext.PoEntry::plural("file", "files", ["fichier", "fichiers"]),
    ])
    let catalog = @moongettext.Catalog::from_po(document)
    assert_eq(catalog.pgettext("button", "Open"), "Ouvrir")
    assert_eq(catalog.ngettext("file", "files", 1), "fichier")
    assert_eq(catalog.ngettext("file", "files", 3), "fichiers")
    assert_eq(catalog.gettext("missing"), "missing")
    }

    Fallback is explicit:

    ///|
    test "fallback catalog" {
    let primary = @moongettext.Catalog::from_po(
    @moongettext.PoFile::new([
    @moongettext.PoEntry::singular("Hello", translation="Salut"),
    ]),
    )
    let fallback = @moongettext.Catalog::from_po(
    @moongettext.PoFile::new([
    @moongettext.PoEntry::singular("Open", translation="Ouvrir"),
    ]),
    )
    assert_eq(primary.gettext("Hello", fallback~), "Salut")
    assert_eq(primary.gettext("Open", fallback~), "Ouvrir")
    assert_eq(primary.gettext("Unknown", fallback~), "Unknown")
    }

    #MO compile and load

    compile_mo_checked first rejects validation errors and then emits deterministic GNU MO revision 0 bytes. Obsolete, fuzzy, and untranslated non-header entries are omitted. parse_mo validates all table ranges, NUL terminators, duplicate keys, and UTF-8 before exposing a document.

    ///|
    test "compile and load big-endian MO" {
    let source =
    #|msgid ""
    #|msgstr "Content-Type: text/plain; charset=UTF-8\n"
    #|
    #|msgid "Save"
    #|msgstr "Enregistrer"
    #|
    let po = @moongettext.parse_po(source)
    let bytes = @moongettext.compile_mo_checked(po, endian=Big)
    assert_true(@moongettext.mo_endian(bytes) == Big)
    let loaded = @moongettext.Catalog::from_mo(bytes)
    assert_eq(loaded.gettext("Save"), "Enregistrer")
    }

    Use compile_mo instead of compile_mo_checked only when a caller intentionally handles validation separately.

    #Plural-Forms

    Supported operators, from high to low precedence:

    1. primary values: integer, n, parentheses;
    2. unary !, unary +, unary -;
    3. *, /, %;
    4. +, -;
    5. <, <=, >, >=;
    6. ==, !=;
    7. &&;
    8. ||;
    9. right-associative ?:.

    Logical operations short-circuit and produce 0 or 1.

    ///|
    test "Russian three-form expression" {
    let rule = @moongettext.parse_plural_forms(
    "nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;",
    )
    assert_eq(rule.select(1), 0)
    assert_eq(rule.select(2), 1)
    assert_eq(rule.select(5), 2)
    assert_eq(rule.select(21), 0)
    }

    Negative counts, division/modulo by zero, malformed syntax, and results outside [0, nplurals) are reported as errors.

    #Validate a catalog

    validate_po returns every finding in stable order. It checks:

    • metadata charset and Plural-Forms;
    • duplicate context-aware keys;
    • plural translation arity;
    • embedded MO separator characters;
    • fuzzy, obsolete, and untranslated entries;
    • representative evaluation of plural rules for counts 0..=200.

    ///|
    test "structured validation report" {
    let invalid = @moongettext.PoFile::new([
    @moongettext.PoEntry::plural("item", "", [""]),
    ])
    let issues = @moongettext.validate_po(invalid)
    assert_true(@moongettext.validation_has_errors(issues))
    let report = @moongettext.validation_report(issues)
    assert_true(report.contains("empty-plural-id"))
    }

    validate_po_strict and compile_mo_checked raise if any IssueError exists. Warnings do not block compilation.

    #Merge a POT template

    merge_template(template, existing) retains translations for matching context/msgid keys, refreshes extracted comments and source references, keeps translator notes, marks changed plural sources fuzzy, and carries removed messages as obsolete.

    ///|
    test "merge template with translated PO" {
    let template = @moongettext.PoFile::new([
    @moongettext.PoEntry::singular("Open"),
    @moongettext.PoEntry::singular("New"),
    ])
    let existing = @moongettext.PoFile::new([
    @moongettext.PoEntry::singular("Open", translation="Ouvrir"),
    @moongettext.PoEntry::singular("Old", translation="Ancien"),
    ])
    let merged = @moongettext.merge_template(template, existing)
    assert_true(merged.stats == { matched: 1, added: 1, obsoleted: 1 })
    assert_eq(merged.document.entries[0].translations[0], "Ouvrir")
    assert_true(merged.document.entries[2].obsolete)
    }

    Set keep_obsolete=false to omit removed entries.

    #CLI

    The native CLI is a reproducible example and a useful catalog probe:

    moon run cmd/main -- demo moon run cmd/main -- validate examples/fr.po moon run cmd/main -- normalize examples/fr.po moon run cmd/main -- compile examples/fr.po _build/fr.mo moon run cmd/main -- compile examples/fr.po _build/fr-be.mo --big-endian moon run cmd/main -- inspect _build/fr.mo

    validate returns exit status 1 for error findings. compile refuses error-level findings and writes no partial output. Run moon run cmd/main -- help for the same command summary.

    The backend-neutral example is:

    moon run examples/basic

    Expected output:

    Ouvrir 0: fichier 1: fichier 2: fichiers 5: fichiers

    #Main public API

    AreaEntry points
    PO/POTparse_po, write_po, write_pot, PoFile::as_template
    MOcompile_mo, compile_mo_checked, parse_mo, mo_endian
    Pluralsparse_plural_forms, evaluate_plural_expression, PluralRule::select
    RuntimeCatalog::from_po, Catalog::from_mo, gettext, pgettext, ngettext, npgettext
    Mergemerge_template, MergeResult, MergeStats
    Validationvalidate_po, validate_po_strict, validation_report
    Queryfind_entry, flags, source_references, statistics, metadata

    Run moon info to regenerate pkg.generated.mbti, which is the exact public surface for the checked-out version.

    #Compatibility and limits

    See docs/compatibility.md for the detailed matrix. Important boundaries:

    • PO output preserves semantics, not original whitespace or line wrapping.
    • MO revision 0 is supported; system-dependent revision extensions are not.
    • MO strings must be UTF-8. Other declared charsets are reported.
    • MO hash tables are accepted as header fields but ignored; emitted files use binary-search-compatible sorted original tables and no hash table.
    • Advanced quoted source-reference filenames are retained as comment text but the convenience source_references tokenizer does not reconstruct them.
    • This package does not extract translatable strings from MoonBit source and does not replace a translation-management platform.

    #Project layout

    model / escape shared PO model and string escaping po_parser / po_writer semantic PO/POT codec plural_lexer / plural_parser Plural-Forms grammar and evaluator mo_reader / mo_writer strict MO revision 0 codec catalog indexed runtime lookup and fallback validation / query / merge tooling APIs cmd/main native file CLI examples/basic backend-neutral runnable example

    #Development and release checks

    moon version --all moon fmt --check moon info --target all moon check --target all --deny-warn --warn-list +73 moon test --target all --deny-warn --warn-list +73 moon build --target native moon run examples/basic moon run cmd/main -- validate examples/fr.po moon package --frozen git diff --exit-code

    CI runs the same format/interface/check/build/test/example/CLI/package gates.

    #Specification and source notice

    This is an original MoonBit implementation informed by the public GNU gettext format specifications and observable behavior. It is not a line-by-line port of an existing codebase:

    The ecosystem survey also found Zhouz-z/moon_l10n, which focuses on an ICU MessageFormat subset and catalog linting. moongettext instead focuses on GNU PO/POT/MO interoperability and gettext-style runtime lookups; the projects have no implementation-source relationship.

    No GNU gettext implementation source, generated code, or third-party catalog fixture is copied into this repository. Test inputs are small original fixtures. The project is licensed under Apache-2.0; see LICENSE.

    GettextError

    pub(all) suberror GettextError {
    PoSyntax(line~ : Int, column~ : Int, message~ : String)
    PluralSyntax(position~ : Int, message~ : String)
    MoFormat(offset~ : Int, message~ : String)
    Validation(message~ : String)
    } derive(Eq,
    Debug
    )

    A structured error raised while reading, compiling, or validating a gettext catalog.

    Catalog

    pub struct Catalog {
    messages : Map[String, PoEntry]
    rule : PluralRule
    }

    An indexed runtime translation catalog.

    Construct catalogs with Catalog::from_po or Catalog::from_mo; fields are intentionally private so duplicate-key and plural-rule checks cannot be bypassed.

    Catalog::from_mo

    fn Catalog::from_mo(bytes : Bytes) -> Catalog raise GettextError

    Build a runtime catalog directly from GNU MO bytes.

    Catalog::from_po

    fn Catalog::from_po(document : PoFile, default_rule? : PluralRule, include_fuzzy? : Bool) -> Catalog raise GettextError

    Build a runtime catalog from a parsed PO document.

    Obsolete entries and the metadata header are excluded. Fuzzy entries are excluded by default and can be explicitly included for preview tooling. When no Plural-Forms header exists, the English rule (or default_rule) is used.

    Catalog::gettext

    fn Catalog::gettext(self : Catalog, msgid : String, fallback? : Catalog) -> String

    Look up a singular message, returning the source msgid when neither this catalog nor the optional fallback has a non-empty translation.

    Catalog::length

    fn Catalog::length(self : Catalog) -> Int

    Number of indexed message entries, excluding the metadata header.

    Catalog::ngettext

    fn Catalog::ngettext(self : Catalog, singular : String, plural : String, n : Int, fallback? : Catalog) -> String raise GettextError

    Look up a plural message using this catalog's Plural-Forms rule.

    If no translation is available, the fallback catalog is tried. Source text finally falls back to singular for n == 1 and plural otherwise.

    Catalog::npgettext

    fn Catalog::npgettext(self : Catalog, context : String, singular : String, plural : String, n : Int, fallback? : Catalog) -> String raise GettextError

    Look up a context-qualified plural message.

    Catalog::pgettext

    fn Catalog::pgettext(self : Catalog, context : String, msgid : String, fallback? : Catalog) -> String

    Look up a context-qualified singular message.

    Catalog::plural_rule

    fn Catalog::plural_rule(self : Catalog) -> PluralRule

    Return the catalog's validated plural rule.

    CommentKind

    pub(all) enum CommentKind {
    Translator
    Extracted
    Reference
    Flag
    Previous
    } derive(Eq,
    Debug
    )

    The semantic kind of a comment attached to a PO entry.

    GNU gettext distinguishes comments by the marker immediately following #: no marker for translator comments, . for extracted comments, : for source references, , for flags, and | for previous values.

    Endian

    pub(all) enum Endian {
    Little
    Big
    } derive(Eq,
    Debug
    )

    Byte order used when writing or reading a GNU MO file.

    IssueSeverity

    pub(all) enum IssueSeverity {
    IssueInfo
    IssueWarning
    IssueError
    } derive(Eq,
    Debug
    )

    Severity assigned to a catalog validation issue.

    MergeResult

    pub(all) struct MergeResult {
    document : PoFile
    stats : MergeStats
    } derive(Eq,
    Debug
    )

    Result of merging a POT-style template with an existing translation.

    MergeStats

    pub(all) struct MergeStats {
    matched : Int
    added : Int
    obsoleted : Int
    } derive(Eq,
    Debug
    )

    Summary of a template/catalog merge.

    PluralRule

    pub struct PluralRule {
    nplurals : Int
    expression : String
    } derive(Eq,
    Debug
    )

    A validated Plural-Forms rule.

    PluralRule::english

    fn PluralRule::english() -> PluralRule

    The conventional English rule: singular only when n == 1.

    PluralRule::select

    fn PluralRule::select(self : PluralRule, n : Int) -> Int raise GettextError

    Select a translation index and ensure the expression stays inside the declared [0, nplurals) range.

    PoComment

    pub(all) struct PoComment {
    kind : CommentKind
    text : String
    } derive(Eq,
    Debug
    )

    A comment attached to a PO entry.

    PoComment::new

    fn PoComment::new(kind : CommentKind, text : String) -> PoComment

    Construct a PO comment.

    PoEntry

    pub(all) struct PoEntry {
    comments : Array[PoComment]
    context : String?
    msgid : String
    msgid_plural : String?
    translations : Array[String]
    obsolete : Bool
    } derive(Eq,
    Debug
    )

    One message entry in a PO or POT catalog.

    translations stores msgstr as element zero for singular entries and stores msgstr[N] values by numeric index for plural entries. Missing indexes are represented by empty strings so indexes remain stable.

    PoEntry::flags

    fn PoEntry::flags(self : PoEntry) -> Array[String]

    Return unique comma-separated flags in first-seen order.

    PoEntry::has_flag

    fn PoEntry::has_flag(self : PoEntry, flag : String) -> Bool

    Return true when a flag comment contains the exact comma-separated flag.

    PoEntry::is_fuzzy

    fn PoEntry::is_fuzzy(self : PoEntry) -> Bool

    Return true when this entry has GNU gettext's fuzzy flag.

    PoEntry::is_header

    fn PoEntry::is_header(self : PoEntry) -> Bool

    Return true when this entry is the conventional metadata header.

    PoEntry::plural

    fn PoEntry::plural(msgid : String, msgid_plural : String, translations : Array[String], context? : String) -> PoEntry

    Construct a plural entry without comments.

    PoEntry::singular

    fn PoEntry::singular(msgid : String, translation? : String, context? : String) -> PoEntry

    Construct a singular entry without comments.

    PoEntry::source_references

    fn PoEntry::source_references(self : PoEntry) -> Array[SourceReference]

    Decode whitespace-separated #: source reference tokens.

    path, path:line, and path:line:column are recognized. GNU PO permits tool-specific quoting for paths containing whitespace; this helper keeps those advanced spellings as separate raw tokens rather than guessing.

    PoEntry::translation

    fn PoEntry::translation(self : PoEntry, index : Int) -> String?

    Return a translation by plural index, or None if it is absent or empty.

    PoFile

    pub(all) struct PoFile {
    entries : Array[PoEntry]
    } derive(Eq,
    Debug
    )

    A parsed PO or POT file.

    PoFile::as_template

    fn PoFile::as_template(self : PoFile) -> PoFile

    Return a POT-style copy with non-header translations cleared.

    The metadata header is retained because template generators commonly store MIME and project fields there. Singular entries receive one empty msgstr; plural entries retain at least two indexed slots.

    PoFile::find_entry

    fn PoFile::find_entry(self : PoFile, msgid : String, context? : String, include_obsolete? : Bool) -> PoEntry?

    Find a context-aware entry. Obsolete entries are ignored by default.

    PoFile::header

    fn PoFile::header(self : PoFile) -> PoEntry?

    Return the first conventional metadata header entry, if present.

    PoFile::metadata

    fn PoFile::metadata(self : PoFile) -> Map[String, String]

    Return parsed fields from the conventional empty-msgid header entry.

    PoFile::new

    fn PoFile::new(entries : Array[PoEntry]) -> PoFile

    Construct a catalog document from entries.

    PoFile::statistics

    fn PoFile::statistics(self : PoFile) -> PoStatistics

    Compute catalog status counts in one pass.

    PoStatistics

    pub(all) struct PoStatistics {
    headers : Int
    messages : Int
    active : Int
    translated : Int
    untranslated : Int
    fuzzy : Int
    obsolete : Int
    singular : Int
    plural : Int
    contextual : Int
    } derive(Eq,
    Debug
    )

    Aggregate status counts for a parsed PO/POT document.

    SourceReference

    pub(all) struct SourceReference {
    path : String
    line : Int?
    column : Int?
    } derive(Eq,
    Debug
    )

    A source location decoded from a #: PO reference comment.

    ValidationIssue

    pub(all) struct ValidationIssue {
    severity : IssueSeverity
    entry : Int?
    code : String
    message : String
    } derive(Eq,
    Debug
    )

    One deterministic catalog validation finding.

    compile_mo

    fn compile_mo(document : PoFile, endian? : Endian) -> Bytes raise GettextError

    Compile a PO document into GNU MO revision 0 bytes.

    Obsolete, fuzzy, and untranslated non-header entries are deliberately omitted, matching the behavior expected from a production message compiler. Original strings are sorted by their UTF-8 byte representation.

    compile_mo_checked

    fn compile_mo_checked(document : PoFile, endian? : Endian) -> Bytes raise GettextError

    Validate a document strictly and then compile it to GNU MO bytes.

    escape_po_string

    fn escape_po_string(input : String) -> String

    Escape a string for the body of a double-quoted PO string.

    The returned value does not include the surrounding quote characters.

    evaluate_plural_expression

    fn evaluate_plural_expression(expression : String, n : Int) -> Int raise GettextError

    Evaluate one GNU gettext plural expression for a non-negative count.

    merge_template

    fn merge_template(template : PoFile, existing : PoFile, keep_obsolete? : Bool) -> MergeResult raise GettextError

    Merge a POT-style template with an existing translated PO document.

    Source comments and references come from the template; translator and previous-value comments come from the existing catalog. Matching translations are retained. A changed plural source is marked fuzzy. Existing entries absent from the template are appended as obsolete unless keep_obsolete=false.

    mo_endian

    fn mo_endian(bytes : Bytes) -> Endian raise GettextError

    Detect byte order from a GNU MO magic word.

    parse_header_fields

    fn parse_header_fields(header : String) -> Map[String, String]

    Parse the translated string of a gettext header entry into metadata fields.

    Lines without a colon are ignored. A line beginning with a space or tab is treated as a continuation of the preceding field.

    parse_mo

    fn parse_mo(bytes : Bytes) -> PoFile raise GettextError

    Parse a GNU MO revision 0 file into the shared catalog model.

    Both little- and big-endian files are accepted. Header/table bounds, NUL terminators, UTF-8 strings, duplicate keys, context separators, and plural separators are checked before a document is returned.

    parse_plural_forms

    fn parse_plural_forms(input : String) -> PluralRule raise GettextError

    Parse a Plural-Forms value or a complete gettext metadata header.

    Both nplurals and plural assignments are required. Assignment order and surrounding whitespace are ignored.

    parse_po

    fn parse_po(source : String) -> PoFile raise GettextError

    Parse a GNU gettext PO or POT document.

    The parser accepts LF and CRLF input, multiline quoted strings, contexts, plurals, the five standard comment classes, and obsolete #~ entries. Parsed output is semantic rather than lossless: whitespace and line wrapping are normalized when the document is written again.

    unescape_po_string

    fn unescape_po_string(input : String) -> String raise GettextError

    Decode C-style escapes used inside a PO quoted string.

    Common named escapes, octal escapes of up to three digits, and hexadecimal escapes are supported. Unknown escapes follow gettext's permissive reader behavior and evaluate to the escaped character itself.

    validate_po

    fn validate_po(document : PoFile) -> Array[ValidationIssue]

    Validate PO/POT semantics used by the parser, MO compiler, and runtime.

    The returned array is stable in document order. Validation checks metadata, plural rules, duplicate context-aware keys, translation arity, fuzzy and obsolete status, embedded NULs, and context separator collisions.

    validate_po_strict

    fn validate_po_strict(document : PoFile) -> Unit raise GettextError

    Raise a single Validation error when a catalog has any error findings.

    validation_has_errors

    fn validation_has_errors(issues : ArrayView[ValidationIssue]) -> Bool

    Return true when at least one validation issue is an error.

    validation_report

    fn validation_report(issues : ArrayView[ValidationIssue]) -> String

    Format findings as a line-oriented CLI report.

    write_po

    fn write_po(document : PoFile) -> String

    Serialize a PO/POT document into deterministic GNU gettext syntax.

    Semantic information is preserved, while whitespace, entry separation, and multiline layout are normalized. Every entry is followed by one blank line.

    write_pot

    fn write_pot(document : PoFile) -> String

    Serialize a document as a translation template.