A port of Pygments (generic syntax highlighter) to MoonBit

syntax-highlighting

pygments

lexer

Download zip

Author

bobzhang

Version

0.2.0

License

BSD-2-Clause

Repository

github.com/bobzhang/pygments.mbt

Last updated

8 hours ago

Downloads

Dependencies

moonbitlang/async0.22.4
moonbitlang/x0.5.5

#pygments.mbt

A port of Pygments 2.21 — the generic syntax highlighter — to MoonBit.

All 615 lexers of upstream Pygments, token-for-token compatible: the 1,508 lexer tests of the upstream test suite plus 6,600 cross-fed inputs produce identical token streams, and every analyse_text/guess_lexer result matches.
A Python-re compatible regex engine (regex/): lookaround, backreferences, conditionals, inline/scoped flags, verbose mode, Unicode \w\d\s and case folding generated from CPython 3.14, a step budget against catastrophic backtracking. Differentially tested against CPython on all 8,838 patterns used by the lexers.
All non-image formatters (HTML, terminal/256/true-color, LaTeX, RTF, SVG, BBCode, IRC, groff, Pango, raw, testcase), all 50 styles and all 8 filters, byte-identical to Python on 5,480 differential cases.
A MoonBit lexer (moonbit, *.mbt, *.mbti, *.mbtx) — not in upstream Pygments.
Command line tools built on moonbitlang/async, runnable with moonx: scat (cat with syntax highlighting) and a Python-compatible pygmentize.

#Usage

///|
test "lex some Python" {
  let lexer = @lexers.get_lexer_by_name("python")
  let tokens = @pygments.lex("print('hi')\n", lexer)
  inspect(
    tokens.map(t => "\{t.0} \{t.1.escape()}").join("\n"),
    content=(
      #|Token.Name.Builtin "print"
      #|Token.Punctuation "("
      #|Token.Literal.String.Single "'"
      #|Token.Literal.String.Single "hi"
      #|Token.Literal.String.Single "'"
      #|Token.Punctuation ")"
      #|Token.Text.Whitespace "\n"
    ),
  )
}

Render with any formatter (html, terminal256, latex, rtf, svg, …):

///|
test "highlight to HTML" {
  let html = @pygments.highlight(
    "print('hi')\n",
    @pygments.get_lexer_by_name("python"),
    @pygments.get_formatter_by_name("html", options={ "noclasses": "True" }),
  )
  inspect(html.has_prefix("<div class=\"highlight\""), content="true")
}

Lexers can be looked up by alias, file name, MIME type or content:

///|
test "lexer lookup" {
  inspect(@lexers.get_lexer_for_filename("main.rs").name(), content="Rust")
  inspect(
    @lexers.guess_lexer("#!/usr/bin/env python3\nimport os\n").name(),
    content="Python",
  )
}

#Command line

scat prints files with syntax highlighting; the language comes from the file name, or is guessed from the contents:

moonx bobzhang/pygments/cmd/scat main.py             # 16-color, follows the terminal theme
moonx bobzhang/pygments/cmd/scat -s monokai -n lib.rs  # a style, with line numbers
cat build.log | moonx bobzhang/pygments/cmd/scat -l console
moonx bobzhang/pygments/cmd/scat --langs            # all 600+ languages

cmd/pygmentize mirrors Python's pygmentize (-l -g -f -O -P -F -o -s -S -a-L -N -C -H -V, --json); a differential script compares both implementations (scripts/cli_compare.py):

# run straight from mooncakes.io, no checkout needed
moonx bobzhang/pygments/cmd/pygmentize -f terminal256 -O style=monokai main.py
# or build it (native or wasm)
moon build --target native --release cmd/pygmentize
./_build/native/release/build/cmd/pygmentize/pygmentize.exe -f terminal256 -O style=monokai main.py

Not supported: -x (loading lexers/formatters from Python files) and the PIL image formatters (img, gif, jpg, bmp), which are listed but raise, as in Python without PIL.

#Layout

Package	Contents
regex/	Python-re compatible backtracking regex engine
token/	token type hierarchy (Token.Name.Builtin, …)
lexer/	lexer runtime: RegexLexer/ExtendedRegexLexer interpreter, DelegatingLexer, do_insertions, options
lexers/	all lexers: tables generated from upstream (gen_*.mbt) plus hand-written parts, registry
styles/, formatters/, filters/	styles, output formatters and token filters
cmd/scat	cat with syntax highlighting
cmd/pygmentize	Python-compatible pygmentize
pystr/	Python string semantics used by the formatters (repr, case mapping, …)
cmd/*_oracle	differential test drivers against Python

#How the port works

scripts/export_lexers.py imports the pinned upstream checkout (.repos/pygments), lets Pygments' own metaclass resolve include, inherit, words(), default and combined, and emits the processed state tables as MoonBit data. Parts that are Python code — callbacks, get_tokens_unprocessed overrides, non-regex lexers, analyse_text — are hand-ported in lexers/<module>.mbt and discovered by name; see PORTING.md and DESIGN.md.

Offsets reported by get_tokens_unprocessed are UTF-16 offsets (MoonBit strings), while matching itself is code-point based like Python.

#Testing against Python

The upstream checkout is pinned in scripts/PYGMENTS_COMMIT; CI runs the same scripts on every push.

moon test                       # unit tests (also --target js / wasm-gc)
scripts/conformance.sh          # differential tests against Python 3.14 + Pygments
scripts/regen.sh                # regenerate all generated MoonBit files

#License

format

source

fn format(tokens : Array[(@bobzhang/pygments/token.TokenType
TokenType, String)], formatter : @bobzhang/pygments/formatters.Formatter
Formatter) -> String raise

Python's pygments.format: renders a token stream with formatter.

get_formatter_by_name

source

fn get_formatter_by_name(name : String, options? : Map[String, String], style? : @bobzhang/pygments/styles.Style
Style) -> @bobzhang/pygments/formatters.Formatter
Formatter raise

Python's get_formatter_by_name: an instance of the formatter with alias, configured with options.

test {
  let f = @formatters.get_formatter_by_name("html", options={ "nowrap": "true" })
  inspect(
    f.format([(@token.keyword, "if")]),
    content="<span class=\"k\">if</span>\n",
  )
}

get_formatter_for_filename

source

fn get_formatter_for_filename(filename : String, options? : Map[String, String], style? : @bobzhang/pygments/styles.Style
Style) -> @bobzhang/pygments/formatters.Formatter
Formatter raise

Python's get_formatter_for_filename: an instance of the formatter whose file name patterns match the base name of filename.

test {
  inspect(
    @formatters.get_formatter_for_filename("out/a.tex").name(),
    content="LaTeX",
  )
}

get_lexer_by_name

source

fn get_lexer_by_name(lexer_alias : String, options? : Map[String, String]) -> @bobzhang/pygments/lexer.Lexer
Lexer raise

Python's get_lexer_by_name.

test {
  let lx = @lexers.get_lexer_by_name("py")
  inspect(lx.name(), content="Python")
}

get_lexer_for_filename

source

fn get_lexer_for_filename(filename : String, code? : String, options? : Map[String, String]) -> @bobzhang/pygments/lexer.Lexer
Lexer raise

Python's get_lexer_for_filename.

get_lexer_for_mimetype

source

fn get_lexer_for_mimetype(mime : String, options? : Map[String, String]) -> @bobzhang/pygments/lexer.Lexer
Lexer raise

Python's get_lexer_for_mimetype.

guess_lexer

source

fn guess_lexer(text : String, options? : Map[String, String]) -> @bobzhang/pygments/lexer.Lexer
Lexer raise

Python's guess_lexer: vim modeline first, then the best analyse_text.

guess_lexer_for_filename

source

fn guess_lexer_for_filename(filename : String, text : String, options? : Map[String, String]) -> @bobzhang/pygments/lexer.Lexer
Lexer raise

Python's guess_lexer_for_filename.

highlight

source

fn highlight(code : String, lexer : @bobzhang/pygments/lexer.Lexer
Lexer, formatter : @bobzhang/pygments/formatters.Formatter
Formatter) -> String raise

Python's pygments.highlight: lexes code with lexer and renders it with formatter.

test {
  let html = @pygments.highlight(
    "x = 1\n",
    @pygments.get_lexer_by_name("python"),
    @pygments.get_formatter_by_name("html"),
  )
  inspect(
    html,
    content=(
      #|<div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">1</span>
      #|</pre></div>
      #|
    ),
  )
}

lex

source

fn lex(code : String, lexer : @bobzhang/pygments/lexer.Lexer
Lexer) -> Array[(@bobzhang/pygments/token.TokenType
TokenType, String)] raise

Python's pygments.lex: the token stream of code (preprocessed and filtered by lexer).

test {
  let lexer = @lexers.get_lexer_by_name("python")
  let tokens = @pygments.lex("x = 1\n", lexer)
  inspect(tokens[0].0, content="Token.Name")
  inspect(tokens.map(t => t.1).join(""), content="x = 1\n")
}