Sign in

    markdown

    Incremental Markdown parser and compiler

    markdown
    parser
    cst
    incremental
    gfm
    Download zip
    Author
    Version
    0.4.9
    License
    MIT
    Last updated
    5 months ago
    Downloads
    21K

    #@mizchi/markdown

    CST-based incremental Markdown parser for JavaScript/MoonBit.

    A cross-platform (JS/WASM/native) Markdown compiler optimized for real-time editing with incremental parsing.

    #Features

    • Fast: Edit-position based incremental updates inspired by CRDTs Go Brrr. Optimized for speed over edge-case correctness CommonMark 207/542
    • Lossless CST: Preserves all whitespace, markers, and formatting
    • Incremental parsing: Re-parses only changed blocks (up to 42x faster)
    • GFM: GitHub Flavored Markdown support (tables, task lists, strikethrough)
    • Cross-platform: Works on JS, WASM-GC, and native targets
    • HTML rendering: Built-in HTML renderer with remark-html compatible output
    • mdast compatible: AST follows mdast specification


    #JavaScript API

    npm install @mizchi/markdown

    #Usage

    import { parse, toHtml, toMarkdown } from "@mizchi/markdown"; // Parse to AST const ast = parse("# Hello\n\n**Bold** text"); console.log(ast.children[0].type); // "heading" // Convert to HTML const html = toHtml("See https://example.com/docs\n"); // => '<p>See <a href="https://example.com/docs">https://example.com/docs</a></p>\n' // Disable bare URL links when you need plain text output const plain = toHtml("See https://example.com/docs\n", { autolink: false }); // => "<p>See https://example.com/docs</p>\n" // Normalize markdown const normalized = toMarkdown("# Hello\n\n\n\nWorld"); // => "# Hello\n\nWorld\n"

    WikiLinks are disabled by default to keep CommonMark-compatible behavior. Pass { wikilinks: true } to parse [[target]] and [[target|label]].

    import { parse, toHtml } from "@mizchi/markdown"; const ast = parse("[[MoonBit#syntax|MoonBit syntax]]", { wikilinks: true }); // ast.children[0].children[0].type === "wikiLink" const html = toHtml("[[MoonBit|MoonBit notes]]", { wikilinks: true }); // => '<p><a href="MoonBit">MoonBit notes</a></p>\n'

    #Incremental Parsing

    For real-time editing scenarios:

    import { createDocument, insertEdit } from "@mizchi/markdown"; // Create document handle const doc = createDocument("# Hello"); // Access AST, HTML, or Markdown console.log(doc.ast); // Parsed AST console.log(doc.toHtml()); // "<h1>Hello</h1>\n" console.log(doc.toMarkdown()); // "# Hello\n" // Incremental update (faster than full re-parse) const edit = insertEdit(7, 6); // Insert 6 chars at position 7 const newDoc = doc.update("# Hello World", edit); // Free resources when done doc.dispose(); newDoc.dispose();

    #TypeScript Support

    Full TypeScript definitions are included:

    import { parse, Document, Block, Inline } from "@mizchi/markdown"; const ast: Document = parse("# Hello"); const heading = ast.children[0] as HeadingBlock; console.log(heading.level); // 1


    #MoonBit API

    #Installation

    moon add mizchi/markdown

    #Usage

    // Parse markdown
    let result = @markdown.parse("# Hello\n\nWorld")
    let doc = result.document

    // Serialize back (lossless)
    let output = @markdown.serialize(doc)

    // Render to HTML
    let html = @markdown.render_html(doc)

    // Or use convenience function
    let html = @markdown.md_to_html("# Hello\n\nWorld")
    let linked = @markdown.md_to_html("See https://example.com/docs\n")
    let plain = @markdown.md_to_html("See https://example.com/docs\n", autolink=false)

    // Enable the WikiLink extension explicitly
    let wiki_html = @markdown.md_to_html("[[MoonBit|MoonBit notes]]", wikilinks=true)

    #Incremental Parsing

    // Initial parse
    let result = @markdown.parse(source)
    let doc = result.document

    // Create edit info
    let edit = @markdown.EditInfo::replace(
    change_start, // Start position
    old_length, // Length of replaced text
    new_length // Length of new text
    )

    // Incremental update (reuses unchanged blocks)
    let inc_result = @markdown.parse_incremental(doc, old_source, new_source, edit)
    let new_doc = inc_result.document


    #Playground

    pnpm install moon build --target js pnpm exec vite

    #Frontend Editor Package

    @mizchi/markdown/editor exports the Luna-based markdown editor without bundling syntax highlighters into the initial module. Code block highlighters are loaded on demand through dynamic imports under @mizchi/markdown/highlight.

    import { SyntaxHighlightEditor } from "@mizchi/markdown/editor"; import "@mizchi/markdown/editor/style.css"; <SyntaxHighlightEditor value={() => markdown} onChange={(next) => setMarkdown(next)} />;

    You can also preload or call a language highlighter explicitly:

    import { loadHighlighter } from "@mizchi/markdown/highlight"; const highlightMoonBit = await loadHighlighter("moonbit"); const html = highlightMoonBit?.("fn main { println(\"hi\") }");

    The currently split lazy highlighter entries are typescript, moonbit, json, html, css, bash, and rust.

    #Performance

    DocumentFull ParseIncrementalSpeedup
    10 paragraphs68.89µs7.36µs9.4x
    50 paragraphs327.99µs8.67µs37.8x
    100 paragraphs651.14µs15.25µs42.7x

    #Documentation

    See docs/markdown.md for detailed architecture and design.

    #CommonMark Compatibility

    This parser handles most common Markdown syntax correctly and works well for typical use cases like documentation, blog posts, and notes.

    However, some edge cases (deeply nested structures, unusual delimiter combinations) are not fully CommonMark compliant. If you need strict CommonMark compliance, consider using cmark.mbt or other fully compliant parsers.

    #Credits

    • Fonts: PlemolJP (SIL Open Font License 1.1) — bundled in playground/public/fonts/

    #License

    MIT

    AgentId

    pub(all) struct AgentId(Int) derive(Eq,
    Debug
    )

    Agent identifier (simplified version: Int)
    impl Show for AgentId

    Block

    pub(all) enum Block {
    ThematicBreak(Char, Int, Span, Trivia, Trivia)
    Heading(Int, HeadingStyle, Array[Inline], Int, Span, Trivia, Trivia)
    Paragraph(Array[Inline], Span, Trivia, Trivia)
    FencedCode(FenceMarker, Int, String, String, Int, Span, Trivia, Trivia)
    IndentedCode(String, Span, Trivia, Trivia)
    Blockquote(Array[Block], Span, Trivia, Trivia)
    BulletList(BulletMarker, Bool, Array[ListItem], Span, Trivia, Trivia)
    OrderedList(Int, OrderedDelimiter, Bool, Array[ListItem], Span, Trivia, Trivia)
    HtmlBlock(String, Span, Trivia, Trivia)
    Table(Array[TableCell], Array[TableAlign], Array[Array[TableCell]], Span, Trivia, Trivia)
    BlankLines(Int, Span)
    FootnoteDefinition(String, Array[Block], Span, Trivia, Trivia)
    }

    Block-level nodes

    BulletMarker

    pub(all) enum BulletMarker {
    Dash
    Asterisk
    Plus
    } derive(Eq,
    Debug
    )

    List marker for unordered lists: -, *, +

    CodeBlockInfo

    pub struct CodeBlockInfo {
    lang : String
    filename : String
    meta : String
    }

    "ts:index.ts {highlight=[1,3]}" → (lang="ts", filename="index.ts", meta="{highlight=[1,3]}")

    CrdtDocument

    pub(all) struct CrdtDocument {
    runs : Array[TextRun]
    gen : IdGenerator
    }

    Experimental document structure

    CrdtDocument::active_count

    fn CrdtDocument::active_count(self : CrdtDocument) -> Int

    Count of active runs

    CrdtDocument::delete

    fn CrdtDocument::delete(self : CrdtDocument, idx : Int) -> Unit

    Delete (mark as tombstone)

    CrdtDocument::get_text

    fn CrdtDocument::get_text(self : CrdtDocument) -> String

    Get active (non-deleted) text

    CrdtDocument::insert

    fn CrdtDocument::insert(self : CrdtDocument, text : String) -> LogicalId

    Insert text

    CrdtDocument::new

    fn CrdtDocument::new(agent : AgentId) -> CrdtDocument

    CrdtDocument::total_count

    fn CrdtDocument::total_count(self : CrdtDocument) -> Int

    Total run count (including tombstones)

    CrdtSpan

    pub(all) struct CrdtSpan {
    start_id : LogicalId
    end_id : LogicalId
    cached_from : Int
    cached_to : Int
    }

    CRDT-aware Span (with cache)

    CrdtSpan::new

    fn CrdtSpan::new(start_id : LogicalId, end_id : LogicalId, from : Int, to : Int) -> CrdtSpan

    Document

    pub(all) struct Document {
    frontmatter : Frontmatter?
    children : Array[Block]
    span : Span
    }

    Document root

    EditInfo

    pub(all) struct EditInfo {
    offset : Int
    old_len : Int
    new_len : Int
    }

    Using #valtype to avoid heap allocation for this small struct

    EditInfo::delete

    fn EditInfo::delete(offset : Int, len : Int) -> EditInfo

    Create an edit info for deletion

    EditInfo::insert

    fn EditInfo::insert(offset : Int, len : Int) -> EditInfo

    Create an edit info for insertion

    EditInfo::replace

    fn EditInfo::replace(offset : Int, old_len : Int, new_len : Int) -> EditInfo

    Create an edit info for replacement

    EmphasisMarker

    pub(all) enum EmphasisMarker {
    Asterisk
    Underscore
    } derive(Eq,
    Debug
    )

    Emphasis marker: * or _

    FenceMarker

    pub(all) enum FenceMarker {
    Backtick
    Tilde
    } derive(Eq,
    Debug
    )

    Fence marker for code blocks: ``` or ~~~
    impl Show for FenceMarker

    Frontmatter

    pub(all) struct Frontmatter {
    raw : String
    entries : Array[(String, String)]
    span : Span
    }

    Frontmatter (YAML)

    HardBreakStyle

    pub(all) enum HardBreakStyle {
    TwoSpaces
    Backslash
    } derive(Eq,
    Debug
    )

    Hard break style

    HeadingStyle

    pub(all) enum HeadingStyle {
    Atx
    Setext
    } derive(Eq,
    Debug
    )

    Heading style

    IdGenerator

    pub(all) struct IdGenerator {
    agent : AgentId
    next_seq : Int
    }

    LogicalId generator

    IdGenerator::new

    fn IdGenerator::new(agent : AgentId) -> IdGenerator

    IdGenerator::next

    fn IdGenerator::next(self : IdGenerator) -> LogicalId

    IncrementalResult

    pub(all) struct IncrementalResult {
    document : Document
    reused_before : Int
    reparsed : Int
    reused_after : Int
    }

    Result of incremental parsing

    Inline

    pub(all) enum Inline {
    Text(String, Span)
    SoftBreak(Span)
    HardBreak(HardBreakStyle, Span)
    Emphasis(EmphasisMarker, Array[Inline], Span)
    Strong(EmphasisMarker, Array[Inline], Span)
    Strikethrough(Array[Inline], Span)
    Code(String, Int, Span)
    WikiLink(String, String, String, Span)
    Link(Array[Inline], String, String, Span)
    RefLink(Array[Inline], String, Span)
    Autolink(String, Bool, Span)
    Image(String, String, String, Span)
    RefImage(String, String, Span)
    HtmlInline(String, Span)
    FootnoteReference(String, Span)
    }

    Inline-level nodes

    LinkDefinition

    pub(all) struct LinkDefinition {
    label : String
    url : String
    title : String
    span : Span
    }

    Link reference definition [label]: url "title"

    ListItem

    pub(all) struct ListItem {
    children : Array[Block]
    checked : Bool?
    marker_offset : Int
    content_offset : Int
    span : Span
    }

    List item

    LogicalId

    pub(all) struct LogicalId {
    agent : AgentId
    seq : Int
    } derive(Eq,
    Debug
    )

    Logical ID: (agent, seq) pair
    impl Show for LogicalId

    LogicalId::compare

    fn LogicalId::compare(self : LogicalId, other : LogicalId) -> Int

    LogicalId ordering comparison (Lamport order)

    LogicalId::le

    fn LogicalId::le(self : LogicalId, other : LogicalId) -> Bool

    LogicalId::lt

    fn LogicalId::lt(self : LogicalId, other : LogicalId) -> Bool

    OrderedDelimiter

    pub(all) enum OrderedDelimiter {
    Dot
    Paren
    } derive(Eq,
    Debug
    )

    Ordered list delimiter: . or )

    ParseResult

    pub(all) struct ParseResult {
    document : Document
    definitions : Array[LinkDefinition]
    }

    Parse result

    RenderOptions

    pub struct RenderOptions {
    code_highlighter : (CodeBlockInfo, String) -> String?
    }

    Render options with plugin support

    RenderOptions::default

    fn RenderOptions::default() -> RenderOptions

    Create default render options

    RenderOptions::with_highlighter

    fn RenderOptions::with_highlighter(highlighter : (CodeBlockInfo, String) -> String) -> RenderOptions

    Create render options with code highlighter

    RenderOptions::with_simple_highlighter

    fn RenderOptions::with_simple_highlighter(highlighter : (String, String) -> String) -> RenderOptions

    Create render options with simple highlighter (lang, code) -> String

    Scanner

    pub(all) struct Scanner {
    source : String
    chars : Array[Char]
    pos : Int
    len : Int
    utf16_offsets : Array[Int]?
    }

    Scanner state - uses Array[Char] for O(1) character access Handles Unicode correctly by tracking UTF-16 offsets for non-BMP characters

    Scanner::advance

    fn Scanner::advance(self : Scanner, n : Int) -> Unit

    Advance position by n characters

    Scanner::consume

    fn Scanner::consume(self : Scanner) -> Char?

    Consume and return current character (O(1))

    Scanner::consume_str

    fn Scanner::consume_str(self : Scanner, s : String) -> Bool

    Match and consume a string

    Scanner::count_char

    fn Scanner::count_char(self : Scanner, c : Char) -> Int

    Count consecutive occurrences of a character from current position - O(1) per char

    Scanner::count_leading_spaces

    fn Scanner::count_leading_spaces(self : Scanner) -> Int

    Count leading spaces (without advancing) - O(1) per char

    Scanner::is_blank_line

    fn Scanner::is_blank_line(self : Scanner) -> Bool

    Check if current line is blank (only whitespace) - O(1) per char

    Scanner::is_eof

    fn Scanner::is_eof(self : Scanner) -> Bool

    Check if at end of input

    Scanner::matches

    fn Scanner::matches(self : Scanner, s : String) -> Bool

    Match a string at current position - optimized with Array[Char]

    Scanner::new

    fn Scanner::new(source : String) -> Scanner

    Create a new scanner

    Scanner::peek

    fn Scanner::peek(self : Scanner) -> Char?

    Peek current character (O(1) with Array[Char])

    Scanner::peek_at

    fn Scanner::peek_at(self : Scanner, offset : Int) -> Char?

    Peek character at offset from current position (O(1))

    Scanner::read_line

    fn Scanner::read_line(self : Scanner) -> String

    Read until end of line (not consuming newline) - O(1) per char

    Scanner::remaining

    fn Scanner::remaining(self : Scanner) -> String

    Get remaining substring from current position

    Scanner::restore

    fn Scanner::restore(self : Scanner, pos : Int) -> Unit

    Restore to saved position

    Scanner::save

    fn Scanner::save(self : Scanner) -> Int

    Save current position

    Scanner::skip_line

    fn Scanner::skip_line(self : Scanner) -> Unit

    Skip to next line (consuming newline if present) - O(1) per char

    Scanner::skip_spaces

    fn Scanner::skip_spaces(self : Scanner) -> Int

    Skip whitespace (space and tab only) - O(1) per char

    Scanner::substring

    fn Scanner::substring(self : Scanner, start : Int, end : Int) -> String

    Get substring from start to end position (code point indices)

    Span

    pub(all) struct Span {
    from : Int
    to : Int
    } derive(Eq,
    Debug
    )

    Using #valtype to avoid heap allocation for this frequently-used small struct
    impl Show for Span

    Span::empty

    fn Span::empty() -> Span

    Empty span (for synthetic nodes)

    Span::new

    fn Span::new(from : Int, to : Int) -> Span

    Create a span

    TableAlign

    pub(all) enum TableAlign {
    Left
    Center
    Right
    None
    } derive(Eq,
    Debug
    )

    Table alignment
    impl Show for TableAlign

    TableCell

    pub(all) struct TableCell {
    children : Array[Inline]
    span : Span
    }

    Table cell

    TextRun

    pub(all) struct TextRun {
    id : LogicalId
    content : String
    deleted : Bool
    }

    TextRun: Compressed representation of contiguous text

    Trivia

    pub(all) struct Trivia {
    content : String
    } derive(Eq,
    Debug
    )

    Trivia represents non-semantic characters that should be preserved
    impl Show for Trivia

    Trivia::empty

    fn Trivia::empty() -> Trivia

    Trivia::new

    fn Trivia::new(content : String) -> Trivia

    char_is

    fn char_is(opt : Char?, c : Char) -> Bool

    Check if char option matches specific char

    char_is_digit

    fn char_is_digit(opt : Char?) -> Bool

    Check if char option is a digit

    is_alphanumeric

    fn is_alphanumeric(c : Char) -> Bool

    Check if character is alphanumeric

    is_digit

    fn is_digit(c : Char) -> Bool

    Check if character is a digit

    is_letter

    fn is_letter(c : Char) -> Bool

    Check if character is ASCII letter

    is_punctuation

    fn is_punctuation(c : Char) -> Bool

    Check if character is a punctuation mark

    is_whitespace

    fn is_whitespace(c : Char) -> Bool

    Check if character is ASCII whitespace

    md_free

    fn md_free(handle : Int) -> Unit

    Free a document handle to release memory

    md_parse

    fn md_parse(source : String) -> Int

    Returns 0 on error

    md_parse_and_render

    fn md_parse_and_render(source : String, strict? : Bool, wikilinks? : Bool) -> String

    When strict=true, uses full CommonMark compliance (slower)

    md_parse_incremental

    fn md_parse_incremental(handle : Int, new_source : String, change_start : Int, old_end : Int, new_end : Int) -> Int

    Returns new document handle, or 0 on error
    fn md_parse_with_wikilinks(source : String) -> Int

    Parse markdown source with wikilinks enabled and return a document handle

    md_render_to_html

    fn md_render_to_html(handle : Int) -> String

    Returns empty string if handle is invalid

    md_render_to_string

    fn md_render_to_string(handle : Int) -> String

    Returns empty string if handle is invalid

    md_to_html

    fn md_to_html(source : String, wikilinks? : Bool, autolink? : Bool) -> String

    Parse markdown and render to HTML
    fn md_to_html_autolink(source : String, wikilinks? : Bool) -> String

    Parse markdown and render to HTML with bare URL text autolinked

    md_to_html_strict

    fn md_to_html_strict(source : String, wikilinks? : Bool, autolink? : Bool) -> String

    Parse markdown and render to HTML (strict mode)

    parse

    fn parse(source : String, strict? : Bool, wikilinks? : Bool) -> ParseResult

    When strict=false (default), uses fast single-pass parser

    parse_code_block_info

    fn parse_code_block_info(info : String) -> CodeBlockInfo

    Parse code block info string into components

    parse_incremental

    fn parse_incremental(old_doc : Document, old_source : String, new_source : String, edit : EditInfo, wikilinks? : Bool) -> IncrementalResult

    Parse incrementally using edit hint

    parse_inlines

    fn parse_inlines(text : String, strict? : Bool, wikilinks? : Bool) -> Array[Inline]

    When strict=false (default), uses fast single-pass parser

    parse_inlines_multipass

    fn parse_inlines_multipass(text : String) -> Array[Inline]

    Parse all inlines using multi-pass approach

    render_html

    fn render_html(doc : Document, autolink? : Bool) -> String

    Render a document to HTML

    serialize

    fn serialize(doc : Document) -> String

    Serialize document to markdown string

    serialize_definitions

    fn serialize_definitions(defs : Array[LinkDefinition]) -> String

    Serialize link definitions