🥮 mooncakes.io
    Sign in

    markdown

    CST-based incremental Markdown parser for MoonBit

    markdown
    parser
    cst
    incremental
    gfm
    Download zip
    Author
    Version
    0.1.1
    License
    MIT
    Repository
    Last updated
    10 months ago
    Downloads
    21K

    #mizchi/markdown

    CST-based incremental Markdown parser for MoonBit.

    JS/WASM/native のクロスターゲットな MoonBit の Markdown コンパイラです。主にリアルタイム編集時のインクリメンタルパースに最適化されています。

    #CommonMark Compatibility

    CommonMark サブセットの実装で、207/542 のテストをパスしています。未対応のものは入れ子のエッジケースが多く、実用上の問題は少ないです。

    完全な CommonMark 準拠が必要な場合は cmark.mbt を推奨します。

    #Features

    • Lossless CST: Preserves all whitespace, markers, and formatting
    • Incremental parsing: Re-parses only changed blocks (up to 42x faster)
    • GFM compatible: GitHub Flavored Markdown support (tables, task lists, strikethrough)
    • Cross-platform: Works on JS, WASM-GC, and native targets
    • HTML rendering: Built-in HTML renderer with remark-html compatible output
    • Plugin system: Custom code block highlighters

    #Installation

    moon add mizchi/markdown

    #Usage

    // Parse markdown
    let doc = @markdown.parse("# Hello\n\nWorld")

    // Serialize back (lossless)
    let output = @markdown.serialize(doc)

    // Incremental update
    let edit = @markdown.EditInfo::new(start, end, new_length)
    let new_doc = @markdown.parse_incremental(doc, new_text, edit)

    #HTML Rendering

    // Direct conversion
    let html = @markdown.md_to_html("# Hello\n\nWorld")

    // Or with handle for incremental updates
    let handle = @markdown.md_parse(source)
    let html = @markdown.md_render_to_html(handle)
    @markdown.md_free(handle)

    #Playground

    pnpm install
moon build --target js
pnpm exec vite

    #Documentation

    See docs/markdown.md for detailed architecture and design.

    #License

    MIT

    AgentId

    source
    pub(all) type AgentId Int

    Agent識別子 (簡易版: Int)
    impl Eq for AgentId
    impl Show for AgentId

    AgentId::inner

    source
    #deprecated("Use `struct T(A)` to declare a newtype and use `.0` access the underlying type instead.")
    fn AgentId::inner(self : AgentId) -> Int
    Convert newtype to its underlying type, automatically derived.

    Block

    source
    pub(all) enum Block {
      ThematicBreak(Char, Int, Span, Trivia, Trivia)
      Heading(Int, HeadingStyle, Array[Inline], Int, Span, Trivia, Trivia)
      Paragraph(Array[Inline], Span, Trivia, Trivia)
      FencedCode(FenceMarker, Int, String, String, Int, Span, Trivia, Trivia)
      IndentedCode(String, Span, Trivia, Trivia)
      Blockquote(Array[Block], Span, Trivia, Trivia)
      BulletList(BulletMarker, Bool, Array[ListItem], Span, Trivia, Trivia)
      OrderedList(Int, OrderedDelimiter, Bool, Array[ListItem], Span, Trivia, Trivia)
      HtmlBlock(String, Span, Trivia, Trivia)
      Table(Array[TableCell], Array[TableAlign], Array[Array[TableCell]], Span, Trivia, Trivia)
      BlankLines(Int, Span)
      FootnoteDefinition(String, Array[Block], Span, Trivia, Trivia)
    }

    Block-level nodes

    BulletMarker

    source
    pub(all) enum BulletMarker {
      Dash
      Asterisk
      Plus
    }

    List marker for unordered lists: -, *, +
    impl Eq for BulletMarker

    CodeBlockInfo

    source
    pub struct CodeBlockInfo {
      lang : String
      filename : String
      meta : String
    }

    "ts:index.ts {highlight=[1,3]}" → (lang="ts", filename="index.ts", meta="{highlight=[1,3]}")

    CrdtDocument

    source
    pub(all) struct CrdtDocument {
      runs : Array[TextRun]
      gen : IdGenerator
    }

    実験用ドキュメント構造

    CrdtDocument::active_count

    source
    fn CrdtDocument::active_count(self : CrdtDocument) -> Int

    アクティブなRun数

    CrdtDocument::delete

    source
    fn CrdtDocument::delete(self : CrdtDocument, idx : Int) -> Unit

    削除（Tombstone化）

    CrdtDocument::get_text

    source
    fn CrdtDocument::get_text(self : CrdtDocument) -> String

    アクティブなテキストを取得

    CrdtDocument::insert

    source
    fn CrdtDocument::insert(self : CrdtDocument, text : String) -> LogicalId

    テキスト挿入

    CrdtDocument::new

    source
    fn CrdtDocument::new(agent : AgentId) -> CrdtDocument

    CrdtDocument::total_count

    source
    fn CrdtDocument::total_count(self : CrdtDocument) -> Int

    全Run数（Tombstone含む）

    CrdtSpan

    source
    pub(all) struct CrdtSpan {
      start_id : LogicalId
      end_id : LogicalId
      cached_from : Int
      cached_to : Int
    }

    CRDT対応Span (キャッシュ付き)

    CrdtSpan::new

    source
    fn CrdtSpan::new(start_id : LogicalId, end_id : LogicalId, from : Int, to : Int) -> CrdtSpan

    Document

    source
    pub(all) struct Document {
      frontmatter : Frontmatter?
      children : Array[Block]
      span : Span
    }

    Document root

    EditInfo

    source
    pub(all) struct EditInfo {
      offset : Int
      old_len : Int
      new_len : Int
    }

    Using #valtype to avoid heap allocation for this small struct

    EditInfo::delete

    source
    fn EditInfo::delete(offset : Int, len : Int) -> EditInfo

    Create an edit info for deletion

    EditInfo::insert

    source
    fn EditInfo::insert(offset : Int, len : Int) -> EditInfo

    Create an edit info for insertion

    EditInfo::replace

    source
    fn EditInfo::replace(offset : Int, old_len : Int, new_len : Int) -> EditInfo

    Create an edit info for replacement

    EmphasisMarker

    source
    pub(all) enum EmphasisMarker {
      Asterisk
      Underscore
    }

    Emphasis marker: * or _

    FenceMarker

    source
    pub(all) enum FenceMarker {
      Backtick
      Tilde
    }

    Fence marker for code blocks: ``` or ~~~
    impl Eq for FenceMarker
    impl Show for FenceMarker

    Frontmatter

    source
    pub(all) struct Frontmatter {
      raw : String
      entries : Array[(String, String)]
      span : Span
    }

    Frontmatter (YAML)

    HardBreakStyle

    source
    pub(all) enum HardBreakStyle {
      TwoSpaces
      Backslash
    }

    Hard break style

    HeadingStyle

    source
    pub(all) enum HeadingStyle {
      Atx
      Setext
    }

    Heading style
    impl Eq for HeadingStyle

    IdGenerator

    source
    pub(all) struct IdGenerator {
      agent : AgentId
      next_seq : Int
    }

    LogicalId生成器

    IdGenerator::new

    source
    fn IdGenerator::new(agent : AgentId) -> IdGenerator

    IdGenerator::next

    source
    fn IdGenerator::next(self : IdGenerator) -> LogicalId

    IncrementalResult

    source
    pub(all) struct IncrementalResult {
      document : Document
      reused_before : Int
      reparsed : Int
      reused_after : Int
    }

    Result of incremental parsing

    Inline

    source
    pub(all) enum Inline {
      Text(String, Span)
      SoftBreak(Span)
      HardBreak(HardBreakStyle, Span)
      Emphasis(EmphasisMarker, Array[Inline], Span)
      Strong(EmphasisMarker, Array[Inline], Span)
      Strikethrough(Array[Inline], Span)
      Code(String, Int, Span)
      Link(Array[Inline], String, String, Span)
      RefLink(Array[Inline], String, Span)
      Autolink(String, Bool, Span)
      Image(String, String, String, Span)
      RefImage(String, String, Span)
      HtmlInline(String, Span)
      FootnoteReference(String, Span)
    }

    Inline-level nodes

    LinkDefinition

    source
    pub(all) struct LinkDefinition {
      label : String
      url : String
      title : String
      span : Span
    }

    Link reference definition [label]: url "title"

    ListItem

    source
    pub(all) struct ListItem {
      children : Array[Block]
      checked : Bool?
      marker_offset : Int
      content_offset : Int
      span : Span
    }

    List item

    LogicalId

    source
    pub(all) struct LogicalId {
      agent : AgentId
      seq : Int
    }

    論理ID: (agent, seq) ペア
    impl Eq for LogicalId
    impl Show for LogicalId

    LogicalId::compare

    source
    fn LogicalId::compare(self : LogicalId, other : LogicalId) -> Int

    LogicalId の順序比較 (Lamport順序)

    LogicalId::le

    source
    fn LogicalId::le(self : LogicalId, other : LogicalId) -> Bool

    LogicalId::lt

    source
    fn LogicalId::lt(self : LogicalId, other : LogicalId) -> Bool

    OrderedDelimiter

    source
    pub(all) enum OrderedDelimiter {
      Dot
      Paren
    }

    Ordered list delimiter: . or )

    ParseResult

    source
    pub(all) struct ParseResult {
      document : Document
      definitions : Array[LinkDefinition]
    }

    Parse result

    RenderOptions

    source
    pub struct RenderOptions {
      code_highlighter : (CodeBlockInfo, String) -> String?
    }

    Render options with plugin support

    RenderOptions::default

    source
    fn RenderOptions::default() -> RenderOptions

    Create default render options

    RenderOptions::with_highlighter

    source
    fn RenderOptions::with_highlighter(highlighter : (CodeBlockInfo, String) -> String) -> RenderOptions

    Create render options with code highlighter

    RenderOptions::with_simple_highlighter

    source
    fn RenderOptions::with_simple_highlighter(highlighter : (String, String) -> String) -> RenderOptions

    Create render options with simple highlighter (lang, code) -> String

    Scanner

    source
    pub(all) struct Scanner {
      source : String
      chars : Array[Char]
      pos : Int
      len : Int
      utf16_offsets : Array[Int]?
    }

    Scanner state - uses Array[Char] for O(1) character access Handles Unicode correctly by tracking UTF-16 offsets for non-BMP characters

    Scanner::advance

    source
    fn Scanner::advance(self : Scanner, n : Int) -> Unit

    Advance position by n characters

    Scanner::consume

    source
    fn Scanner::consume(self : Scanner) -> Char?

    Consume and return current character (O(1))

    Scanner::consume_str

    source
    fn Scanner::consume_str(self : Scanner, s : String) -> Bool

    Match and consume a string

    Scanner::count_char

    source
    fn Scanner::count_char(self : Scanner, c : Char) -> Int

    Count consecutive occurrences of a character from current position - O(1) per char

    Scanner::count_leading_spaces

    source
    fn Scanner::count_leading_spaces(self : Scanner) -> Int

    Count leading spaces (without advancing) - O(1) per char

    Scanner::is_blank_line

    source
    fn Scanner::is_blank_line(self : Scanner) -> Bool

    Check if current line is blank (only whitespace) - O(1) per char

    Scanner::is_eof

    source
    fn Scanner::is_eof(self : Scanner) -> Bool

    Check if at end of input

    Scanner::matches

    source
    fn Scanner::matches(self : Scanner, s : String) -> Bool

    Match a string at current position - optimized with Array[Char]

    Scanner::new

    source
    fn Scanner::new(source : String) -> Scanner

    Create a new scanner

    Scanner::peek

    source
    fn Scanner::peek(self : Scanner) -> Char?

    Peek current character (O(1) with Array[Char])

    Scanner::peek_at

    source
    fn Scanner::peek_at(self : Scanner, offset : Int) -> Char?

    Peek character at offset from current position (O(1))

    Scanner::read_line

    source
    fn Scanner::read_line(self : Scanner) -> String

    Read until end of line (not consuming newline) - O(1) per char

    Scanner::remaining

    source
    fn Scanner::remaining(self : Scanner) -> String

    Get remaining substring from current position

    Scanner::restore

    source
    fn Scanner::restore(self : Scanner, pos : Int) -> Unit

    Restore to saved position

    Scanner::save

    source
    fn Scanner::save(self : Scanner) -> Int

    Save current position

    Scanner::skip_line

    source
    fn Scanner::skip_line(self : Scanner) -> Unit

    Skip to next line (consuming newline if present) - O(1) per char

    Scanner::skip_spaces

    source
    fn Scanner::skip_spaces(self : Scanner) -> Int

    Skip whitespace (space and tab only) - O(1) per char

    Scanner::substring

    source
    fn Scanner::substring(self : Scanner, start : Int, end : Int) -> String

    Get substring from start to end position (code point indices)

    Span

    source
    pub(all) struct Span {
      from : Int
      to : Int
    }

    Using #valtype to avoid heap allocation for this frequently-used small struct
    impl Eq for Span
    impl Show for Span

    Span::empty

    source
    fn Span::empty() -> Span

    Empty span (for synthetic nodes)

    Span::new

    source
    fn Span::new(from : Int, to : Int) -> Span

    Create a span

    TableAlign

    source
    pub(all) enum TableAlign {
      Left
      Center
      Right
      None
    }

    Table alignment
    impl Eq for TableAlign
    impl Show for TableAlign

    TableCell

    source
    pub(all) struct TableCell {
      children : Array[Inline]
      span : Span
    }

    Table cell

    TextRun

    source
    pub(all) struct TextRun {
      id : LogicalId
      content : String
      deleted : Bool
    }

    TextRun: 連続テキストの圧縮表現

    Trivia

    source
    pub(all) struct Trivia {
      content : String
    }

    Trivia represents non-semantic characters that should be preserved
    impl Eq for Trivia
    impl Show for Trivia

    Trivia::empty

    source
    fn Trivia::empty() -> Trivia

    Trivia::new

    source
    fn Trivia::new(content : String) -> Trivia

    char_is

    source
    fn char_is(opt : Char?, c : Char) -> Bool

    Check if char option matches specific char

    char_is_digit

    source
    fn char_is_digit(opt : Char?) -> Bool

    Check if char option is a digit

    is_alphanumeric

    source
    fn is_alphanumeric(c : Char) -> Bool

    Check if character is alphanumeric

    is_digit

    source
    fn is_digit(c : Char) -> Bool

    Check if character is a digit

    is_letter

    source
    fn is_letter(c : Char) -> Bool

    Check if character is ASCII letter

    is_punctuation

    source
    fn is_punctuation(c : Char) -> Bool

    Check if character is a punctuation mark

    is_whitespace

    source
    fn is_whitespace(c : Char) -> Bool

    Check if character is ASCII whitespace

    md_free

    source
    fn md_free(handle : Int) -> Unit

    Free a document handle to release memory

    md_parse

    source
    fn md_parse(source : String) -> Int

    Returns 0 on error

    md_parse_and_render

    source
    fn md_parse_and_render(source : String, strict? : Bool) -> String

    When strict=true, uses full CommonMark compliance (slower)

    md_parse_incremental

    source
    fn md_parse_incremental(handle : Int, new_source : String, change_start : Int, old_end : Int, new_end : Int) -> Int

    Returns new document handle, or 0 on error

    md_render_to_html

    source
    fn md_render_to_html(handle : Int) -> String

    Returns empty string if handle is invalid

    md_render_to_string

    source
    fn md_render_to_string(handle : Int) -> String

    Returns empty string if handle is invalid

    md_to_html

    source
    fn md_to_html(source : String) -> String

    Parse markdown and render to HTML

    md_to_html_strict

    source
    fn md_to_html_strict(source : String) -> String

    Parse markdown and render to HTML (strict mode)

    parse

    source
    fn parse(source : String, strict? : Bool) -> ParseResult

    When strict=false (default), uses fast single-pass parser

    parse_code_block_info

    source
    fn parse_code_block_info(info : String) -> CodeBlockInfo

    Parse code block info string into components

    parse_incremental

    source
    fn parse_incremental(old_doc : Document, old_source : String, new_source : String, edit : EditInfo) -> IncrementalResult

    Parse incrementally using edit hint

    parse_inlines

    source
    fn parse_inlines(text : String, strict? : Bool) -> Array[Inline]

    When strict=false (default), uses fast single-pass parser

    parse_inlines_multipass

    source
    fn parse_inlines_multipass(text : String) -> Array[Inline]

    Parse all inlines using multi-pass approach

    render_html

    source
    fn render_html(doc : Document) -> String

    Render a document to HTML

    serialize

    source
    fn serialize(doc : Document) -> String

    Serialize document to markdown string

    serialize_definitions

    source
    fn serialize_definitions(defs : Array[LinkDefinition]) -> String

    Serialize link definitions

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io