🥮 mooncakes.io

    moonmd

    Markdown for MoonBit: CommonMark and GFM to a syntax tree, HTML or Markdown, whole or as it streams in.

    markdown
    commonmark
    commonmark-0.31.2
    gfm
    github-flavored-markdown
    md
    cmark
    parser
    syntax-tree
    ast
    html
    renderer
    html-renderer
    markdown-writer
    markdown-to-html
    markdown-parser
    commonmark-parser
    sourcepos
    streaming
    incremental
    table
    footnotes
    tagfilter
    strikethrough
    autolink
    task-list
    readme
    documentation
    moonbit
    Download zip
    Version
    0.2.0
    License
    Apache-2.0
    Repository
    Last updated
    12 hours ago
    Downloads
    426

    No dependencies

    #moonmd

    #Install

    moon add moonbitstack/moonmd

    #Read a document

    parse reads a document once and hands back a Document. Nothing is rendered until you ask for it, and both renderings come from the same tree.

    test {
      let document = @moonmd.parse("# Title\n\nA *short* paragraph with a [link](/a).\n")
      assert_eq(
        document.html(safe=false),
        "<h1>Title</h1>\n<p>A <em>short</em> paragraph with a <a href=\"/a\">link</a>.</p>\n",
      )
      assert_eq(document.markdown(), "# Title\n\nA *short* paragraph with a [link](/a).\n")
    }

    html escapes raw HTML and refuses dangerous destinations unless you pass safe=false; it closes the elements that carry nothing (<br />, <hr />, <img />) unless you pass xhtml=false. markdown writes the tree back as Markdown that reads the same way it did.

    #The tree

    A Document holds top level Blocks in order. There are nine block kinds and ten inline kinds; each of the types and their fields is listed in the API reference. walk visits every node, each container before its children:

    test {
      let document = @moonmd.parse("# One\n\nA [link](/a) and a *word*.\n")
      let counts = [0, 0]
      @moonmd.walk(document, fn(node) {
        match node {
          @moonmd.Block(@moonmd.Heading(_)) => counts[0] = counts[0] + 1
          @moonmd.Inline(@moonmd.Link(_)) => counts[1] = counts[1] + 1
          _ => ()
        }
      })
      assert_eq(counts, [1, 1])
    }

    #Where a block came from

    lines gives the lines of the source each block at the top of the document was read from, counted from one, in the same order as blocks(). Blank lines between blocks belong to neither, and a container reports the lines of everything inside it.

    test {
      let document = @moonmd.parse("# One\n\ntwo\nthree\n")
      let spans = document.lines()
      assert_eq((spans[0].first, spans[0].last), (1, 1))
      assert_eq((spans[1].first, spans[1].last), (3, 4))
    }

    That is what cutting a document by heading needs: the lines of a heading open the next piece, and the block after it starts the one before. A document assembled from blocks with of_blocks was not read from text, so it reports no lines.

    #Footnotes

    A [^label] in the text is a footnote reference; a line that opens with [^label]: is its definition. The reference becomes a numbered marker, the definition is moved to the end of the document as a notes list, and a link in each note points back to its reference.

    test {
      let document = @moonmd.parse("A claim[^note]\n\n[^note]: the support\n")
      let expected =
        "<p>A claim<sup class=\"footnote-ref\"><a href=\"#fn-note\" " +
        "id=\"fnref-note\" data-footnote-ref>1</a></sup></p>\n" +
        "<section class=\"footnotes\" data-footnotes>\n" +
        "<ol>\n" +
        "<li id=\"fn-note\">\n" +
        "<p>the support <a href=\"#fnref-note\" class=\"footnote-backref\" " +
        "data-footnote-backref data-footnote-backref-idx=\"1\" " +
        "aria-label=\"Back to reference 1\">\u{21A9}</a></p>\n" +
        "</li>\n</ol>\n</section>\n"
      assert_eq(document.html(safe=false), expected)
      assert_eq(document.markdown(), "A claim[^note]\n\n[^note]:\n    the support\n")
    }

    The number follows the first reference, not the definition, and a reference with no definition stays the text it was written as. The block is a Footnote; the reference is a FootnoteRef. Footnotes are part of gfm and off in commonmark.

    #Flavors

    Flavor is a plain record. gfm is the default; commonmark turns every extension off, and any field can be changed on its own:

    test {
      assert_eq(@moonmd.parse("~~struck~~\n").html(safe=false), "<p><del>struck</del></p>\n")
      assert_eq(
        @moonmd.parse("~~struck~~\n", flavor=@moonmd.commonmark).html(safe=false),
        "<p>~~struck~~</p>\n",
      )
    }

    smart is off in both presets, because it writes the text differently rather than reading the structure differently: quotes curl towards what they wrap, -- and --- become dashes, ... becomes an ellipsis. A code span, an escaped character, an entity and a destination keep what they were written with.

    test {
      let typed : @moonmd.Flavor = { ..@moonmd.gfm, smart: true }
      assert_eq(
        @moonmd.parse("\"quoted\" -- and...\n"[:], flavor=typed).html(safe=false),
        "<p>\u{201C}quoted\u{201D} \u{2013} and\u{2026}</p>\n",
      )
      assert_eq(
        @moonmd.parse("`\"--\"`\n"[:], flavor=typed).html(safe=false),
        "<p><code>&quot;--&quot;</code></p>\n",
      )
    }

    breaks is the other one, and it is also off: a line ending inside a paragraph is a hard break, the way a message box reads it, instead of the soft break the specification gives it.

    test {
      let boxed : @moonmd.Flavor = { ..@moonmd.gfm, breaks: true }
      assert_eq(
        @moonmd.parse("one\ntwo\n"[:], flavor=boxed).html(safe=false),
        "<p>one<br />\ntwo</p>\n",
      )
    }

    #Streaming

    A model writes a document a token at a time. Stream::feed takes whatever has arrived and returns the blocks that no later text can change; finish returns the rest. A block is settled when a following line has closed it, so a chunk may end anywhere, including in the middle of a line:

    test {
      let stream = @moonmd.Stream::new()
      let counts = [
        stream.feed("# Hi"[:]).length(),
        stream.feed("\n\n"[:]).length(),
        stream.feed("a paragraph"[:]).length(),
        stream.finish().length(),
      ]
      assert_eq(counts, [0, 1, 0, 1])
    }

    Link reference definitions apply to the whole document, so a block that mentions [ waits until finish by default; passing defer_links=false hands such a block out as soon as it closes.

    #Safety

    html defaults to safe=true: raw HTML is escaped, and a link whose destination would run script keeps its text and loses its anchor.

    test {
      assert_eq(@moonmd.parse("<div>hi</div>\n").html(), "&lt;div&gt;hi&lt;/div&gt;\n")
      assert_eq(@moonmd.parse("[bad](javascript:alert(1))\n").html(), "<p>bad</p>\n")
    }

    With safe=false the document is written as it was read.

    A document that nests deeper than a hundred blocks, or a hundred inline spans inside each other, stops there: the marker that would open the next one stays in the text. Both the tree and the HTML are produced by walking containers, so a document that nests deeper than that would end the process rather than the document.

    #Conformance

    The test suite replays the examples of the CommonMark 0.31.2 specification and the extension corpus of the GitHub implementation. Both are generated into the test package by scripts/gen_spec.py, with the version pinned.

    CorpusExamplesPassing
    CommonMark 0.31.2 spec.json652652
    GitHub Flavored Markdown extensions2929

    Every example in both corpora produces the exact HTML each expects, including the three footnote examples (23, 24 and 25). Two further properties hold over the same corpora:

    • writing a tree back as Markdown and reading it again gives the same HTML, for all 652 CommonMark examples and all 29 extension examples;
    • feeding an example in pieces of one, two, three, five and eight characters gives the same blocks as reading it whole, for the same 652 and 29 examples.

    The same documents are read twice more by the reference implementation of the specification, once as they are and once as this library writes them back. 670 of the 681 agree in HTML, and 679 of the 681 agree after a round trip. The eleven that do not are examples where CommonMark 0.31.2 changed a rule and the reference still follows 0.29:

    ExampleWhat the reference does instead
    28turns a character reference above U+10FFFF into U+FFFD; 0.31.2 keeps it as text
    354counts a currency sign as punctuation, so *£* closes emphasis where 0.31.2 leaves it open
    389collapses __foo, __bar__, baz__ into one strong span
    417drops the runs of foo******bar*********baz down to one strong span
    425collapses __foo __bar__ baz__ into a single strong span
    426collapses ____foo__ bar__ into a single strong span
    427collapses **foo **bar**** into a single strong span
    464collapses ****foo**** into a single strong span
    465collapses ____foo____ into a single strong span
    466collapses ******foo****** into a single strong span
    468collapses _____foo_____ into one strong span inside an emphasis

    #License

    Apache-2.0.

    Align

    source
    pub(all) enum Align {
      Default
      Left
      Center
      Right
    }

    How a table column lines up its cells, taken from the delimiter row.

    Block

    source
    pub(all) enum Block {
      Heading(Heading)
      Paragraph(Array[Inline])
      Code(Code)
      Quote(Array[Block])
      List(ListBlock)
      Table(Table)
      Html(String)
      Break
      Footnote(Footnote)
    }

    A block level node.

    Break

    source
    pub(all) enum Break {
      Soft
      Hard
    }

    A line break inside a paragraph.

    Code

    source
    pub(all) struct Code {
      info : String
      text : String
      fenced : Bool
    }

    A code block, fenced or indented. info is the word after the opening fence, empty for indented code.

    Document

    source
    pub struct Document {
      // private fields
    }

    A parsed document: the blocks in order, and the flavor it was read with, which the HTML renderer needs for the tag filter.

    Document::blocks

    source
    fn Document::blocks(self : Document) -> Array[Block]

    The top level blocks of the document, in order.

    Document::flavor

    source
    fn Document::flavor(self : Document) -> Flavor

    The flavor this document was parsed with.

    Document::html

    source
    fn Document::html(self : Document, safe? : Bool, xhtml? : Bool) -> String

    Render the document as HTML. With safe set, which is the default, raw HTML is escaped and links to dangerous destinations lose their anchor; pass safe=false to write a document that may embed HTML of its own.

    xhtml closes the elements that carry nothing (<br />, <hr />, <img />) the way the reference implementation does; pass xhtml=false for the form a document written to HTML5 uses.

    Document::lines

    source
    fn Document::lines(self : Document) -> Array[LineSpan]

    The lines of the source each top level block was read from, counted from one, one entry per block of blocks() and in the same order. Blank lines between blocks belong to neither. A document assembled with of_blocks was not read from text, so its list is empty.

    test {
      let document = @moonmd.parse("# One\n\ntwo\n")
      let spans = document.lines()
      assert_eq((spans[0].first, spans[0].last), (1, 1))
      assert_eq((spans[1].first, spans[1].last), (3, 3))
    }

    Document::markdown

    source
    fn Document::markdown(self : Document) -> String

    Render the document as Markdown.

    Document::of_blocks

    source
    fn Document::of_blocks(blocks : Array[Block], flavor? : Flavor) -> Document

    Assemble a document out of blocks, which is what a caller collecting the output of a Stream needs in order to render it. Footnote references are resolved against the definitions in the blocks, so a definition read after the reference that uses it still applies.

    Flavor

    source
    pub(all) struct Flavor {
      tables : Bool
      strikethrough : Bool
      autolinks : Bool
      task_lists : Bool
      tag_filter : Bool
      footnotes : Bool
      smart : Bool
      breaks : Bool
    }

    What a parser accepts beyond the CommonMark core. The presets commonmark and gfm cover the two dialects this library is asked for; the fields are public so a caller can take one and drop what it does not want.

    Footnote

    source
    pub(all) struct Footnote {
      label : String
      blocks : Array[Block]
      number : Int
      backrefs : Int
    }

    A footnote, the body of one [^label]: definition. label is the label as it was written. number is the order the footnote is shown with, assigned in the order its first reference appears in the text; backrefs is how many references point at it.

    FootnoteRef

    source
    pub(all) struct FootnoteRef {
      label : String
      number : Int
      ref_ix : Int
    }

    A footnote reference, [^label], and where it lands: the definition's label, the number that definition is shown with, and ref_ix, the order of this reference among the ones that point at the definition.

    Heading

    source
    pub(all) struct Heading {
      level : Int
      content : Array[Inline]
    }

    A heading, level counting the ATX # marks or the setext underline.

    Inline

    source
    pub(all) enum Inline {
      Text(String)
      Code(String)
      Emphasis(Array[Inline])
      Strong(Array[Inline])
      Link(Link)
      Image(Link)
      Html(String)
      Strike(Array[Inline])
      Break(Break)
      FootnoteRef(FootnoteRef)
    }

    An inline level node.

    Item

    source
    pub(all) struct Item {
      blocks : Array[Block]
      task : Task
    }

    One item of a list, with the task marker that introduced it if any.

    LineSpan

    source
    pub(all) struct LineSpan {
      first : Int
      last : Int
    }

    A run of source lines, counted from one, that a block was read from.
    pub(all) struct Link {
      dest : String
      title : String
      content : Array[Inline]
    }

    The destination, title and content of a link or image. For an image the content is the alternative text.

    ListBlock

    source
    pub(all) struct ListBlock {
      kind : ListKind
      tight : Bool
      items : Array[Item]
    }

    A list, with its items and whether it is tight (no blank lines between items, so their paragraphs are rendered without <p>).

    ListKind

    source
    pub(all) enum ListKind {
      Bullet(Char)
      Ordered(Int, Char)
    }

    What kind of list a ListBlock is, together with the marks needed to write it back unchanged.

    Node

    source
    pub(all) enum Node {
      Block(Block)
      Inline(Inline)
    }

    A node as walk hands it to its visitor: either a block or an inline.

    Stream

    source
    pub struct Stream {
      // private fields
    }

    A parser that reads a document in pieces. Text that has arrived is parsed again on every call, but only the part that has not been handed out yet, so the work per chunk is proportional to the unfinished tail rather than to the whole document.

    Stream::feed

    source
    fn Stream::feed(self : Stream, chunk : StringView) -> Array[Block]

    Take the next piece of the document and return the blocks it completed.

    Stream::finish

    source
    fn Stream::finish(self : Stream) -> Array[Block]

    Close the stream and return whatever was still open.

    Stream::new

    source
    fn Stream::new(flavor? : Flavor, defer_links? : Bool) -> Stream

    Start a stream.

    defer_links keeps a block that mentions [ back until the whole document has arrived, because a link reference definition written later still applies to it. With it off such a block is handed out as soon as it closes, at the cost of a forward reference not resolving; the blocks are then the same as a whole parse for any document that defines its labels before using them.

    Table

    source
    pub(all) struct Table {
      align : Array[Align]
      head : Array[Array[Inline]]
      body : Array[Array[Array[Inline]]]
    }

    A table: the alignment of each column, the header row, then the body rows. A row holds one cell per column, and a cell holds the inlines inside it.

    Task

    source
    pub(all) enum Task {
      Plain
      Unchecked
      Checked
    }

    Whether a list item is a task list item, and if so whether it is done.

    commonmark

    source
    let commonmark : Flavor

    CommonMark without any of the extensions.

    flavor

    source
    let flavor : Flavor

    The preset parse uses when the caller does not pass one: gfm.

    gfm

    source
    let gfm : Flavor

    CommonMark as GitHub reads it: tables, strikethrough, bare links, task list items, the tag filter and footnotes. This is the default.

    parse

    source
    fn parse(text : StringView, flavor? : Flavor) -> Document

    Parse a Markdown document.

    test {
      let document = @moonmd.parse("# Title")
      assert_eq(document.html(safe=false), "<h1>Title</h1>\n")
    }

    walk

    source
    fn walk(document : Document, visit : (Node) -> Unit) -> Unit

    Visit every node of the document, each container before its children and siblings in order.

    Source Files