moonmd

    Markdown for MoonBit: CommonMark and GFM to a syntax tree, HTML or Markdown, whole or as it streams in.

    markdown
    commonmark
    commonmark-0.31.2
    gfm
    github-flavored-markdown
    md
    cmark
    parser
    syntax-tree
    ast
    html
    renderer
    html-renderer
    markdown-writer
    streaming
    incremental
    table
    strikethrough
    autolink
    task-list
    readme
    documentation
    moonbit
    Download zip
    Version
    0.1.1
    License
    Apache-2.0
    Last updated
    12 hours ago
    Downloads
    65

    No dependencies

    #moonmd

    #Install

    moon add moonbitstack/moonmd

    #Read a document

    parse reads a document once and hands back a Document. Nothing is rendered until you ask for it, and both renderings come from the same tree.

    test {
    let document = @moonmd.parse("# Title\n\nA *short* paragraph with a [link](/a).\n")
    assert_eq(
    document.html(safe=false),
    "<h1>Title</h1>\n<p>A <em>short</em> paragraph with a <a href=\"/a\">link</a>.</p>\n",
    )
    assert_eq(document.markdown(), "# Title\n\nA *short* paragraph with a [link](/a).\n")
    }

    html escapes raw HTML and refuses dangerous destinations unless you pass safe=false; markdown writes the tree back as Markdown that reads the same way it did.

    #The tree

    A Document holds top level Blocks in order. There are nine block kinds and ten inline kinds; each of the types and their fields is listed in the API reference. walk visits every node, each container before its children:

    test {
    let document = @moonmd.parse("# One\n\nA [link](/a) and a *word*.\n")
    let counts = [0, 0]
    @moonmd.walk(document, fn(node) {
    match node {
    @moonmd.Block(@moonmd.Heading(_)) => counts[0] = counts[0] + 1
    @moonmd.Inline(@moonmd.Link(_)) => counts[1] = counts[1] + 1
    _ => ()
    }
    })
    assert_eq(counts, [1, 1])
    }

    #Footnotes

    A [^label] in the text is a footnote reference; a line that opens with [^label]: is its definition. The reference becomes a numbered marker, the definition is moved to the end of the document as a notes list, and a link in each note points back to its reference.

    test {
    let document = @moonmd.parse("A claim[^note]\n\n[^note]: the support\n")
    let expected =
    "<p>A claim<sup class=\"footnote-ref\"><a href=\"#fn-note\" " +
    "id=\"fnref-note\" data-footnote-ref>1</a></sup></p>\n" +
    "<section class=\"footnotes\" data-footnotes>\n" +
    "<ol>\n" +
    "<li id=\"fn-note\">\n" +
    "<p>the support <a href=\"#fnref-note\" class=\"footnote-backref\" " +
    "data-footnote-backref data-footnote-backref-idx=\"1\" " +
    "aria-label=\"Back to reference 1\">\u{21A9}</a></p>\n" +
    "</li>\n</ol>\n</section>\n"
    assert_eq(document.html(safe=false), expected)
    assert_eq(document.markdown(), "A claim[^note]\n\n[^note]:\n the support\n")
    }

    The number follows the first reference, not the definition, and a reference with no definition stays the text it was written as. The block is a Footnote; the reference is a FootnoteRef. Footnotes are part of gfm and off in commonmark.

    #Flavors

    Flavor is a plain record. gfm is the default; commonmark turns every extension off, and any field can be changed on its own:

    test {
    assert_eq(@moonmd.parse("~~struck~~\n").html(safe=false), "<p><del>struck</del></p>\n")
    assert_eq(
    @moonmd.parse("~~struck~~\n", flavor=@moonmd.commonmark).html(safe=false),
    "<p>~~struck~~</p>\n",
    )
    }

    #Streaming

    A model writes a document a token at a time. Stream::feed takes whatever has arrived and returns the blocks that no later text can change; finish returns the rest. A block is settled when a following line has closed it, so a chunk may end anywhere, including in the middle of a line:

    test {
    let stream = @moonmd.Stream::new()
    let counts = [
    stream.feed("# Hi"[:]).length(),
    stream.feed("\n\n"[:]).length(),
    stream.feed("a paragraph"[:]).length(),
    stream.finish().length(),
    ]
    assert_eq(counts, [0, 1, 0, 1])
    }

    Link reference definitions apply to the whole document, so a block that mentions [ waits until finish by default; passing defer_links=false hands such a block out as soon as it closes.

    #Safety

    html defaults to safe=true: raw HTML is escaped, and a link whose destination would run script keeps its text and loses its anchor.

    test {
    assert_eq(@moonmd.parse("<div>hi</div>\n").html(), "&lt;div&gt;hi&lt;/div&gt;\n")
    assert_eq(@moonmd.parse("[bad](javascript:alert(1))\n").html(), "<p>bad</p>\n")
    }

    With safe=false the document is written as it was read.

    #Conformance

    The test suite replays the examples of the CommonMark 0.31.2 specification and the extension corpus of the GitHub implementation. Both are generated into the test package by scripts/gen_spec.py, with the version pinned.

    CorpusExamplesPassing
    CommonMark 0.31.2 spec.json652652
    GitHub Flavored Markdown extensions2929

    Every example in both corpora produces the exact HTML each expects, including the three footnote examples (23, 24 and 25). Two further properties hold over the same corpora:

    • writing a tree back as Markdown and reading it again gives the same HTML, for all 652 CommonMark examples and all 29 extension examples;
    • feeding an example in pieces of one, two, three, five and eight characters gives the same blocks as reading it whole, for the same 652 and 29 examples.

    #License

    Apache-2.0.

    Align

    pub(all) enum Align {
    Default
    Left
    Center
    Right
    }

    How a table column lines up its cells, taken from the delimiter row.

    Block

    pub(all) enum Block {
    Heading(Heading)
    Paragraph(Array[Inline])
    Code(Code)
    Quote(Array[Block])
    List(ListBlock)
    Table(Table)
    Html(String)
    Break
    Footnote(Footnote)
    }

    A block level node.

    Break

    pub(all) enum Break {
    Soft
    Hard
    }

    A line break inside a paragraph.

    Code

    pub(all) struct Code {
    info : String
    text : String
    fenced : Bool
    }

    A code block, fenced or indented. info is the word after the opening fence, empty for indented code.

    Document

    pub struct Document {
    // private fields
    }

    A parsed document: the blocks in order, and the flavor it was read with, which the HTML renderer needs for the tag filter.

    Document::blocks

    fn Document::blocks(self : Document) -> Array[Block]

    The top level blocks of the document, in order.

    Document::flavor

    fn Document::flavor(self : Document) -> Flavor

    The flavor this document was parsed with.

    Document::html

    fn Document::html(self : Document, safe? : Bool) -> String

    Render the document as HTML. With safe set, which is the default, raw HTML is escaped and links to dangerous destinations lose their anchor; pass safe=false to write a document that may embed HTML of its own.

    Document::markdown

    fn Document::markdown(self : Document) -> String

    Render the document as Markdown.

    Document::of_blocks

    fn Document::of_blocks(blocks : Array[Block], flavor? : Flavor) -> Document

    Assemble a document out of blocks, which is what a caller collecting the output of a Stream needs in order to render it. Footnote references are resolved against the definitions in the blocks, so a definition read after the reference that uses it still applies.

    Flavor

    pub(all) struct Flavor {
    tables : Bool
    strikethrough : Bool
    autolinks : Bool
    task_lists : Bool
    tag_filter : Bool
    footnotes : Bool
    }

    What a parser accepts beyond the CommonMark core. The presets commonmark and gfm cover the two dialects this library is asked for; the fields are public so a caller can take one and drop what it does not want.

    Footnote

    pub(all) struct Footnote {
    label : String
    blocks : Array[Block]
    number : Int
    backrefs : Int
    }

    A footnote, the body of one [^label]: definition. label is the label as it was written. number is the order the footnote is shown with, assigned in the order its first reference appears in the text; backrefs is how many references point at it.

    FootnoteRef

    pub(all) struct FootnoteRef {
    label : String
    number : Int
    ref_ix : Int
    }

    A footnote reference, [^label], and where it lands: the definition's label, the number that definition is shown with, and ref_ix, the order of this reference among the ones that point at the definition.

    Heading

    pub(all) struct Heading {
    level : Int
    content : Array[Inline]
    }

    A heading, level counting the ATX # marks or the setext underline.

    Inline

    pub(all) enum Inline {
    Text(String)
    Code(String)
    Emphasis(Array[Inline])
    Strong(Array[Inline])
    Link(Link)
    Image(Link)
    Html(String)
    Strike(Array[Inline])
    Break(Break)
    FootnoteRef(FootnoteRef)
    }

    An inline level node.

    Item

    pub(all) struct Item {
    blocks : Array[Block]
    task : Task
    }

    One item of a list, with the task marker that introduced it if any.
    pub(all) struct Link {
    dest : String
    title : String
    content : Array[Inline]
    }

    The destination, title and content of a link or image. For an image the content is the alternative text.

    ListBlock

    pub(all) struct ListBlock {
    kind : ListKind
    tight : Bool
    items : Array[Item]
    }

    A list, with its items and whether it is tight (no blank lines between items, so their paragraphs are rendered without <p>).

    ListKind

    pub(all) enum ListKind {
    Bullet(Char)
    Ordered(Int, Char)
    }

    What kind of list a ListBlock is, together with the marks needed to write it back unchanged.

    Node

    pub(all) enum Node {
    Block(Block)
    Inline(Inline)
    }

    A node as walk hands it to its visitor: either a block or an inline.

    Stream

    pub struct Stream {
    // private fields
    }

    A parser that reads a document in pieces. Text that has arrived is parsed again on every call, but only the part that has not been handed out yet, so the work per chunk is proportional to the unfinished tail rather than to the whole document.

    Stream::feed

    fn Stream::feed(self : Stream, chunk : StringView) -> Array[Block]

    Take the next piece of the document and return the blocks it completed.

    Stream::finish

    fn Stream::finish(self : Stream) -> Array[Block]

    Close the stream and return whatever was still open.

    Stream::new

    fn Stream::new(flavor? : Flavor, defer_links? : Bool) -> Stream

    Start a stream.

    defer_links keeps a block that mentions [ back until the whole document has arrived, because a link reference definition written later still applies to it. With it off such a block is handed out as soon as it closes, at the cost of a forward reference not resolving; the blocks are then the same as a whole parse for any document that defines its labels before using them.

    Table

    pub(all) struct Table {
    align : Array[Align]
    head : Array[Array[Inline]]
    body : Array[Array[Array[Inline]]]
    }

    A table: the alignment of each column, the header row, then the body rows. A row holds one cell per column, and a cell holds the inlines inside it.

    Task

    pub(all) enum Task {
    Plain
    Unchecked
    Checked
    }

    Whether a list item is a task list item, and if so whether it is done.

    commonmark

    let commonmark : Flavor

    CommonMark without any of the extensions.

    flavor

    let flavor : Flavor

    The preset parse uses when the caller does not pass one: gfm.

    gfm

    let gfm : Flavor

    CommonMark as GitHub reads it: tables, strikethrough, bare links, task list items, the tag filter and footnotes. This is the default.

    parse

    fn parse(text : StringView, flavor? : Flavor) -> Document

    Parse a Markdown document.

    test {
    let document = @moonmd.parse("# Title")
    assert_eq(document.html(safe=false), "<h1>Title</h1>\n")
    }

    walk

    fn walk(document : Document, visit : (Node) -> Unit) -> Unit

    Visit every node of the document, each container before its children and siblings in order.