🥮 mooncakes.io
    Sign in

    moonurl

    URLs for MoonBit: WHATWG and RFC 3986 parsing, IDNA, URI templates, percent coding.

    url
    uri
    rfc3986
    whatwg
    idna
    punycode
    uri-template
    rfc6570
    percent-encoding
    data-url
    urlencoded
    moonbit
    Download zip
    Version
    0.2.1
    License
    Apache-2.0
    Repository
    Last updated
    2 hours ago
    Downloads
    320

    #moonurl

    URLs for MoonBit, read the way a browser reads them and the way RFC 3986 says to read them: one type, two grammars. Host names are internationalised, so münchen.example and xn--mnchen-3ya.example are the same host. URI templates fill in and read back. Form bodies and data URLs ride on the same percent coding. No sockets.

    // A string a browser and a generic parser disagree about.
    @moonurl.Url::decode("HTTP://EXAMPLE.test:80/a/../b").encode()
    // "HTTP://EXAMPLE.test:80/a/../b"

    @moonurl.Url::decode("HTTP://EXAMPLE.test:80/a/../b", syntax=@moonurl.Whatwg).encode()
    // "http://example.test/b"

    // Host names, in and out of ASCII.
    @moonurl.domain("MÜNCHEN.example")                       // "xn--mnchen-3ya.example"
    @moonurl.Url::decode("http://192.0x00A80001/", syntax=@moonurl.Whatwg).host // Some("192.168.0.1")

    // A template filled in, and a URI read back.
    @template.Template::decode("/users/{id}{?page}").expand(values)   // "/users/42?page=2"
    @template.Template::decode("/users/{id}{?page}").capture("/users/42?page=2")

    moon run src/examples/tour takes a reference apart, resolves the RFC 3986 §5.4 examples, escapes one string for four components and builds a query. moon run src/examples/whatwg reads eight strings both ways, resolves five references against a base, converts four host names and sets five components. moon run src/examples/uritemplate fills in templates at every level and reads a URI back. moon run src/examples/data reads the four examples RFC 2397 prints.

    #Packages

    PackageWhatSpecification
    moonurlUrl: parsing, recomposing, resolving, normalising, origins, settersRFC 3986, WHATWG URL Standard
    moonurlPercent coding by component, KindRFC 3986 §2
    moonurlQuery: the pairs of a query or a form bodyWHATWG URL Standard §5
    moonurldomain: a host name in ASCIIUTS 46
    moonurl/idnaPunycode, and the mapping and validity rules behind itRFC 3492, UTS 46
    moonurl/templateURI templates: parse, expand, captureRFC 6570
    moonurl/dataurlA whole representation written inside the reference to itRFC 2397

    #Two grammars, one type

    RFC 3986 describes a reference as five components and says nothing about what a scheme means. A browser has to know: http has a port and file does not, \ is a solidus, a host may be a number in any of its spellings. Both readings produce the same Url, so a program that has to accept what people type can switch grammar for one string and keep working with the result.

    let url = @moonurl.Url::decode("https://leo:pw@example.test:8443/a/b?x=1#top")
    url.scheme      // Some("https")
    url.host        // Some("example.test")
    url.port        // Some(8443)
    url.query       // Some("x=1")
    url.origin()    // "https://example.test:8443"

    syntax picks the grammar, and rfc3986 is what it defaults to, so nothing written against the earlier version of this package changes. base resolves a reference in either grammar.

    The WHATWG reading adds what a browser needs and RFC 3986 deliberately leaves out: the special schemes and their default ports (http://h:80/ loses the port), paths normalised on the way in, hosts as names, as IPv4 in decimal, octal or hexadecimal, as IPv6, or as an opaque string under a scheme of your own. Url::origin answers which URLs are the same site; the nine setters — set_scheme, set_username, set_password, set_host, set_hostname, set_port, set_path, set_query, set_fragment — change one component and leave the rest where it was, tools and all.

    #Internationalised host names

    A host name is a sequence of labels, and a label may be any word in any script. moonurl/idna maps a name to its ASCII form and back: the mapping table, NFC, the hyphen rules, the joiner rules of ContextJ, the bidi rule of RFC 5893 and the DNS lengths. to_ascii and to_unicode return the name the standard converts and what it objected to, because a browser shows the rest of a name it cannot finish converting. moonurl.domain is the one a URL parser uses: an ASCII string is taken as it stands, and everything else goes through UTS 46.

    let outcome = @idna.to_ascii("münchen.example")
    outcome.name      // "xn--mnchen-3ya.example"
    outcome.is_ok()   // true

    // Two settings, for the two callers there are.
    @idna.to_ascii("Faß.de").name                                  // "xn--fa-hia.de"
    @idna.to_ascii("Faß.de", options={ ..@idna.options, transitional: true }).name // "fass.de"

    #URI templates

    RFC 6570 puts holes in a URI and names them. All four levels are here: the simple and reserved expansions, the label, path and parameter operators, prefixes and exploded lists and pairs. capture goes the other way, reading a URI into the names that made it.

    let values : Map[String, @template.Value] = Map([])
    values.set("list", @template.Value::List(["red", "green"]))
    values.set("q", @template.Value::Text("moon bit"))

    @template.Template::decode("{/list*}{?q}").expand(values)   // "/red/green?q=moon%20bit"
    @template.Template::decode("{/list*}{?q}").capture("/red/green?q=moon%20bit")

    Values come from anything that implements Variables; a Map[String, Value] already does, and so can a request, a configuration record or a database row.

    #Form encoding

    Query is a URL's query or a form body as ordered pairs, repeated names kept. + is a space there and nowhere else in a URL, and the serialiser keeps only letters, digits and *-._, so ~ goes out as %7E although RFC 3986 leaves it alone.

    Query::decode refuses the whole input past its limits: a thousand pairs of up to 1 MiB each by default. Change one with Limits::new(pairs=100) or { ..@moonurl.limits, pairs: 100 }; the two are the same value. A URL's query is the same thing, read the same way: url.params().

    #Interface

    ItemShapeWhat it does
    Url::decode(text, syntax?, base?)Url raise RefusedRead a reference, in either grammar
    Url::encode()StringWrite it back
    Url::resolve(reference)UrlResolve against this base (RFC 3986)
    Url::normalize()UrlCase, escapes and dot segments (§6.2.2)
    Url::origin()Stringscheme://host:port, or "null"
    Url::authority()String?The authority, brackets restored
    Url::is_absolute()BoolA scheme and no fragment
    Url::params()Query raise RefusedThe query as pairs
    Url::set_*UrlOne component changed, the rest kept
    Url::new(..)UrlA reference built piece by piece
    domain(text, strict?)String raise RefusedA host name in ASCII
    escape / escape_textStringPercent-encode octets or text
    unescape / unescape_textBytes / StringUndo it
    escaped(text, kind?)BoolWhether every escape is complete
    normalize_escapes(text)StringUpper-case hex, unreserved octets plain
    remove_dots(path)StringThe . and .. segments out
    Query::decode(bytes, limits?)Query raise RefusedA form body or a query
    Query::encode()StringThe same body written back
    idna::to_ascii / to_unicodeOutcomeA name in ASCII, or in Unicode
    idna::encode / decodeString raise RefusedOne label of Punycode
    Template::decode(text)Template raise RefusedRead a template
    Template::expand(values)StringFill it in
    Template::capture(uri)Array[(String, String)]?Read a URI back into names
    dataurl::Url::decode(text)Url raise RefusedA data URL

    Errors are one enum per package — Refused — and every offset in one is at.

    #Limits

    Query::decode bounds a form body; Url and Template do not bound their input, since a caller that has read the bytes usually knows how long they are allowed to be. Neither does the IDNA conversion, which the DNS limits can cut short (strict turns them on).

    #Licence

    Apache-2.0. See LICENSE.

    Refused

    source
    pub(all) suberror Refused {
      Port(at~ : Int)
      Host(at~ : Int)
      Name(at~ : Int)
      Address(at~ : Int)
      Scheme(at~ : Int)
      Pairs(limit~ : Int)
      Pair(limit~ : Int, got~ : Int)
    } derive(Eq, 
    Debug    )

    Why a string is not a URI reference. Only the authority can fail: §B splits any string at all into the five top-level components, and does not split that one.

    Refused::equal

    source
    fn Refused::equal(Refused, Refused) -> Bool

    Refused::not_equal

    source
    fn Refused::not_equal(x : Refused, y : Refused) -> Bool

    Refused::to_repr

    source

    Kind

    source
    pub(all) enum Kind {
      Bare
      Host
      Userinfo
      Segment
      Path
      Query
      Fragment
      Whole
    } derive(Eq, 
    Debug    )

    Which component an octet is going into, and so which octets may stand for themselves (RFC 3986 §2.2, §3).

    & is ordinary inside a path and a separator inside a query. The grammar answers that, not the caller.

    Kind::equal

    source
    fn Kind::equal(Kind, Kind) -> Bool

    Kind::not_equal

    source
    fn Kind::not_equal(x : Kind, y : Kind) -> Bool

    Kind::to_repr

    source

    Limits

    source
    pub(all) struct Limits {
      pairs : Int
      size : Int
    } derive(Eq, 
    Debug    )

    What a decode accepts. Past either bound the whole input is refused rather than cut short, so a handler never acts on part of a form.

    Limits::equal

    source
    fn Limits::equal(Limits, Limits) -> Bool

    Limits::new

    source
    fn Limits::new(pairs? : Int, size? : Int) -> Limits

    Limits by name, each defaulting to the preset's.

    Limits::not_equal

    source
    fn Limits::not_equal(x : Limits, y : Limits) -> Bool

    Limits::to_repr

    source

    Query

    source
    pub struct Query {
      // private fields
    } derive(Eq, 
    Debug    )

    The name–value pairs of an application/x-www-form-urlencoded string: a URL's query, or a form body (WHATWG URL Standard §5). Order and repeated names are kept, as URLSearchParams and Go's url.Values keep them.

    Query::add

    source
    fn Query::add(self : Query, name : String, value : String) -> Unit

    Add a pair at the end, keeping any already under name.

    Query::decode

    source
    fn Query::decode(raw : BytesView, limits? : Limits) -> Query raise Refused

    Read an application/x-www-form-urlencoded string (WHATWG URL Standard §5.1).

    & separates pairs and the first = separates a name from its value; + is a space here and nowhere else in a URL. Escapes are undone to octets and the octets read as UTF-8, a broken sequence becoming U+FFFD. An empty piece between two & is no pair at all.

    Query::encode

    source
    fn Query::encode(self : Query) -> String

    Write the pairs as application/x-www-form-urlencoded (WHATWG URL Standard §5.2): letters, digits and *-._ stand for themselves, a space is +, and every other octet of the UTF-8 is %XX.

    Query::entries

    source
    fn Query::entries(self : Query) -> Array[(String, String)]

    Every pair, in order.

    Query::equal

    source
    fn Query::equal(Query, Query) -> Bool

    Query::get

    source
    fn Query::get(self : Query, name : StringView) -> String?

    The first value under name.

    Query::has

    source
    fn Query::has(self : Query, name : StringView) -> Bool

    Whether any pair is named name.

    Query::new

    source
    fn Query::new(pairs? : Array[(String, String)]) -> Query

    A query holding pairs, in order.

    Query::not_equal

    source
    fn Query::not_equal(x : Query, y : Query) -> Bool

    Query::remove

    source
    fn Query::remove(self : Query, name : StringView) -> Unit

    Remove every pair under name.

    Query::set

    source
    fn Query::set(self : Query, name : String, value : String) -> Unit

    Leave exactly one pair under name, holding value: the first one's place if there was one, the end if not.

    Query::to_repr

    source

    Query::values

    source
    fn Query::values(self : Query, name : StringView) -> Array[String]

    Every value under name, in order.

    Syntax

    source
    pub(all) enum Syntax {
      Rfc3986
      Whatwg
    } derive(Eq, 
    Debug    )

    Which grammar decides what a string means. The two disagree about almost everything a browser has to get right, and both are in the standard library of every language that talks to the web.

    Syntax::equal

    source
    fn Syntax::equal(Syntax, Syntax) -> Bool

    Syntax::not_equal

    source
    fn Syntax::not_equal(x : Syntax, y : Syntax) -> Bool

    Syntax::to_repr

    source

    Url

    source
    pub(all) struct Url {
      scheme : String?
      userinfo : String?
      host : String?
      port : Int?
      path : String
      query : String?
      fragment : String?
    } derive(Eq, 
    Debug    )

    A URI reference taken apart (RFC 3986 §3).

    No scheme makes it relative (§4.2). No host means no authority, which is not an empty host — file:///tmp has one, mailto:x@y has none.

    Fields stay percent-encoded. Decoding on the way in would lose the difference between a / that separates two segments and a %2F inside one.

    Url::authority

    source
    fn Url::authority(self : Url) -> String?

    The authority as one string, brackets restored around an IP-literal (§3.2).

    Url::decode

    source
    fn Url::decode(text : StringView, syntax? : Syntax, base? : Url) -> Url raise Refused

    Take a reference apart.

    Under Rfc3986 any string splits (§3), and only the authority can be malformed. Under Whatwg the string is what a browser would make of it: a reference that does not parse is refused rather than guessed at.

    base resolves a relative reference against one, in whichever grammar is asked for. The WHATWG algorithm takes the base as text and parses it again, so a base that is itself not a URL is refused here too.

    test {
      // RFC 3986 keeps the reference as it was written.
      let url = @moonurl.Url::decode("HTTP://EXAMPLE.test:80/a/../b?q#f")
      assert_eq(url.encode(), "HTTP://EXAMPLE.test:80/a/../b?q#f")
      let whatwg = @moonurl.Url::decode(
        "HTTP://EXAMPLE.test:80/a/../b",
        syntax=@moonurl.Whatwg,
      )
      assert_eq(whatwg.encode(), "http://example.test/b")
    }

    Url::encode

    source
    fn Url::encode(self : Url) -> String

    Put the reference back together (RFC 3986 §5.3).

    Url::equal

    source
    fn Url::equal(Url, Url) -> Bool

    Url::is_absolute

    source
    fn Url::is_absolute(self : Url) -> Bool

    A reference with a scheme and no fragment is a URI (§3); anything else is a relative reference or a same-document one.

    Url::new

    source
    fn Url::new(scheme? : String, userinfo? : String, host? : String, port? : Int, path? : String, query? : String, fragment? : String) -> Url

    A reference this shape, for building one rather than reading one.

    Url::normalize

    source
    fn Url::normalize(self : Url) -> Url

    Syntax-based normalisation (RFC 3986 §6.2.2): scheme and host to lower case, escapes normalised, dot segments out.

    Not §6.2.3: dropping :80 is true of http and false in general, and this package holds no register of schemes.

    Url::not_equal

    source
    fn Url::not_equal(x : Url, y : Url) -> Bool

    Url::origin

    source
    fn Url::origin(self : Url) -> String

    The origin of a URL (§4.7): what two URLs have to share to be the same site, written scheme://host:port.

    A blob: URL carries the origin of the URL inside it. Everything else that is not a scheme with an authority of its own — file:, data:, mailto: — has an origin no other URL shares, which is written "null".

    test {
      let url = @moonurl.Url::decode(
        "https://example.test/a",
        syntax=@moonurl.Whatwg,
      )
      assert_eq(url.origin(), "https://example.test")
      let file = @moonurl.Url::decode("file:///tmp/x", syntax=@moonurl.Whatwg)
      assert_eq(file.origin(), "null")
    }

    Url::params

    source
    fn Url::params(self : Url) -> Query raise Refused

    The query as pairs, read the way URLSearchParams reads it (WHATWG URL Standard §5.1).

    test {
      let url = @moonurl.Url::decode(
        "https://h/?a=1&a=2&b=x+y",
        syntax=@moonurl.Whatwg,
      )
      assert_eq(url.params().values("a").length(), 2)
      assert_eq(url.params().get("b").unwrap(), "x y")
    }

    Url::resolve

    source
    fn Url::resolve(self : Url, reference : Url) -> Url

    Resolve reference against this base (RFC 3986 §5.2.2, strict).

    Strict: a reference that repeats the base's scheme stays absolute. The non-strict reading is for pre-1998 parsers and turns http:g into something it does not say.

    Url::set_fragment

    source
    fn Url::set_fragment(self : Url, value : StringView) -> Url

    This URL with its fragment replaced, the leading # of the value optional. An empty value takes the fragment away.

    Url::set_host

    source
    fn Url::set_host(self : Url, value : StringView) -> Url

    This URL with its host replaced, and its port too if the value carries one.

    An empty value is a host with nothing in it, not a missing host: the URL keeps its shape and loses its authority's middle.

    Url::set_hostname

    source
    fn Url::set_hostname(self : Url, value : StringView) -> Url

    This URL with its host replaced and its port left alone.

    Url::set_password

    source
    fn Url::set_password(self : Url, value : StringView) -> Url

    This URL with its password replaced.

    Url::set_path

    source
    fn Url::set_path(self : Url, value : StringView) -> Url

    This URL with its path replaced. A URL whose path is opaque — mailto:, data: — has no path to replace, and keeps the one it has.

    Url::set_port

    source
    fn Url::set_port(self : Url, value : StringView) -> Url

    This URL with its port replaced. An empty value takes the port away, which is not the same as setting it to zero.

    Url::set_query

    source
    fn Url::set_query(self : Url, value : StringView) -> Url

    This URL with its query replaced, the leading ? of the value optional. An empty value takes the query away.

    Url::set_scheme

    source
    fn Url::set_scheme(self : Url, value : StringView) -> Url

    This URL with its scheme replaced, or this URL if the value is not a scheme.

    A URL may change between special and non-special schemes only through another special one, and only a URL with no credentials, port or file host can become file; the setter leaves anything else as it was, as the standard's own setter does.

    test {
      let url = @moonurl.Url::decode(
        "http://example.test/a",
        syntax=@moonurl.Whatwg,
      )
      assert_eq(url.set_scheme("https").encode(), "https://example.test/a")
      assert_eq(url.set_scheme("1bad").encode(), "http://example.test/a")
    }

    Url::set_username

    source
    fn Url::set_username(self : Url, value : StringView) -> Url

    This URL with its username replaced.

    Url::to_repr

    source

    domain

    source
    fn domain(text : StringView, strict? : Bool) -> String raise Refused

    The domain parser of §3.3: the host name a browser would make of text.

    Strict is for a name that has to be one: CheckHyphens, the STD 3 rules and the DNS lengths are all on, and invalid Punycode is refused.

    Lenient is the browser: an ASCII string is taken as it stands, lowercased, whatever IDNA would say about the label — a name that reads as Punycode but is not still addresses something. Everything else goes through UTS 46.

    test {
      assert_eq(@moonurl.domain("MÜNCHEN.example"), "xn--mnchen-3ya.example")
      assert_eq(@moonurl.domain("xn--a"), "xn--a")
    }

    escape

    source
    fn escape(raw : BytesView, kind? : Kind, spare? : StringView) -> String

    Percent-encode octets for kind (RFC 3986 §2.1).

    Octets, not text: what travels inside a URI is whatever the caller encoded it as. [escape_text] is the one that assumes UTF-8.

    spare is for percent coding that is not going into a URI at all — gRPC's grpc-message keeps all of printable ASCII but %, which is no component of RFC 3986's. A URI component names its kind and leaves spare alone.

    escape_text

    source
    fn escape_text(text : StringView, kind? : Kind, spare? : StringView) -> String

    Percent-encode text as UTF-8 (RFC 3986 §2.5).

    escaped

    source
    fn escaped(text : StringView, kind? : Kind, spare? : StringView) -> Bool

    Whether every % is a complete escape and nothing outside kind's set is there unescaped — the check [unescape] does not make.

    kind

    source
    let kind : Kind

    The strictest component, for when nothing says. Escaping an octet that did not need it changes nothing; leaving one that did changes where the component ends.

    limits

    source
    let limits : Limits

    A thousand pairs of up to 1 MiB each: Starlette's max_fields and max_part_size, and the parameterLimit of Node's qs.

    normalize_escapes

    source
    fn normalize_escapes(text : StringView) -> String

    Percent-encoding normalised (RFC 3986 §6.2.2.2): hex digits in upper case, and escapes of unreserved octets written out plainly.

    remove_dots

    source
    fn remove_dots(path : String) -> String

    Take out the . and .. segments (RFC 3986 §5.2.4).

    The section's loop rather than a split-and-rejoin: the two differ at exactly the inputs that matter — a trailing .., a path that climbs above its own root, /..//.

    rfc3986

    source
    let rfc3986 : Syntax

    The grammar [Url::decode] uses when nothing says.

    unescape

    source
    fn unescape(raw : BytesView) -> Bytes

    Undo percent-encoding (RFC 3986 §2.1).

    A % without two hex digits after it stands for itself. This runs on request targets and form bodies, where refusing the whole message over one stray % helps nobody; [escaped] is for the caller who would rather be told.

    unescape_text

    source
    fn unescape_text(text : StringView) -> String

    Undo percent-encoding and read the result as UTF-8, replacing what is not.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io