HTTP support for moonbitlang/async.

    #Making simple HTTP request

    Simple HTTP request can be made in just one line:

    ///|
    #cfg(any(target="native", target="js"))
    async test {
    let (response, body) = @http.get("https://www.moonbitlang.com")
    inspect(response.code, content="200")
    assert_true(body.text().has_prefix("<!doctype html>"))
    }

    You can use use body.text() to get a String (decoded via UTF8) or body.json() for a Json from the response body.

    #Generic HTTP client

    Sometimes, the simple one-time @http.get etc. is insufficient, for example you need to reuse the same connection for multiple requests, or the request/response body is very large and need to be processed lazily. In this case, you can use the @http.Client type. @http.Client can be created via @http.Client(uri).

    The workflow of performing a request with @http.Client is:

    1. initiate the request via client.request(..)
    2. send the request body by using @http.Client as a @io.Writer
    3. complete the request and obtain response header from the server via client.end_request()
    4. read the response body by using @http.Client as a @io.Reader, or use client.read_all() to obtain the whole response body. Yon can also ignore the body via client.skip_response_body()

    The helpers client.get(..), client.put(..) etc. can be used to perform step (1)-(3) above.

    A complete example:

    ///|
    #cfg(any(target="native", target="js"))
    async test {
    let client = @http.Client("https://www.moonbitlang.com")
    defer client.close()
    let response = client..request(Get, "/").end_request()
    inspect(response.code, content="200")
    let body = client.read_all()
    assert_true(body.text().has_prefix("<!doctype html>"))
    }

    All HTTP client API, including the simple request helpers @http.get etc., are supported on JavaScript backend using fetch API, meaning that you can use these API in browser environment. Notice that some feature, such as proxy, is not supported on JavaScript backend.

    #Writing HTTP servers

    The recommended way to create HTTP servers is @http.Server::run_forever(..). A HTTP server should first get created via @http.Server(..), after that, server.run_forever(f) automatically start and run the server. The callback function f is used to handle HTTP requests. It receives three parameters:

    • the request to process
    • a &@io.Reader that can be used to read request body
    • a @http.ServerConnection that can be used to send response. The procedure to send a response is:
      1. initiate a response and send response header via .send_response()
      2. send response body by using the @http.ServerConnection as @io.Writer
      3. (optional) complete the response via .end_response(). If .end_response() is not called, it will be called automatically after f returns normally.

    Here's an example server that returns 404 to every request:

    ///|
    #cfg(target="native")
    pub async fn server(listen_addr : @socket.Addr) -> Unit {
    @http.Server(listen_addr).run_forever((request, _body, conn) => {
    conn..send_response(404, "NotFound").write("`\{request.path}` not found")
    })
    }

    Headers

    type Headers = Map[CaseInsensitiveString, String]

    Type for HTTP headers. The names of headers are case insensitive.

    The RFC requires that multiple header entries with the same name should be semantically equivalent to concatenating values with comma, so a Map with unique keys can be used to represent headers.

    One important exception to the above rule is the Set-Cookie header, which is handled specially via the Cookie type.

    HttpProtocolError

    pub suberror HttpProtocolError {
    BadRequest
    HttpVersionNotSupported(String)
    NotImplemented
    } derive(ToJson,
    Debug
    )

    IncorrectBodyLength

    pub suberror IncorrectBodyLength derive(ToJson,
    Debug
    )

    ProxyError

    pub suberror ProxyError {
    ProxyError(Response)
    } derive(ToJson,
    Debug
    )

    Error raised when the proxy responded with a non-2XX response code
    impl Show for ProxyError

    URIParseError

    pub suberror URIParseError {
    InvalidFormat
    UnsupportedProtocol(String)
    } derive(ToJson,
    Debug
    )

    CaseInsensitiveString

    pub(all) struct CaseInsensitiveString(String) derive(ToJson)

    String type with case insensitive comparison (in the ASCII sense). This type is implicitly convertible from String & string literal.

    CaseInsensitiveString::to_string

    fn CaseInsensitiveString::to_string(self : CaseInsensitiveString) -> String

    Client

    type Client

    Simple HTTP client which connect to a remote host via TCP
    impl Reader for Client
    impl Writer for Client

    Client::Client

    #alias(new, deprecated="`new` is deprecated, use `Client` instead")
    async fn Client::Client(uri : String, headers? : Map[CaseInsensitiveString, String], proxy? : Client, verify? : Bool, trust? :
    TrustedRoot
    ) -> Client

    Create a new HTTP client by connecting to a remote host. Host should be specified via protocol://host[:port], where protocol is one of http or https. If protocol is https, a TLS connection will be established, and the certificate of the remote peer will be verified.

    If the protocol is https, trust will determine the trusted root for cert validation. See @tls.TrustedRoot for more details.

    headers can be used to specify persistent headers for the client, i.e. all requests made from this client will share these headers. The ownership of headers will be transferred to the new client, so headers should not be used by the caller later. The following headers is automatically set, and must not be specified in headers:

    • Host
    • Content-Length, Transfer-Encoding

    If proxy is present, it should be another HTTP client in a clean state. The new client will send a CONNECT request via the proxy client and try to establish a tunnel via the proxy client. All subsequent requests made by the new client will go through the proxy tunnel. The ownership of the proxy client is transferred to the new client, so it must not be used nor closed anymore by the caller. Using another HTTP client as proxy allows advanced features such as proxy authentication and https CONNECT proxy.

    Client::close

    fn Client::close(self : Client) -> Unit

    Close a HTTP client and release underlying resource. In particular close the underlying TCP connection. This function is idempotent: it is safe to call .close() multiple times, only the first .close() call takes effect.

    Client::drop

    async fn Client::drop(self : Client, len : Int) -> Int

    Client::end_request

    async fn Client::end_request(self : Client, on_informational_response? : async (Response) -> Unit) -> Response

    End the body of the request currently being sent, and obtain response from the server. Should be called immediately after request body is fully sent.

    Only the header of the response will be received and returned, the body of the response can be extracted by using Client as a @io.Reader.

    If the body of the last response is still not consumed, it will be discarded.

    1XX response, except for 101 Switching Protocol, will be automatically processed and discarded, and the returned response is always the final response (or 101). on_informational_response, if present, will be invoked for every skipped 1XX informational response.

    Client::enter_passthrough_mode

    async fn Client::enter_passthrough_mode(self : Client) -> Unit

    Let the client enter "pass through" mode, where the client serve as a TCP tunnel (maybe TLS encrypted) directly. This is useful for the special CONNECT HTTP request and HTTP protocol upgrade.

    In passthrough mode, Read/write the client becomes direct read/write on the underlying connection, and all API except @io.Reader and @io.Writer must not be used anymore.

    When entering pass through mode, the client must be in a clean state (i.e. not in the middle of sending a request). Unread data from the body of the last response will be discarded.

    Client::flush

    async fn Client::flush(self : Client) -> Unit

    Flush buffered data in the request body being sent, if any.

    Client::get

    async fn Client::get(self : Client, path : String, extra_headers? : Map[CaseInsensitiveString, String], body? : &
    Data
    ) -> Response

    Perform a GET request to the server, see Client::request for more details.

    Client::post

    async fn Client::post(self : Client, path : String, body : &
    Data
    , extra_headers? : Map[CaseInsensitiveString, String]) -> Response

    Perform a POST request to the server, see Client::request for more details.

    Client::put

    async fn Client::put(self : Client, path : String, body : &
    Data
    , extra_headers? : Map[CaseInsensitiveString, String]) -> Response

    Perform a PUT request to the server, see Client::request for more details.

    Client::read

    async fn Client::read(self : Client, dst : FixedArray[Byte], offset? : Int, max_len? : Int) -> Int

    Client::read_all

    async fn Client::read_all(self : Client) -> &
    Data

    Client::read_exactly

    async fn Client::read_exactly(self : Client, len : Int) -> Bytes

    Client::read_some

    async fn Client::read_some(self : Client, max_len? : Int) -> Bytes?

    Client::read_until

    async fn Client::read_until(self : Client, sep : StringView) -> String?

    Client::request

    async fn Client::request(self : Client, meth : RequestMethod, path : StringView, extra_headers? : Map[CaseInsensitiveString, String]) -> Unit

    Send a HTTP request to the server. Only the header of the request will be sent, request body can be sent by using Client as a @io.Writer. Once request body has been sent, end_request must be called to complete the request and obtain response from the server.

    After performing a request, the next request MUST NOT be made before the request is completed via end_request.

    In addition to headers in Client::connect, extra HTTP headers can be passed via extra_headers. The following headers is automatically set by request, and must not be specified in extra_headers:

    • Host
    • Transfer-Encoding

    The following headers can only be set globally for each client on initialization, and must not be set on a per-request basis:

    • User-Agent

    If Content-Length is present in extra_headers, it should be the total length of the request body. The request body can still be sent incrementally, but if the actual body being sent is shorter and longer than the provided length, an error will be raised.

    Client::skip_response_body

    async fn Client::skip_response_body(self : Client) -> Unit

    Skip the body of the response currently being produced, so that the next request can be made.

    Client::trailer

    fn Client::trailer(self : Client) -> Map[CaseInsensitiveString, String]

    Get the trailer of current response message.

    MUST be called AFTER all content of current message body has been read, and BEFORE reading the next response, otherwise this method will crash immediately.

    If the message is not chunk encoded, an empty header map will be returned.

    Client::write

    async fn Client::write(self : Client, data : &
    Data
    ) -> Unit

    Client::write_once

    async fn Client::write_once(self : Client, buf : Bytes, offset~ : Int, len~ : Int) -> Int

    Client::write_reader

    async fn Client::write_reader(self : Client, reader : &
    Reader
    ) -> Unit

    pub struct Cookie {
    name : String
    value : String
    path : String?
    expires_raw : String?
    max_age : Int64?
    domain : String?
    secure : Bool
    http_only : Bool
    extensions : Array[String]
    } derive(ToJson,
    Debug
    )

    Cookie::Cookie

    #alias(new, deprecated="`new` is deprecated, use `Cookie` instead")
    fn Cookie::Cookie(name : String, value : String, path? : String, expires_raw? : String, max_age? : Int64, domain? : String, secure? : Bool, http_only? : Bool, extensions? : Array[String]) -> Cookie

    Protocol

    pub(all) enum Protocol {
    Http
    Https
    } derive(Compare, Eq, Hash, ToJson,
    Debug
    )

    impl Show for Protocol

    Protocol::default_port

    fn Protocol::default_port(p : Protocol) -> Int

    Request

    pub(all) struct Request {
    meth : RequestMethod
    path : String
    headers : Map[CaseInsensitiveString, String]
    } derive(ToJson,
    Debug
    )

    impl Show for Request

    RequestMethod

    pub(all) enum RequestMethod {
    Get
    Head
    Post
    Put
    Delete
    Connect
    Options
    Trace
    Patch
    } derive(Compare, Eq, Hash, ToJson,
    Debug
    )

    Response

    pub(all) struct Response {
    code : Int
    reason : String
    headers : Map[CaseInsensitiveString, String]
    cookies : Array[Cookie]
    } derive(ToJson,
    Debug
    )

    impl Show for Response

    Server

    pub struct Server {
    addr :
    Addr

    // private fields
    }

    A HTTP server

    Server::Server

    #alias(new, deprecated="`new` is deprecated, use `Server` instead")
    async fn Server::Server(addr :
    Addr
    , dual_stack? : Bool, reuse_addr? : Bool, headers? : Map[CaseInsensitiveString, String]) -> Server

    Create a new HTTP server listening on addr.

    The meaning of dual_stack and reuse_addr is the same as @socket.TcpServer::new(), see there for more details.

    headers can be used to specify common headers shared by all responses sent by this server.

    Server::accept

    Accept a new connection from a HTTP server. Return the new HTTP connection and the address of peer.

    Server::addr

    #deprecated("use `.addr` instead")
    fn Server::addr(self : Server) ->
    Addr

    Get the listen address of the server

    Server::close

    fn Server::close(self : Server) -> Unit

    Server::run_forever

    async fn Server::run_forever(self : Server, f : async (Request, &
    Reader
    , ServerConnection) -> Unit, allow_failure? : Bool, max_connections? : Int) -> Unit

    Start the main loop of a HTTP server, the server will keep listening for new connections and new requests from existing connections. Requests are handled by the callback function f.

    The callback function f accepts three arguments:
    • the request to process
    • a &@io.Reader that can be used to read request body
    • a @http.ServerConnection that can be used to send response. The procedure to send a response is:
      1. initiate a response and send response header via .send_response()
      2. send response body by using the @http.ServerConnection as @io.Writer
      3. (optional) complete the response via .end_response(). If .end_response() is not called, it will be called automatically after f returns.

    If allow_failure is true (true by default), error raised by f will not crash the whole server. In this case, run_forever will only terminate when cancelled externally. Note that if f fails, the connection used by f will still get aborted.

    If max_connections is present, at most max_connections clients are allowed in parallel. New clients will only get handled after a previous client terminates.

    If f fails without closing the connection, run_forever will close each connection automatically. If f manually closes the connection, the connection will be dropped after f returns. On error or cancellation, the server will be closed automatically.

    ServerConnection

    type ServerConnection

    A single HTTP server connection

    ServerConnection::client_addr

    Get the address of client from a HTTP server connection

    ServerConnection::close

    fn ServerConnection::close(self : ServerConnection) -> Unit

    Close the connection, the underlying TCP connection will be closed as well. This function is idempotent: it is safe to call .close() multiple times, only the first .close() call takes effect.

    ServerConnection::drop

    async fn ServerConnection::drop(self : ServerConnection, len : Int) -> Int

    ServerConnection::end_response

    async fn ServerConnection::end_response(self : ServerConnection) -> Unit

    End the body of the response currently being sent. Should be called immediately after response body is fully sent.

    ServerConnection::enter_passthrough_mode

    async fn ServerConnection::enter_passthrough_mode(self : ServerConnection) -> Unit

    Let the server enter "pass through" mode, where the server serve as a TCP tunnel (maybe TLS encrypted) directly. This is useful for handling the special CONNECT HTTP request and HTTP protcol upgrade.

    In passthrough mode, Read/write the server becomes direct read/write on the underlying connection, and all API except @io.Reader and @io.Writer must not be used anymore.

    When entering pass through mode, the server must be in a clean state (i.e. not in the middle of sending a response). Unread data from the body of the last request will be discarded.

    ServerConnection::flush

    async fn ServerConnection::flush(self : ServerConnection) -> Unit

    Flush buffered data in the response body being sent, if any.

    ServerConnection::new

    Create a new HTTP server connection from a TCP connection.

    headers can be used to specify persistent headers for the connection, i.e. all response sent by this connection will share these headers. The following headers is automatically set, and must not be specified in headers:

    • Content-Length, Transfer-Encoding

    ServerConnection::read

    async fn ServerConnection::read(self : ServerConnection, dst : FixedArray[Byte], offset? : Int, max_len? : Int) -> Int

    ServerConnection::read_all

    ServerConnection::read_exactly

    async fn ServerConnection::read_exactly(self : ServerConnection, len : Int) -> Bytes

    ServerConnection::read_request

    async fn ServerConnection::read_request(self : ServerConnection) -> Request

    Read a single request from the connection. If the body of the last request is not consumed yet, it will be discarded.

    After calling read_request, the body of the request can be obtained by using ServerConnection as a @io.Reader.

    ServerConnection::read_some

    async fn ServerConnection::read_some(self : ServerConnection, max_len? : Int) -> Bytes?

    ServerConnection::read_until

    async fn ServerConnection::read_until(self : ServerConnection, sep : StringView) -> String?

    ServerConnection::send_response

    async fn ServerConnection::send_response(self : ServerConnection, code : Int, reason : String, extra_headers? : Map[CaseInsensitiveString, String], cookies? : Array[Cookie]) -> Unit

    Send a response to the peer. The ServerConnection can be used as a @io.Writer later for sending response body. After calling send_response, end_response must be called before sending the next response.

    cookies, if present, specify a list of cookies that the server wish the client to set. They will be transferred to the client via the Set-Cookie HTTP header. Users should always set cookies via cookies instead of filling Set-Cookie header directly.

    If Content-Length is present in extra_headers, it should be the total length of the request body. This useful for, for example, file servers to allow clients to know the progress of download, perform multipart download etc.

    The request body can still be sent incrementally, but if the actual body being sent is shorter and longer than the provided length, an error will be raised.

    ServerConnection::skip_request_body

    async fn ServerConnection::skip_request_body(self : ServerConnection) -> Unit

    Manually discard the body of the request currently being processed.

    ServerConnection::trailer

    Get the trailer of current request message.

    MUST be called AFTER all content of current message body has been read, and BEFORE reading the next request, otherwise this method will crash immediately.

    If the message is not chunk encoded, an empty header map will be returned.

    ServerConnection::write

    async fn ServerConnection::write(self : ServerConnection, data : &
    Data
    ) -> Unit

    ServerConnection::write_once

    async fn ServerConnection::write_once(self : ServerConnection, buf : Bytes, offset~ : Int, len~ : Int) -> Int

    ServerConnection::write_reader

    async fn ServerConnection::write_reader(self : ServerConnection, reader : &
    Reader
    ) -> Unit

    ServerConnection::write_string

    async fn ServerConnection::write_string(self : ServerConnection, s : String) -> Unit

    Introduced so that the sugar writer <+ "hello \{world}" can work on ServerConnection. Performance will be improved in the future

    ServerConnection::write_string_interpolation

    async fn ServerConnection::write_string_interpolation(self : ServerConnection, data : &
    Data
    ) -> Unit

    get

    async fn get(uri : String, headers? : Map[CaseInsensitiveString, String], body? : &
    Data
    , proxy? : Client) -> (Response, &
    Data
    )

    Perform a HTTP GET request to uri. Supported protocols are http:// and https://. The HTTP response message and the whole response body will be returned.

    proxy, if present, specifies the proxy to use for this request. See @http.Client::new for more details. proxy is not supported on JavaScript backend.

    See Client::request for more details.

    get_stream

    async fn get_stream(uri : String, headers? : Map[CaseInsensitiveString, String], body? : &
    Data
    , proxy? : Client) -> (Response, Client)

    Similar to @http.get, but allow reading response body streamingly. A pair (response, client) will be returned, where response is the response header from the server, and client is the HTTP client that performs the request. client can be used to read the content of response body via @io.Reader, see @http.Client for more details.

    Note that the returned client must be manually closed via .close() to close the underlying connection used for the request.

    post

    async fn post(uri : String, content : &
    Data
    , headers? : Map[CaseInsensitiveString, String], proxy? : Client) -> (Response, &
    Data
    )

    Similar to get, but performs a POST request instead.

    post_stream

    async fn post_stream(uri : String, headers? : Map[CaseInsensitiveString, String], proxy? : Client) -> Client

    Similar to @http.post, but allow writing request body streamingly. The return value client is the HTTP client that performs the request, it can be used to write the content of request body via @io.Writer. Notice that writing to @http.Client is buffered, so if you need to send data to the server immediately, .flush() must be called. After writing all the content, .end_request() must be called to complete the request and obtain response from the server. After that, the response body from the server can be obtained by using client as a @io.Reader. See @http.Client for more details.

    Note that the returned client must be manually closed via .close() to close the underlying connection used for the request.

    put

    async fn put(uri : String, content : &
    Data
    , headers? : Map[CaseInsensitiveString, String], proxy? : Client) -> (Response, &
    Data
    )

    Similar to get, but performs a PUT request instead.

    put_stream

    async fn put_stream(uri : String, headers? : Map[CaseInsensitiveString, String], proxy? : Client) -> Client

    Similar to @http.put, but allow writing request body streamingly. The return value client is the HTTP client that performs the request, it can be used to write the content of request body via @io.Writer. Notice that writing to @http.Client is buffered, so if you need to send data to the server immediately, .flush() must be called. After writing all the content, .end_request() must be called to complete the request and obtain response from the server. After that, the response body from the server can be obtained by using client as a @io.Reader. See @http.Client for more details.

    Note that the returned client must be manually closed via .close() to close the underlying connection used for the request.

    request

    async fn request(uri : String, meth : RequestMethod, headers : Map[CaseInsensitiveString, String], body : &
    Data
    , proxy? : Client) -> (Response, &
    Data
    )

    Perform a single HTTP request to uri. Supported protocols are http:// and https://. The HTTP response message and the whole response body will be returned.

    proxy, if present, specifies the proxy to use for this request. See @http.Client::new for more details. proxy is not supported on JavaScript backend.

    See Client::request for more details.