moonpostgres

    Pure-MoonBit PostgreSQL wire-protocol driver implementing @moondb.AsyncDriver — no C, like asyncpg.

    postgres
    driver
    moondb
    wire
    database
    Download zip
    Version
    0.4.0
    License
    Apache-2.0
    Last updated
    3 hours ago
    Downloads
    10

    #moonpostgres

    Moved on mooncakes from Lfan-ke/moonpostgres to moonbitstack/moonpostgres.

    A pure-MoonBit PostgreSQL driver — the v3 frontend/backend wire protocol spoken directly over async TCP, with zero C and no libpq. Like asyncpg or pg8000, it talks to a real PostgreSQL server itself. PgConn implements the @moondb.AsyncDriver contract, so a moondb-based stack (e.g. moonorm) can sit on top of it.

    moon add moonbitstack/moonpostgres

    Previously published as Lfan-ke/moon-postgres.

    Native-only: the connection layer rides moonbitlang/async sockets, which have no JS/wasm backend.

    #Quickstart

    The real driver is the async PgConn (asyncpg-shaped), used inside an event loop (async fn main / async test):

    async fn run() -> Unit raise {
    let conn = @moonpostgres.PgConn::connect(
    "127.0.0.1", 5432, "postgres", "postgres", "test",
    )
    conn.execute("CREATE TABLE hero (id int, name text)", []) |> ignore

    // '?' placeholders are translated to '$1','$2' and bound out-of-band
    // (never spliced into the SQL) — injection-safe to the wire.
    conn.execute("INSERT INTO hero VALUES (?, ?)", [
    @moondb.Int(1), @moondb.Text("Boromir"),
    ]) |> ignore

    let rows = conn.query("SELECT id, name FROM hero WHERE id = ?", [@moondb.Int(1)])
    println(rows[0].text_by("name")) // Boromir

    conn.begin()
    conn.execute("UPDATE hero SET name = ? WHERE id = ?", [
    @moondb.Text("Faramir"), @moondb.Int(1),
    ]) |> ignore
    conn.commit()

    conn.close()
    }

    #What it speaks

    sequenceDiagram participant C as PgConn (client) participant S as PostgreSQL C->>S: StartupMessage (user, database, client_encoding) S-->>C: Authentication (Ok / MD5 / cleartext) C->>S: PasswordMessage (md5… token) %% if requested S-->>C: AuthenticationOk S-->>C: ParameterStatus*, BackendKeyData, ReadyForQuery Note over C,S: simple query (no params) C->>S: Query 'Q' S-->>C: RowDescription 'T', DataRow 'D'*, CommandComplete 'C', ReadyForQuery 'Z' Note over C,S: extended query (bound params, ?→$n) C->>S: Parse 'P', Bind 'B', Describe 'D', Execute 'E', Sync 'S' S-->>C: ParseComplete, BindComplete, RowDescription, DataRow*, CommandComplete, ReadyForQuery

    • Framing — every message is a 1-byte type tag, a big-endian Int32 self-inclusive length, and a payload (the untagged StartupMessage aside).
    • AuthAuthenticationOk (trust), cleartext, and MD5 (md5(md5(password+user)+salt), with a self-contained pure-MoonBit MD5). SCRAM-SHA-256 is on the roadmap.
    • Simple query (Q) when there are no parameters; extended query (Parse/Bind/Describe/Execute/Sync) when there are, with ?$n translation and out-of-band text-format binding.
    • DecodingRowDescription + DataRow decode into @moondb.Row; column type OIDs map int2/int4Int, int8Int64, float4/float8Double, boolBool, byteaBlob, everything else→Text.

    #Why the driver is async

    @moondb.Driver's methods are synchronous (fn execute(...) raise DbError), which fits an FFI-backed driver like moonsqlite whose C calls block. PostgreSQL is reached over TCP, and MoonBit's only socket stack (moonbitlang/async) is async-only: an async fn cannot be called from a synchronous one, and the runtime exposes no public "run this async thunk to completion" bridge (with_event_loop lives in an import-blocked internal package). A synchronous method therefore cannot perform a PostgreSQL round trip.

    That is why moondb has a second seam. PgConn implements [@moondb.AsyncDriver] — the same eight operations as Driver, every one of them awaited — and PgDriver is now just the connection descriptor you open one from. Use it inside an event loop (async test / async fn main):

    async fn main {
    let conn = PgDriver::new(host="127.0.0.1", user="postgres", database="app").connect()
    defer conn.close()
    let drv : &@moondb.AsyncDriver = conn
    let rows = drv.query("SELECT id, name FROM users WHERE id = ?", [@moondb.Int(1)])
    }

    Before this, PgDriver implemented the synchronous Driver by raising from every row-touching method. That was worse than useless in one specific way: ping is a defaulted method built on query, so it swallowed the raise and returned false, and any Pool with pre_ping judged every connection permanently unhealthy. The raising façade is gone.

    This is the language-idiomatic equivalent, not a shortcut: asyncpg is async for exactly the same reason.

    #Testing

    • Unit (moon test) — MD5 vectors, ?$n translation (literals/identifiers/comments/dollar-quotes/??), value encode/decode by OID, and message framing + RowDescription/DataRow decode against synthetic backend messages.
    • Integration (moon test, gated on MOON_PG_TEST) — connect / DDL / parameterised INSERT / SELECT round trip and transaction commit+rollback against a real PostgreSQL. Skipped when MOON_PG_TEST is unset (local checkout without a database); CI stands up a postgres:16 service and runs it. Connection parameters come from the standard PGHOST/PGPORT/PGUSER/PGPASSWORD/PGDATABASE variables.

    The DataRow decoder is mutation-checked: breaking it makes the decode test fail with a type error.

    #Roadmap

    This round covers connect + auth (MD5/trust) + simple query + text decode + ?$n + @moondb.AsyncDriver conformance + a real round trip. Later rounds:

    • SCRAM-SHA-256 authentication (the default for PostgreSQL 14+ with a password set).
    • Binary result/parameter format and a type-OID registry (numeric, dates/times, arrays, JSON, UUID).
    • Prepared statements (named, cached) and a streaming Rows cursor.
    • RETURNING-based last_insert_id, COPY, LISTEN/NOTIFY, TLS, and a connection pool.

    #License

    Apache-2.0 © Leo Cheng.

    DbError

    The single database error type every fallible operation raises, aliased from moondb so driver signatures read raise DbError.

    ExecResult

    The outcome of a non-row statement, aliased from moondb.

    Row

    using @moonbitstack/moondb { type Row }

    One decoded result row (column names + values), aliased from moondb.

    Value

    The dialect-neutral value type this driver binds and decodes. Aliased so the wire code reads Value rather than @moondb.Value throughout; the enum constructors (Int, Text, …) are still spelled @moondb.Int where a value is built.

    PgConn

    pub struct PgConn {
    tcp :
    Tcp

    closed : Bool
    streaming : Bool
    }

    A live connection to a PostgreSQL backend, already past the startup handshake and sitting at ReadyForQuery. Runs the v3 frontend/backend protocol over a single @socket.Tcp. Not safe for concurrent use by multiple tasks: one statement must complete (drain to ReadyForQuery) before the next begins, as the wire protocol is a strict request/response pipeline per connection.

    PgConn::begin

    async fn PgConn::begin(self : PgConn) -> Unit raise
    DbError

    Begin an explicit transaction (BEGIN).

    PgConn::close

    async fn PgConn::close(self : PgConn) -> Unit

    Close the connection: best-effort Terminate, then close the socket. Idempotent — a second call is a no-op.

    PgConn::commit

    async fn PgConn::commit(self : PgConn) -> Unit raise
    DbError

    Commit the current transaction (COMMIT).

    PgConn::connect

    async fn PgConn::connect(host : String, port : Int, user : String, password : String, database : String) -> PgConn raise
    DbError

    Open a connection and complete the startup handshake: send StartupMessage, satisfy the authentication request (trust/AuthenticationOk, cleartext, MD5, or SASL/SCRAM-SHA-256), and drain server parameters up to the first ReadyForQuery. host may be a hostname or a literal IP; port is typically 5432. Raises ConnectError on any transport or handshake failure, including an unsupported auth method.

    PgConn::execute

    Run a non-row statement (INSERT/UPDATE/DELETE/DDL) and report rows changed. last_insert_id is 0: the simple/extended protocols do not surface a generated key without a RETURNING clause (roadmap).

    PgConn::query

    Run a row-returning statement and materialise every row. Uses the simple Query protocol when params is empty, and the extended Parse/Bind/Execute protocol (with ?$n translation and out-of-band text-format binding) when parameters are supplied.

    PgConn::query_stream

    async fn PgConn::query_stream(self : PgConn, sql : String, params : Array[
    Value
    ]) -> PgRowStream raise
    DbError

    Send sql (with bound params) and return a streaming cursor over its rows.

    PgConn::rollback

    async fn PgConn::rollback(self : PgConn) -> Unit raise
    DbError

    Roll back the current transaction (ROLLBACK).

    PgDriver

    pub struct PgDriver {
    host : String
    port : Int
    user : String
    password : String
    database : String
    }

    The parameters needed to reach a PostgreSQL backend — a connection descriptor, not a connection. connect opens the live [PgConn], which is the thing that implements the driver seam.

    It exists so a caller can carry connection settings around (a config value, a pool factory) and open a connection from them on demand, the way a DSN string works elsewhere.

    PgDriver::connect

    async fn PgDriver::connect(self : PgDriver) -> PgConn raise
    DbError

    Open the connection this descriptor points at. Call it inside an event loop (async test / async fn main); the returned [PgConn] is an [@moondb.AsyncDriver].

    PgDriver::new

    fn PgDriver::new(host~ : String, port? : Int, user~ : String, password? : String, database~ : String) -> PgDriver

    Build a connection descriptor. Does not connect: PostgreSQL I/O is async, so the connection is opened by connect inside an event loop.

    PgRowStream

    pub struct PgRowStream {
    conn : PgConn
    col_names : Array[String]
    col_oids : Array[Int]
    done : Bool
    }

    A forward-only cursor that reads a query's rows off the wire on demand instead of buffering them — the streaming counterpart to [query], for results too large to materialise (asyncpg's cursor, SQLAlchemy's stream_results). It is bound to its PgConn, which is a single-statement pipeline: the stream must be drained (next returns None) or closed before another statement runs on that connection.

    PgRowStream::close

    async fn PgRowStream::close(self : PgRowStream) -> Unit raise
    DbError

    Abandon the stream early, draining any unread response to ReadyForQuery so the connection can run the next statement. Idempotent once the stream is done.

    PgRowStream::next

    The next row, or None once the result is exhausted. Reads messages until a DataRow (updating the column metadata from any RowDescription first), and on ReadyForQuery marks the stream done. An ErrorResponse is drained to ReadyForQuery before raising, so the connection stays usable.

    OID_BOOL

    let OID_BOOL : Int

    OID_BYTEA

    let OID_BYTEA : Int

    OID_FLOAT4

    let OID_FLOAT4 : Int

    OID_FLOAT8

    let OID_FLOAT8 : Int

    OID_INT2

    let OID_INT2 : Int

    OID_INT4

    let OID_INT4 : Int

    OID_INT8

    let OID_INT8 : Int

    base64_decode

    fn base64_decode(s : String) -> Bytes

    Standard base64 decode (RFC 4648); = padding is ignored.

    base64_encode

    fn base64_encode(data : Bytes) -> String

    Standard base64 encoding (RFC 4648) with = padding.

    build_bind

    fn build_bind(params : Array[
    Value
    ]) -> Bytes

    Bind ('B'): bind params (all text format) to the unnamed statement, producing the unnamed portal, and request all result columns in text format.

    build_describe_portal

    fn build_describe_portal() -> Bytes

    Describe ('D') the unnamed portal, so the server sends a RowDescription before the DataRows (giving column names + type OIDs for decoding).

    build_execute

    fn build_execute() -> Bytes

    Execute ('E') the unnamed portal with no row limit (0 = all rows).

    build_parse

    fn build_parse(sql : String) -> Bytes

    Parse ('P'): prepare the unnamed statement from sql (with $n placeholders). Zero declared parameter types — the server infers them.

    build_password

    fn build_password(token : String) -> Bytes

    PasswordMessage ('p'): the auth response token (cleartext, or the md5… digest), as a C string.

    build_query

    fn build_query(sql : String) -> Bytes

    Simple Query ('Q'): one SQL string, no bound parameters.

    build_sasl_initial

    fn build_sasl_initial(mechanism : String, client_first : String) -> Bytes

    SASLInitialResponse ('p'): the chosen mechanism name, then the length-prefixed client-first SCRAM message (PG protocol §55.2.1).

    build_sasl_response

    fn build_sasl_response(client_final : String) -> Bytes

    SASLResponse ('p'): the raw client-final SCRAM message, no length prefix.

    build_startup

    fn build_startup(user : String, database : String) -> Bytes

    StartupMessage: protocol version 3.0 (196608) plus the user / database parameters and a client_encoding=UTF8 request, terminated by an empty key.

    build_sync

    fn build_sync() -> Bytes

    Sync ('S'): close the extended-query batch; the server replies ReadyForQuery.

    build_terminate

    fn build_terminate() -> Bytes

    Terminate ('X'): ask the backend to close the connection.

    concat_bytes

    fn concat_bytes(a : Bytes, b : Bytes) -> Bytes

    Concatenate two byte strings. A small helper the MD5 auth path and message framing lean on; MoonBit's Bytes is immutable so this allocates once.

    count_placeholders

    fn count_placeholders(sql : String) -> Int

    The number of ? placeholders [translate_placeholders] would consume — the count of parameters a statement expects. Shares the same scanner discipline so ?? escapes and quoted/comment spans do not count.

    decode_bytea

    fn decode_bytea(s : String) -> Bytes

    Decode PostgreSQL bytea hex text (\xDEADBEEF) back to raw bytes. A value not in hex form (legacy escape format) rides back as its UTF-8 bytes.

    decode_value

    fn decode_value(oid : Int, is_null : Bool, raw : String) ->
    Value

    Decode a text-format result cell (raw, already UTF-8) tagged with its column oid into a [Value]. is_null marks a wire NULL (length -1), which decodes to Null regardless of type. Numeric and boolean OIDs decode to their typed cases; everything else — including numeric, dates, and unknown OIDs — rides back as Text, exactly the dialect-neutral contract moondb documents (temporal/numeric typing is a roadmap item).

    encode_param

    fn encode_param(v :
    Value
    ) -> Bytes?

    Encode a bound parameter to its text-format bytes, or None for SQL NULL (which the Bind message sends as a length of -1). Binding is out-of-band — the value never touches the SQL string — so this is the injection-safe path.

    • integers / doubles / bools render to their canonical PostgreSQL text literals (t/f for booleans);
    • Text passes through UTF-8;
    • Blob uses the bytea hex format (\x + lowercase hex), which the server accepts for a text-format bytea parameter.

    hex_lower

    fn hex_lower(data : Bytes) -> String

    Lowercase hex of data, e.g. the 16-byte MD5 digest rendered as its 32-char hex string — the form PostgreSQL's MD5 auth concatenates and re-hashes.

    hmac_sha256

    fn hmac_sha256(key : Bytes, msg : Bytes) -> Bytes

    HMAC-SHA256 (RFC 2104).

    md5

    fn md5(msg : Bytes) -> Bytes

    The MD5 digest of msg as 16 raw bytes (RFC 1321). Used only for PostgreSQL's AuthenticationMD5Password handshake; not a general-purpose hashing API. Runs the standard little-endian padding + four-round compression over 64-byte blocks.

    parse_double

    fn parse_double(s : String) -> Double?

    Parse a floating-point literal from PostgreSQL text (123.45, -1e10, Infinity, NaN), or None if malformed. Handles sign, fraction, and a base-10 exponent; the special IEEE tokens PostgreSQL emits are recognised.

    parse_int

    fn parse_int(s : String) -> Int?

    Parse a base-10 Int, or None if s is not a well-formed integer. Used to decode int2/int4 result text; a parse failure falls back to Text so a surprising server rendering never silently becomes a wrong number.

    parse_int64

    fn parse_int64(s : String) -> Int64?

    Parse a base-10 Int64, or None on any non-digit (after an optional sign).

    pbkdf2_sha256

    fn pbkdf2_sha256(password : Bytes, salt : Bytes, iterations : Int) -> Bytes

    PBKDF2-HMAC-SHA256 producing a 32-byte key (SCRAM's Hi, dkLen = hLen so only the first block is computed): T = U1 ^ U2 ^ … ^ Uc, U1 = HMAC(pw, salt‖INT32(1)).

    pg_md5_password

    fn pg_md5_password(user : String, password : String, salt : Bytes) -> String

    The AuthenticationMD5Password response token: "md5" ++hex(md5(hex(md5(password ++ user)) ++ salt)), exactly per the PostgreSQL frontend/backend protocol. salt is the four bytes the server sent.

    scram_client_final

    fn scram_client_final(password : Bytes, client_first_bare : String, server_first : String) -> (String, Bytes) raise
    DbError

    Build the SCRAM client-final message and the expected server signature from the server's first message (RFC 5802). client_first_bare is n=…,r=clientnonce; server_first is r=nonce,s=salt,i=iters. Returns (client_final_message,server_signature) — the client sends the first and verifies the server's v= against the base64 of the second.

    scram_client_proof

    fn scram_client_proof(salted : Bytes, auth_message : Bytes) -> Bytes

    The SCRAM ClientProof for the AuthMessage: ClientKey XOR HMAC(StoredKey,AuthMessage), where ClientKey = HMAC(SaltedPassword, "Client Key") and StoredKey = SHA256(ClientKey) (RFC 5802 §3).

    scram_server_signature

    fn scram_server_signature(salted : Bytes, auth_message : Bytes) -> Bytes

    The SCRAM ServerSignature: HMAC(ServerKey, AuthMessage), ServerKey = HMAC(SaltedPassword, "Server Key"). The client verifies the server's v= against this to authenticate the server (RFC 5802 §3).

    sha256

    fn sha256(msg : Bytes) -> Bytes

    SHA-256 (FIPS 180-4).

    translate_placeholders

    fn translate_placeholders(sql : String) -> String

    Rewrite moondb's dialect-neutral ? positional placeholders into PostgreSQL's numbered $1, $2, … form. moondb fixes the calling convention (an ordered params array) but leaves the placeholder spelling to the driver; PostgreSQL's extended-query protocol requires $n, so a query layer that emits ? for portability is translated here before Parse.

    A ? is only a placeholder in SQL text — never inside a single-quoted string literal, a dollar-quoted string, a "-quoted identifier, or a -- / /* */ comment. Those spans are scanned through verbatim so a literal ? in data or a comment is left untouched and does not shift the parameter numbering. A literal ? an application genuinely needs in output can be written ??, which collapses to a single ? (mirroring JDBC-style escaping).

    utf8_decode

    fn utf8_decode(data : Bytes) -> String

    Decode UTF-8 bytes to a String. PostgreSQL text values, column names, and error fields all arrive UTF-8 (the startup message negotiates client_encoding). Decodes the ASCII fast path directly and multi-byte sequences by code point; malformed input yields U+FFFD rather than raising, matching a lenient text codec.