moon-mysql
Pure-MoonBit MySQL wire-protocol driver implementing @moondb.Driver — no C, like PyMySQL.
mysql
driver
moondb
wire
database
protocol
moon add Lfan-ke/moon-mysql@0.2.0
Download zipAuthor
Version
0.2.0
License
Apache-2.0
Repository
Last updated
17 days ago
Downloads
8
Dependencies
- Lfan-ke/moondb0.1.5
- moonbitlang/async0.20.3
README#moon-mysql
A pure-MoonBit MySQL / MariaDB wire-protocol driver — no C, no bindings, like
PyMySQL. It speaks the MySQL client/server protocol (version 10 handshake,
mysql_native_password auth, the text COM_QUERY protocol) directly over a raw
TCP socket, and implements the @moondb.Driver
contract so a moondb-based query layer (moonorm) can talk to MySQL through the
same seam as the SQLite and Postgres drivers.
MariaDB speaks the same wire protocol and is a first-class target: the
handshake parser recognises MariaDB's 5.5.5- version sentinel (recovering the
real version and reporting server_kind = MariaDB), reads past its extended
capability bits without disturbing the MySQL-8 offsets, and authenticates over
the shared mysql_native_password path. CI runs the same integration suite
against MySQL 8, MariaDB 11, and MariaDB 10.11.
$ moon add Lfan-ke/moon-mysql- API reference: https://lfan-ke.github.io/moon-mysql/
- License: Apache-2.0
#Quickstart
The synchronous @moondb.Driver adapter is the one-call-per-statement surface:
let db = @client.MysqlDriver::new(
user="root", password="root", database="test",
host="127.0.0.1", port=3306,
)
db.execute("CREATE TABLE t (id INT PRIMARY KEY, name TEXT)", [])
db.execute("INSERT INTO t VALUES (?, ?)", [@moondb.Int(1), @moondb.Text("a")])
let rows = db.query("SELECT id, name FROM t", [])
rows[0].int_by("id") // => 1
rows[0].text_by("name") // => "a"For long-lived connections and real transactions, use the asynchronous
MysqlConn directly (see The async wall below):
async fn run() -> Unit raise {
let conn = @client.MysqlConn::connect("127.0.0.1", 3306, "root", "root", "test")
conn.begin()
conn.execute("INSERT INTO t VALUES (?, ?)", [@moondb.Int(2), @moondb.Text("b")])
conn.commit()
let rows = conn.query("SELECT * FROM t", [])
conn.close()
}#Architecture
The library is split so the entire wire format is testable off any backend, and
only the socket transport is native-bound.
flowchart TD
subgraph root["Lfan-ke/moon-mysql · pure codec · target: all backends"]
sha1["sha1 — FIPS 180-4"]
packet["packet — lenenc / fixed-int / string cursor"]
handshake["handshake — parse + scramble + response"]
response["response — OK/ERR/EOF + text-row decode"]
binding["binding — ? placeholders → escaped literals"]
end
subgraph client["Lfan-ke/moon-mysql/client · native only"]
conn["MysqlConn — async transport over @socket.Tcp"]
driver["MysqlDriver — sync @moondb.Driver adapter"]
end
handshake --> conn
response --> conn
binding --> conn
packet --> handshake
packet --> response
sha1 --> handshake
conn --> driver
driver -. implements .-> moondb["@moondb.Driver"]%% root (pure, all backends) client (native)
%% sha1 ─┐ MysqlConn ── async socket transport
%% packet┼─ handshake ─┐ │
%% └─ response ──┼──────────────► │
%% binding ───┘ MysqlDriver ── impl @moondb.DriverEvery packet field type — the 3-byte length + sequence-id framing, length-encoded
and fixed-width little-endian integers, NUL- and length-encoded strings — is
parsed through an in-memory cursor in the pure half, so the codec is unit-tested
on wasm, wasm-gc, js, and native. The client package adds the async socket
transport and the driver, and is native-only because moonbitlang/async/socket
has no JS/wasm backend.
#The async wall
moonbitlang/async sockets are asynchronous, but @moondb.Driver's methods are
synchronous, and MoonBit forbids calling an async function from a non-async
one. The synchronous adapter bridges each call through @async.run_async_main,
which runs an async body to completion on a fresh event loop.
A socket does not survive across two such event loops (each tears its file
descriptors down — verified empirically), so the synchronous MysqlDriver opens
a fresh connection, authenticates, runs one statement, and closes, on every
execute/query. This is correct and efficient enough for autocommit work
(which is exactly what a moondb roundtrip is), but it means a transaction cannot
be held open across separate Driver calls. begin/commit/rollback on the
synchronous adapter therefore raise a clear DbError directing you to the
asynchronous MysqlConn, which owns its socket for its whole lifetime and
brackets real server-side transactions. This is a language constraint, not a
shortcut.
#Injection safety
Values never reach the server as SQL text unescaped. The text protocol has no
out-of-band parameter binding (that arrives with prepared statements — see the
roadmap), so this round renders each bound @moondb.Value as an escaped SQL
literal: strings are single-quoted with every metacharacter backslash-escaped,
blobs use the x'…' hex form, and the placeholder scanner tracks single-,
double-, and backtick-quoted spans so a literal ? inside a string is never
mistaken for a placeholder.
#Type mapping (text protocol)
| MySQL column type | @moondb.Value |
|---|---|
| TINYINT / SMALLINT / INT / MEDIUMINT / YEAR | Int |
| BIGINT | Int64 |
| FLOAT / DOUBLE | Double |
| binary-collation columns (BLOB, VARBINARY, …) | Blob |
| VARCHAR / TEXT / DECIMAL / JSON / temporal (ISO text) | Text |
| SQL NULL | Null |
#Tested against a real MySQL and MariaDB
CI (.github/workflows/ci.yml) starts a real MySQL 8 (pinned to
mysql_native_password) and, in separate jobs, MariaDB 11 and MariaDB
10.11, then runs moon check/build/test on native against each. The one
integration test — gated by MYSQL_TEST so a local moon test without a
database skips it — connects, authenticates, creates a table, inserts
parameter-bound rows (integers, UTF-8 text, an embedded quote), and asserts the
decoded result set cell by cell; the same suite is green on all three servers.
The pure codec additionally runs on every backend, and parse_server_version
has a unit test covering both MariaDB's 5.5.5- sentinel and a plain MySQL
version string.
#MariaDB auth negotiation
mysql_native_password is the tested path on both servers. If a server's default
plugin differs but it still offers pluggable auth, the client advertises
mysql_native_password and answers the server's AuthSwitchRequest for it — the
down-negotiation MariaDB may require. A switch to any other plugin, or a
caching_sha2_password full-auth exchange, raises a clear UnsupportedError
pointing here rather than silently mis-authenticating.
#Roadmap
This round covers connect + mysql_native_password auth + the text protocol +
value decode + the @moondb.Driver adapter + a real roundtrip. Planned next:
- Prepared statements — COM_STMT_PREPARE/COM_STMT_EXECUTE, the binary protocol, and true out-of-band parameter binding.
- caching_sha2_password — MySQL 8's default plugin, including the RSA full-auth exchange over a plain connection.
- client_ed25519 — MariaDB's native Ed25519 auth plugin, for MariaDB accounts not using mysql_native_password.
- Full type coverage — binary-protocol temporal/decimal/bit decoding, and a dedicated temporal Value case once moondb grows one.
- Connection pooling and a streaming Rows cursor.
- TLS once moonbitlang/async exposes a client-side entry point.
#MysqlError
pub(all) suberror MysqlError {
ProtocolError(String)
ServerError(Int, String, String)
UnsupportedError(String)
} derive(Eq)A failure raised while speaking the MySQL wire protocol. It is deliberately
distinct from @moondb.DbError: the pure codec and the socket transport raise
MysqlError (which carries wire-level specifics — a server error code, an
unsupported auth plugin, a malformed packet), and the @moondb.Driver
adapter maps it onto the coarse DbError cases at the public boundary.
- ProtocolError — a packet did not parse: short read, bad prefix byte, an out-of-place packet in the result-set state machine.
- ServerError — the server sent an ERR packet. Carries the numeric error code, the 5-char SQLSTATE, and the human message verbatim.
- UnsupportedError — a code path this round does not implement (e.g. the caching_sha2_password full-auth exchange, a LOCAL INFILE request).
impl Show for MysqlError
#MysqlError::to_string
A one-line rendering, e.g. ServerError(1146, 42S02): Table 'test.t' doesn't exist.
#ColumnDef
One result-set column's definition, reduced to what text decoding needs: the
projected name, the MySQL type code, the collation (to tell text from binary),
and the column flags.
#Handshake
pub struct Handshake {
server_version : String
server_kind : ServerKind
connection_id : Int
salt : Bytes
capability : Int
mariadb_capability : Int
charset : Int
status : Int
auth_plugin : String
}The server's initial Handshake packet (protocol version 10), decoded down to
the fields the client needs: the 20-byte auth salt, the negotiated
capabilities, and the auth-plugin name that selects the scramble.
#OkPacket
pub(all) struct OkPacket {
affected_rows : Int64
last_insert_id : Int64
status : Int
warnings : Int
} derive(Eq)An OK packet: a statement that returned no result set (INSERT/UPDATE/DELETE/DDL)
or the terminator of a successful command.
#PacketReader
pub struct PacketReader {
data : Bytes
pos : Int
}A cursor over one decoded MySQL packet payload. Every field type the protocol
uses — fixed-width little-endian integers, length-encoded integers and strings,
NUL-terminated strings, and raw byte runs — is read through this, advancing a
position and raising [MysqlError::ProtocolError] on any short read. It holds
no socket: the transport reads a full payload into Bytes first, then parses
it here, which is what lets the whole codec be tested off any backend.
#PacketReader::at_end
Whether the cursor has consumed the whole payload.
#PacketReader::bytes
Read n raw bytes.
#PacketReader::lenenc_bytes
Read a length-encoded string (the string<lenenc> type), returning None
for the 0xFB NULL sentinel that appears in text-protocol rows.
#PacketReader::lenenc_uint
Read a length-encoded unsigned integer (the int<lenenc> type). The 0xFB
NULL sentinel and 0xFF are rejected here — NULL only has meaning inside a
row, which [lenenc_bytes] handles.
#PacketReader::new
Wrap a decoded packet payload for reading from the start.
#PacketReader::peek
Peek the next byte without advancing; -1 at end of payload.
#PacketReader::remaining
Bytes not yet consumed.
#PacketReader::rest
Read everything left in the payload (an EOF-terminated string field).
#PacketReader::skip
Skip n bytes.
#PacketReader::string_nul
Read a NUL-terminated string, consuming the terminator.
#PacketReader::u8
Read one byte as an Int in 0..=255.
#PacketReader::uint_le
Read an n-byte little-endian unsigned integer.
#ServerKind
Which server dialect answered the handshake. Both speak the MySQL wire
protocol; they diverge in the version string and the extended capabilities.
impl Show for ServerKind#bind_params
Substitute the ordered params for the ? placeholders in sql, producing
the UTF-8 query bytes a COM_QUERY carries.
The text protocol has no out-of-band parameter binding — that arrives with
prepared statements (COM_STMT_PREPARE, binary protocol) in a later round —
so values are rendered as escaped literals here. The scanner tracks single-,
double-, and backtick-quoted spans (honouring backslash escapes) so a literal
? inside a string or identifier is never mistaken for a placeholder, and it
raises when the placeholder and parameter counts disagree.
#build_handshake_response
fn build_handshake_response(handshake : Handshake, user : String, password : Bytes, database : String) -> Bytes raise MysqlErrorBuild the client's HandshakeResponse41 payload for user/password/database.
mysql_native_password and caching_sha2_password (MySQL 8's default, fast-path
scramble here and full auth driven in [MysqlConn::connect]) are both handled
directly. For any other plugin the client advertises mysql_native_password and
sends a native token, so a native-capable account authenticates and otherwise
the server drives an AuthSwitchRequest the connection layer answers (native or
caching_sha2) or rejects. Only a server that requires a non-native plugin and
does not offer pluggable auth is rejected here. client_ed25519 remains on the
README roadmap.
#build_text_rows
Build the materialised rows of a text-protocol result set from its column
definitions and the raw row packets. This is the pure heart of query: the
socket transport collects columns and row_payloads, and every cell decode
happens here, off any backend.
#bytes_to_string
fn bytes_to_string(b : Bytes) -> StringBytes → String, lossily (server version, error text, text-protocol cells).
#caching_sha2_full_auth_token
fn caching_sha2_full_auth_token(password : Bytes, nonce : Bytes, pem : String, seed : Bytes) -> Bytes raise MysqlErrorThe caching_sha2_password full-auth token: the NUL-terminated password XORed
with the nonce (cycled), then RSA-OAEP encrypted under the server's public key.
#caching_sha2_scramble
fn caching_sha2_scramble(password : Bytes, nonce : Bytes) -> BytesThe caching_sha2_password fast-auth scramble:
SHA256(pw) XOR SHA256( SHA256(SHA256(pw)) ‖ nonce ), 32 bytes. An empty
password sends an empty token.
#decode_text_value
Map one text-protocol cell (None = the 0xFB NULL sentinel) onto a
[@moondb.Value] using the column's declared type. Integers narrow to Int
except BIGINT, which keeps 64 bits; FLOAT/DOUBLE become Double;
binary-collation strings become Blob; everything else (VARCHAR, TEXT,
DECIMAL, temporal types as ISO text, JSON) becomes Text.
#ed25519_sign
fn ed25519_sign(secret : Bytes, msg : Bytes) -> BytesEd25519 signing (RFC 8032 §5.1.6), deterministic, keyed by secret (expanded
via [ed_expand]). Returns the 64-byte R || S: r = SHA-512(prefix || M) modl, R = [r]B, k = SHA-512(R || A || M) mod l, S = (r + k·s) mod l.
#is_auth_switch_request
fn is_auth_switch_request(payload : Bytes) -> BoolWhether payload is an AuthSwitchRequest (0xFE lead byte with a body — the
length is what tells it apart from a 5-byte EOF, exactly as [is_eof_packet]
keys off < 9).
#is_eof_packet
fn is_eof_packet(payload : Bytes) -> BoolWhether payload is an EOF packet (first byte 0xFE, fewer than 9 bytes —
which is what tells it apart from a row whose first cell is an 8-byte
length-encoded value).
#is_err_packet
fn is_err_packet(payload : Bytes) -> BoolWhether payload is an ERR packet (first byte 0xFF).
#is_ok_packet
fn is_ok_packet(payload : Bytes) -> BoolWhether payload is an OK packet (first byte 0x00, at least 7 bytes).
#mariadb_ed25519_response
fn mariadb_ed25519_response(password : Bytes, scramble : Bytes) -> BytesThe MariaDB client_ed25519 response: the 64-byte Ed25519 signature of the
server's 32-byte challenge, keyed by the password. Unlike mysql_native_password,
the ed25519 plugin signs unconditionally — an empty password expands to
SHA-512("") and still yields a valid signature.
#native_password_scramble
fn native_password_scramble(password : Bytes, salt : Bytes) -> BytesThe mysql_native_password challenge response:
SHA1(password) XOR SHA1(salt ++ SHA1(SHA1(password))), 20 bytes. An empty
password produces an empty response (the server accepts a zero-length token).
#parse_auth_switch_request
Decode an AuthSwitchRequest into the plugin name the server wants and the fresh
20-byte auth salt for it. The trailing NUL that follows the scramble is
dropped so the salt feeds [native_password_scramble] directly.
#parse_column_def
Decode a column-definition packet (protocol 41). Only the fields that steer
text decoding are kept; catalog/schema/table names and the length/decimals
fields are read past.
#parse_ed25519_challenge
The 32-byte nonce from a client_ed25519 AuthSwitchRequest. MariaDB sends
exactly NONCE_BYTES (= 32) with no NUL terminator and the client signs all of
them, so — unlike the native scramble — the full 32 bytes are read raw rather
than through the NUL-stripping [parse_auth_switch_request].
#parse_err
Decode a full ERR packet (including its 0xFF marker) into the
[MysqlError::ServerError] it represents.
#parse_handshake
Parse a protocol-10 initial Handshake payload. An ERR packet in its place
(0xFF, e.g. "Host is blocked" / "Too many connections") is surfaced as a
[MysqlError::ServerError].
#parse_ok
Decode an OK packet (including its 0x00 marker).
#parse_rsa_public_key
Parse a PEM SubjectPublicKeyInfo (-----BEGIN PUBLIC KEY-----) into the RSA
modulus and public exponent. Walks SEQUENCE { AlgorithmIdentifier, BIT STRING{ RSAPublicKey { INTEGER n, INTEGER e } } }.
#parse_server_version
Classify a raw handshake version string and recover the real version.
MariaDB 10+ prefixes its version with the fake 5.5.5- sentinel (MySQL 5.5.5
was never released), so pre-10 clients that gate on the leading 5. still
connect. Strip it to get the true version — e.g.
5.5.5-11.4.2-MariaDB-ubu2404 → 11.4.2-MariaDB-ubu2404 as
[ServerKind::MariaDB]; a plain 8.0.35 is returned unchanged as
[ServerKind::MySQL].
#put_lenenc_bytes
Append a length-encoded string (its lenenc length prefix, then the bytes).
#put_lenenc_uint
Append a length-encoded unsigned integer.
#put_string_nul
Append raw bytes followed by a NUL terminator.
#put_uint_le
Append an n-byte little-endian unsigned integer.
#rsa_oaep_sha1_encrypt
fn rsa_oaep_sha1_encrypt(msg : Bytes, n : BigInt, e : BigInt, seed : Bytes) -> Bytes raise MysqlErrorEME-OAEP encode msg (SHA-1, empty label) to k octets with the given seed,
then RSA-encrypt: EM^e mod n. seed must be 20 random bytes (RFC 8017 §7.1.1).
#sha1
fn sha1(msg : Bytes) -> BytesSHA-1 (FIPS 180-4) over msg, returning the 20-byte digest.
MoonBit's core ships no SHA-1, but mysql_native_password's challenge-response
is built entirely on it (SHA1(password) XOR SHA1(salt + SHA1(SHA1(password)))),
so the driver carries its own. Pure and allocation-simple; verified against
the FIPS/NIST test vectors in the suite.
#sha512
fn sha512(msg : Bytes) -> BytesSHA-512 (FIPS 180-4), 64-bit words over 80 rounds. Messages here are far under
2^64 bits, so the 128-bit length field's high half is always zero.
Pure-MoonBit MySQL wire-protocol driver implementing @moondb.Driver — no C, like PyMySQL.
mysqldrivermoondbwiredatabaseprotocol
0.2.0
Apache-2.0
Updated 17 days ago
8 downloadsDependencies
- Lfan-ke/moondb0.1.5
- moonbitlang/async0.20.3