🥮 mooncakes.io
    Sign in

    moonkv

    An append-only key-value storage engine in MoonBit based on Bitcask architecture.

    kv
    database
    storage
    bitcask
    Download zip
    Version
    0.1.3
    License
    Apache-2.0
    Repository
    Last updated
    2 months ago
    Downloads
    53

    #MoonKV package guide

    MoonKV is an embedded append-only key-value store implemented in MoonBit. It provides a small Bitcask-style storage engine with an in-memory key directory, checksummed records, logical TTL, atomic write batches, segment rolling, and hint-file assisted compaction recovery.

    #Installation

    From another MoonBit module:

    moon add wallll-wal6/moonkv

    Then import the root package:

    import {
      "wallll-wal6/moonkv" @moonkv,
    }

    The package is published as wallll-wal6/moonkv, uses Apache-2.0, and is tested against MoonBit 0.10.3 in CI.

    #Core API

    let db = @moonkv.DB::open("my_db", 10 * 1024 * 1024)
    db.put("user_name", @moonkv.string_to_bytes("Alice"))
    let value = db.get("user_name")
    println(@moonkv.bytes_to_string(value))
    db.close()

    The max_file_size argument must be positive. put, get, delete, put_ttl, merge, and WriteBatch::commit may raise MoonKVError.

    #TTL and atomic batches

    TTL is a logical write clock: a positive TTL expires after the corresponding number of later logical write steps, rather than after wall-clock seconds.

    db.put_ttl("temporary", @moonkv.string_to_bytes("value"), 5L)
    let batch = db.new_write_batch()
    batch.put("balance", @moonkv.string_to_bytes("100"))
    batch.delete("old_balance")
    batch.commit()

    put_ttl requires a positive TTL. A batch validates every operation before it writes the first transaction record, so an invalid TTL or a reserved key does not create a partial batch.

    #Ordered queries and diagnostics

    The query API is intended for cache indexes, local queues, and metadata tools:

    ///|
    let page = db.scan_prefix("user:", 100)

    ///|
    let next_page = db.scan_after("user:", page[page.length() - 1].0, 100)

    ///|
    let range = db.scan_range("user:", "user;", 100)

    ///|
    let page = db.scan_page("user:", "", 100)

    ///|
    let stats = db.stats()

    ///|
    let verified = db.verify()

    ///|
    let fingerprint = db.fingerprint()

    Scans are lexicographically ordered and omit deleted or logically expired keys. scan_after takes an exclusive cursor, while scan_range uses a half-open interval. scan_page returns rows, has_more, and an optional continuation cursor. Limits must be positive. delete_prefix uses one atomic write batch and is bounded by its limit. verify reads every visible value through the normal corruption checks. fingerprint is a deterministic FNV-1a smoke-check value, not a cryptographic digest.

    #CLI

    moon run cmd/main -- ./data_dir put mykey "Hello MoonBit"
moon run cmd/main -- ./data_dir get mykey
moon run cmd/main -- ./data_dir put-ttl token "temporary" 2
moon run cmd/main -- ./data_dir delete mykey
moon run cmd/main -- ./data_dir merge
moon run cmd/main -- ./data_dir scan user: 100
moon run cmd/main -- ./data_dir delete-prefix session: 100
moon run cmd/main -- ./data_dir stats
moon run cmd/main -- ./data_dir verify
moon run cmd/main -- ./data_dir fingerprint

    The standalone benchmark uses one native process to write and read a deterministic workload:

    bash scripts/benchmark.sh 10000
# Windows PowerShell:
.\scripts\benchmark.ps1 -Records 10000

    #Key and value boundaries

    Values are immutable Bytes; string_to_bytes and bytes_to_string are convenience helpers for simple text examples. Empty keys and empty values are valid. Keys beginning with __tx__: and __tx_commit__: are reserved for the transaction journal and are rejected. A database directory should have one writer at a time.

    #Recovery and file format

    Records contain a 28-byte header followed by the key and value. The checksum is verified while reading. A compaction writes live records to a data segment and a hint segment, removes obsolete files, and creates the next empty active data file before returning. This ensures that a new process writes after the hint segment rather than appending into it. The independent-process regression is:

    bash scripts/cross-process-recovery.sh

    #Test and quality gates

    moon fmt --check
moon check --target all --deny-warn
moon build --target all
moon test --target all --deny-warn
moon info
git diff --exit-code

    The CI matrix covers Linux, macOS, and Windows with MoonBit 0.10.3. See README.md, CHANGELOG.md, and CONTRIBUTING.md for the repository-level acceptance and maintenance information.

    MoonKVError

    source
    pub suberror MoonKVError {
      IOError(String)
      KeyNotFound(String)
      CorruptedDatabase(String)
      InvalidRecord(String)
    }

    Custom error types for moonkv database operations.
    pub struct DB {
      dir : String
      keydir : Keydir
      active_file_id : Int
      active_file_offset : Int
      max_file_size : Int
      logical_clock : Int64
    }

    Main database handle.

    DB::close

    source
    fn DB::close(self : DB) -> Unit

    Helper to close/clean resources (currently a no-op).

    DB::contains

    source
    fn DB::contains(self : DB, key : String) -> Bool

    Returns whether a key is currently visible.

    DB::delete

    source
    fn DB::delete(self : DB, key : String) -> Unit raise MoonKVError

    Deletes a key-value pair from the database by writing a tombstone.

    DB::delete_prefix

    source
    fn DB::delete_prefix(self : DB, prefix : String, limit : Int) -> Int raise MoonKVError

    Deletes up to limit visible keys with the given prefix atomically.

    The prefix may be empty to clear the visible keyspace. The operation first collects matching keys, then commits one write batch, so an invalid limit cannot leave a partial deletion.

    Callers can repeat the operation with the same prefix to drain large sets.

    DB::fingerprint

    source
    fn DB::fingerprint(self : DB) -> Int raise MoonKVError

    Computes a deterministic FNV-1a fingerprint of the visible keyspace.

    The fingerprint is useful for smoke-checking replicas or detecting an unexpected change between maintenance runs. It is not a cryptographic digest and must not be used as an integrity or authentication boundary.

    DB::get

    source
    fn DB::get(self : DB, key : String) -> Bytes raise MoonKVError

    Retrieves a value from the database by its key.

    DB::key_count

    source
    fn DB::key_count(self : DB) -> Int

    Returns the number of currently indexed keys.

    Expired entries remain lazily represented until accessed, so use verify or a scan when a fully materialized visible count is required.

    DB::live_value_bytes

    source
    fn DB::live_value_bytes(self : DB) -> Int64

    Returns the approximate bytes occupied by currently visible values.

    This is metadata from the key directory, so it excludes record headers and stale log records. It is intended for admission control and dashboards.

    Values are measured in bytes and remain exact for the current keydir view.

    DB::merge

    source
    fn DB::merge(self : DB) -> Unit raise MoonKVError

    Writes consolidated active records to a new data and hint file, and deletes older files.

    DB::new_write_batch

    source
    fn DB::new_write_batch(self : DB) -> WriteBatch

    Creates a new WriteBatch for the database.

    DB::open

    source
    fn DB::open(dir : String, max_file_size : Int) -> DB raise MoonKVError

    Rebuilds the in-memory index from existing data files.

    DB::put

    source
    fn DB::put(self : DB, key : String, value : Bytes) -> Unit raise MoonKVError

    Writes a key-value pair to the database.

    DB::put_ttl

    source
    fn DB::put_ttl(self : DB, key : String, value : Bytes, ttl : Int64) -> Unit raise MoonKVError

    Writes a key-value pair to the database with a logical Time-To-Live (TTL).

    DB::scan_after

    source
    fn DB::scan_after(self : DB, prefix : String, cursor : String, limit : Int) -> Array[(String, Bytes)] raise MoonKVError

    Returns at most limit keys with prefix after an exclusive cursor.

    The returned keys are sorted lexicographically. Pass the last returned key as cursor to fetch the next page. An empty cursor starts at the beginning.

    DB::scan_page

    source
    fn DB::scan_page(self : DB, prefix : String, cursor : String, limit : Int) -> ScanPage raise MoonKVError

    Returns one page and an exclusive cursor for the following page.

    Unlike inferring pagination from a short result array, has_more is determined by reading one sentinel row. This makes an exact page at the end of the keyspace unambiguous to HTTP, CLI, and replication adapters.

    DB::scan_prefix

    source
    fn DB::scan_prefix(self : DB, prefix : String, limit : Int) -> Array[(String, Bytes)] raise MoonKVError

    A stable, ordered view of the keys currently visible in the database.

    Scans are built from the in-memory key directory and then read through the normal record validation path. They therefore never expose transaction markers, tombstones, or partially written records.

    DB::scan_range

    source
    fn DB::scan_range(self : DB, start_key : String, end_key : String, limit : Int) -> Array[(String, Bytes)] raise MoonKVError

    Returns keys in the half-open range [start_key, end_key).

    DB::stats

    source
    fn DB::stats(self : DB) -> DBStats

    Returns storage metadata without reading every value from disk. This is safe to call from periodic monitoring loops. The values describe the current process-local view. Reopen the database to refresh it after an external writer.

    DB::verify

    source
    fn DB::verify(self : DB) -> Int raise MoonKVError

    Reads every visible key and returns the number successfully verified.

    This is intended for startup diagnostics and maintenance tooling. It uses the same bounds and checksum validation as get, so a corrupted value is reported instead of being silently skipped.

    DBStats

    source
    pub struct DBStats {
      key_count : Int
      live_value_bytes : Int64
      active_file_id : Int
      active_file_offset : Int
      max_file_size : Int
      logical_clock : Int64
    }

    A compact health and capacity snapshot for monitoring integrations.

    HintRecord

    source
    pub struct HintRecord {
      timestamp : Int64
      expires_at : Int64
      key_sz : Int
      val_sz : Int
      value_pos : Int64
      key : Bytes
    }

    Container for metadata read from a Hint File.

    KeyEntry

    source
    pub struct KeyEntry {
      file_id : Int
      value_sz : Int
      value_pos : Int64
      timestamp : Int64
      expires_at : Int64
    }

    KeyEntry metadata representing a record's location on disk.

    KeyEntry::new

    source
    fn KeyEntry::new(file_id : Int, value_sz : Int, value_pos : Int64, timestamp : Int64, expires_at : Int64) -> KeyEntry

    Creates a new KeyEntry.

    Keydir

    source
    pub struct Keydir {
      entries : Map[String, KeyEntry]
    }

    In-memory key directory mapping keys to their metadata.

    Keydir::clear

    source
    fn Keydir::clear(self : Keydir) -> Unit

    Clears all entries from the directory.

    Keydir::contains

    source
    fn Keydir::contains(self : Keydir, key : String) -> Bool

    Checks if the directory contains a key.

    Keydir::delete

    source
    fn Keydir::delete(self : Keydir, key : String) -> Unit

    Removes a key from the directory.

    Keydir::get

    source
    fn Keydir::get(self : Keydir, key : String) -> KeyEntry?

    Retrieves a KeyEntry from the directory.

    Keydir::keys

    source
    fn Keydir::keys(self : Keydir) -> Array[String]

    Returns all keys in the directory.

    Keydir::new

    source
    fn Keydir::new() -> Keydir

    Creates a new Keydir.

    Keydir::put

    source
    fn Keydir::put(self : Keydir, key : String, entry : KeyEntry) -> Unit

    Inserts a KeyEntry into the directory.

    LogRecord

    source
    pub struct LogRecord {
      record : Record
      offset : Int
      size : Int
    }

    Container for a record retrieved from a WAL data file.

    Record

    source
    pub struct Record {
      checksum : Int
      timestamp : Int64
      expires_at : Int64
      key : Bytes
      value : Bytes
      is_tombstone : Bool
    }

    Represents a single key-value record stored on disk.

    Record::new

    source
    fn Record::new(key : Bytes, value : Bytes, timestamp : Int64) -> Record

    Creates a new active Record.

    Record::new_tombstone

    source
    fn Record::new_tombstone(key : Bytes, timestamp : Int64) -> Record

    Creates a new tombstone (deleted) Record.

    Record::new_ttl

    source
    fn Record::new_ttl(key : Bytes, value : Bytes, timestamp : Int64, ttl : Int64) -> Record

    Creates a new active Record with a Time-To-Live (expires at timestamp + ttl).

    Record::serialize

    source
    fn Record::serialize(self : Record) -> Bytes

    Serializes the Record to its on-disk binary representation.

    ScanPage

    source
    pub struct ScanPage {
      rows : Array[(String, Bytes)]
      next_cursor : String?
      has_more : Bool
    }

    A page of ordered scan results for clients that need explicit continuation.

    WriteBatch

    source
    pub struct WriteBatch {
      db : DB
      ops : Array[(String, Bytes?, Int64, Bool)]
    }

    WriteBatch stores a sequence of database operations to be committed atomically.

    WriteBatch::commit

    source
    fn WriteBatch::commit(self : WriteBatch) -> Unit raise MoonKVError

    Commits all operations in the batch atomically to the WAL log.

    WriteBatch::delete

    source
    fn WriteBatch::delete(self : WriteBatch, key : String) -> Unit

    Adds a Delete operation to the batch.

    WriteBatch::put

    source
    fn WriteBatch::put(self : WriteBatch, key : String, value : Bytes) -> Unit

    Adds a Put operation to the batch.

    WriteBatch::put_ttl

    source
    fn WriteBatch::put_ttl(self : WriteBatch, key : String, value : Bytes, ttl : Int64) -> Unit

    Adds a Put operation with TTL to the batch.

    append_bytes_to_file

    source
    fn append_bytes_to_file(path : String, content : Bytes) -> Unit raise MoonKVError

    Appends a Bytes payload to the file at path for Native target.

    bytes_to_string

    source
    fn bytes_to_string(bytes : Bytes) -> String

    Converts UTF-8/ASCII Bytes to a String.

    decode_header

    source
    fn decode_header(header : Bytes) -> (Int, Int64, Int64, Int, Int) raise MoonKVError

    Returns (checksum, timestamp, expires_at, key_sz, val_sz).

    deserialize_record

    source
    fn deserialize_record(bytes : Bytes) -> Record raise MoonKVError

    Deserializes a complete Record from raw bytes, verifying its checksum.

    fnv1a

    source
    fn fnv1a(bytes : Array[Byte], offset : Int, len : Int) -> Int

    Computes the hash for a slice of bytes from offset to offset + len.

    fnv1a_bytes

    source
    fn fnv1a_bytes(bytes : Bytes, offset : Int, len : Int) -> Int

    Computes the hash for a slice of bytes from offset to offset + len.

    get_int32

    source
    fn get_int32(bytes : Bytes, offset : Int) -> Int

    Reads a 32-bit signed integer from Bytes starting at offset in Big-Endian format.

    get_int64

    source
    fn get_int64(bytes : Bytes, offset : Int) -> Int64

    Reads a 64-bit signed integer from Bytes starting at offset in Big-Endian format.

    merge_bytes

    source
    fn merge_bytes(b1 : Bytes, b2 : Bytes) -> Bytes

    Merges two Bytes objects into a single new Bytes object.

    read_hint_records_from_file

    source
    fn read_hint_records_from_file(path : String) -> Array[HintRecord] raise MoonKVError

    Reads and parses all hint records from a Hint File.

    read_records_from_file

    source
    fn read_records_from_file(path : String) -> Array[LogRecord] raise MoonKVError

    Sequentially reads and parses all records from a WAL data file.

    serialize_hint_record

    source
    fn serialize_hint_record(timestamp : Int64, expires_at : Int64, key_sz : Int, val_sz : Int, value_pos : Int64, key : Bytes) -> Bytes

    Layout: timestamp (8B) | expires_at (8B) | key_sz (4B) | val_sz (4B) | value_pos (8B) | key

    set_int32

    source
    fn set_int32(bytes : Array[Byte], offset : Int, value : Int) -> Unit

    Writes a 32-bit signed integer into an Array[Byte] starting at offset in Big-Endian format.

    set_int64

    source
    fn set_int64(bytes : Array[Byte], offset : Int, value : Int64) -> Unit

    Writes a 64-bit signed integer into an Array[Byte] starting at offset in Big-Endian format.

    string_to_bytes

    source
    fn string_to_bytes(s : String) -> Bytes

    Converts a String to UTF-8/ASCII Bytes.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io