🥮 mooncakes.io

    vcdb

    High-performance vector database with multiple ANN algorithms

    vector-database
    ann
    hnsw
    ivf
    similarity-search
    Download zip
    Author
    Version
    0.3.2
    License
    Apache-2.0
    Repository
    Last updated
    3 months ago
    Downloads
    681

    #vcdb

    High-performance vector database with multiple ANN algorithms for MoonBit.

    #Features

    • Multiple ANN Algorithms: HNSW, IVF, and Brute-force search
    • Flexible Storage: Pluggable backends (memory, S3)
    • Persistence: WAL-based durability with segment management
    • Attribute Filtering: Metadata-based vector filtering
    • Gateway Execution Layer: API handling reusable across native and JS runtimes

    #Overview

    vcdb is structured as a layered architecture:

    #Core Layers

    PackagePurpose
    core/annANN algorithms (HNSW, IVF, Bruteforce)
    core/storeVector data storage
    core/storageAbstract storage interface
    core/persistenceWAL and segment management
    core/attrAttribute indexing and filtering

    #API Layers

    PackagePurpose
    gatewayTransport-agnostic API execution core
    httpHTTP transport adapter over gateway
    jsJS runtime adapter, npm distribution, and HTTP server
    cmd/native-gatewayNative gateway execution entrypoint
    cmd/native-serveNative HTTP server over gateway
    cliCommand-line interface
    libLibrary entry point

    #Getting Started

    // Create a collection with HNSW index
    let store = @vcdb.CoreStore::new()
    let collection = store.create_collection("my_vectors", dim=128, ann_type=HNSW)

    // Add vectors with optional attributes
    collection.upsert([
      { id: "vec1", vector: [...], attrs: { "category": "A" } },
      { id: "vec2", vector: [...], attrs: { "category": "B" } },
    ])

    // Search with filtering
    let results = collection.search(
      query_vector,
      top_k=10,
      filter={ "category": "A" }
    )

    #Usage

    #Native Gateway Execution

    Execute gateway requests directly from MoonBit:

    moon run cmd/native-gateway -- healthz
moon run cmd/native-gateway -- collections create demo --dim 3

    #HTTP Serving via JS Adapter

    Start the JavaScript transport adapter:

    cd js
npm run build
node dist/server.js --host 127.0.0.1 --port 6333 --storage ../.local-storage

    #HTTP Serving via Native Adapter

    Start the MoonBit native transport adapter:

    moon run cmd/native-serve -- --host 127.0.0.1 --port 6333

    #Create Collection

    curl -X POST http://localhost:8080/collections/my_collection \
  -H "Content-Type: application/json" \
  -d '{"dim": 128, "ann_type": "hnsw"}'

    #Upsert Vectors

    curl -X PUT http://localhost:8080/collections/my_collection/points \
  -H "Content-Type: application/json" \
  -d '{"points": [{"id": "1", "vector": [...]}]}'

    #Search

    curl -X POST http://localhost:8080/collections/my_collection/search \
  -H "Content-Type: application/json" \
  -d '{"vector": [...], "top_k": 10}'

    #Installation

    #From Package Manager

    moon add trkbt10/vcdb

    #From Source

    git clone https://github.com/trkbt10/vcdb_mbt
cd vcdb_mbt
moon build

    #Requirements

    • MoonBit toolchain (moon >= 0.1.0)

    gateway is the API execution core. Long-lived HTTP serving is now available in both the JS adapter and the native async/http adapter.

    #License

    See LICENSE for details.

    PersistentDB

    source
    pub struct PersistentDB[W, S] {
      engine : VectorDB
      wal : 
    AsyncWalRuntime    [W]
      snapshot_storage : S
      base_path : String
      name : String
      checkpoint_threshold : Int
      checkpoint_bytes : Int
    }

    PersistentDB::add

    source

    Add a single vector. Fails if the ID already exists (not tombstoned).

    This is the WAL-backed equivalent of VectorDB::add. Existence check is performed BEFORE WAL append to prevent orphaned WAL records when the ID already exists.

    PersistentDB::checkpoint

    source

    Checkpoint: snapshot to snapshot_storage, then truncate WAL. Crash safe: snapshot-first, WAL truncate second.

    PersistentDB::compact

    source

    Compact HNSW index (removes tombstones). Returns removed count. Triggers checkpoint after compaction.

    PersistentDB::count_filtered

    source
    fn[W, S] PersistentDB::count_filtered(self : PersistentDB[W, S], expr? : 
    FilterExpr    ?) -> Int

    Count vectors matching filter.

    PersistentDB::deserialize

    source

    Deserialize from snapshot bytes, wrapping with in-memory storage.

    Used for loading a serialized database without persistence backends.

    PersistentDB::dim

    source
    fn[W, S] PersistentDB::dim(self : PersistentDB[W, S]) -> Int

    Vector dimension.

    PersistentDB::find

    source

    Find the single best match.

    PersistentDB::from_snapshot

    source
    fn[S] PersistentDB::from_snapshot(data : Bytes, storage : S, base_path : String, name : String, checkpoint_threshold? : Int, checkpoint_bytes? : Int) -> PersistentDB[S, S]

    Load from snapshot bytes with explicit storage backends.

    Used by gateway's load_collection to restore from a serialized snapshot and attach real storage backends for subsequent WAL operations.

    PersistentDB::get

    source

    Get a single vector by ID.

    PersistentDB::has

    source
    fn[W, S] PersistentDB::has(self : PersistentDB[W, S], id : 
    VectorId    ) -> Bool

    Check if a vector exists.

    PersistentDB::in_memory

    source

    Create an in-memory PersistentDB (no durable persistence).

    Uses MemoryStorage for both WAL and snapshot backends. Since MemoryStorage resolves all I/O synchronously, initialization completes immediately — but the function is async to maintain a uniform interface.

    PersistentDB::in_memory_bruteforce

    source

    Create an in-memory PersistentDB with Bruteforce strategy.

    PersistentDB::in_memory_hnsw

    source

    Create an in-memory PersistentDB with HNSW strategy.

    PersistentDB::in_memory_ivf

    source

    Create an in-memory PersistentDB with IVF strategy.

    PersistentDB::init

    source
    async fn[W : 
    AsyncStorage    , S : 
    AsyncStorage    ] PersistentDB::init(wal_storage : W, snapshot_storage : S, base_path : String, name : String, dim : Int, capacity : Int, metric? : 
    Metric    , strategy? : 
    Strategy    , checkpoint_threshold? : Int, checkpoint_bytes? : Int) -> PersistentDB[W, S]

    Initialize from storage: load WAL + snapshot, replay, build engine.

    Call sequence:
    1. Load snapshot from snapshot_storage
    2. Load WAL from wal_storage into in-memory buffer
    3. Deserialize snapshot -> CoreStore (or create empty)
    4. Replay WAL records onto CoreStore
    5. Build VectorDB from CoreStore

    PersistentDB::metric

    source

    Similarity metric.

    PersistentDB::raw_size

    source
    fn[W, S] PersistentDB::raw_size(self : PersistentDB[W, S]) -> Int

    Raw size including tombstones.

    PersistentDB::remove

    source

    Remove a vector with WAL-before-state guarantee.

    PersistentDB::scroll

    source

    Scroll through vectors in ascending ID order.

    PersistentDB::scroll_filtered

    source

    Scroll with filter expression.

    PersistentDB::search

    source

    Search by vector similarity with optional filter expression.

    PersistentDB::search_with_expr

    source

    Search with explicit filter expression, index override, and strategy.

    PersistentDB::search_with_filter

    source

    Search by vector similarity with callback filter.

    PersistentDB::serialize_snapshot

    source
    fn[W, S] PersistentDB::serialize_snapshot(self : PersistentDB[W, S]) -> Bytes

    Serialize current state as snapshot bytes.

    PersistentDB::size

    source
    fn[W, S] PersistentDB::size(self : PersistentDB[W, S]) -> Int

    Database size (excluding tombstones).

    PersistentDB::store

    source

    Access the underlying CoreStore (for advanced/distributed use).

    PersistentDB::strategy

    source

    ANN strategy.

    PersistentDB::train

    source
    fn[W, S] PersistentDB::train(self : PersistentDB[W, S], iterations? : Int) -> Unit

    Train IVF index (IVF strategy only).

    PersistentDB::update_attrs

    source

    Update attributes with WAL-before-state guarantee.

    PersistentDB::upsert

    source

    Upsert points with WAL-before-state guarantee.

    Order:
    1. Encode WAL records (sync)
    2. Persist WAL to storage (async, awaited)
    3. Update engine in-memory (sync)
    4. Auto-checkpoint if thresholds exceeded

    PersistentDB::wal_byte_size

    source
    fn[W, S] PersistentDB::wal_byte_size(self : PersistentDB[W, S]) -> Int

    Current WAL byte size (for diagnostics).

    PersistentDB::wal_record_count

    source
    fn[W, S] PersistentDB::wal_record_count(self : PersistentDB[W, S]) -> Int

    Current WAL record count (for diagnostics).

    VectorDB

    source

    VectorDB - Internal ANN index engine.

    Provides unified access to all ANN strategies: Bruteforce, HNSW, and IVF. The engine automatically handles:
    • Vector normalization for cosine similarity
    • Index updates on add/remove operations
    • Tombstone management for HNSW deletions

    This is an internal implementation detail. The public API is VectorDB[W, S].

    VectorDB::add

    source

    Add a new vector to the database.

    For HNSW mode, allows re-adding tombstoned IDs (clears tombstone and re-indexes). Fails if the ID already exists and is not tombstoned.

    Parameters:
    • id: Unique identifier for the vector
    • vector: The embedding vector (must match database dimension)
    • attrs: Metadata attributes for filtering

    VectorDB::compact

    source
    fn VectorDB::compact(self : VectorDB) -> (VectorDB, Int)

    Compact the database by removing tombstoned vectors (HNSW only).

    HNSW uses soft-delete (tombstones) for remove(). Over time, tombstones accumulate and waste memory. compact() rebuilds the HNSW graph without tombstoned entries.

    For Bruteforce and IVF, this is a no-op (they physically remove).

    Returns: A new VectorDB with the compacted state, and the number of removed tombstones.

    VectorDB::count_filtered

    source
    fn VectorDB::count_filtered(self : VectorDB, expr? : 
    FilterExpr    ?) -> Int

    Count vectors matching an optional filter expression.

    When expr is None, counts all non-tombstoned vectors. When expr is Some, resolves the expression to a candidate set via the B+ tree index, then subtracts tombstoned entries. No vector data is touched.

    Parameters:
    • expr: Optional filter expression to count

    Returns: Number of matching, non-tombstoned vectors

    VectorDB::deserialize

    source
    fn VectorDB::deserialize(data : Bytes) -> VectorDB

    Deserialize a database from bytes.

    Restores all vectors, attributes, metadata index state, and ANN index state.

    Parameters:
    • data: Binary data from serialize()

    Returns: The restored VectorDB instance

    VectorDB::dim

    source
    fn VectorDB::dim(self : VectorDB) -> Int

    Get the dimension of vectors in this database.

    VectorDB::export_jsonl

    source
    fn VectorDB::export_jsonl(self : VectorDB) -> Array[String]

    Export all points as JSONL lines.

    VectorDB::find

    source

    Find the single best match for a query.

    Convenience method equivalent to search(query, 1, filter).

    Parameters:
    • query: The query vector
    • filter: Optional filter predicate

    Returns: The best matching SearchHit, or None if no matches

    VectorDB::from_store

    source

    Create VectorDB from an existing CoreStore (for loading from distributed storage)

    VectorDB::get

    source

    Get a vector and its attributes by ID.

    For HNSW mode, tombstoned IDs return None.

    Parameters:
    • id: The vector ID to retrieve

    Returns: The vector record if found and accessible, None otherwise

    VectorDB::has

    source

    Check if an id exists and is accessible.

    For HNSW mode, tombstoned IDs are considered non-existent.

    Parameters:
    • id: The vector ID to check

    Returns: true if the vector exists and is not tombstoned

    VectorDB::metric

    source

    Get the similarity metric used by this database.

    VectorDB::new

    source

    Create a new VectorDB with the given options.

    Parameters:
    • options: Configuration including dimension, metric, capacity, and strategy

    Returns: A new VectorDB instance configured according to options

    VectorDB::raw_size

    source
    fn VectorDB::raw_size(self : VectorDB) -> Int

    Get the raw size including tombstoned vectors. This is primarily for internal use and serialization.

    VectorDB::remove

    source

    Remove a vector by ID.

    Behavior differs by strategy:
    • Bruteforce/IVF: Physically removes the vector
    • HNSW: Marks as tombstone (soft delete) to maintain graph connectivity

    For HNSW, use compact() to reclaim space from tombstoned vectors.

    Parameters:
    • id: The vector ID to remove

    Returns: true if the vector was removed, false if not found or already deleted

    VectorDB::scroll

    source

    Scroll points in ascending ID order, similar to Qdrant scroll.

    Thin wrapper over scroll_filtered with no filter expression. Offset semantics: returns entries with ID strictly greater than offset. Pass the last ID from previous page as offset to get the next page.

    VectorDB::scroll_filtered

    source

    Scroll points in ascending ID order with optional filter expression.

    When expr is None, returns all (non-tombstoned) entries. When expr is Some, resolves candidates via the B+ tree attribute index — every expression variant is handled by the index with a single execution path.

    Offset semantics: returns entries with ID strictly greater than offset (cursor = last ID seen by caller).

    Parameters:
    • expr: Optional filter expression to match against attributes
    • offset: Cursor — only IDs strictly greater than this value are returned
    • limit: Maximum number of entries to return

    Returns: Array of (VectorId, VectorRecord) sorted by ID ascending

    VectorDB::search

    source

    Search for k nearest neighbors to a query vector.

    Parameters:
    • query: The query vector (must match database dimension)
    • k: Number of results to return
    • filter: Optional predicate to filter results by ID and attributes

    Returns: Array of SearchHit sorted by score (highest/most similar first)

    VectorDB::search_with_expr

    source

    Search with filter expressions and metadata index support.

    Supports two filtering strategies:
    • PreFilter: Use attr index to get candidate IDs, then score only those
    • PostFilter: Score all vectors, then filter results
    • Auto: Automatically choose based on estimated selectivity

    PreFilter is more efficient when the filter is selective (<50% of vectors). PostFilter is better for broad filters or when no attr index is available.

    Parameters:
    • query: The query vector
    • k: Number of results to return
    • expr: Optional filter expression (Must/MustNot/Should/Leaf)
    • attr_index: Optional B+ tree index override for pre-filtering
    • strategy: Filtering strategy (PreFilter/PostFilter/Auto)

    Returns: Array of SearchHit matching the filter criteria

    VectorDB::serialize

    source
    fn VectorDB::serialize(self : VectorDB) -> Bytes

    Serialize the database to bytes for persistence.

    Includes all vectors, attributes, metadata index state, and ANN index state. Use deserialize() to restore.

    Returns: Binary representation of the database

    VectorDB::size

    source
    fn VectorDB::size(self : VectorDB) -> Int

    Get the number of accessible vectors in the database.

    For HNSW, this excludes tombstoned (deleted) vectors. Use raw_size() to get the total count including tombstones.

    VectorDB::store

    source

    Get access to the underlying CoreStore (for distributed storage operations)

    VectorDB::strategy

    source

    Get the ANN strategy used by this database.

    VectorDB::train

    source
    fn VectorDB::train(self : VectorDB, iterations? : Int) -> Unit

    Train the IVF index for optimal search performance.

    Only applicable for IVF strategy. Call this after adding a representative sample of vectors to build the centroid clusters.

    Parameters:
    • iterations: Number of k-means iterations (default: 10)

    VectorDB::update_attrs

    source

    Update only the attributes for a vector (without changing the embedding).

    Parameters:
    • id: The vector ID to update
    • attrs: New attributes to set

    Returns: true if updated, false if ID not found

    VectorDB::upsert

    source

    Add or update a vector (upsert operation).

    If the ID exists, updates the vector and reindexes. If the ID doesn't exist, adds a new vector.

    Parameters:
    • id: Unique identifier for the vector
    • vector: The embedding vector
    • attrs: Metadata attributes

    Returns: true if a new vector was created, false if existing was updated

    VectorDB::with_dim

    source
    fn VectorDB::with_dim(dim : Int) -> VectorDB

    Create a VectorDB with default options (Bruteforce strategy).

    Parameters:
    • dim: Vector dimension

    Returns: A VectorDB with Bruteforce strategy and Cosine metric

    VectorDB::with_hnsw

    source
    fn VectorDB::with_hnsw(dim : Int, metric? : 
    Metric    ) -> VectorDB

    Create a VectorDB with HNSW strategy.

    HNSW (Hierarchical Navigable Small World) is recommended for most use cases. It provides O(log n) search time with high recall.

    Parameters:
    • dim: Vector dimension
    • metric: Similarity metric (default: Cosine)

    Returns: A VectorDB configured with HNSW index

    VectorDB::with_ivf

    source
    fn VectorDB::with_ivf(dim : Int, metric? : 
    Metric    ) -> VectorDB

    Create a VectorDB with IVF strategy.

    IVF (Inverted File Index) is best for very large datasets. Requires training with train() before optimal search performance.

    Parameters:
    • dim: Vector dimension
    • metric: Similarity metric (default: Cosine)

    Returns: A VectorDB configured with IVF index

    empty_attrs

    source

    Re-export factory functions for convenience

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io