moon-s3-server

    A small, deployable S3-compatible object storage server written in MoonBit.

    s3
    object-storage
    http
    native
    Download zip
    Author
    Version
    0.1.0
    License
    Apache-2.0
    Last updated
    11 hours ago
    Downloads
    3

    #moon-s3-server

    A small S3-compatible object storage server written in MoonBit, for local development, CI, and private deployments. Stores data on local disk and uses path-style URLs.

    Status: early development (v0.1); not production-ready.

    #Features

    • Bucket and object CRUD; batch deletion; ListObjectsV2 with prefixes, pagination, and delimiters.
    • Optional SigV4 authentication and presigned GET / HEAD / PUT URLs.
    • Multipart uploads with resumable parts, pagination, abort, and expiration.
    • Range downloads, conditional reads/writes, ETags, and Content-Type metadata.
    • Server-side CopyObject across buckets, with source conditions and COPY / REPLACE Content-Type.
    • Write/delete transaction recovery, consistent concurrent reads, and a data directory lock.
    • Persistent bucket CORS rules with preflight matching; a GET /healthz endpoint.
    • Offline storage checks, SHA-256 backups, and verified restores with multipart resume.
    • Bucket versioning, historical reads, delete markers, version listing, and version restore by copy.
    • Bucket public-read/private switching through a restricted S3 Bucket Policy API.
    • Durable JSON webhooks for object creation and deletion, with retries and optional HMAC signatures.
    • Standard-library management CLI for files, versions, access, CORS, multipart cleanup, and offline administration.
    • Web console with a separate admin login, file browsing, transfers, version restore, and bucket settings.
    • JSON configuration, graceful shutdown, structured logs, authenticated metrics, and Docker packaging.

    Fixed-version client acceptance covers botocore 1.44.0 (including a 100 MiB multipart round trip), AWS CLI 2.22.35, MinIO mc over TLS, and selected Ceph s3-tests. See the compatibility matrix and reproduction steps.

    #Quick start

    Requires the MoonBit CLI (tested with moon 0.1.20261102; CI uses the hash-locked 0.1.20261104 snapshot).

    git clone https://github.com/zzSmall6/moon-s3-server.git cd moon-s3-server moon build cmd/main --target native MOON_S3_ROOT=./data MOON_S3_LISTEN=127.0.0.1:9000 \ _build/native/debug/build/cmd/main/main.exe

    In another terminal, create a bucket and upload a file:

    curl --noproxy '*' -X PUT http://127.0.0.1:9000/demo printf 'Hello MoonBit!\n' > hello.txt curl --noproxy '*' -X PUT --data-binary @hello.txt http://127.0.0.1:9000/demo/hello.txt curl --noproxy '*' http://127.0.0.1:9000/demo/hello.txt curl --noproxy '*' 'http://127.0.0.1:9000/demo?list-type=2'

    Or run bash scripts/smoke.sh for a self-contained example. For the management CLI, run python3 scripts/moon_s3.py --help; see command-line management for examples. To open the Web console, set MOON_S3_CONSOLE_PASSWORD and run python3 scripts/console.py; visit http://127.0.0.1:9001.

    #Configuration

    Environment variableDefaultPurpose
    MOON_S3_CONFIGUnsetJSON configuration file; environment variables override file values
    MOON_S3_ROOT./dataData directory
    MOON_S3_LISTEN127.0.0.1:9000Listen address
    MOON_S3_ACCESS_KEYUnsetEnable SigV4 authentication
    MOON_S3_SECRET_KEYUnsetSecret paired with the access key
    MOON_S3_CORS_ORIGIN*Fallback origin for buckets without CORS rules
    MOON_S3_MULTIPART_TTL_SECONDS604800Upload lifetime from creation; 1–31536000 seconds
    MOON_S3_WEBHOOK_URLUnsetEnable object-event delivery to an HTTP(S) endpoint
    MOON_S3_WEBHOOK_SECRETUnsetSign webhook bodies with HMAC-SHA256
    MOON_S3_WEBHOOK_TIMEOUT_MS5000Callback timeout; 100–60000 milliseconds
    MOON_S3_SHUTDOWN_TIMEOUT_MS10000Drain active requests on SIGINT/SIGTERM; 100–300000 milliseconds
    MOON_S3_MAX_REQUEST_BYTES67108864Maximum buffered request body; 1–536870912 bytes
    MOON_S3_MAX_BUFFERED_BYTES134217728Shared body budget; at least the per-request limit, up to 536870912 bytes
    MOON_S3_MAX_CONNECTIONS128Maximum simultaneous connections; 1–4096
    MOON_S3_REQUEST_TIMEOUT_MS30000Header-read and request-handling deadlines; 100–3600000 ms

    Set both credentials to nonempty values to enable authentication; leaving both unset allows anonymous access. Use a trusted network and a TLS reverse proxy for deployments beyond localhost. See deployment and operations for configuration files, Docker/Compose, shutdown behavior, JSON logs, and authenticated GET /metrics. With credentials enabled, buckets default to private. Bucket policies can grant anonymous object reads while keeping writes and management authenticated.

    SDK clients should use SigV4, path-style addressing, and disable optional checksums (request_checksum_calculation="when_required" and response_checksum_validation="when_required" in botocore). For temporary links, configure matching credentials and use python3 scripts/presign.py --help. Use the SDK's copy_object operation to copy without downloading; copy headers must be signed when authentication is enabled. Use put_bucket_cors / get_bucket_cors / delete_bucket_cors for bucket rules; see CORS configuration for matching and browser behavior. See versioning for enable/suspend behavior and historical object operations. See event webhooks for payloads, retries, and receiver deduplication.

    #Limits and recovery

    • Object keys support 1–1024 UTF-8 bytes; invalid URL encoding is rejected.
    • Upload requests are buffered in memory with per-request and shared limits; choose multipart sizes accordingly. Parts are numbered 1–10000 and must be at least 5 MiB except the last.
    • Multipart state survives restarts. Expiration cleanup runs at startup and on multipart traffic; upload activity does not extend the lifetime.
    • One process per data directory. Transaction recovery runs before serving requests; corrupt or incomplete committed records prevent startup.
    • Recovery is tested on Linux for process crashes; power loss and hardware faults need further validation.
    • Bucket policies support public object-read grants only; no general IAM/ACL, encryption, or online backups yet. Only single-range downloads are supported.
    • No STS tokens, AWS streaming signatures, checksums beyond MD5/CRC32, UploadPartCopy, or delimiter grouping for multipart upload listings.
    • CopyObject supports sources up to 5 GiB and Content-Type metadata only; tags, ACLs, and custom metadata are unsupported.
    • Unsupported subresources, encryption and extra checksum options are rejected. Content-MD5 and request CRC32 are verified; Content-Type is limited to 1024 bytes.

    #Backup and restore

    Stop the server first. On Linux with Python 3.9+, run:

    python3 scripts/admin.py check --root ./data python3 scripts/admin.py backup --root ./data --output ./backup.tar python3 scripts/admin.py restore --archive ./backup.tar --root ./restored-data

    Restore requires a new directory and verifies all files before publishing it. See administration for guarantees, limits, and recovery.

    #Development

    moon test --target native moon info && moon fmt bash scripts/smoke.sh for suite in auth list object presigned multipart storage copy admin cors version policy event cli console operations safety delete; do python3 "scripts/${suite}_regression.py" done

    Regression scripts use Python's standard library. With botocore installed, add --sdk to the presigned, multipart, copy, CORS, version, or policy regression script for SDK checks. The console regression also supports --browser with Playwright Chromium installed.

    Pushes to main and pull requests run native CI and all local regression suites. External client acceptance and release packaging are manual Actions workflows. See release bundles and validation for installation and publishing. Run the deployment recovery drill before delivery or storage migration.

    #License

    ParsedAuth

    pub(all) struct ParsedAuth {
    access_key : String
    date : String
    region : String
    service : String
    signed_headers : String
    signature : String
    }

    QueryParam

    pub(all) struct QueryParam {
    name : String
    value : String
    }

    canonical_headers

    fn canonical_headers(signed_headers : StringView, headers : Array[(String, String)]) -> String

    canonical_query

    fn canonical_query(params : Array[QueryParam]) -> String

    filename_to_key

    fn filename_to_key(name : StringView) -> String?

    hmac_sha256

    fn hmac_sha256(key : BytesView, data : BytesView) -> FixedArray[Byte]

    hmac_sha256_hex

    fn hmac_sha256_hex(key : BytesView, data : BytesView) -> String

    key_to_filename

    fn key_to_filename(key : StringView) -> String

    md5_hex

    fn md5_hex(data : BytesView) -> String

    parse_authorization

    fn parse_authorization(value : StringView) -> ParsedAuth?

    percent_decode

    fn percent_decode(input : StringView) -> String

    query_pairs

    fn query_pairs(query : StringView) -> Array[QueryParam]

    query_value

    fn query_value(params : Array[QueryParam], name : StringView) -> String?

    sha256_hex

    fn sha256_hex(data : BytesView) -> String

    signing_key

    fn signing_key(secret : StringView, date : StringView, region : StringView, service : StringView) -> FixedArray[Byte]

    split_bucket_key

    fn split_bucket_key(path : StringView) -> (String?, String?)

    split_raw_path

    fn split_raw_path(path : StringView) -> (String, String)

    uri_encode

    fn uri_encode(input : StringView, allow_slash? : Bool) -> String

    valid_bucket_name

    fn valid_bucket_name(name : StringView) -> Bool

    verify_sigv4

    fn verify_sigv4(http_method : StringView, canonical_uri : StringView, query : Array[QueryParam], signed_header_pairs : Array[(String, String)], payload_hash : StringView, authorization : StringView, access_key : StringView, secret_key : StringView) -> Bool

    xml_escape

    fn xml_escape(value : StringView) -> String