Sign in

    moonbagit

    BagIt 1.0 archive package creation and validation for MoonBit

    bagit
    rfc8493
    preservation
    integrity
    Download zip
    Author
    Version
    0.1.0
    License
    MIT
    Last updated
    yesterday
    Downloads
    5

    #MoonBagIt

    CI

    BagIt 1.0 archive-directory creation and validation for MoonBit, based on RFC 8493.

    The input is a directory of opaque files, such as a research dataset or a digital-library export. The output is a standard BagIt directory, not a custom JSON evidence pack, backup database, ZIP file, or network transfer service.

    #Quick Start

    Install the MoonBit toolchain and Node.js 22 or later. Use MoonBit moon 0.1.20260920 or later. This release was tested with moonc v0.10.14 and Node.js 24.14.1.

    moon check --target js moon test --target js moon build --target js node bin/moonbagit.mjs create examples/payload example-bag --both --metadata examples/metadata.json node bin/moonbagit.mjs validate example-bag

    example-bag must not already exist. Creation leaves the source unchanged. Default creation uses SHA-512; --sha256 and --both select other supported manifest sets. Metadata is an ordered JSON array of { "label", "value" } objects, allowing repeated labels and multi-line values. Payload-Oxum is computed by MoonBagIt and cannot be supplied by the caller.

    example-bag/ bagit.txt bag-info.txt manifest-sha256.txt manifest-sha512.txt tagmanifest-sha256.txt tagmanifest-sha512.txt data/ readme.txt nested/measurement.csv

    Validation returns JSON with complete, valid, payload counts, verified checksum counts and diagnostics containing code, path, message and severity. Int64 counts use decimal strings to preserve integer precision. Exit codes: 0 valid/success, 1 invalid or unsupported bag, 2 I/O or CLI failure. Invalid UTF-8 is an I/O decoding failure, not silently replaced text.

    Changing payload bytes without changing its length leaves the structure complete but makes it invalid. Adding an unlisted payload file makes the bag incomplete. SHA digests detect corruption; they do not authenticate the sender or protect against an attacker who can rewrite files and manifests together.

    #Library

    The root MoonBit package exports typed models, parse_declaration, parse_manifest, parse_metadata, parse_fetch, encode_path, decode_path, create_tags, create_tag_manifests, payload_oxum, and validate. It also exports parse_profile, validate_profile, plan_upgrade, and lint_portability.

    After publication, add the library to an independent MoonBit project:

    moon add zbhzs1/moonbagit@0.1.0

    Add "zbhzs1/moonbagit" @bag to the consumer package's import block. Then call @bag.parse_declaration(...), @bag.create_tags(...) or @bag.validate(...). The registry library does not install a global CLI: clone this repository and build the JS target for real-filesystem commands.

    BagView separates the rules engine from its filesystem. It contains file records with relative paths, Int64 sizes and raw-byte digests, directory names, and strictly decoded text tags. Callers must supply accurate records and hashes. validate does not read files or compute hashes itself. The CLI supplies these through a small Node.js adapter using streaming node:crypto hashes; there is no custom cryptographic implementation. The pure core also runs on WASM-GC.

    Creation uses two stages: create_tags generates the declaration, metadata and payload manifests; after the caller hashes these exact UTF-8 bytes, create_tag_manifests generates tag manifests. See the runnable CLI and generated interface for the full signatures.

    Run the in-memory library example without Node filesystem integration:

    moon run examples/core --target wasm-gc

    #Supported Scope

    AreaCurrent behavior
    DeclarationBagIt 1.0, UTF-8 only; exact field order; no BOM
    Payload manifestsSHA-256 and SHA-512; every manifest covers every payload file exactly once
    Tag manifestsRequired payload-manifest coverage; forbid payload and tag-manifest references; check listed arbitrary tags
    MetadataOrdered/repeated fields, continuations; unique and numeric Payload-Oxum
    Paths/ separators, normative %25, %0D, %0A; no traversal, absolute, NUL, drive or backslash paths
    ChecksumsVerify every supported manifest, including all tag manifests
    FetchParse and check local coverage/length; missing remote files remain incomplete; no download or URI execution
    Additional tagsOpaque content; verify if listed; unprotected tags produce warnings
    Host safetyReject symlinks and special files; no source overwrite; new output only
    Receiving profilesBagIt Profiles 1.4.0 rules for metadata, algorithms, fetch, files/directories and directory serialization constraints
    Digest upgradeAdd supported manifests to a new output; retain payload, metadata and existing payload manifests; regenerate every tag manifest
    Portability lintAdvisory device-name, trailing-character, case-collision and path-length diagnostics

    #Receiving and Upgrading

    The included research-data scenario supplies receiving requirements, rather than generic JSON schema validation. Profile JSON is read from a caller-selected local file; profile identifiers and fetch URLs are never executed or fetched.

    node bin/moonbagit.mjs create examples/payload research-bag --both --metadata examples/research-metadata.json node bin/moonbagit.mjs validate research-bag --profile examples/research-profile.json node bin/moonbagit.mjs lint research-bag node bin/moonbagit.mjs create examples/payload sha256-bag --sha256 node bin/moonbagit.mjs upgrade sha256-bag upgraded-bag --sha512 node bin/moonbagit.mjs validate upgraded-bag

    Profile results expose conforms, the separate base bag report, and profile diagnostics. A checksum-valid bag can fail a receiving profile. Unknown profile keys and older profile specifications are rejected, not ignored. Only * is a wildcard and matches zero or more characters, including /. Serialized input is currently not opened: a profile requiring serialization rejects a directory bag. optional permits this directory input; its accepted MIME types describe serialized input, not the directory. See profiles.

    Upgrade checks the original bag, copies it to sibling staging, rechecks the copy, adds missing manifests and regenerates all old/new tag manifests so that they cover every payload manifest. Existing corrupt bags are refused; upgrade is not a repair or authenticity service. The original bag is never modified. Existing valid manifests are preserved byte-for-byte; repeating an upgrade produces the same manifest content when tag coverage was already complete. Previously unprotected arbitrary tags are included in regenerated tag manifests. The pure plan_upgrade API only prepares payload manifests: the host must hash their exact bytes before calling create_tag_manifests.

    #Compatibility Boundaries

    This is a defined subset, not a claim of complete RFC conformance. Legacy BagIt versions, MD5/SHA-1, non-UTF-8 tags, automatic fetch, Unicode normalization matching, and general RFC 3986 URI validation are future compatibility work. The fetch parser currently verifies an absolute URI scheme and token structure, not every URI grammar rule. Case-only collisions produce warnings. Portable path restrictions intentionally reject some otherwise legal Unix filenames.

    The CLI supports 100,000 files and 100,000 directories, 16 MiB per standard text tag and 64 MiB total standard text. Payload hashing is streaming. File sizes must fit the Node.js safe-integer range; the core uses Int64. File mutation is checked while reading, but this is not a hostile-filesystem sandbox or lock service. Keep the source, destination and scanned bag quiescent during an operation. Creation uses a sibling staging directory and renames it only after validation; failed staging directories are retained for inspection. It does not promise crash durability or race-free publication against another writer.

    #Verification

    moon test --target js moon test --target wasm-gc python -m pip install bagit==1.9.0 python scripts/interop.py git -c core.autocrlf=false clone https://github.com/LibraryOfCongress/bagit-conformance-suite.git ../bagit-conformance-suite git -C ../bagit-conformance-suite checkout 9ab48705c06664f9f419ca740c728c9f89a61e7c python scripts/conformance.py ../bagit-conformance-suite python scripts/benchmark.py --files 10000 --bytes 1024

    The independent Library of Congress bagit-python tool validates generated bags, and MoonBagIt validates bags generated by it. The released tool creates BagIt 0.97, so the reverse test upgrades the declaration to 1.0 and asks that independent tool to regenerate tag manifests. No MoonBagIt creation or hashing is used to regenerate those independent manifests. The same suite was also run against the pinned upstream snapshot listed in the support document, which natively creates 1.0 bags and needs no such upgrade.

    Known upstream compatibility issue: bagit-python does not decode %25 in manifest filenames. The percent-filename test records this limitation explicitly; it is not counted as successful cross-tool interoperability. The normative MoonBagIt encoding is retained. Newline-filename and symlink tests are skipped where Windows permissions or filesystem restrictions prevent them; Linux CI also runs the suite. See support details.

    #Competition Status

    The public repository contains the implementation, CI workflow, executable examples and independently reproducible tests. The current delivered scope is the documented BagIt subset and local receiving/upgrade workflows, not a backup, evidence-ledger or network-transfer service. Verification details separate observed checks from future work. Competition acceptance is decided by the organizers, not by test count or a configured workflow.

    #License

    MIT. The implementation is written against RFC 8493, not copied from adjacent MoonBit archive/evidence projects. bagit-python is a test-only independent dependency distributed under CC0; its source is not part of this package. The independently maintained conformance fixtures and BagIt Profiles specification are also CC0. See third-party references.

    FormatError

    pub(all) suberror FormatError {
    Invalid(String)
    } derive(Eq,
    Debug
    )

    FormatError::equal

    fn FormatError::equal(FormatError, FormatError) -> Bool

    FormatError::not_equal

    fn FormatError::not_equal(x : FormatError, y : FormatError) -> Bool

    Algorithm

    pub(all) enum Algorithm {
    Sha256
    Sha512
    } derive(Eq,
    Debug
    )

    Algorithm::equal

    fn Algorithm::equal(Algorithm, Algorithm) -> Bool

    Algorithm::hex_length

    fn Algorithm::hex_length(self : Algorithm) -> Int

    Algorithm::name

    fn Algorithm::name(self : Algorithm) -> String

    Algorithm::not_equal

    fn Algorithm::not_equal(x : Algorithm, y : Algorithm) -> Bool

    BagView

    pub(all) struct BagView {
    files : Array[FileRecord]
    directories : Array[String]
    texts : Map[String, String]
    } derive(
    Debug
    ,
    FromJson
    )

    Declaration

    pub(all) struct Declaration {
    version : String
    encoding : String
    } derive(Eq,
    Debug
    )

    Declaration::equal

    fn Declaration::equal(Declaration, Declaration) -> Bool

    Declaration::not_equal

    fn Declaration::not_equal(x : Declaration, y : Declaration) -> Bool

    Diagnostic

    pub(all) struct Diagnostic {
    severity : Severity
    code : String
    path : String
    message : String
    } derive(ToJson,
    Debug
    )

    Diagnostic::to_json

    fn Diagnostic::to_json(Diagnostic) -> Json

    Entry

    pub(all) struct Entry {
    path : String
    checksum : String
    } derive(Eq,
    Debug
    )

    Entry::equal

    fn Entry::equal(Entry, Entry) -> Bool

    Entry::not_equal

    fn Entry::not_equal(x : Entry, y : Entry) -> Bool

    Entry::to_repr

    FetchEntry

    pub(all) struct FetchEntry {
    url : String
    size : Int64?
    path : String
    } derive(Eq,
    Debug
    )

    FetchEntry::equal

    fn FetchEntry::equal(FetchEntry, FetchEntry) -> Bool

    FetchEntry::not_equal

    fn FetchEntry::not_equal(x : FetchEntry, y : FetchEntry) -> Bool

    FileRecord

    pub(all) struct FileRecord {
    path : String
    size : Int64
    digests : Map[String, String]
    } derive(ToJson,
    Debug
    ,
    FromJson
    )

    The host supplies hashes of raw file bytes, not hashes of decoded text.

    FileRecord::to_json

    fn FileRecord::to_json(FileRecord) -> Json

    Manifest

    pub(all) struct Manifest {
    name : String
    algorithm : Algorithm
    tag : Bool
    entries : Array[Entry]
    } derive(Eq,
    Debug
    )

    Manifest::equal

    fn Manifest::equal(Manifest, Manifest) -> Bool

    Manifest::not_equal

    fn Manifest::not_equal(x : Manifest, y : Manifest) -> Bool

    Manifest::to_repr

    Metadata

    pub(all) struct Metadata {
    label : String
    value : String
    } derive(Eq, ToJson,
    Debug
    ,
    FromJson
    )

    Metadata::equal

    fn Metadata::equal(Metadata, Metadata) -> Bool

    Metadata::not_equal

    fn Metadata::not_equal(x : Metadata, y : Metadata) -> Bool

    Metadata::to_json

    fn Metadata::to_json(Metadata) -> Json

    Metadata::to_repr

    MetadataRule

    pub(all) struct MetadataRule {
    label : String
    required : Bool
    repeatable : Bool
    values : Array[String]
    } derive(
    Debug
    )

    Profile

    pub(all) struct Profile {
    identifier : String
    version : String
    metadata : Array[MetadataRule]
    manifests_required : Array[String]
    manifests_allowed : Array[String]?
    tag_manifests_required : Array[String]
    tag_manifests_allowed : Array[String]?
    allow_fetch : Bool
    require_fetch : Bool
    data_empty : Bool
    serialization : String
    accepted_serializations : Array[String]
    bag_versions : Array[String]
    tags_required : Array[String]
    tags_allowed : Array[String]
    payload_required : Array[String]
    payload_allowed : Array[String]
    } derive(
    Debug
    )

    Profile::to_repr

    ProfileReport

    pub(all) struct ProfileReport {
    conforms : Bool
    bag : Report
    diagnostics : Array[Diagnostic]
    } derive(ToJson,
    Debug
    )

    ProfileReport::to_json

    Report

    pub(all) struct Report {
    complete : Bool
    valid : Bool
    payload_files : Int
    payload_bytes : Int64
    verified_checksums : Int
    diagnostics : Array[Diagnostic]
    } derive(ToJson,
    Debug
    )

    Report::has_code

    fn Report::has_code(self : Report, code : String) -> Bool

    Report::to_json

    fn Report::to_json(Report) -> Json

    Report::to_repr

    Severity

    pub(all) enum Severity {
    Error
    Warning
    } derive(Eq, ToJson,
    Debug
    )

    Severity::equal

    fn Severity::equal(Severity, Severity) -> Bool

    Severity::not_equal

    fn Severity::not_equal(x : Severity, y : Severity) -> Bool

    Severity::to_json

    fn Severity::to_json(Severity) -> Json

    Severity::to_repr

    UpgradePlan

    pub(all) struct UpgradePlan {
    payload_manifests : Map[String, String]
    tag_algorithms : Array[Algorithm]
    } derive(
    Debug
    )

    algorithm

    fn algorithm(name : String) -> Algorithm raise FormatError

    check_path

    fn check_path(path : String) -> Unit raise FormatError

    Conservative portable paths also rule out drive letters and UNC separators.

    create_tag_manifests

    fn create_tag_manifests(tags : Array[FileRecord], algorithms? : Array[Algorithm]) -> Map[String, String] raise FormatError

    create_tags

    fn create_tags(payload : Array[FileRecord], metadata? : Array[Metadata], algorithms? : Array[Algorithm]) -> Map[String, String] raise FormatError

    Returns initial tag text; callers then hash these bytes to build tag manifests.

    decode_path

    fn decode_path(path : String) -> String raise FormatError

    encode_path

    fn encode_path(path : String) -> String

    lint_portability

    fn lint_portability(view : BagView) -> Array[Diagnostic]

    Advisory destination portability checks, distinct from BagIt validity.

    parse_declaration

    fn parse_declaration(text : String) -> Declaration raise FormatError

    parse_fetch

    fn parse_fetch(text : String) -> Array[FetchEntry] raise FormatError

    parse_manifest

    fn parse_manifest(name : String, text : String) -> Manifest raise FormatError

    parse_metadata

    fn parse_metadata(text : String) -> Array[Metadata] raise FormatError

    parse_profile

    fn parse_profile(json : Json) -> Profile raise FormatError

    Strict BagIt Profiles 1.4 JSON. Older versions and unknown constraints fail closed.

    payload_oxum

    fn payload_oxum(files : Array[FileRecord]) -> String raise FormatError

    plan_upgrade

    fn plan_upgrade(view : BagView, algorithms : Array[Algorithm]) -> UpgradePlan raise FormatError

    Refuses invalid sources. The caller must rehash all tags after writing the plan.

    profile_path_matches

    fn profile_path_matches(pattern : String, path : String) -> Bool

    Only '*' is special. It matches zero or more code units, including '/'.

    render_manifest

    fn render_manifest(entries : Array[Entry]) -> String raise FormatError

    render_metadata

    fn render_metadata(fields : Array[Metadata]) -> String raise FormatError

    validate

    fn validate(view : BagView) -> Report

    validate_profile

    fn validate_profile(view : BagView, profile : Profile) -> ProfileReport

    Validates a directory bag and receiving rules; no profile URL or payload is fetched.

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io