sos
Simple Object Storage
storage
opendal
filesystem
indexeddb
blob
Author
Version
0.1.2
License
Apache-2.0
Repository
Last updated
17 hours ago
Downloads
5
Dependencies
- moonbitlang/async0.21.0
#sos
Simple Object Storage: one storage interface, many backends.
sos is a small storage abstraction for MoonBit, modelled on
Apache OpenDAL: application code reads and writes
blobs through one interface, and does not care whether the bytes live in a
Map, on disk, or in a browser database.
@sos the interface, the types, the path rules all targets
@sos/memory a Map all targets
@sos/fs a real directory tree native
@sos/idbcore the IndexedDB semantics, over a `Conn` all targets
@sos/indexeddb IndexedDB, reached by Promise js
@sos/idbwasm IndexedDB, reached by token and export wasm-gc *
@sos/testing one conformance suite, run by all of them all targets* @sos/idbwasm is in the repository but not in the published module. Any
call to %async.suspend targeting wasm-gc crashes moonc as of
v0.10.12+1634b282e, and the token registry that backend parks calls in cannot
work without it. moon check passes, because the failure is in code generation
rather than typing. See docs/wasm-gc-async-suspend-ice.md
for the repro; the exclusion in moon.mod comes out when the compiler can build
it again.
There are two ways into IndexedDB from MoonBit and they have nothing in common:
on js an IDBRequest becomes a Promise; on wasm-gc a closure cannot cross
into JavaScript at all, so a call hands JavaScript a token and JavaScript calls
back into an export with it. What they have in common is everything else —
implicit directories, the key bound, the range delete, the ancestor walk — so
that lives once in @sos/idbcore, over a six-method Conn seam, for the reason
listing.mbt gives about the listing. A Conn over a Map is then a store the
suite can run under moon test, which is how semantics that used to be
reachable only from a browser got unit tests.
#The shape of it
Two layers, the way OpenDAL splits Access from Operator:
- Store is what a backend implements — eight async methods taking explicit option structs.
- Operator is what everyone else calls — labelled optional arguments, path validation, and capability checks before anything reaches the backend.
let op = @sos.Operator::new(@memory.MemoryStore::new())
op.write("notes/march.md", b"# March")
op.read("notes/march.md") // Bytes
op.read("notes/march.md", range=@sos.Range::prefix(7L))
op.list("notes/", recursive=true) // Array[Entry]
op.delete("notes/", recursive=true)Also on Operator: read_text, write_text, stat, exists, create_dir,
copy, rename.
Every method is async and raises SosError. Nothing from a backend's own
dependencies escapes: a caller's catch sees the same error type whether the
failure came from a Map, from errno, or from an IDBRequest.
#Paths
Relative to the store root, always /, and a trailing / means directory. The
root is "", never "/". Operator normalises and validates every path before
a backend sees it, which is what makes it safe for FsStore to concatenate one
onto its root: .., a leading /, \ and U+FFFF are rejected outright.
Ordering is by @sos.path_compare, not by <. MoonBit's Compare for String
is length-first, so "b/" < "a.txt" — a fine total order, and not the one a
listing promises. start_after is only a resumable cursor if ascending means
what every other store means by it.
#Capabilities
A backend declares what it can do, and Operator refuses the rest before the
call lands:
| memory | fs | both IndexedDB | |
|---|---|---|---|
| ranged reads | yes | yes | yes |
| append | yes | yes | no — two transactions would race |
| content type | yes | no — nowhere to keep one | yes |
| shared between processes | no | yes | yes |
The conformance suite reads these flags to decide which checks to run, and
asserts the inverse as well: every flag that is false must produce
Unsupported, so a backend cannot under-declare its way out of a check.
#Testing a backend
// A store that never suspends, on any target, with no scheduler:
@testing.run_sync(@memory.MemoryStore::new())
// A store that does, under moonbitlang/async:
@testing.run(store)run_sync uses the %async.run intrinsic, which drives an async computation on
the current stack. That is why the memory backend's conformance run works on
wasm-gc, where moonbitlang/async's event loop is unimplemented.
Note that an async test in a package that does not import moonbitlang/async
compiles, never runs, and passes — moon's generated driver falls back to a
no-op. Packages here that have no scheduler use run_sync instead.
#Running the tests
moon test --target nativeCore, memory and the filesystem backend.
moon test --target wasm-gc -p marianoguerra/sos \
-p marianoguerra/sos/memory -p marianoguerra/sos/idbcoreCore, memory and the IndexedDB semantics, on the backend with no event loop.
The packages are named one by one because a bare moon test --target wasm-gc
still reaches idbwasm and dies in the compiler; see the note at the top.
moon check --target allNeither IndexedDB backend has a moon test: node has no indexedDB. Each is
verified in a real browser instead, against the same suite —
moon build --target js --release indexeddb/harness
python3 -m http.server 8731Then open http://localhost:8731/indexeddb/harness/index.html. The page loads
the build output straight out of _build/ and must report 33 passed, 1 skipped,
0 failed, the skip being append.
The wasm-gc harness is built the same way —
moon build --target wasm-gc --release idbwasm/harness— and reported the same 33 passed, 1 skipped, 0 failed when it was written. It
does not build on the current compiler; see the note above.
idbwasm/harness is also the reference for embedding that backend. A host needs
three things and they are all in it: the two exports on_sos_reply /
on_sos_error in main.mbt, the sosidb import object, and a run_async to
start from — there is no event loop on wasm-gc, so the call that starts a run
finishes long before the run does.
The import object itself is idbwasm/loader.mjs, and it is the package's
contract rather than the harness's: copy it beside your page and spread
createSosImports(getExports) into the imports you instantiate with. A page is
served from one directory and an import that climbs out of it is a 403, which is
why it is copied rather than reached for where it lies.
#Demo
moon run cmd/main --target native#What is not here
Deliberately, for now: streaming readers and writers, presign, versioning,
conditional headers, and layers (retry, logging, metrics). read and write
take and return whole Bytes.
Store
pub(open) trait Store {
fn info(Self) -> StoreInfo
async fn read(Self, String, ReadOptions) -> Bytes raise SosError
async fn write(Self, String, Bytes, WriteOptions) -> Unit raise SosError
async fn stat(Self, String) -> Metadata raise SosError
async fn list(Self, String, ListOptions) -> Array[Entry] raise SosError
async fn delete(Self, String, DeleteOptions) -> Unit raise SosError
async fn create_dir(Self, String) -> Unit raise SosError
async fn copy(Self, String, String) -> Unit raise SosError = _
async fn rename(Self, String, String) -> Unit raise SosError = _
}What a storage backend must be able to do.
This is the raw seam -- OpenDAL's Access rather than its Operator. Two
obligations on an implementor, and both are things a hand-written backend
gets wrong:
- Paths arrive normalised and validated. Operator runs every path through validate before it gets here: no leading /, no .., no //, no ., and a trailing / exactly when the caller meant a directory. A backend must not re-normalise, and may concatenate the path onto its own root without escaping it -- which is precisely why validation is not optional and not the backend's job.
- Nothing from the backend's own dependencies escapes. Every failure leaves as a SosError, or a caller's catch sees a type from a package it never imported.
Implement it and hand the value to Operator::new; callers go through the
operator, not through this.
SosError
pub(all) suberror SosError {
SosError(kind~ : ErrorKind, operation~ : String, path~ : String, message~ : String)
}Every failure this library raises.
One error type, not one per backend: the point of the Store seam is that a
caller's catch sees the same thing whether the bytes came from a Map, a
disk or a browser database. A backend that lets @os_error.OSError or a JS
error escape has broken the abstraction, so each translates at its boundary.
SosError::is_not_found
True when the store said "there is nothing here".
The one kind callers branch on often enough to deserve a name; the rest go
through kind().
SosError::new
Capability
pub(all) struct Capability {
read : Bool
read_with_range : Bool
write : Bool
write_can_append : Bool
write_with_content_type : Bool
write_with_if_not_exists : Bool
stat : Bool
list : Bool
list_with_recursive : Bool
list_with_limit : Bool
list_with_start_after : Bool
delete : Bool
delete_with_recursive : Bool
create_dir : Bool
copy : Bool
rename : Bool
shared : Bool
} derive(Eq, Debug)What a backend can do.
Declared rather than discovered, and checked by Operator before the call
reaches the store, so an unsupported option fails with the operation's own
name instead of somewhere inside a backend.
The conformance suite reads this to decide which checks to run, and asserts
the inverse too: every false flag must produce Unsupported. A backend
cannot under-declare its way out of a check.
Capability::none
Everything off.
Build a real one with struct update, so adding a field to Capability does
not silently turn it on everywhere:
{ ..Capability::none(), read: true, write: true }.
DeleteOptions
Entry
One result of list: a path and what is known about it.
path is relative to the store root and is the exact string read and
stat will accept. A directory entry's path ends in /; a file's does not.
That invariant is what lets a caller recurse without a second stat.
EntryMode
What a path denotes.
Unknown is not a placeholder for "we did not look": it is what a filesystem
answers for a socket, a fifo or a device node, which a store can see and
cannot serve.
ErrorKind
What went wrong, in terms a caller can branch on.
The set is OpenDAL's, minus the kinds only a network backend can produce,
plus two this library needs and OpenDAL folds into Unexpected:
- InvalidPath, because a rejected path is the caller's bug, and a caller that cannot tell it from an I/O failure will retry it forever.
- DirectoryNotEmpty, because the only other honest answer is Unexpected, and then a caller cannot tell "pass recursive" from "something broke".
ListOptions
Metadata
What a store knows about one path.
Everything but mode is optional, because backends genuinely differ: a
filesystem has no content type and no etag, an object store has both. A
caller that needs one must check the backend's Capability first, or
tolerate None.
Metadata::file
Operator
type OperatorThe front door.
Holds a &Store and does three things the trait deliberately does not:
normalises and validates every path, turns labelled optional arguments into
the option structs, and refuses up front whatever Capability says the
backend cannot do. A backend author writes Store; everyone else writes
this.
Operator::copy
Operator::create_dir
Operator::delete
Operator::exists
Does it resolve? stat reduced to the question most callers were asking.
Operator::list
Operator::read
Operator::read_text
read, decoded as UTF-8.
Operator::rename
Operator::stat
Operator::store
The store underneath, for a caller that needs the raw seam.
Operator::write
Range
A half-open byte range, [start, end).
Half-open rather than inclusive because every slice API in MoonBit is, and a
library that flips the convention at its own boundary buys one off-by-one per
caller. end of None means "to the end of the object".
Range::resolve
Where this range lands in an object of size bytes, as (offset, length).
None means RangeNotSatisfied: the range starts before zero, or past the
end. An end beyond the object clamps rather than failing, which is what HTTP
range semantics do and what a caller asking for "the next 4KB" wants.
ReadOptions
StoreInfo
pub(all) struct StoreInfo {
scheme : String
root : String
name : String
capability : Capability
} derive(Eq, Debug)Who a store is.
Cheap and pure: Operator::new calls Store::info once at construction and
caches the answer, so it must not do I/O.
WriteOptions
ancestors
Every strict ancestor directory, shallowest first, in directory form.
ancestors("a/b/c") == ["a/", "a/b/"], ancestors("a") == [].
This is what the object-store backends walk to materialise the directory
records a filesystem gets for free from mkdir -p.
apply_list_options
Order a listing and apply start_after and limit, in that order.
Ascending by path, always: start_after is only a resumable cursor if the
order it resumes into is the order it was produced in, and every backend
promises the same one.
basename
fn basename(path : String) -> StringThe last segment, keeping the / for a directory.
basename("a/b/c") == "c", basename("a/b/") == "b/", basename("") == "".
depth
fn depth(path : String) -> IntNumber of /-separated segments. depth("") == 0, depth("a/b/") == 2.
ensure_dir_path
fn ensure_dir_path(path : String) -> StringAppend a / unless the path is the root or already has one.
group_flat_listing
Directory listing over a flat key space.
entries is every record the backend holds, in any order: files, and the
directory markers it chose to materialise. Returns what list(prefix, opts)
should answer, ordered and filtered.
Non-recursive, a child that has a / left in it after the prefix comes back
as one Dir entry for its first segment, however many files are under it.
Recursive, every descendant comes back, plus a Dir entry for every
directory implied along the way -- so a caller sees the same tree whether or
not the backend bothered to store markers for it.
A real record always wins over a synthesised one: the marker knows its
last_modified, the synthesised entry does not.
is_dir_path
fn is_dir_path(path : String) -> BoolTrue for the root ("") and for anything ending in /.
join
fn join(base : String, child : String) -> StringJoin, treating base as a directory whether or not it says so.
join("a/b", "c") == "a/b/c", join("", "c") == "c", join("a/", "") == "a/".
normalize
fn normalize(path : String) -> StringReduce path to canonical form. Total; never fails.
- runs of / collapse to one
- . segments drop
- a leading / drops
- a trailing / survives, because that is what says "directory" -- except that the root is "", so "/", ".", "./" and "//" all become ""
.. is NOT resolved. validate rejects it instead, because resolving it
would make a/../../b mean something and there is nothing above the root of
a store for it to mean.
parent
fn parent(path : String) -> StringThe containing directory, always in directory form.
parent("a/b/c") == "a/b/", parent("a/b/") == "a/", parent("a") == "",
parent("") == "".
path_compare
fn path_compare(a : String, b : String) -> IntOrder two store paths, ascending, by UTF-16 code unit.
MoonBit's Compare for String is length-first: it compares lengths and
only then contents, so "b/" < "a.txt". That is a fine total order and it is
not the one a listing promises -- pagination with start_after is only
resumable if "ascending" means what every other store means by it. So every
ordering decision in this library goes through here, and none through <.
Code units rather than code points, deliberately, because that is what
IndexedDB compares by. The IdbStore cursor is bounded by
prefix + "\u{FFFF}", and in code-unit order every astral character starts
with a surrogate below 0xFFFF and so stays inside the bound. In code-point
order it would not, and a key with an emoji in it would silently vanish from
a listing.
path_lt
fn path_lt(a : String, b : String) -> Boolpath_compare(a, b) < 0, for readability at call sites.
strip_dir_path
fn strip_dir_path(path : String) -> StringDrop a trailing /. The root stays the root.
validate
normalize plus the rules a store path must obey, or InvalidPath.
Rejects any .. segment, a NUL, a \, a Windows drive prefix (C:), and
U+FFFF.
That last one looks arbitrary and is not: the IndexedDB backend bounds its
prefix cursor with prefix + "\u{FFFF}", and the bound is only sound if no
stored key can contain the character. One rule in the core buys a correct
range scan in a backend.
validate_dir
validate, then ensure_dir_path, so a caller may write list("logs") and
mean the directory.
For list and create_dir.
Install
Download zipSimple Object Storage
storageopendalfilesystemindexeddbblob
0.1.2
Apache-2.0
Updated 17 hours ago
5 downloadsDependencies
- moonbitlang/async0.21.0