Delthin/moon_repomix/metrics

Delthin/moon_repomix/metrics does not have a README file

ResourceError

pub suberror ResourceError {
  ResourceError(String)
} derive(@moonbitlang/core/debug.Debug
Debug)

Invalid resource path, checksum, or resource contents. Filesystem errors and cancellation propagate unchanged from the async filesystem layer.

ResourceError::to_repr

source

#deprecated("`ResourceError::to_repr` is deprecated, use `@moonbitlang/core/debug.Debug::to_repr` instead.")
fn ResourceError::to_repr(ResourceError) -> @moonbitlang/core/debug.Repr
Repr

Encoder

source

type Encoder

A validated o200k_base encoder. Initialization performs all resource I/O; encoding is deterministic, does not normalize text, and treats special-token spellings as ordinary text. Instances contain no mutable encoding cache.

Encoder::count

source

fn Encoder::count(self : Encoder, text : String) -> Int

Count one complete text. Counts of separate fragments are not additive: callers must count the final rendered document for an exact total.

Encoder::encode

source

fn Encoder::encode(self : Encoder, text : String) -> Array[Int]

Encode complete text with o200k_base's ordinary-text policy. Token IDs are the original OpenAI mergeable ranks, not remapped vocabulary indices.

Encoder::load

source

async fn Encoder::load(path : String) -> Encoder

Load the bundled ranks from an absolute path and Unicode categories from tokenizer-unicode-16.0.0.txt in the same directory. Both SHA-256 digests are checked before parsing. No network access or cwd-based lookup occurs.