Encoding package extracted from moonbitlang/x

Author

Version

0.1.3

License

Apache-2.0

Repository

github.com/moonbitlang/x

Last updated

last year

Downloads

No dependencies

#tonyfettes/encoding

This package is extracting from moonbitlang/x.

#Moonbit/Core Encoding

#Overview

The @encoding package provides an implementation for encoding and decoding strings using various character encodings (e.g., UTF-8).

It supports both streaming and non-streaming (bulk) operations, making it flexible for different use-cases.

#Supported Encoding

UTF8
UTF16 // alias for UTF16LE
UTF16LE
UTF16BE

#Usage

#Decoding

Decode a UTF-8 byte stream:

// Initialize a streaming UTF-8 decoder
let decoder = @encoding.decoder(UTF8)

// Consume byte chunks
let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
inspect!(decoder.consume!(inputs[0]), content="abc")
inspect!(decoder.consume!(inputs[1]), content="")
inspect!(decoder.consume!(inputs[2]), content="🐰")

// Finish decoding
assert_true!(decoder.finish!().is_empty())

#Encoding

Encode a string to UTF-8 bytes:

// Encode a string to UTF-8
let src = "你好👀"
let bytes = @encoding.encode(UTF8, src)
inspect!(
  bytes,
  content=
    #|b"\xe4\xbd\xa0\xe5\xa5\xbd\xf0\x9f\x91\x80"
  ,
)

Encode a single character to UTF-8 bytes:

inspect!(
  @encoding.to_utf8_bytes('A'),
  content=
    #|b"\x41"
  ,
)

Encode

source

pub trait Encode {
  size_hint(Self, encoding~ : Encoding) -> Int
  encode(Self, encoding~ : Encoding) -> Bytes
  encode_to(Self, @moonbitlang/core/buffer.T
T, encoding~ : Encoding) -> Unit
}

impl Encode for Char

impl Encode for String

impl Encode for StringView

fn encode_to(value : @moonbitlang/core/string.StringView
StringView, buffer : @moonbitlang/core/buffer.T
T, encoding~ : Encoding) -> Unit

source

fn size_hint(value : @moonbitlang/core/string.StringView
StringView, encoding~ : Encoding) -> Int

source

Decoder

source

type Decoder

Decoder::decode

source

fn Decoder::decode(self : Decoder, input : Bytes, stream~ : Bool = ..) -> @moonbitlang/core/string.StringView
StringView!DecodingError

Decode the given byte sequence using the specified Decoder and return the resulting string.

This function can work in streaming mode where bytes are consumed incrementally. When stream is false, it indicates the end of the input and triggers the final decoding step.

Parameters

self: The Decoder instance used to decode the byte sequence.
input: The byte sequence to be decoded.
stream~: A boolean indicating whether more bytes will be supplied for decoding. It defaults to false.

Returns

A String representing the decoded content from the input byte sequence.

Errors

MalformedError: when the byte sequence is not properly formatted according to the specified encoding. TruncatedError: when the byte sequence ends prematurely, implying that more data is expected for complete decoding.

Examples

let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
let decoder = @encoding.decoder(UTF8)
inspect!(decoder.decode!(inputs[0], stream=true), content="abc")
inspect!(decoder.decode!(inputs[1], stream=true), content="")
inspect!(decoder.decode!(inputs[2], stream=false), content="🐰")

Decoder::decode_to

source

fn Decoder::decode_to(self : Decoder, input : Bytes, output : StringBuilder, stream~ : Bool = ..) -> Unit!DecodingError

Decodes the given byte sequence using the specified decoder and writes the result directly to a StringBuilder. Similar to decode!, but writes the result to an existing StringBuilder instead of creating a new String.

Parameters:

decoder : The decoder instance used to decode the byte sequence.
input : The byte sequence to be decoded.
output : The StringBuilder where the decoded content will be written to.

Throws a MalformedError when the byte sequence is not properly formatted according to the specified encoding.

Example:

test "decode_to" {
  let decoder = decoder(UTF8)
  let buf = StringBuilder::new()
  decoder.decode_to!(b"Hello", buf)
  inspect!(buf.to_string(), content="Hello")
}

DecodingError

source

pub enum DecodingError {
  Malformed(Bytes)
  Truncated(Bytes)
}

impl Show for DecodingError

Encoding

source

pub(all) enum Encoding {
  UTF8
  UTF16LE
  UTF16BE
}

U_REP

source

let U_REP : Char

The Unicode Replacement Character, which is used to replace invalid or unrecognized sequences during lossy decoding. https://unicode.org/charts/nameslist/n_FFF0.html

decode

source

fn decode(bytes : Bytes, encoding~ : Encoding) -> @moonbitlang/core/string.StringView
StringView!DecodingError

decode_lossy

source

fn decode_lossy(input : Bytes, encoding~ : Encoding) -> @moonbitlang/core/string.StringView
StringView

decode_lossy_to

source

fn decode_lossy_to(input : Bytes, output : StringBuilder, encoding~ : Encoding) -> Unit

decode_to

source

fn decode_to(input : Bytes, output : StringBuilder, encoding~ : Encoding) -> Unit!Error

encode

source

fn encode[Encode : Encode](string : Encode, encoding~ : Encoding) -> Bytes

Encode a given string to the specified character encoding and returns the resulting bytes.

Parameters

encoding : The target encoding format.
src: The input string to be encoded.

Returns

A bytes representing the encoded string in the selected format.

Examples

let src = "Hello, World!"
let encoded_bytes = encode(UTF8, src)

encode_to

source

fn encode_to[Encode : Encode](src : Encode, buffer : @moonbitlang/core/buffer.T
T, encoding~ : Encoding) -> Unit

Encodes a string into the specified character encoding and writes the result directly into a buffer.

Parameters:

string : The input string to be encoded.
buffer : The buffer where the encoded bytes will be written to.
encoding : The target encoding format. Defaults to UTF8 if not specified.

Example:

test "encode_to" {
  let buf = @buffer.new()
  let text = "Hello, world"
  @encoding.encode_to(text, buf, encoding=UTF16)
  inspect!(buf.to_string(), content="Hello, world")
}

Source Files

Site source Report issue Packages Build queue Skills Statistics

encoding

#tonyfettes/encoding

#Moonbit/Core Encoding

#Overview

#Supported Encoding

#Usage

#Decoding

#Encoding

#Encode

#Decoder

#Decoder::decode

Parameters

Returns

Errors

Examples

#Decoder::decode_to

#DecodingError

#Encoding

#U_REP

#decode

#decode_lossy

#decode_lossy_to

#decode_to

#encode

Parameters

Returns

Examples

#encode_to

Source Files

Encode

Decoder

Decoder::decode

Decoder::decode_to

DecodingError

Encoding

U_REP

decode

decode_lossy

decode_lossy_to

decode_to

encode

encode_to