Encoding package extracted from moonbitlang/x

Author

Version

0.3.3

License

Apache-2.0

Repository

github.com/moonbitlang/x

Last updated

last year

Downloads

No dependencies

#tonyfettes/encoding

This package is extracting from moonbitlang/x.

#Moonbit/Core Encoding

#Overview

The @encoding package provides an implementation for encoding and decoding strings using various character encodings (e.g., UTF-8).

It supports both streaming and non-streaming (bulk) operations, making it flexible for different use-cases.

#Supported Encoding

UTF8
UTF16 // alias for UTF16LE
UTF16LE
UTF16BE

#Usage

#Decoding

Decode a UTF-8 byte stream:

// Initialize a streaming UTF-8 decoder
let decoder = @encoding.decoder(UTF8)

// Consume byte chunks
let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
inspect!(decoder.consume!(inputs[0]), content="abc")
inspect!(decoder.consume!(inputs[1]), content="")
inspect!(decoder.consume!(inputs[2]), content="🐰")

// Finish decoding
assert_true!(decoder.finish!().is_empty())

#Encoding

Encode a string to UTF-8 bytes:

// Encode a string to UTF-8
let src = "你好👀"
let bytes = @encoding.encode(UTF8, src)
inspect!(
  bytes,
  content=
    #|b"\xe4\xbd\xa0\xe5\xa5\xbd\xf0\x9f\x91\x80"
  ,
)

Encode a single character to UTF-8 bytes:

inspect!(
  @encoding.to_utf8_bytes('A'),
  content=
    #|b"\x41"
  ,
)

Encode

source

pub trait Encode {
  size_hint(Self, encoding~ : Encoding) -> Int
  encode(Self, encoding~ : Encoding) -> Bytes
  encode_to(Self, @moonbitlang/core/buffer.T
T, encoding~ : Encoding) -> Unit
}

impl Encode for Char

impl Encode for String

impl Encode for StringView

fn encode_to(value : @moonbitlang/core/string.StringView
StringView, buffer : @moonbitlang/core/buffer.T
T, encoding~ : Encoding) -> Unit

source

fn size_hint(value : @moonbitlang/core/string.StringView
StringView, encoding~ : Encoding) -> Int

source

Decoder

source

type Decoder

Decoder::consume

source

fn Decoder::consume(self : Decoder, input : @moonbitlang/core/bytes.View
View) -> String raise Error

Consume the given byte sequence using the specified Decoder and return the resulting string incrementally.

This function calls decode! with the stream parameter set to true, indicating that more bytes will follow for decoding.

Parameters

self: The Decoder instance used to consume the byte sequence.
input: The byte sequence to be consumed and decoded incrementally.

Returns

A String representing the partially decoded content from the input byte sequence, as more bytes are expected.

Errors

MalformedError: when the byte sequence is not properly formatted according to the specified encoding. TruncatedError: when the byte sequence ends prematurely, implying that more data is expected for complete decoding.

Decoder::decode

source

fn Decoder::decode(self : Decoder, input : @moonbitlang/core/bytes.View
View, stream~ : Bool = ..) -> String raise DecodingError

Decode the given byte sequence using the specified Decoder and return the resulting string.

This function can work in streaming mode where bytes are consumed incrementally. When stream is false, it indicates the end of the input and triggers the final decoding step.

Parameters

self: The Decoder instance used to decode the byte sequence.
input: The byte sequence to be decoded.
stream~: A boolean indicating whether more bytes will be supplied for decoding. It defaults to false.

Returns

A String representing the decoded content from the input byte sequence.

Errors

Examples

let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
let decoder = @encoding.decoder(UTF8)
inspect(decoder.decode(inputs[0], stream=true), content="abc")
inspect(decoder.decode(inputs[1], stream=true), content="")
inspect(decoder.decode(inputs[2], stream=false), content="🐰")

Decoder::decode_lossy

source

fn Decoder::decode_lossy(self : Decoder, input : @moonbitlang/core/bytes.View
View, stream~ : Bool = ..) -> String

Decode the given byte sequence using the specified Decoder and return the resulting string, replacing any invalid sequences with the Unicode Replacement Character (U+FFFD).

This function can work in streaming mode where bytes are consumed incrementally. When stream is false, it indicates the end of the input and triggers the final decoding step.

Parameters

self: The Decoder instance used to decode the byte sequence.
input: The byte sequence to be decoded.
stream~: A boolean indicating whether more bytes will be supplied for decoding. It defaults to false.

Returns

A String representing the decoded content from the input byte sequence, with any invalid sequences replaced by the Unicode Replacement Character (U+FFFD).

Decoder::decode_lossy_to

source

fn Decoder::decode_lossy_to(self : Decoder, input : @moonbitlang/core/bytes.View
View, output : StringBuilder, stream~ : Bool = ..) -> Unit

Decoder::decode_to

source

fn Decoder::decode_to(self : Decoder, input : @moonbitlang/core/bytes.View
View, output : StringBuilder, stream~ : Bool = ..) -> Unit raise DecodingError

Decodes the given byte sequence using the specified decoder and writes the result directly to a StringBuilder. Similar to decode!, but writes the result to an existing StringBuilder instead of creating a new String.

Parameters:

decoder : The decoder instance used to decode the byte sequence.
input : The byte sequence to be decoded.
output : The StringBuilder where the decoded content will be written to.

Throws a MalformedError when the byte sequence is not properly formatted according to the specified encoding.

Example:

let decoder = decoder(UTF8)
let buf = StringBuilder::new()
decoder.decode_to(b"Hello", buf)
inspect(buf.to_string(), content="Hello")

Decoder::finish

source

fn Decoder::finish(self : Decoder) -> String raise Error

Finalize the decoding process and return the remaining decoded string.

This function calls decode! with the stream parameter set to false, indicating that no more bytes will be supplied and triggering the final decoding step to produce the remaining output.

Parameters

self: The Decoder instance used to finalize the decoding process.

Returns

A String representing the final part of the decoded content, after all byte sequences have been processed.

Errors

MalformedError: This error is raised if the remaining byte sequence is not properly formatted according to the specified encoding. TruncatedError: This error is raised if the remaining byte sequence ends prematurely, implying that more data was expected for complete decoding.

Decoder::lossy_consume

source

fn Decoder::lossy_consume(self : Decoder, input : @moonbitlang/core/bytes.View
View) -> String

Consume the given byte sequence using the specified Decoder and return the resulting string incrementally, replacing any invalid sequences with the Unicode Replacement Character (U+FFFD).

This function calls decode_lossy with the stream parameter set to true, indicating that more bytes will follow for decoding.

Parameters

self: The Decoder instance used to consume and decode the byte sequence.
input: The byte sequence to be consumed and decoded incrementally.

Returns

A String representing the partially decoded content from the input byte sequence, with any invalid sequences replaced by the Unicode Replacement Character (U+FFFD), as more bytes are expected.

Decoder::lossy_finish

source

fn Decoder::lossy_finish(self : Decoder) -> String

Finalize the lossy decoding process and return the remaining decoded string, replacing any invalid sequences with the Unicode Replacement Character (U+FFFD).

This function calls decode_lossy with the stream parameter set to false, indicating that no more bytes will be supplied and triggering the final decoding step to produce the remaining output.

Parameters

self: The Decoder instance used to finalize the lossy decoding process.

Returns

A String representing the final part of the decoded content, with any invalid sequences replaced by the Unicode Replacement Character (U+FFFD), after all byte sequences have been processed.

DecodingError

source

pub enum DecodingError {
  Malformed(Bytes)
  Truncated(Bytes)
}

impl Show for DecodingError

Encoding

source

pub(all) enum Encoding {
  UTF8
  UTF16LE
  UTF16BE
}

U_REP

source

let U_REP : Char

The Unicode Replacement Character, which is used to replace invalid or unrecognized sequences during lossy decoding. https://unicode.org/charts/nameslist/n_FFF0.html

decode

source

fn decode(bytes : @moonbitlang/core/bytes.View
View, encoding~ : Encoding) -> String raise DecodingError

decode_lossy

source

fn decode_lossy(input : @moonbitlang/core/bytes.View
View, encoding~ : Encoding) -> String

decode_lossy_to

source

fn decode_lossy_to(input : @moonbitlang/core/bytes.View
View, output : StringBuilder, encoding~ : Encoding) -> Unit

decode_to

source

fn decode_to(input : @moonbitlang/core/bytes.View
View, output : StringBuilder, encoding~ : Encoding) -> Unit raise Error

decoder

source

fn decoder(encoding : Encoding) -> Decoder

Create and return a Decoder for the specified character encoding.

The Decoder consumes byte sequences and decodes them into the original string format.

Parameters

encoding: The character encoding format to be used for decoding the input byte sequences.

Returns

A Decoder instance that can be used to decode byte sequences into strings.

Examples

let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
let decoder = decoder(UTF8)
inspect!(decoder.consume!(inputs[0]), content="abc")
inspect!(decoder.consume!(inputs[1]), content="")
inspect!(decoder.consume!(inputs[2]), content="🐰")
assert_true!(decoder.finish!().is_empty())

encode

source

fn[Encode : Encode] encode(string : Encode, encoding~ : Encoding) -> Bytes

Encode a given string to the specified character encoding and returns the resulting bytes.

Parameters

encoding : The target encoding format.
src: The input string to be encoded.

Returns

A bytes representing the encoded string in the selected format.

Examples

let src = "Hello, World!"
inspect(encode(encoding=UTF8, src), content=
  #|b"\x48\x65\x6c\x6c\x6f\x2c\x20\x57\x6f\x72\x6c\x64\x21"
)

encode_to

source

fn[Encode : Encode] encode_to(src : Encode, buffer : @moonbitlang/core/buffer.T
T, encoding~ : Encoding) -> Unit

Encodes a string into the specified character encoding and writes the result directly into a buffer.

Parameters:

string : The input string to be encoded.
buffer : The buffer where the encoded bytes will be written to.
encoding : The target encoding format. Defaults to UTF8 if not specified.

Example:

let buf = @buffer.new()
let text = "Hello, world"
@encoding.encode_to(text, buf, encoding=UTF16LE)
inspect(buf.to_string(), content="Hello, world")

Source Files

Site source Report issue Packages Build queue Skills Statistics

encoding

#tonyfettes/encoding

#Moonbit/Core Encoding

#Overview

#Supported Encoding

#Usage

#Decoding

#Encoding

#Encode

#Decoder

#Decoder::consume

Parameters

Returns

Errors

#Decoder::decode

Parameters

Returns

Errors

Examples

#Decoder::decode_lossy

Parameters

Returns

#Decoder::decode_lossy_to

#Decoder::decode_to

#Decoder::finish

Parameters

Returns

Errors

#Decoder::lossy_consume

Parameters

Returns

#Decoder::lossy_finish

Parameters

Returns

#DecodingError

#Encoding

#U_REP

#decode

#decode_lossy

#decode_lossy_to

#decode_to

#decoder

Parameters

Returns

Examples

#encode

Parameters

Returns

Examples

#encode_to

Source Files

Encode

Decoder

Decoder::consume

Decoder::decode

Decoder::decode_lossy

Decoder::decode_lossy_to

Decoder::decode_to

Decoder::finish

Decoder::lossy_consume

Decoder::lossy_finish

DecodingError

Encoding

U_REP

decode

decode_lossy

decode_lossy_to

decode_to

decoder

encode

encode_to