🥮 mooncakes.io

encoding

Encoding package extracted from moonbitlang/x

moon add tonyfettes/encoding@0.3.9
Download zip
Version
0.3.9
License
Apache-2.0
Repository
Last updated
9 months ago
Downloads
4K
README

#tonyfettes/encoding

This package is extracting from moonbitlang/x.

#Moonbit/Core Encoding

#Overview

The @encoding package provides an implementation for encoding and decoding strings using various character encodings (e.g., UTF-8).

It supports both streaming and non-streaming (bulk) operations, making it flexible for different use-cases.

#Supported Encoding

  • UTF8
  • UTF16 // alias for UTF16LE
  • UTF16LE
  • UTF16BE

#Usage

#Decoding

Decode a UTF-8 byte stream:

test {
  // Initialize a streaming UTF-8 decoder
  let decoder = @encoding.decoder(UTF8)

  // Consume byte chunks
  let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
  inspect(decoder.consume(inputs[0]), content="abc")
  inspect(decoder.consume(inputs[1]), content="")
  inspect(decoder.consume(inputs[2]), content="🐰")

  // Finish decoding
  assert_true(decoder.finish().is_empty())
}

#Encoding

Encode a string to UTF-8 bytes:

test {
  // Encode a string to UTF-8
  let src = "你好👀"
  let bytes = @encoding.encode(src, encoding=UTF8)
  inspect(
    bytes,
    content="b\"\\xe4\\xbd\\xa0\\xe5\\xa5\\xbd\\xf0\\x9f\\x91\\x80\"",
  )
}

#
Encode

source
pub trait Encode {
  size_hint(Self, encoding~ : Encoding) -> Int
  encode(Self, encoding~ : Encoding) -> Bytes
  encode_to(Self, 
Buffer, encoding~ : Encoding) -> Unit
}

impl Encode for Char
impl Encode for String

#
DecodingError

source
pub suberror DecodingError {
  Malformed(Bytes)
  Truncated(Bytes)
}

#
Decoder

source
type Decoder

#
Decoder::consume

source
#as_free_fn
fn Decoder::consume(self : Decoder, input : BytesView) -> String raise

Consume the given byte sequence using the specified Decoder and return the resulting string incrementally.

This function calls decode! with the stream parameter set to true, indicating that more bytes will follow for decoding.

Parameters

  • self: The Decoder instance used to consume the byte sequence.
  • input: The byte sequence to be consumed and decoded incrementally.

Returns

A String representing the partially decoded content from the input byte sequence, as more bytes are expected.

Errors

MalformedError: when the byte sequence is not properly formatted according to the specified encoding. TruncatedError: when the byte sequence ends prematurely, implying that more data is expected for complete decoding.

#
Decoder::decode

source
fn Decoder::decode(self : Decoder, input : BytesView, stream? : Bool) -> String raise DecodingError

Decode the given byte sequence using the specified Decoder and return the resulting string.

This function can work in streaming mode where bytes are consumed incrementally. When stream is false, it indicates the end of the input and triggers the final decoding step.

Parameters

  • self: The Decoder instance used to decode the byte sequence.
  • input: The byte sequence to be decoded.
  • stream~: A boolean indicating whether more bytes will be supplied for decoding. It defaults to false.

Returns

A String representing the decoded content from the input byte sequence.

Errors

MalformedError: when the byte sequence is not properly formatted according to the specified encoding. TruncatedError: when the byte sequence ends prematurely, implying that more data is expected for complete decoding.

Examples

let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
let decoder = @encoding.decoder(UTF8)
inspect(decoder.decode(inputs[0], stream=true), content="abc")
inspect(decoder.decode(inputs[1], stream=true), content="")
inspect(decoder.decode(inputs[2], stream=false), content="🐰")

#
Decoder::decode_lossy

source
fn Decoder::decode_lossy(self : Decoder, input : BytesView, stream? : Bool) -> String

Decode the given byte sequence using the specified Decoder and return the resulting string, replacing any invalid sequences with the Unicode Replacement Character (U+FFFD).

This function can work in streaming mode where bytes are consumed incrementally. When stream is false, it indicates the end of the input and triggers the final decoding step.

Parameters

  • self: The Decoder instance used to decode the byte sequence.
  • input: The byte sequence to be decoded.
  • stream~: A boolean indicating whether more bytes will be supplied for decoding. It defaults to false.

Returns

A String representing the decoded content from the input byte sequence, with any invalid sequences replaced by the Unicode Replacement Character (U+FFFD).

#
Decoder::decode_lossy_to

source
fn Decoder::decode_lossy_to(self : Decoder, input : BytesView, output : StringBuilder, stream? : Bool) -> Unit

#
Decoder::decode_to

source
fn Decoder::decode_to(self : Decoder, input : BytesView, output : StringBuilder, stream? : Bool) -> Unit raise DecodingError

Decodes the given byte sequence using the specified decoder and writes the result directly to a StringBuilder. Similar to decode!, but writes the result to an existing StringBuilder instead of creating a new String.

Parameters:

  • decoder : The decoder instance used to decode the byte sequence.
  • input : The byte sequence to be decoded.
  • output : The StringBuilder where the decoded content will be written to.

Throws a MalformedError when the byte sequence is not properly formatted according to the specified encoding.

Example:

let decoder = decoder(UTF8)
let buf = StringBuilder::new()
decoder.decode_to(b"Hello", buf)
inspect(buf.to_string(), content="Hello")

#
Decoder::finish

source
#as_free_fn
fn Decoder::finish(self : Decoder) -> String raise

Finalize the decoding process and return the remaining decoded string.

This function calls decode! with the stream parameter set to false, indicating that no more bytes will be supplied and triggering the final decoding step to produce the remaining output.

Parameters

  • self: The Decoder instance used to finalize the decoding process.

Returns

A String representing the final part of the decoded content, after all byte sequences have been processed.

Errors

MalformedError: This error is raised if the remaining byte sequence is not properly formatted according to the specified encoding. TruncatedError: This error is raised if the remaining byte sequence ends prematurely, implying that more data was expected for complete decoding.

#
Decoder::lossy_consume

source
#as_free_fn
fn Decoder::lossy_consume(self : Decoder, input : BytesView) -> String

Consume the given byte sequence using the specified Decoder and return the resulting string incrementally, replacing any invalid sequences with the Unicode Replacement Character (U+FFFD).

This function calls decode_lossy with the stream parameter set to true, indicating that more bytes will follow for decoding.

Parameters

  • self: The Decoder instance used to consume and decode the byte sequence.
  • input: The byte sequence to be consumed and decoded incrementally.

Returns

A String representing the partially decoded content from the input byte sequence, with any invalid sequences replaced by the Unicode Replacement Character (U+FFFD), as more bytes are expected.

#
Decoder::lossy_finish

source
#as_free_fn
fn Decoder::lossy_finish(self : Decoder) -> String

Finalize the lossy decoding process and return the remaining decoded string, replacing any invalid sequences with the Unicode Replacement Character (U+FFFD).

This function calls decode_lossy with the stream parameter set to false, indicating that no more bytes will be supplied and triggering the final decoding step to produce the remaining output.

Parameters

  • self: The Decoder instance used to finalize the lossy decoding process.

Returns

A String representing the final part of the decoded content, with any invalid sequences replaced by the Unicode Replacement Character (U+FFFD), after all byte sequences have been processed.

#
Encoding

source
pub(all) enum Encoding {
  UTF8
  UTF16LE
  UTF16BE
}

#
U_REP

source
let U_REP : Char
The Unicode Replacement Character, which is used to replace invalid or unrecognized sequences during lossy decoding. https://unicode.org/charts/nameslist/n_FFF0.html

#
decode

source
fn decode(bytes : BytesView, encoding~ : Encoding) -> String raise DecodingError

#
decode_lossy

source
fn decode_lossy(input : BytesView, encoding~ : Encoding) -> String

#
decode_lossy_to

source
fn decode_lossy_to(input : BytesView, output : StringBuilder, encoding~ : Encoding) -> Unit

#
decode_to

source
fn decode_to(input : BytesView, output : StringBuilder, encoding~ : Encoding) -> Unit raise

#
decoder

source
fn decoder(encoding : Encoding) -> Decoder

Create and return a Decoder for the specified character encoding.

The Decoder consumes byte sequences and decodes them into the original string format.

Parameters

  • encoding: The character encoding format to be used for decoding the input byte sequences.

Returns

A Decoder instance that can be used to decode byte sequences into strings.

Examples

let inputs = [b"abc", b"\xf0", b"\x9f\x90\xb0"] // UTF8(🐰) == <F09F 90B0>
let decoder = decoder(UTF8)
inspect(decoder.consume(inputs[0]), content="abc")
inspect(decoder.consume(inputs[1]), content="")
inspect(decoder.consume(inputs[2]), content="🐰")
assert_true(decoder.finish().is_empty())

#
encode

source
fn[Encode : Encode] encode(string : Encode, encoding~ : Encoding) -> Bytes

Encode a given string to the specified character encoding and returns the resulting bytes.

Parameters

  • encoding : The target encoding format.
  • src: The input string to be encoded.

Returns

A bytes representing the encoded string in the selected format.

Examples

let src = "Hello, World!"
@json.inspect(encode(encoding=UTF8, src).to_array(), content=
  [72,101,108,108,111,44,32,87,111,114,108,100,33]
)

#
encode_to

source
fn[Encode : Encode] encode_to(src : Encode, buffer : 
Buffer, encoding~ : Encoding) -> Unit

Encodes a string into the specified character encoding and writes the result directly into a buffer.

Parameters:

  • string : The input string to be encoded.
  • buffer : The buffer where the encoded bytes will be written to.
  • encoding : The target encoding format. Defaults to UTF8 if not specified.

Example:

let buf = @buffer.new()
let text = "Hello, world"
@encoding.encode_to(text, buf, encoding=UTF16LE)
inspect(buf.to_string(), content="Hello, world")

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io