mooncakes.io

marianoguerra/pure-py/basic does not have a README file

Pos

pub(all) struct Pos {
  line : Int
  col : Int
  offset : Int
} derive(Compare, Eq, @moonbitlang/core/debug.Debug
Debug)

A position in a Python source file.

Three numbers in three units, and keeping them apart is the first thing this port has to get right, because Python itself does not:

line is 1-based, as everything that reports a Python position is.
col counts CODE POINTS from the start of the line, 0-based. This is what the tokenize module reports and what a person means by a column. CPython's ast nodes carry col_offset, which counts UTF-8 BYTES; the reference checker prints that number, so a message in its format is rendered through Source::byte_col and nothing else converts.
offset counts UTF-16 code units from the start of the source, 0-based. It exists to slice a MoonBit String and to hand a span to error-report, and it is never reported.

Pos::to_display

source

fn Pos::to_display(self : Pos) -> String

line:col, in the format the reference's messages use.

Source

source

pub(all) struct Source {
  name : String
  text : String
  chars : Array[Char]
  offsets : Array[Int]
  line_starts : Array[Int]
}

A source file: its name, its text, and the three indexes everything else needs to talk about a position in it.

The text is Python's, after what Python calls universal newlines: \r\n and a lone \r become \n. That is done here, once, so that no lexer rule has to think about it -- and it is done to the SOURCE only. A string literal that spells \r still means a carriage return.

A leading byte-order mark is dropped, as CPython's tokenizer drops it.

Source::byte_col

source

fn Source::byte_col(self : Source, pos : Pos) -> Int

The column CPython's ast would report for this position: UTF-8 BYTES from the start of the line, not code points.

The reference checker prints node.col_offset, so every message in its format goes through here. Nothing else in this port converts.

Source::index_of_offset

source

fn Source::index_of_offset(self : Source, offset : Int) -> Int

The code-point index of a UTF-16 offset: the inverse of offset_of.

A Pos carries both, and everything that walks the source walks code points, so a caller handed a Pos from somewhere else needs the way back. A binary search rather than a scan, because offsets is sorted by construction -- it is a running total.

Source::length

source

fn Source::length(self : Source) -> Int

How many code points the source has. One past the last is where the tokenizer's end marker sits.

Source::line_count

source

fn Source::line_count(self : Source) -> Int

Source::line_text

source

fn Source::line_text(self : Source, line : Int) -> String

The text of a 1-based line, without its newline. Out of range gives "".

Source::new

source

fn Source::new(text : String, name? : String) -> Source

Source::offset_of

source

fn Source::offset_of(self : Source, index : Int) -> Int

The UTF-16 offset of a code-point index.

Source::slice

source

fn Source::slice(self : Source, span : Span) -> String

The text a span covers.

Span

source

pub(all) struct Span {
  start : Pos
  end : Pos
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

A half-open range of a source file.

Span::at

source

fn Span::at(p : Pos) -> Span

The empty span at p, for something with no extent -- an unexpected end of input, or a token the parser wanted and did not get.

Span::merge

source

fn Span::merge(self : Span, other : Span) -> Span

The smallest span covering both.

Span::to_display

source

fn Span::to_display(self : Span) -> String

l:c-l:c, for a token dump.

Span::units

source

fn Span::units(self : Span) -> Int

Length in UTF-16 code units -- the unit error-report spans are in.

hex4

source

fn hex4(n : Int) -> String

Four lowercase hexadecimal digits, for a JSON escape or a code point.

is_printable

source

fn is_printable(c : Char) -> Bool

Python's str.isprintable, exactly below U+0100 and optimistically above.

lexical_compare

source

fn lexical_compare(a : String, b : String) -> Int

Python's ordering of strings, which is not MoonBit's.

Python compares strings by CODE POINT, so "ab" < "b". MoonBit's default String::compare compares by length first, so "b" < "ab". The difference is visible wherever the reference reaches for min or sorted to decide which of several names a message reports -- the captured-then-reassigned name, the clashing inherited field, the shadowed submodule, the list of constructor keywords -- so every such place goes through here.

min_name

source

fn min_name(names : Array[String]) -> String?

Python's min of a collection of names: the one a message reports when several qualify. None for an empty collection, which no caller passes.

nowhere

source

let nowhere : Span

A span standing for "nowhere": what the builders of ast give a node that a code generator made up rather than read.

origin

source

let origin : Pos

The position before the first character.

py_bytes_repr

source

fn py_bytes_repr(b : Bytes) -> String

Python's repr of a bytes object.

The same shape as a string's, with two differences: the b prefix, and "printable" meaning printable ASCII -- every byte outside 0x20..0x7E is \xNN, whatever a code point of that value would be.

py_float_layout

source

fn py_float_layout(d : Double, dot_zero~ : Bool) -> String

The same layout without the forced .0, which is how CPython writes the imaginary part of a complex: repr(4j) is 4j and not 4.0j. The flag is CPython's own Py_DTSF_ADD_DOT_0, set for a float and clear for a complex.

py_float_repr

source

fn py_float_repr(d : Double) -> String

Python's repr of a float.

MoonBit's Double::to_string already produces the shortest digits that round-trip, and agrees with CPython on the digits for every value tested. What it does not agree on is the LAYOUT, in six ways: 1 for 1.0, 10000000000000000 for 1e16, 0 for -0.0, Infinity, NaN, and 2.5e-7 where Python writes 2.5e-07. So the digits are taken from MoonBit and laid out again here, by CPython's rules (Python/pystrtod.c, format_float_short with mode r):

nan, inf, -inf, and -0.0 with its sign from the bit pattern.
With the value written as 0.D × 10^decpt: exponent notation when decpt <= -4 || decpt > 16, and positional otherwise. That is why 1e15 prints in full and 1e16 does not.
Positional always carries a point and at least one digit after it.
The exponent always carries a sign and at least two digits.

The same layout is what str() produces: Python 3 has one float format.

py_repr

source

fn py_repr(s : String) -> String

Python's repr of a string.

It lives here, below everything, because three consumers need the same escaping: the token dump, the AST dump and the runtime's repr of a value. One implementation, one set of rules:

The quote is ', unless the string contains a ' and no ".
\\, \n, \r, \t and the chosen quote are escaped by name; the OTHER quote never is.
A code point Python calls unprintable becomes \xNN, \uNNNN or \UNNNNNNNN, by size.

Printability is exact below U+0100 -- the control ranges, U+007F, the C1 block, U+00A0 and U+00AD -- and everything above is treated as printable. That is decision 8 of the plan: the exact answer is the Unicode general category tables, the conformance suite is ASCII, and the difference shows only on a format character or an unassigned code point in a string that a program prints back out.

sort_names

source

fn sort_names(names : Array[String]) -> Unit

Sort in place, the way Python's sorted would.

Source Files

Site source Report issue Packages Build queue Skills Statistics

#Pos

#Pos::to_display

#Source

#Source::byte_col

#Source::index_of_offset

#Source::length

#Source::line_count

#Source::line_text

#Source::new

#Source::offset_of

#Source::slice

#Span

#Span::at

#Span::merge

#Span::to_display

#Span::units

#hex4

#is_printable

#lexical_compare

#min_name

#nowhere

#origin

#py_bytes_repr

#py_float_layout

#py_float_repr

#py_repr

#sort_names

Source Files

Pos

Pos::to_display

Source

Source::byte_col

Source::index_of_offset

Source::length

Source::line_count

Source::line_text

Source::new

Source::offset_of

Source::slice

Span

Span::at

Span::merge

Span::to_display

Span::units

hex4

is_printable

lexical_compare

min_name

nowhere

origin

py_bytes_repr

py_float_layout

py_float_repr

py_repr

sort_names