#TOML Parser for MoonBit

#Highlights

• ✅ TOML 1.0 compliant - Matches the official specification
• ✅ All core data types - Strings, integers, floats, booleans, arrays, tables, and datetimes
• ✅ Validation helpers - Enforces array homogeneity and validates nested structures
• ✅ Round-trip formatting - Convert TomlValue back to TOML via to_string()
• ✅ Helpful errors - Line/column locations in error messages
• ✅ Thorough tests - 8000+ lines plus the official TOML test suite

#Supported TOML Features

#Data Types

• Strings: basic, literal, and multiline strings
• Integers: 42, -17
• Floats: 3.14, -0.01, inf, -inf, nan
• Booleans: true, false
• Arrays: [1, 2, 3], ["a", "b", "c"] (homogeneous types only)
• Inline Tables: {name = "John", age = 30}
• Date & Time Types:
  • Offset Date-Time: 1979-05-27T07:32:00Z, 1979-05-27T07:32:00+01:00
  • Local Date-Time: 1979-05-27T07:32:00
  • Local Date: 1979-05-27
  • Local Time: 07:32:00

#Tables and Structure

• Table headers: [section], [section.subsection]
• Array of tables: [[products]] for repeated table structures
• Nested keys: dotted key notation (a.b.c = value)

#Basic Syntax

• Key-value pairs: key = value
• Comments: # This is a comment (line comments)
• Multi-line strings with proper whitespace handling

#Escape Sequences

• Basic Escapes: \n (newline), \t (tab), \r (carriage return)
• Quote Escapes: \" (double quote), \' (single quote), \\ (backslash)
• Control Characters: \b (backspace), \f (form feed)
• Unicode Escapes: \uXXXX (4-digit hex), \UXXXXXXXX (8-digit hex)
• Line Continuation: \ at end of line in multiline strings

#Installation

moon add bobzhang/toml

{
  "deps": {
    "bobzhang/toml": "^0.1.7"
  }
}

#Quick Start

///|
test {
  // Quick start example - parsing and accessing values
  let config = @toml.parse(
    (
      #|title = "My App"
      #|version = "1.0.0"
      #|[database]
      #|host = "localhost"
      #|port = 5432
      #|
    ),
  )

  // Access values using nested pattern matching with literal values
  guard config
    is TomlTable(
      {
        "title": TomlString("My App"),
        "version": TomlString("1.0.0"),
        "database": TomlTable(
          { "host": TomlString("localhost"), "port": TomlInteger(5432L), .. }
        ),
        ..
      }
    ) else {
    fail("Expected config with title, version, and database")
  }
}

#API Reference

#Core Functions

• parse(input : String) -> TomlValue raise - Parse TOML string and return structured data. Raises errors with location information on invalid syntax.

• TomlValue::validate(self : TomlValue) -> Bool - Validate parsed TOML structure for correctness (array homogeneity, etc.)

• TomlValue::to_string(self : TomlValue) -> String - Convert TomlValue back to a human-readable string representation

#TomlValue - The Core Data Type

///|
pub(all) enum TomlValue {
  TomlString(String)
  TomlInteger(Int64)
  TomlFloat(Double)
  TomlBoolean(Bool)
  TomlArray(Array[TomlValue])
  TomlTable(Map[String, TomlValue])
  TomlDateTime(TomlDateTime)
} derive(Eq, ToJson)

///|
test {
  // Constructing TomlValue programmatically
  let table : Map[String, @toml.TomlValue] = {
    "name": @toml.TomlString("Alice"),
    "age": @toml.TomlInteger(30L),
    "active": @toml.TomlBoolean(true),
    "scores": @toml.TomlArray([
      @toml.TomlInteger(95L),
      @toml.TomlInteger(87L),
      @toml.TomlInteger(92L),
    ]),
  }
  let value = @toml.TomlTable(table)

  // Validate and convert back to TOML string
  assert_true(value.validate())
  inspect(
    value.to_string(),
    content=(
      #|name = "Alice"
      #|age = 30
      #|active = true
      #|
      #|scores = [95, 87, 92]
      #|
    ),
  )
}

#Variant Summary

#DateTime Variants

• OffsetDateTime - With timezone (e.g., 1979-05-27T07:32:00Z)
• LocalDateTime - Without timezone (e.g., 1979-05-27T07:32:00)
• LocalDate - Date only (e.g., 1979-05-27)
• LocalTime - Time only (e.g., 07:32:00)

#Error Handling

///|
test {
  // Error handling with try-catch
  let invalid_toml = "invalid = [unclosed"
  let config = @toml.parse(invalid_toml) catch {
    _ => @toml.TomlTable(Map::new()) // Return default value on error
  }
  guard config is TomlTable(table) else { fail("Expected table") }
  assert_eq(table.length(), 0) // Empty table from error handler

  // Error handling with try? - converts to Result type
  let result = try! @toml.parse("key = \"value\"")
  guard result is Ok(TomlTable({ "key": _, .. })) else {
    fail("Should have parsed successfully with key")
  }

  // Parsing error example
  let bad_result = try! @toml.parse("bad syntax here")
  assert_true(bad_result is Err(_))
}

#Behavior Notes

• Validation: TomlValue::validate() enforces array homogeneity and checks nested values
• Round-trip formatting: TomlValue::to_string() renders normalized TOML
• Special Float Values: Support for inf, -inf, nan per TOML spec
• Comments: Full support for # comment syntax (line comments only)
• Unicode Keys: International character support in keys (e.g., "café" = "coffee")
• Error Location Tracking: Precise line/column error reporting for debugging

#Examples

#Basic Key-Value Pairs

///|
test {
  // basic key-value pairs example
  let toml =
    #|title = "My Application"
    #|version = "1.0.0"
    #|debug = true
    #|max_connections = 100
    #|
  @json.inspect(@toml.parse(toml), content=[
    "TomlTable",
    {
      "title": ["TomlString", "My Application"],
      "version": ["TomlString", "1.0.0"],
      "debug": ["TomlBoolean", true],
      "max_connections": ["TomlInteger", "100"],
    },
  ])
}

#Arrays with Validation

///|
test {
  // arrays with validation example
  let toml_arrays =
    #|numbers = [1, 2, 3, 4, 5]
    #|strings = ["red", "green", "blue"]
    #|booleans = [true, false, true]
    #|
  @json.inspect(@toml.parse(toml_arrays), content=[
    "TomlTable",
    {
      "numbers": [
        "TomlArray",
        [
          ["TomlInteger", "1"],
          ["TomlInteger", "2"],
          ["TomlInteger", "3"],
          ["TomlInteger", "4"],
          ["TomlInteger", "5"],
        ],
      ],
      "strings": [
        "TomlArray",
        [["TomlString", "red"], ["TomlString", "green"], ["TomlString", "blue"]],
      ],
      "booleans": [
        "TomlArray",
        [["TomlBoolean", true], ["TomlBoolean", false], ["TomlBoolean", true]],
      ],
    },
  ])
}

#Table Headers and Array of Tables

///|
test {
  // table headers and array of tables example
  let toml_tables =
    #|title = "Configuration Example"
    #|
    #|[database]
    #|server = "192.168.1.1"
    #|port = 5432
    #|
    #|[[products]]
    #|name = "Hammer"
    #|sku = 738594937
    #|
    #|[[products]]
    #|name = "Nail"
    #|sku = 284758393
    #|
  @json.inspect(@toml.parse(toml_tables), content=[
    "TomlTable",
    {
      "title": ["TomlString", "Configuration Example"],
      "database": [
        "TomlTable",
        {
          "server": ["TomlString", "192.168.1.1"],
          "port": ["TomlInteger", "5432"],
        },
      ],
      "products": [
        "TomlArray",
        [
          [
            "TomlTable",
            {
              "name": ["TomlString", "Hammer"],
              "sku": ["TomlInteger", "738594937"],
            },
          ],
          [
            "TomlTable",
            {
              "name": ["TomlString", "Nail"],
              "sku": ["TomlInteger", "284758393"],
            },
          ],
        ],
      ],
    },
  ])
}

#DateTime Support

///|
test {
  // datetime support example
  let toml_datetime =
    #|created_at = 2023-01-01T00:00:00Z
    #|updated_at = 2023-01-02T12:30:45
    #|birth_date = 1990-05-15
    #|meeting_time = 14:30:00
    #|
  @json.inspect(@toml.parse(toml_datetime), content=[
    "TomlTable",
    {
      "created_at": ["TomlDateTime", ["OffsetDateTime", "2023-01-01T00:00:00Z"]],
      "updated_at": ["TomlDateTime", ["LocalDateTime", "2023-01-02T12:30:45"]],
      "birth_date": ["TomlDateTime", ["LocalDate", "1990-05-15"]],
      "meeting_time": ["TomlDateTime", ["LocalTime", "14:30:00"]],
    },
  ])
}

#Inline Tables

///|
test {
  // inline tables example
  let toml_inline =
    #|database = {server = "localhost", port = 5432}
    #|cache = {enabled = true, ttl = 300}
    #|
  @json.inspect(@toml.parse(toml_inline), content=[
    "TomlTable",
    {
      "database": [
        "TomlTable",
        {
          "server": ["TomlString", "localhost"],
          "port": ["TomlInteger", "5432"],
        },
      ],
      "cache": [
        "TomlTable",
        { "enabled": ["TomlBoolean", true], "ttl": ["TomlInteger", "300"] },
      ],
    },
  ])
}

#Complex Configuration

///|
test {
  // complex configuration example
  let config =
    #|service_name = "user-service"
    #|version = "2.1.0"
    #|deployed_at = 2023-06-15T14:30:00+00:00
    #|
    #|# Special float values
    #|timeout = 30.0
    #|max_retry = inf
    #|error_rate = nan
    #|
    #|http = {port = 8080, host = "0.0.0.0", timeout = 30.0}
    #|database = {url = "postgresql://localhost:5432/users", max_connections = 20}
    #|
    #|maintenance_schedule = [
    #|  2023-06-20T02:00:00,
    #|  2023-07-20T02:00:00,
    #|  2023-08-20T02:00:00
    #|]
    #|
  let result = @toml.parse(config)
  assert_true(result.validate())
  // Verify the structure contains expected keys
  guard result
    is TomlTable(
      {
        "service_name": _,
        "http": TomlTable({ "port": TomlInteger(_), .. }),
        "database": _,
        "maintenance_schedule": _,
        ..
      }
    ) else {
    fail("Expected table")
  }
}

#Special Values and Advanced Features

///|
test {
  // special values and advanced features example
  let toml_advanced =
    #|# Configuration with comments
    #|app_name = "TOML Demo"
    #|version = "1.0.0"
    #|
    #|# Special float values
    #|max_timeout = inf
    #|min_timeout = -inf
    #|error_rate = nan
    #|
    #|# Unicode keys and values
    #|"café" = "☕ Coffee shop"
    #|"数量" = 42
    #|
    #|# Complex nested structure
    #|[server.database]
    #|host = "localhost"
    #|port = 5432
    #|
    #|[[server.replicas]]
    #|name = "primary"
    #|weight = 1.0
    #|
    #|[[server.replicas]]
    #|name = "secondary"
    #|weight = 0.5
    #|
  @json.inspect(@toml.parse(toml_advanced), content=[
    "TomlTable",
    {
      "app_name": ["TomlString", "TOML Demo"],
      "version": ["TomlString", "1.0.0"],
      "max_timeout": ["TomlFloat", @double.infinity],
      "min_timeout": ["TomlFloat", @double.neg_infinity],
      "error_rate": ["TomlFloat", @double.not_a_number],
      "café": ["TomlString", "☕ Coffee shop"],
      "数量": ["TomlInteger", "42"],
      "server": [
        "TomlTable",
        {
          "database": [
            "TomlTable",
            {
              "host": ["TomlString", "localhost"],
              "port": ["TomlInteger", "5432"],
            },
          ],
          "replicas": [
            "TomlArray",
            [
              [
                "TomlTable",
                {
                  "name": ["TomlString", "primary"],
                  "weight": ["TomlFloat", 1],
                },
              ],
              [
                "TomlTable",
                {
                  "name": ["TomlString", "secondary"],
                  "weight": ["TomlFloat", 0.5],
                },
              ],
            ],
          ],
        },
      ],
    },
  ])
}

#Project Structure

toml-parser/
├── toml.mbt                              # Core TOML types and API
├── parser.mbt                            # TOML parsing logic
├── moon.mod.json                         # Project configuration
├── moon.pkg.json                         # Package configuration
├── LICENSE                               # Apache-2.0 license
├── README.mbt.md                         # Documentation with executable examples
├── todo.md                               # Development roadmap
│
├── internal/                             # Internal modules
│   └── tokenize/                         # Tokenization and lexical analysis
│       ├── tokenize.mbt                  # Main tokenizer implementation
│       ├── token.mbt                     # Token definitions
│       ├── lexer_test.mbt                # Lexer unit tests
│       ├── lexer_bug_test.mbt            # Lexer regression tests
│       └── moon.pkg.json                 # Tokenizer package config
│
├── cmd/                                  # Command-line tools and demos
│   └── main/                             # Demo application
│       ├── main.mbt                      # Interactive TOML parser demo
│       ├── main_test.mbt                 # Demo tests
│       └── moon.pkg.json                 # Main package config
└── target/                               # Build artifacts (generated)
    ├── wasm-gc/                          # WebAssembly build outputs
    └── packages.json                     # Package dependency info

#Test Suite

• toml_test.mbt - Basic TOML functionality tests
• parser_test.mbt - Parser-specific tests
• datetime_test.mbt - DateTime parsing tests
• datetime_extended_test.mbt - Extended datetime scenarios
• comprehensive_test.mbt - Complex real-world scenarios
• coverage_improvement_test.mbt - Coverage enhancement tests
• coverage_improvement_comprehensive_test.mbt - Advanced coverage tests
• official_toml_test_suite_test.mbt - Official TOML spec compliance
• additional_official_tests_test.mbt - Extended official tests
• special_value_validation_test.mbt - Special values (inf, nan) tests

#Key Components

• Core Parser (toml.mbt, parser.mbt) - Main TOML parsing implementation with full TOML 1.0 specification support, recursive descent parser with error recovery
• Tokenizer (internal/tokenize/) - Lexical analysis and tokenization engine supporting all TOML tokens including special float values (inf, nan)
• Utilities (toml_utils.mbt) - Helper functions for table creation, dotted key handling, and nested structure management
• Demo Application (cmd/main/) - Interactive command-line examples demonstrating parser capabilities
• Comprehensive Test Suite (8000+ lines) - Extensive coverage including:
  • Unit tests for all data types and features
  • Official TOML specification compliance tests
  • Edge case and boundary value tests
  • Real-world configuration file examples
• Executable Documentation (README.mbt.md) - This file contains runnable MoonBit code examples that are automatically tested

#Usage Tips

#Working with Tables

///|
test {
  // Working with nested tables
  let parsed_toml = @toml.parse(
    (
      #|[database]
      #|host = "localhost"
      #|port = 5432
      #|enabled = true
      #|
    ),
  )

  // Use guard with nested pattern matching and literal values
  guard parsed_toml
    is TomlTable(
      {
        "database": TomlTable(
          {
            "host": TomlString("localhost"),
            "port": TomlInteger(5432L),
            "enabled": TomlBoolean(true),
            ..
          }
        ),
        ..
      }
    ) else {
    fail(
      "Expected database configuration with host=localhost, port=5432, enabled=true",
    )
  }
}

#Validating Arrays

///|
test {
  // Valid: all integers
  let valid = @toml.parse("numbers = [1, 2, 3]")
  assert_true(valid.validate())

  // Mixed types are allowed during parsing but fail validation
  let mixed = @toml.parse("mixed = [1, \"two\", 3.0]")
  assert_false(mixed.validate()) // Validation catches the type mismatch
}

#Working with DateTime Values

///|
test {
  // Working with all 4 datetime types
  let parsed_toml = @toml.parse(
    (
      #|offset_dt = 2023-01-15T10:30:00Z
      #|local_dt = 2023-01-15T10:30:00
      #|local_date = 2023-01-15
      #|local_time = 10:30:00
      #|
    ),
  )

  // Verify all datetime types are parsed correctly
  @json.inspect(parsed_toml, content=[
    "TomlTable",
    {
      "offset_dt": ["TomlDateTime", ["OffsetDateTime", "2023-01-15T10:30:00Z"]],
      "local_dt": ["TomlDateTime", ["LocalDateTime", "2023-01-15T10:30:00"]],
      "local_date": ["TomlDateTime", ["LocalDate", "2023-01-15"]],
      "local_time": ["TomlDateTime", ["LocalTime", "10:30:00"]],
    },
  ])
}

#Development Status

#Recent Improvements

• ✅ Enhanced error location tracking in lexer and parser
• ✅ Comprehensive test suite expansion (8000+ lines)
• ✅ Official TOML test suite integration
• ✅ Unicode key support and complex escape sequences
• ✅ Special float values support (inf, -inf, nan)
• ✅ Advanced string escaping and literal strings
• ✅ Robust table and array of tables implementation

#Running Tests

moon test

• Basic TOML data types
• DateTime functionality (all 4 types)
• Array homogeneity validation
• TOML 1.0 specification compliance
• Edge cases and real-world scenarios
• Error handling with location tracking
• Official TOML test suite integration
• Unicode key support and escape sequences
• Complex nested structures

#Running the Demo

moon run cmd/main

#Building

moon build

#Common Pitfalls and Best Practices

#✅ DO:

1. Always validate after parsing:

   let toml = @toml.parse(input)
   assert_true(toml.validate())  // Ensures data integrity

2. Use pattern matching for type safety:

   match config {
     TomlTable(table) => {
       match table.get("port") {
         Some(TomlInteger(port)) => use_port(port)
         _ => use_default_port()
       }
     }
     _ => ()
   }

3. Handle errors appropriately:

   ///|
   let config = @toml.parse(file_content) catch {
     err => {
       logger.error("Config parse failed: \{err}")
       load_default_config()
     }
   }

#❌ DON'T:

1. Don't mix types in arrays - TOML requires homogeneous arrays:

   # ❌ Invalid
   mixed = [1, "two", 3.0]
   
   # ✅ Valid
   integers = [1, 2, 3]
   strings = ["one", "two", "three"]

2. Don't redefine keys - Each key must be unique within its scope:

   # ❌ Invalid
   name = "foo"
   name = "bar"  # Error: key redefined
   
   # ✅ Valid
   first_name = "foo"
   last_name = "bar"

3. Don't mix inline tables with later updates:

   # ❌ Invalid - inline tables cannot be extended
   server = { host = "localhost", port = 8080 }
   [server]
   tls = true
   
   # ✅ Valid - use a standard table
   [server]
   host = "localhost"
   port = 8080
   tls = true

#TOML 1.0 Specification Compliance

• ✅ All basic data types (strings, integers, floats, booleans)
• ✅ Complete datetime support (4 types: offset datetime, local datetime, local date, local time)
• ✅ Array homogeneity validation (arrays must contain single type)
• ✅ Inline tables with proper nesting
• ✅ RFC 3339 datetime format compliance
• ✅ Recursive validation for complex structures
• ✅ Enhanced error reporting with line/column location tracking
• ✅ Unicode key support for international characters
• ✅ Comprehensive test coverage with official TOML test suite integration

#Roadmap

• ✅ Completed
• ✅ Completed
• ✅ Completed
• ✅ Completed
• ✅ Completed
• ✅ Completed
• ✅ Completed
• ✅ Completed
• ✅ Completed

#Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass with moon test
5. Submit a pull request

#License

#References

• TOML Specification
• RFC 3339 (Date and Time on the Internet)
• MoonBit Language Guide