lexical_edit

    Download zip
    Version
    0.1.1
    License
    Apache-2.0
    Last updated
    16 days ago
    Downloads
    40

    #moonbit-community/lexical_edit

    Replaces the first matching MoonBit token sequence from left to right. Whitespace, line breaks, comments, and automatically inserted semicolons between tokens are ignored; explicit semicolons remain significant.

    ///|
    test "replace across whitespace and comments" {
    let result = @lexical_edit.lexical_edit(
    old="a + b",
    new="sum",
    "let x = a // comment\n + b\n",
    )
    assert_eq(result, "let x = sum\n")
    }

    Leading and trailing whitespace and comments in old do not affect match locations. The replacement range starts at the beginning of the first matched token and ends at the end of the last matched token. Whitespace and comments within the range are removed with the replacement; text outside it is preserved verbatim. new is inserted verbatim, without parsing or indentation adjustment; an empty string deletes the match.

    ///|
    test "delete the first match" {
    assert_eq(
    @lexical_edit.lexical_edit(old=" + b", new="", "a + b + b"),
    "a + b",
    )
    }

    Matching requires identical token kinds and original text, except that trailing lexer-recognized whitespace in attribute tokens is ignored, including Unicode whitespace. These trailing characters remain part of the replacement range and are removed with the attribute. ASCII line endings (LF, CRLF, or CR) after a final attribute are outside its token and are preserved.

    ///|
    test "replace an attribute with different trailing whitespace" {
    assert_eq(
    @lexical_edit.lexical_edit(
    old="#deprecated(\"reason\") ",
    new="#deprecated(\"updated\")",
    "#deprecated(\"reason\")\t\u3000\nfn f {}",
    ),
    "#deprecated(\"updated\")\nfn f {}",
    )
    }

    Literal values are not normalized. Whitespace inside strings, interpolated strings, and the remaining attribute text must match exactly, and these tokens are not searched recursively. Ordinary MoonBit lexical rules apply. Code fragments need not form complete syntax, and wildcards are not supported.

    The function raises LEditError:

    ErrorMeaning
    EmptyOldold contains no meaningful tokens after ignoring whitespace, comments, and automatic semicolons.
    OldLexError(offset~, message~)old contains a lexical error.
    InputLexError(offset~, message~)The entire input contains a lexical error, including errors after the match.
    NotFoundNo contiguous token sequence matches.

    First, old is tokenized and validated, then the entire input is tokenized and validated, and finally a match is sought. Lexical errors use the first diagnostic returned by the lexer, without using recovery tokens or falling back to text matching. offset is a UTF-16 code unit offset relative to the start of the corresponding StringView, and message preserves the lexer's diagnostic text.

    ///|
    test "handle an empty pattern" {
    try @lexical_edit.lexical_edit(old="// comment", new="x", "a") catch {
    error => assert_true(error is @lexical_edit.LEditError::EmptyOld)
    } noraise {
    _ => fail("expected EmptyOld")
    }
    }

    LEditError

    pub suberror LEditError {
    EmptyOld
    OldLexError(offset~ : Int, message~ : String)
    InputLexError(offset~ : Int, message~ : String)
    NotFound
    }

    Errors raised by lexical_edit when validating the pattern or input, or finding a match.

    Validation proceeds in this order: check old for lexical errors, check that old contains meaningful tokens, check the entire input for lexical errors, then search for a match. Even if the beginning of the input matches, a later lexical error still raises InputLexError; no partial replacement is returned.

    Both lexical errors preserve the first diagnostic returned by the lexer. offset is a zero-based UTF-16 code unit offset from the start of the relevant StringView, not a UTF-8 byte count, Unicode character count, or line/column position. For a slice, the offset is relative to the slice rather than the entire underlying string. For example, 😀 occupies two UTF-16 code units. message is the lexer's original diagnostic text.

    test "handle a missing token sequence" {
    try @lexical_edit.lexical_edit(old="foo", new="bar", "foobar") catch {
    @lexical_edit.LEditError::NotFound => ()
    _ => fail("expected NotFound")
    } noraise {
    _ => fail("expected NotFound")
    }
    }

    lexical_edit

    fn lexical_edit(old~ : StringView, new~ : StringView, input : StringView) -> String raise LEditError

    Replaces the first contiguous MoonBit token sequence in input that matches old, scanning from left to right, with new.

    Parameters and return value

    • old: A required named argument specifying the code fragment to find. Leading and trailing whitespace and comments do not affect match locations. At least one token must remain after ignored items are removed. This is not a regular expression, and wildcards are not supported.
    • new: A required named argument specifying the text to insert verbatim. An empty string deletes the matched range. No lexical or syntactic validation is performed, and indentation, line breaks, and escapes are not adjusted. No separators are added.
    • input: A positional argument specifying the source text to edit, either complete source code or a string slice.

    Returns a String consisting of the original text before the matched range, new, and the original text after the matched range. Only the supplied view is processed; underlying string content outside the view is not included. The arguments are not modified. Each call replaces at most one occurrence, leaving other matches unchanged. If no match is found, NotFound is raised instead of returning the original text as a successful result.

    Matching rules

    old and input are tokenized using ordinary MoonBit lexical rules. Code fragments need not form complete syntax. Comparisons ignore whitespace, line breaks, comments, and automatically inserted semicolons between tokens. Explicit ; tokens must match: for example, a b does not match a;b.

    Each token's kind and original text must match exactly, except that trailing lexer-recognized whitespace in attribute tokens is ignored, including Unicode whitespace. Matching is case-sensitive, and literal values are not normalized: foo does not match foobar, 1 does not match 01, and 1.0 does not match 1.00. Individual tokens such as strings, interpolated strings, and attributes are compared as a whole, with no recursive search inside them. Their internal whitespace must also match. Compound tokens such as .field include the leading dot, so field does not match .field, and replacing .field also replaces the dot.

    The replacement range starts at the beginning of the first matched token and ends at the end of the last matched token. Whitespace and comments inside this range are removed; text outside it is preserved verbatim. Attributes retain their original lexer spans, so their ignored trailing whitespace is removed with the attribute. ASCII line endings (LF, CRLF, or CR) after a final attribute are outside its token and are preserved. The result is not tokenized again after insertion. It may contain lexical errors or join adjacent tokens together. Any required spaces or other separators must be supplied explicitly in new.

    Errors

    Checks and raises LEditError in the following order, without using lexical recovery tokens or falling back to text matching:

    1. OldLexError: old contains a lexical error.
    2. EmptyOld: old is lexically valid but contains no tokens to match.
    3. InputLexError: The entire input contains a lexical error, including errors after the first match.
    4. NotFound: Validation succeeds, but no contiguous token sequence matches old.

    Lexical errors report only the first lexer diagnostic in the relevant view. Offsets count UTF-16 code units from the start of that view; see LEditError for details.

    Examples

    Match across whitespace and comments, replace only the first occurrence, and preserve formatting outside the matched range:

    test "replace the first token match" {
    let result = @lexical_edit.lexical_edit(
    old="a + b",
    new="sum",
    " a // inside\n + b // after\n a+b",
    )
    assert_eq(result, " sum // after\n a+b")
    }

    An empty replacement deletes the matched range without merging spaces outside it:

    test "delete tokens without trimming surrounding text" {
    assert_eq(
    @lexical_edit.lexical_edit(old=" + b", new="", "a + b + b"),
    "a + b",
    )
    }

    Source Files