Sign in

    #Rolling log files

    The rolling package (@rolling) decides which file a log record goes to, when a new file starts and which old files to delete. It is a pure state machine: it does no I/O and reads no clock. The host file adapter lists the log directory, opens, writes and deletes files, and gives the results to this package. The package has the same API on wasm, wasm-gc, js and native, and imports only the MoonBit core library and @formatting.

    ///|
    import {
    "moonrockz/moon-lander-logging/formatting",
    "moonrockz/moon-lander-logging/rolling",
    }

    #Settings

    Each setting with rules is a type with a constructor that checks the rules and raises a RollingError that names the input and the rule.

    TypeRuleDefault
    LogFileStem1 to 64 characters: an ASCII letter, then ASCII letters, digits, _ and -. Not a Windows device name (CON, PRN, AUX, NUL, COM0-COM9, LPT0-LPT9, any letter case). Not _ or - and digits at the endNone
    FileByteLimitMore than 0 bytes10 MiB
    RetainedFileCountMore than 0 files31
    RollingPolicyDaily, Size(limit) or DailyAndSize(limit)DailyAndSize(10 MiB)
    RetentionKeepNewest(count) or KeepAllKeepNewest(31)
    FileFormatText (.log) or JsonLines (.jsonl)Text

    The stem cannot end with _ or - and digits because the file names of a shorter stem have that form. For example, the stem acme_001 would make acme_001.log, which is the second file of the stem acme.

    #File names

    A FileSet is the files of one stem and one format, with one policy. The periods are UTC days. For the stem acme:

    PolicyFirst fileNext files in the period
    Daily or DailyAndSizeacme-20261027.logacme-20261027_001.log, acme-20261027_002.log
    Sizeacme.logacme_001.log, acme_002.log

    JsonLines uses .jsonl in place of .log, so the two formats of one stem can share a directory. A sequence number has at least three digits, and gets more digits after 999 (_1000). After the largest Int, the plan raises SequenceExhausted; it never uses a name again.

    parse_name accepts only the complete form of an owned name. The ownership marker, for example acme.log.lock, is not an owned name:

    ///|
    test "the file set owns only its exact names" {
    let stem = @rolling.LogFileStem::parse("acme")
    let set = @rolling.FileSet::new(stem, Text, @rolling.RollingPolicy::default())
    assert_true(set.parse_name("acme-20261027_001.log") is Some(_))
    assert_true(set.parse_name("acme-old.log") is None)
    assert_true(set.parse_name("acme-20261027.log.bak") is None)
    assert_true(set.parse_name("acme-20261027.jsonl") is None)
    assert_eq(set.marker_name(), "acme.log.lock")
    assert_true(set.parse_name(set.marker_name()) is None)
    }

    #Plan each record

    RollingState::plan gives a FilePlan for one record of a number of bytes at the writer time:

    PlanAdapter action
    Open(name, existing_size~)No file is open. Open name to append (never truncate), then write
    Append(name)Write to the open file
    Roll(name, existing_size~)Close the active file, open name to append, write, then run retention

    After each successful action, the adapter calls opened(name, size) and wrote(bytes). A failed action does not change the state. Sizes are ByteCount values: parse a file size from the file system with ByteCount::of at the adapter, and count a record with ByteCount::of_text. The rules are:

    • The size is the UTF-8 bytes of the rendered record, including its LF and its error lines.
    • A nonempty file rolls before a record that does not fit. An exact fit appends.
    • An empty file takes a record larger than the limit, and the next record rolls. A record never splits across files.
    • A daily policy rolls at the first record of a later UTC day. That plan reads the listing, as at a restart, so a file of the new day from an earlier run is appended to when it has room. needs_listing(now) tells the adapter when to list the directory. When the clock moves back, the writer stays in the day of the active file.
    • Size does not roll at a new day. Daily does not roll at any size.
    • At the first open, the plan reads the directory listing. It appends to the newest owned regular file of the current period when that file has room, and else it opens the next sequence number.

    ///|
    test "a day with two size rolls" {
    let stem = @rolling.LogFileStem::parse("acme")
    let limit = @rolling.FileByteLimit::bytes(100L)
    let set = @rolling.FileSet::new(stem, JsonLines, DailyAndSize(limit))
    let now = @formatting.LogTimestamp::from_epoch_millis(1793110987024L)
    let mut state = @rolling.RollingState::new(set)
    let names = []
    for size in [60L, 40L, 1L, 99L, 1L] {
    let bytes = @rolling.ByteCount::of(size)
    let plan = state.plan(now, bytes, [])
    match plan {
    Append(name) => names.push("append \{name}")
    Open(name, existing_size~) => {
    state = state.opened(name, existing_size)
    names.push("open \{name}")
    }
    Roll(name, existing_size~) => {
    state = state.opened(name, existing_size)
    names.push("roll \{name}")
    }
    }
    state = state.wrote(bytes)
    }
    inspect(names.join("\n"), content=(
    #|open acme-20261027.jsonl
    #|append acme-20261027.jsonl
    #|roll acme-20261027_001.jsonl
    #|append acme-20261027_001.jsonl
    #|roll acme-20261027_002.jsonl
    ))
    }

    At a restart, the adapter gives the directory entries. The plan appends to the existing file and gives its size, so the size limit counts the old bytes:

    ///|
    test "a restart appends to the newest file of the day" {
    let stem = @rolling.LogFileStem::parse("acme")
    let set = @rolling.FileSet::new(stem, Text, @rolling.RollingPolicy::default())
    let now = @formatting.LogTimestamp::from_epoch_millis(1793110987024L)
    let listing : Array[@rolling.ListedFile] = [
    { name: "acme-20261027.log", size: @rolling.ByteCount::of(10485760L), regular: true },
    { name: "acme-20261027_001.log", size: @rolling.ByteCount::of(2048L), regular: true },
    { name: "acme.log.lock", size: @rolling.ByteCount::zero(), regular: true },
    ]
    let plan = @rolling.RollingState::new(set).plan(
    now,
    @rolling.ByteCount::of(120L),
    listing,
    )
    guard plan is Open(name, existing_size~) else { fail("no open plan") }
    assert_eq(name.to_string(), "acme-20261027_001.log")
    assert_eq(existing_size.to_int64(), 2048L)
    }

    #Retention

    After a new file opens, FileSet::retention gives the names of the files to delete. It counts only regular entries whose names have the complete owned form, so it never deletes directories, links, the marker, another stem, another format or a similar name such as acme-20261027.log.bak. KeepNewest(n) keeps the active file and the newest n - 1 other owned files. It sorts by UTC day and then by sequence number, not by modification time. KeepAll deletes nothing.

    ///|
    test "keep the newest three files" {
    let stem = @rolling.LogFileStem::parse("acme")
    let set = @rolling.FileSet::new(stem, Text, Daily)
    let names = [
    "acme-20261024.log", "acme-20261025.log", "acme-20261026.log", "acme-20261027.log",
    "acme-20261027.log.bak",
    ]
    let listing = names.map(name => @rolling.ListedFile::{
    name,
    size: @rolling.ByteCount::of(1L),
    regular: true,
    })
    guard set.parse_name("acme-20261027.log") is Some(active) else {
    fail("not owned")
    }
    let retention = @rolling.Retention::KeepNewest(
    @rolling.RetainedFileCount::count(3),
    )
    assert_eq(set.retention(retention, listing, active).map(name => name.to_string()), ["acme-20261024.log"])
    }

    The adapter deletes the names only after the new file opens. A failed delete is a retention failure; it does not stop the active file.

    RollingError

    pub suberror RollingError {
    InvalidStem(input~ : String, rule~ : StemRule)
    NotPositiveByteLimit(bytes~ : Int64)
    NotPositiveFileCount(count~ : Int)
    NegativeByteCount(bytes~ : Int64)
    SequenceExhausted(last~ : String)
    } derive(
    Debug
    )

    A rolling input that breaks a rule, or a file set that has no next file name. Callers can match the cases, but only this package makes them.

    RollingError::message

    fn RollingError::message(self : RollingError) -> String

    Text for people that describes the fault.

    ByteCount

    pub struct ByteCount {
    // private fields
    } derive(Compare, Eq, Hash,
    Debug
    )

    A count of bytes: a file size or the encoded size of a record. It is zero or more.
    impl Show for ByteCount

    ByteCount::compare

    fn ByteCount::compare(ByteCount, ByteCount) -> Int

    ByteCount::equal

    fn ByteCount::equal(ByteCount, ByteCount) -> Bool

    ByteCount::hash

    fn ByteCount::hash(self : ByteCount) -> Int

    ByteCount::hash_combine

    fn ByteCount::hash_combine(ByteCount, Hasher) -> Unit

    ByteCount::not_equal

    fn ByteCount::not_equal(x : ByteCount, y : ByteCount) -> Bool

    ByteCount::of

    fn ByteCount::of(bytes : Int64) -> ByteCount raise RollingError

    Make a byte count, for example a file size from the file system. It raises NegativeByteCount when bytes is less than zero.

    ByteCount::of_text

    fn ByteCount::of_text(text : String) -> ByteCount

    The count of UTF-8 bytes of text, for example a rendered record with its LF.

    ByteCount::op_ge

    fn ByteCount::op_ge(x : ByteCount, y : ByteCount) -> Bool

    ByteCount::op_gt

    fn ByteCount::op_gt(x : ByteCount, y : ByteCount) -> Bool

    ByteCount::op_le

    fn ByteCount::op_le(x : ByteCount, y : ByteCount) -> Bool

    ByteCount::op_lt

    fn ByteCount::op_lt(x : ByteCount, y : ByteCount) -> Bool

    ByteCount::output

    fn ByteCount::output(self : ByteCount, logger : &Logger) -> Unit

    ByteCount::to_int64

    fn ByteCount::to_int64(self : ByteCount) -> Int64

    The count as a number.

    ByteCount::to_string

    fn ByteCount::to_string(self : ByteCount) -> String

    ByteCount::zero

    fn ByteCount::zero() -> ByteCount

    No bytes.

    FileByteLimit

    pub struct FileByteLimit {
    // private fields
    } derive(Compare, Eq, Hash,
    Debug
    )

    The size at which a file rolls: a positive number of bytes.

    FileByteLimit::bytes

    fn FileByteLimit::bytes(bytes : Int64) -> FileByteLimit raise RollingError

    Make a limit of bytes bytes. The value must be greater than zero.

    FileByteLimit::compare

    FileByteLimit::default

    fn FileByteLimit::default() -> FileByteLimit

    The default limit: 10 MiB (10 485 760 bytes).

    FileByteLimit::equal

    FileByteLimit::hash

    fn FileByteLimit::hash(self : FileByteLimit) -> Int

    FileByteLimit::hash_combine

    fn FileByteLimit::hash_combine(FileByteLimit, Hasher) -> Unit

    FileByteLimit::not_equal

    fn FileByteLimit::not_equal(x : FileByteLimit, y : FileByteLimit) -> Bool

    FileByteLimit::op_ge

    fn FileByteLimit::op_ge(x : FileByteLimit, y : FileByteLimit) -> Bool

    FileByteLimit::op_gt

    fn FileByteLimit::op_gt(x : FileByteLimit, y : FileByteLimit) -> Bool

    FileByteLimit::op_le

    fn FileByteLimit::op_le(x : FileByteLimit, y : FileByteLimit) -> Bool

    FileByteLimit::op_lt

    fn FileByteLimit::op_lt(x : FileByteLimit, y : FileByteLimit) -> Bool

    FileByteLimit::to_int64

    fn FileByteLimit::to_int64(self : FileByteLimit) -> Int64

    The limit in bytes, greater than zero.

    FileFormat

    pub(all) enum FileFormat {
    Text
    JsonLines
    } derive(Eq,
    Debug
    )

    The format of the records in the files of a file set.

    FileFormat::default

    fn FileFormat::default() -> FileFormat

    The default format: readable text.

    FileFormat::equal

    fn FileFormat::equal(FileFormat, FileFormat) -> Bool

    FileFormat::extension

    fn FileFormat::extension(self : FileFormat) -> String

    The file name extension of the format, without the dot: log or jsonl.

    FileFormat::not_equal

    fn FileFormat::not_equal(x : FileFormat, y : FileFormat) -> Bool

    FilePlan

    pub(all) enum FilePlan {
    Append(LogFileName)
    Open(LogFileName, existing_size~ : ByteCount)
    Roll(LogFileName, existing_size~ : ByteCount)
    } derive(Eq,
    Debug
    )

    The file action for the next record. The adapter does the action, and then tells the state the result with opened and wrote.

    FilePlan::equal

    fn FilePlan::equal(FilePlan, FilePlan) -> Bool

    FilePlan::not_equal

    fn FilePlan::not_equal(x : FilePlan, y : FilePlan) -> Bool

    FilePlan::to_repr

    FileSet

    pub struct FileSet {
    // private fields
    } derive(Eq,
    Debug
    )

    The files of one stem and one format in a log directory, with one rolling policy. A file set owns only the names of its complete naming scheme. For the stem acme:

    PolicyFirst fileNext files
    Daily or DailyAndSizeacme-20261027.logacme-20261027_001.log, acme-20261027_002.log
    Sizeacme.logacme_001.log, acme_002.log

    JsonLines uses .jsonl in place of .log. A sequence number has at least three digits, and has more digits after 999 (_1000).

    FileSet::equal

    fn FileSet::equal(FileSet, FileSet) -> Bool

    FileSet::format

    fn FileSet::format(self : FileSet) -> FileFormat

    The format of the records.

    FileSet::marker_name

    fn FileSet::marker_name(self : FileSet) -> String

    The name of the ownership marker of the file set, for example acme.log.lock. It does not have the form of an owned file name, so retention never deletes it. The same stem and format with a different policy has the same marker, so one directory cannot mix policies for one file set.

    FileSet::new

    fn FileSet::new(stem : LogFileStem, format : FileFormat, policy : RollingPolicy) -> FileSet

    Make the file set of stem and format that rolls by policy.

    FileSet::not_equal

    fn FileSet::not_equal(x : FileSet, y : FileSet) -> Bool

    FileSet::parse_name

    fn FileSet::parse_name(self : FileSet, text : String) -> LogFileName?

    The owned name text, or None when text does not have the complete form of a name of this file set. The check is exact: the stem with the same letter case, a real date for a daily policy, a sequence number in its one text form (_001, not _01, _0001 or _000) and the extension of the format. So acme-old.log, acme-20261027.log.bak, acmex.log, the names of another stem or format and the marker are not owned.

    FileSet::policy

    fn FileSet::policy(self : FileSet) -> RollingPolicy

    The rolling policy.

    FileSet::retention

    fn FileSet::retention(self : FileSet, retention : Retention, listing : Array[ListedFile], active : LogFileName) -> Array[LogFileName]

    The owned files to delete under retention, after active opened. Only regular entries of listing whose names have the complete owned form count, so directories, links, the marker and other names stay. The active file is never in the result. KeepNewest(n) keeps the active file and the newest n - 1 other owned files, by UTC day and then by sequence number, not by modification time. The result is oldest first. KeepAll deletes nothing.

    FileSet::stem

    fn FileSet::stem(self : FileSet) -> LogFileStem

    The stem of the file names.

    FileSet::to_repr

    ListedFile

    pub(all) struct ListedFile {
    name : String
    size : ByteCount
    regular : Bool
    } derive(Eq,
    Debug
    )

    One entry of a log directory, as the host adapter found it. The adapter gives the entries; this package does no I/O.

    ListedFile::equal

    fn ListedFile::equal(ListedFile, ListedFile) -> Bool

    ListedFile::not_equal

    fn ListedFile::not_equal(x : ListedFile, y : ListedFile) -> Bool

    LogFileName

    pub struct LogFileName {
    // private fields
    } derive(Compare, Eq, Hash,
    Debug
    )

    The name of one file that a file set owns: an optional UTC day and a sequence number, where 0 is the name with no number. Names of one file set sort by day and then by sequence number.
    impl Show for LogFileName

    LogFileName::compare

    fn LogFileName::compare(LogFileName, LogFileName) -> Int

    LogFileName::epoch_day

    fn LogFileName::epoch_day(self : LogFileName) -> Int64?

    The UTC day of the name as days after 1970-01-01, as LogTimestamp::epoch_day gives it, or None for a name with no date.

    LogFileName::equal

    fn LogFileName::equal(LogFileName, LogFileName) -> Bool

    LogFileName::hash

    fn LogFileName::hash(self : LogFileName) -> Int

    LogFileName::hash_combine

    fn LogFileName::hash_combine(LogFileName, Hasher) -> Unit

    LogFileName::not_equal

    fn LogFileName::not_equal(x : LogFileName, y : LogFileName) -> Bool

    LogFileName::op_ge

    fn LogFileName::op_ge(x : LogFileName, y : LogFileName) -> Bool

    LogFileName::op_gt

    fn LogFileName::op_gt(x : LogFileName, y : LogFileName) -> Bool

    LogFileName::op_le

    fn LogFileName::op_le(x : LogFileName, y : LogFileName) -> Bool

    LogFileName::op_lt

    fn LogFileName::op_lt(x : LogFileName, y : LogFileName) -> Bool

    LogFileName::output

    fn LogFileName::output(self : LogFileName, logger : &Logger) -> Unit

    LogFileName::sequence

    fn LogFileName::sequence(self : LogFileName) -> Int

    The sequence number in the period: 0 for the name with no number, then 1 for _001 and so on.

    LogFileName::to_string

    fn LogFileName::to_string(self : LogFileName) -> String

    LogFileStem

    pub struct LogFileStem {
    // private fields
    } derive(Compare, Eq, Hash,
    Debug
    )

    The first part of each file name of a file set, for example acme in acme-20261027.log. It has 1 to 64 characters: an ASCII letter, then ASCII letters, ASCII digits, _ and -. It does not end with _ or - and digits, and it is not a Windows device name. So the file names are valid on each OS, and two stems never make the same file name.
    impl Show for LogFileStem

    LogFileStem::compare

    fn LogFileStem::compare(LogFileStem, LogFileStem) -> Int

    LogFileStem::equal

    fn LogFileStem::equal(LogFileStem, LogFileStem) -> Bool

    LogFileStem::hash

    fn LogFileStem::hash(self : LogFileStem) -> Int

    LogFileStem::hash_combine

    fn LogFileStem::hash_combine(LogFileStem, Hasher) -> Unit

    LogFileStem::not_equal

    fn LogFileStem::not_equal(x : LogFileStem, y : LogFileStem) -> Bool

    LogFileStem::op_ge

    fn LogFileStem::op_ge(x : LogFileStem, y : LogFileStem) -> Bool

    LogFileStem::op_gt

    fn LogFileStem::op_gt(x : LogFileStem, y : LogFileStem) -> Bool

    LogFileStem::op_le

    fn LogFileStem::op_le(x : LogFileStem, y : LogFileStem) -> Bool

    LogFileStem::op_lt

    fn LogFileStem::op_lt(x : LogFileStem, y : LogFileStem) -> Bool

    LogFileStem::output

    fn LogFileStem::output(self : LogFileStem, logger : &Logger) -> Unit

    LogFileStem::parse

    fn LogFileStem::parse(text : String) -> LogFileStem raise RollingError

    Parse a file stem. The checks are in the order of the cases of StemRule.

    LogFileStem::to_string

    fn LogFileStem::to_string(self : LogFileStem) -> String

    RetainedFileCount

    pub struct RetainedFileCount {
    // private fields
    } derive(Compare, Eq, Hash,
    Debug
    )

    The number of newest files that retention keeps, including the active file: a positive count.

    RetainedFileCount::compare

    RetainedFileCount::count

    fn RetainedFileCount::count(count : Int) -> RetainedFileCount raise RollingError

    Make a count of count files. The value must be greater than zero.

    RetainedFileCount::default

    The default count: 31 files.

    RetainedFileCount::equal

    RetainedFileCount::hash

    fn RetainedFileCount::hash(self : RetainedFileCount) -> Int

    RetainedFileCount::hash_combine

    fn RetainedFileCount::hash_combine(RetainedFileCount, Hasher) -> Unit

    RetainedFileCount::not_equal

    fn RetainedFileCount::not_equal(x : RetainedFileCount, y : RetainedFileCount) -> Bool

    RetainedFileCount::op_ge

    RetainedFileCount::op_gt

    RetainedFileCount::op_le

    RetainedFileCount::op_lt

    RetainedFileCount::to_int

    fn RetainedFileCount::to_int(self : RetainedFileCount) -> Int

    The count, greater than zero.

    Retention

    pub(all) enum Retention {
    KeepNewest(RetainedFileCount)
    KeepAll
    } derive(Eq,
    Debug
    )

    Which old files of a file set stay after a roll.

    Retention::default

    fn Retention::default() -> Retention

    The default retention: the newest 31 files, including the active file.

    Retention::equal

    fn Retention::equal(Retention, Retention) -> Bool

    Retention::not_equal

    fn Retention::not_equal(x : Retention, y : Retention) -> Bool

    RollingPolicy

    pub(all) enum RollingPolicy {
    Daily
    Size(FileByteLimit)
    DailyAndSize(FileByteLimit)
    } derive(Eq,
    Debug
    )

    When a file set starts a new file. Periods are UTC days.

    RollingPolicy::default

    fn RollingPolicy::default() -> RollingPolicy

    The default policy: a new file for each UTC day and at 10 MiB.

    RollingPolicy::equal

    RollingPolicy::not_equal

    fn RollingPolicy::not_equal(x : RollingPolicy, y : RollingPolicy) -> Bool

    RollingState

    pub struct RollingState {
    // private fields
    } derive(Eq,
    Debug
    )

    The rolling state of one writer of a file set: the active file and its size. The value is immutable: opened and wrote give a new state. The adapter calls them only after its I/O succeeds, so a failed action does not move the state.

    RollingState::active

    fn RollingState::active(self : RollingState) -> LogFileName?

    The open active file, or None before the first open.

    RollingState::equal

    RollingState::file_set

    fn RollingState::file_set(self : RollingState) -> FileSet

    The file set of the writer.

    RollingState::needs_listing

    True when the next plan at now reads the directory listing: no file is open yet, or a daily policy moves to a later UTC day. Read the clock once and give the same now to plan.

    RollingState::new

    The state of a writer of set that has no open file.

    RollingState::not_equal

    fn RollingState::not_equal(x : RollingState, y : RollingState) -> Bool

    RollingState::opened

    fn RollingState::opened(self : RollingState, name : LogFileName, size : ByteCount) -> RollingState

    The state after the adapter opened name to append, with size bytes in it after the open.

    RollingState::plan

    The file action for a record of record_bytes bytes (the UTF-8 bytes of the rendered record, including its LF) at the writer time now.

    • With no open file, the plan reads listing, the entries of the log directory. It finds the newest owned file of the current period (the UTC day of now for a daily policy, all owned files for Size). When that is a regular file with room for the record, the plan opens it to append. When it is not, the plan opens the next sequence number. With no owned file, it opens the first name of the period.
    • With an open file and a later UTC day, a daily policy rolls. It reads listing for the files of the new day, as at a restart, so a file that an earlier run left for that day is appended to when it has room, else the next sequence number opens. A clock that moves back does not roll: the writer stays in the day of the active file.
    • Otherwise the plan does not read listing; needs_listing tells when it does.
    • A size limit rolls a nonempty file when the record does not fit. An empty file takes a record larger than the limit, and the next record rolls. The plan never splits a record.

    Raises SequenceExhausted when the period has no next name.

    RollingState::size

    fn RollingState::size(self : RollingState) -> ByteCount

    The bytes in the active file: its size at the open and each write after it.

    RollingState::wrote

    fn RollingState::wrote(self : RollingState, bytes : ByteCount) -> RollingState

    The state after the adapter wrote bytes bytes to the active file.

    StemRule

    pub enum StemRule {
    Empty
    TooLong
    NotLetterStart
    InvalidCharacter(Char)
    NumericSuffix
    ReservedDeviceName
    } derive(Eq,
    Debug
    )

    The rule of a file stem that an input breaks.

    StemRule::equal

    fn StemRule::equal(StemRule, StemRule) -> Bool

    StemRule::not_equal

    fn StemRule::not_equal(x : StemRule, y : StemRule) -> Bool

    StemRule::to_repr