moon-warc

    Streaming WARC 1.1 parser, writer, validator, indexer and audit toolkit for MoonBit.

    warc
    web-archive
    archive
    parser
    streaming
    iso28500
    webarchiving
    moonbit
    Download zip
    Version
    0.1.0
    License
    Apache-2.0
    Last updated
    2 months ago
    Downloads
    29

    No dependencies

    #moon-warc

    A streaming WARC 1.1 (ISO 28500) parser, writer, validator, indexer and audit toolkit for MoonBit.

    • Module: xiaojing012/moon-warc
    • Version: 0.1.0
    • Repository: https://github.com/xiaojing012/moon-warc
    • Applicant / Maintainer: 宋晓静 / xiaojing012
    • Mooncakes: xiaojing012/moon-warc

    #Overview

    WARC (Web ARChive) is the ISO-standard container format (ISO 28500:2017) used by web archives, crawlers and digital-preservation systems to store billions of harvested resources. A WARC file concatenates any number of WARC records; each record carries a set of named fields plus an arbitrary binary content block.

    moon-warc implements the record framing and semantic layer of WARC 1.1 in pure MoonBit: binary-safe record framing, ordered named fields, buffered and incremental streaming parsing, deterministic writing, semantic validation, segmentation support, in-memory indexing, and an advisory audit engine — with structured errors, resource limits and cross-target tests along the way.

    #Install

    moon add xiaojing012/moon-warc

    Add the package alias to the consumer's moon.pkg:

    import {
    "xiaojing012/moon-warc" @warc,
    }

    Parse an in-memory WARC archive with structured limits and errors:

    match @warc.parse_archive(data, @warc.Limits::default()) {
    Ok(archive) => println("records: \{archive.record_count()}")
    Err(error) => println(error.to_string())
    }

    #Why WARC

    • Web archive interoperability — read and write the format used by the Internet Archive, national libraries and Common Crawl;
    • Binary-safe streaming — blocks are arbitrary bytes; parsing never depends on scanning for version strings;
    • Archival data processing — record framing, validation, querying and statistics for offline analysis pipelines;
    • Reproducible data pipelines — a deterministic writer/builder with no hidden clock or network access;
    • MoonBit ecosystem infrastructure — a foundation for crawler storage layers, migration tools and archive inspection utilities.

    #Record Structure

    warc-record = header CRLF block CRLF CRLF header = version warc-fields version = "WARC/1.1" CRLF warc-fields = *named-field CRLF block = *OCTET

    #CLI and Examples

    A small command-line entry point (cli/) operates on a built-in demo archive (this build has no file-system I/O):

    moon run cli help moon run cli parse | validate | inspect | stats | audit moon run cli query http://example.org/a moon run cli build

    Runnable examples (examples/) demonstrate each slice of the library:

    moon run examples parse | build | validate | index | audit

    #Development Status

    Released version 0.1.0. The complete record model, framing, buffered parser, streaming decoder, writer/builder, semantic validator, segmentation, indexing and statistics, and the audit engine are implemented and covered by 208 tests on all four MoonBit targets (wasm, wasm-gc, js, native); see docs/ for the specification map, architecture notes, limitations and the reproducibility guide. verify_all.ps1 runs the whole verification in one go.

    #License

    Apache License 2.0. See LICENSE.

    ArchiveStats

    pub struct ArchiveStats {
    record_count : Int
    total_record_bytes : Int64
    total_block_bytes : Int64
    type_counts : Array[(String, Int)]
    distinct_targets : Array[String]
    earliest : WarcDate?
    latest : WarcDate?
    }

    Aggregate statistics over an archive.

    ArchiveStats::distinct_target_count

    fn ArchiveStats::distinct_target_count(self : ArchiveStats) -> Int

    The number of distinct WARC-Target-URI values.

    ArchiveStats::earliest_date

    fn ArchiveStats::earliest_date(self : ArchiveStats) -> WarcDate?

    The earliest WARC-Date seen, when any record carried a parseable one.

    ArchiveStats::latest_date

    fn ArchiveStats::latest_date(self : ArchiveStats) -> WarcDate?

    The latest WARC-Date seen, when any record carried a parseable one.

    ArchiveStats::record_count

    fn ArchiveStats::record_count(self : ArchiveStats) -> Int

    The number of records in the archive.

    ArchiveStats::total_block_bytes

    fn ArchiveStats::total_block_bytes(self : ArchiveStats) -> Int64

    The total size of all content blocks in octets.

    ArchiveStats::total_record_bytes

    fn ArchiveStats::total_record_bytes(self : ArchiveStats) -> Int64

    The total serialized size of all records in octets.

    ArchiveStats::type_count

    fn ArchiveStats::type_count(self : ArchiveStats, name : String) -> Int

    The number of records of the given type name (lowercase); records of unknown WARC-Type count as unknown.

    FindingSeverity

    pub enum FindingSeverity {
    Info
    Warning
    } derive(Eq,
    Debug
    )

    The severity of an audit finding.

    FindingSeverity::severity_name

    fn FindingSeverity::severity_name(self : FindingSeverity) -> String

    The lowercase machine-readable severity name.

    Limits

    pub struct Limits {
    max_archive_bytes : Int64
    max_record_bytes : Int64
    max_header_bytes : Int64
    max_header_count : Int
    max_field_name_bytes : Int
    max_field_value_bytes : Int
    max_block_bytes : Int64
    max_records : Int
    max_content_length_digits : Int
    }

    Limits::default

    fn Limits::default() -> Limits

    Balanced defaults for general-purpose archive processing.

    Limits::permissive

    fn Limits::permissive() -> Limits

    Near-unbounded ceilings for controlled offline processing.

    Limits::strict

    fn Limits::strict() -> Limits

    Tight bounds for untrusted input and memory-constrained targets.

    Limits::with_max_archive_bytes

    fn Limits::with_max_archive_bytes(self : Limits, n : Int64) -> Limits

    A copy of these limits with max_archive_bytes replaced.

    Limits::with_max_block_bytes

    fn Limits::with_max_block_bytes(self : Limits, n : Int64) -> Limits

    A copy of these limits with max_block_bytes replaced.

    Limits::with_max_content_length_digits

    fn Limits::with_max_content_length_digits(self : Limits, n : Int) -> Limits

    A copy of these limits with max_content_length_digits replaced.

    Limits::with_max_field_name_bytes

    fn Limits::with_max_field_name_bytes(self : Limits, n : Int) -> Limits

    A copy of these limits with max_field_name_bytes replaced.

    Limits::with_max_field_value_bytes

    fn Limits::with_max_field_value_bytes(self : Limits, n : Int) -> Limits

    A copy of these limits with max_field_value_bytes replaced.

    Limits::with_max_header_bytes

    fn Limits::with_max_header_bytes(self : Limits, n : Int64) -> Limits

    A copy of these limits with max_header_bytes replaced.

    Limits::with_max_header_count

    fn Limits::with_max_header_count(self : Limits, n : Int) -> Limits

    A copy of these limits with max_header_count replaced.

    Limits::with_max_record_bytes

    fn Limits::with_max_record_bytes(self : Limits, n : Int64) -> Limits

    A copy of these limits with max_record_bytes replaced.

    Limits::with_max_records

    fn Limits::with_max_records(self : Limits, n : Int) -> Limits

    A copy of these limits with max_records replaced.

    PendingRecord

    type PendingRecord

    A partially parsed record whose header is complete but whose block and trailing CRLF CRLF have not fully arrived.

    SegmentInfo

    pub struct SegmentInfo {
    number : Int64?
    origin_id : String?
    total_length : Int64?
    }

    The parsed segmentation fields of a record. Absent fields are None.

    SegmentInfo::new

    fn SegmentInfo::new(number : Int64?, origin_id : String?, total_length : Int64?) -> SegmentInfo

    Construct segment info from optional parts.

    SegmentInfo::number

    fn SegmentInfo::number(self : SegmentInfo) -> Int64?

    The WARC-Segment-Number value, when present.

    SegmentInfo::origin_id

    fn SegmentInfo::origin_id(self : SegmentInfo) -> String?

    The interior URI of WARC-Segment-Origin-ID, when present.

    SegmentInfo::total_length

    fn SegmentInfo::total_length(self : SegmentInfo) -> Int64?

    The WARC-Segment-Total-Length value, when present.

    WarcArchive

    pub struct WarcArchive {
    records : Array[WarcRecord]
    }

    A fully parsed in-memory WARC archive: the records in file order.

    WarcArchive::record

    fn WarcArchive::record(self : WarcArchive, index : Int) -> WarcRecord?

    The record at index, or None when out of range.

    WarcArchive::record_count

    fn WarcArchive::record_count(self : WarcArchive) -> Int

    The number of records in the archive.

    WarcArchive::total_block_bytes

    fn WarcArchive::total_block_bytes(self : WarcArchive) -> Int64

    The total number of octets across all content blocks.

    WarcBuilder

    pub struct WarcBuilder {
    version : String
    fields : Array[WarcField]
    }

    An incremental WARC record builder.

    WarcBuilder::build

    fn WarcBuilder::build(self : WarcBuilder, block : Bytes) -> Result[WarcRecord, WarcError]

    Build the record around a content block.

    A missing Content-Length is generated from the block size; a present one must match it exactly (and must not be repeated). Field names must be legal tokens and values must be single-line, so the result always serializes to a parseable record.

    WarcBuilder::field

    fn WarcBuilder::field(self : WarcBuilder, name : String, value : String) -> WarcBuilder

    Append a named field. The original spelling is preserved; names are validated at build time and values must fit on one header line.

    WarcBuilder::new

    Create a builder for a WARC/1.1 record with no fields.

    WarcBuilder::record_id

    fn WarcBuilder::record_id(self : WarcBuilder, uri : String) -> WarcBuilder

    Append a WARC-Record-ID field (value in <uri> form).

    WarcBuilder::target_uri

    fn WarcBuilder::target_uri(self : WarcBuilder, uri : String) -> WarcBuilder

    Append a WARC-Target-URI field (value in <uri> form).

    WarcBuilder::version

    fn WarcBuilder::version(self : WarcBuilder, v : String) -> WarcBuilder

    Set the version line. Only WARC/1.1 is accepted at build time.

    WarcBuilder::warc_date

    fn WarcBuilder::warc_date(self : WarcBuilder, date : String) -> WarcBuilder

    Append a WARC-Date field (W3CDTF UTC timestamp).

    WarcBuilder::with_type

    fn WarcBuilder::with_type(self : WarcBuilder, t : String) -> WarcBuilder

    Append a WARC-Type field.

    WarcDate

    pub struct WarcDate {
    year : Int
    month : Int?
    day : Int?
    hour : Int?
    minute : Int?
    second : Int?
    fraction : String
    } derive(Eq,
    Debug
    )

    A parsed WARC-Date. Optional components are absent at coarser granularities; fraction is the exact fractional-second digit string ("" when there are no fractional seconds).

    WarcDate::day

    fn WarcDate::day(self : WarcDate) -> Int?

    The day, present at YYYY-MM-DD granularity and finer.

    WarcDate::has_fraction

    fn WarcDate::has_fraction(self : WarcDate) -> Bool

    Whether this is the finest granularity: date plus time plus fractional seconds.

    WarcDate::has_time

    fn WarcDate::has_time(self : WarcDate) -> Bool

    Whether a time of day is present.

    WarcDate::month

    fn WarcDate::month(self : WarcDate) -> Int?

    The month, present at YYYY-MM granularity and finer.

    WarcDate::year

    fn WarcDate::year(self : WarcDate) -> Int

    The four-digit year.

    WarcDecoder

    pub struct WarcDecoder {
    limits : Limits
    bytes : Bytes
    start : Int
    pos : Int64
    records_done : Int
    total_fed : Int64
    pending : PendingRecord?
    error : WarcError?
    }

    A streaming WARC decoder.

    WarcDecoder::feed

    fn WarcDecoder::feed(self : WarcDecoder, chunk : Bytes) -> Result[Array[WarcRecord], WarcError]

    Feed one chunk of the input stream.

    Returns the records completed by this chunk (possibly none). The first error makes the decoder terminal: later feed/finish calls return the same error. Enforces max_archive_bytes across the whole stream, max_records at each record boundary and all per-record limits through the shared parsing functions.

    WarcDecoder::finish

    fn WarcDecoder::finish(self : WarcDecoder) -> Result[Unit, WarcError]

    Signal end of input.

    Returns Ok when the stream ended exactly at a record boundary and every record was complete. Bytes that remain at end of input — a truncated record or anything after the last complete record that cannot start a new one — are reported with the same trailing-garbage semantics as parse_archive.

    WarcDecoder::new

    fn WarcDecoder::new(limits : Limits) -> WarcDecoder

    Create a decoder with the given limits.

    WarcDigest

    pub struct WarcDigest {
    algorithm : String
    value : String
    }

    A parsed labelled digest such as sha1:B2QYELGDQWXWAWJ4OA2GSCBXDAOB7UHR.

    WarcDigest::algorithm

    fn WarcDigest::algorithm(self : WarcDigest) -> String

    The algorithm label (e.g. sha1).

    WarcDigest::new

    fn WarcDigest::new(algorithm : String, value : String) -> WarcDigest

    Construct a labelled digest from its parts.

    WarcDigest::to_string

    fn WarcDigest::to_string(self : WarcDigest) -> String

    The canonical algorithm:value rendering.

    WarcDigest::value

    fn WarcDigest::value(self : WarcDigest) -> String

    The encoded digest value.

    WarcError

    pub struct WarcError {
    stage : WarcErrorStage
    kind : WarcErrorKind
    byte_offset : Int64
    record_index : Int64
    context : String
    }

    A structured WARC error.

    WarcError::new

    fn WarcError::new(stage : WarcErrorStage, kind : WarcErrorKind, byte_offset : Int64, record_index : Int64, context : String) -> WarcError

    Create an error at a byte offset within a specific record.

    WarcError::to_string

    fn WarcError::to_string(self : WarcError) -> String

    A human-readable one-line description of the error.

    WarcErrorKind

    pub enum WarcErrorKind {
    UnexpectedEof
    InvalidVersion
    MissingColon
    InvalidFieldName
    InvalidUtf8
    InvalidContentLength
    IntegerOverflow
    ContentLengthMismatch
    MissingRequiredField
    DuplicateField
    InvalidDate
    InvalidUri
    InvalidDigest
    InvalidRecordType
    InvalidSeparator
    TrailingGarbage
    LimitExceeded
    InvalidFieldValue
    MisplacedField
    InvalidIpAddress
    } derive(Eq,
    Debug
    )

    The precise failure kind. Parse-time failures and semantic validation failures share this model; advisory findings are reported separately by the audit engine.

    WarcErrorKind::kind_name

    fn WarcErrorKind::kind_name(self : WarcErrorKind) -> String

    The lowercase machine-readable kind name.

    WarcErrorStage

    pub enum WarcErrorStage {
    Input
    Version
    Header
    Field
    Number
    ContentLength
    Block
    Separator
    Date
    Uri
    Digest
    Record
    Archive
    Segment
    Limit
    Builder
    } derive(Eq,
    Debug
    )

    The processing stage in which an error occurred.

    WarcErrorStage::stage_name

    fn WarcErrorStage::stage_name(self : WarcErrorStage) -> String

    The lowercase machine-readable stage name.

    WarcField

    pub struct WarcField {
    name : String
    value : String
    }

    One named field of a WARC record header.

    Per ISO 28500 clause 5, field names are compared case-insensitively; the original spelling is kept so callers can inspect or reproduce it.

    WarcField::new

    fn WarcField::new(name : String, value : String) -> WarcField

    Construct a named field from its raw spelling and value.

    WarcFinding

    pub struct WarcFinding {
    severity : FindingSeverity
    code : String
    record_index : Int64
    context : String
    }

    An advisory audit finding anchored to a record.

    WarcFinding::code

    fn WarcFinding::code(self : WarcFinding) -> String

    The machine-readable finding code.

    WarcFinding::context

    fn WarcFinding::context(self : WarcFinding) -> String

    A human-readable description of the finding.

    WarcFinding::new

    fn WarcFinding::new(severity : FindingSeverity, code : String, record_index : Int64, context : String) -> WarcFinding

    Construct a finding.

    WarcFinding::record_index

    fn WarcFinding::record_index(self : WarcFinding) -> Int64

    The record the finding is anchored to.

    WarcFinding::severity

    fn WarcFinding::severity(self : WarcFinding) -> FindingSeverity

    The finding's severity.

    WarcFinding::to_string

    fn WarcFinding::to_string(self : WarcFinding) -> String

    A one-line rendering of the finding.

    WarcIndex

    pub struct WarcIndex {
    by_id : Array[(String, Int)]
    by_target : Array[(String, Int)]
    by_type : Array[(String, Int)]
    }

    An in-memory index over an archive's records.

    WarcIndex::build

    fn WarcIndex::build(a : WarcArchive) -> WarcIndex

    Build an index over an archive. Records whose keys are absent or malformed are simply not indexed.

    WarcIndex::by_record_id

    fn WarcIndex::by_record_id(self : WarcIndex, id : String) -> Int?

    The index of the first record whose WARC-Record-ID interior is id, or None when no record carries it.

    WarcIndex::by_target_uri

    fn WarcIndex::by_target_uri(self : WarcIndex, uri : String) -> Array[Int]

    The indices of all records whose WARC-Target-URI interior is uri, in file order.

    WarcIndex::by_type

    fn WarcIndex::by_type(self : WarcIndex, t : WarcRecordType) -> Array[Int]

    The indices of all records of the given type, in file order.

    WarcIndex::size

    fn WarcIndex::size(self : WarcIndex) -> Int

    The total number of indexed keys across all three key kinds.

    WarcRecord

    pub struct WarcRecord {
    version : String
    fields : Array[WarcField]
    block : Bytes
    }

    A fully framed WARC record: version line, ordered named fields and the raw binary content block.

    WarcRecord::block_length

    fn WarcRecord::block_length(self : WarcRecord) -> Int

    The number of octets in the record's content block.

    WarcRecord::declared_content_length

    fn WarcRecord::declared_content_length(self : WarcRecord) -> String?

    The raw Content-Length field value, when present.

    WarcRecord::field_all

    fn WarcRecord::field_all(self : WarcRecord, name : String) -> Array[String]

    All values of fields with a case-insensitive name match, in input order (relevant for repeatable fields such as WARC-Concurrent-To).

    WarcRecord::field_count

    fn WarcRecord::field_count(self : WarcRecord, name : String) -> Int

    How many fields carry a case-insensitive name match.

    WarcRecord::field_first

    fn WarcRecord::field_first(self : WarcRecord, name : String) -> String?

    The value of the first field with a case-insensitive name match.

    WarcRecord::field_has

    fn WarcRecord::field_has(self : WarcRecord, name : String) -> Bool

    Whether at least one field carries a case-insensitive name match.

    WarcRecord::new

    fn WarcRecord::new(version : String, fields : Array[WarcField], block : Bytes) -> WarcRecord

    Construct a WARC record from its parts.

    WarcRecord::record_id

    fn WarcRecord::record_id(self : WarcRecord, record_index : Int64) -> Result[String, WarcError]

    The WARC-Record-ID of the record: the mandatory field, parsed as a <uri> reference. Returns the URI inside the angle brackets.

    WarcRecord::record_type

    fn WarcRecord::record_type(self : WarcRecord) -> WarcRecordType?

    The record type declared by WARC-Type, or None when the field is absent or carries an unknown type.

    WarcRecord::require_field

    fn WarcRecord::require_field(self : WarcRecord, name : String, record_index : Int64) -> Result[String, WarcError]

    The value of a mandatory field, or a structured error naming it.

    WarcRecord::segment_info

    fn WarcRecord::segment_info(self : WarcRecord, index : Int64) -> Result[SegmentInfo, WarcError]

    Parse the segmentation fields of a record.

    Fails when a field is present but malformed: a non-decimal or non-positive WARC-Segment-Number, a WARC-Segment-Origin-ID that is not a <uri>, or a non-decimal WARC-Segment-Total-Length.

    WarcRecord::target_uri

    fn WarcRecord::target_uri(self : WarcRecord, record_index : Int64) -> Result[String, WarcError]

    The WARC-Target-URI of the record, parsed as a <uri> reference. Callers decide whether the record type requires the field; this helper parses it whenever it is present.

    WarcRecord::to_bytes

    fn WarcRecord::to_bytes(self : WarcRecord) -> Bytes

    Serialize a record to its canonical byte form.

    The output is deterministic: field order and spelling are preserved, a single space follows each colon, and the content block is written byte-exact followed by CRLF CRLF. The block boundary is expressed solely by the Content-Length field, as the specification requires.

    WarcRecord::warc_date

    fn WarcRecord::warc_date(self : WarcRecord, record_index : Int64) -> Result[WarcDate, WarcError]

    The WARC-Date of the record, parsed to a structured date.

    WarcRecordType

    pub enum WarcRecordType {
    Warcinfo
    Response
    Resource
    Request
    Metadata
    Revisit
    Conversion
    Continuation
    } derive(Eq,
    Debug
    )

    The eight standard WARC record types of clause 6.

    WarcRecordType::parse

    fn WarcRecordType::parse(s : String) -> WarcRecordType?

    Parse a WARC-Type field value case-insensitively; unknown types yield None.

    WarcRecordType::type_name

    fn WarcRecordType::type_name(self : WarcRecordType) -> String

    The canonical lowercase spelling of the type.

    WARC_FIELDS_MIME_TYPE

    let WARC_FIELDS_MIME_TYPE : String

    The MIME type of the warcinfo field block (ISO 28500 clause 8).

    WARC_MIME_TYPE

    let WARC_MIME_TYPE : String

    The MIME type of a whole archive (ISO 28500 clause 8).

    WARC_VERSION_LINE

    let WARC_VERSION_LINE : String

    The version string that begins every WARC 1.1 record.

    all_digits

    fn all_digits(data : Bytes, start : Int, end : Int) -> Bool

    Whether data[start:end] consists solely of US-ASCII digits.

    all_kind_codes

    fn all_kind_codes() -> Array[String]

    Every kind code in declaration order. Stable across versions.

    all_record_type_codes

    fn all_record_type_codes() -> Array[String]

    Every record type code in declaration order (lowercase, as written in the specification).

    all_severity_codes

    fn all_severity_codes() -> Array[String]

    Every severity code in declaration order.

    all_stage_codes

    fn all_stage_codes() -> Array[String]

    Every stage code in declaration order. Stable across versions so callers can render error documentation without matching enums.

    append_decimal_digit

    fn append_decimal_digit(value : Int64, d : Int) -> Int64?

    Append one decimal digit to value, returning None on overflow. This is also the primitive used by the incremental streaming decoder when a Content-Length is split across input chunks.

    archive_stats

    fn archive_stats(a : WarcArchive) -> ArchiveStats

    Compute aggregate statistics over an archive.

    audit_archive

    fn audit_archive(a : WarcArchive) -> Array[WarcFinding]

    Run the audit over a whole archive and return every finding in file order.

    digit_count

    fn digit_count(data : Bytes, start : Int, end : Int) -> Int

    Number of leading ASCII digits in data[start:end].

    digit_value

    fn digit_value(b : Byte) -> Int

    The decimal value of an ASCII digit byte, or -1 if it is no digit.

    eq_bytes

    fn eq_bytes(data : Bytes, start : Int, end : Int, needle : String) -> Bool

    Whether data[start:end] byte-equals the UTF-8 encoding of needle (exact case-sensitive comparison).

    eq_ignore_ascii_case

    fn eq_ignore_ascii_case(data : Bytes, start : Int, end : Int, needle : String) -> Bool

    Whether data[start:end] byte-equals needle ignoring ASCII case.

    find_crlf

    fn find_crlf(data : Bytes, from : Int) -> Int

    Index of the CR that starts a CRLF pair at or after from, or -1.

    A trailing CR without a following LF is never reported: it cannot form a line ending and the caller must treat it as data (or EOF).

    index_of_byte

    fn index_of_byte(data : Bytes, from : Int, needle : Byte) -> Int

    Index of the first needle byte at or after from, or -1.

    is_cr

    fn is_cr(b : Byte) -> Bool

    ASCII carriage return (13).

    is_ctl

    fn is_ctl(b : Byte) -> Bool

    US-ASCII control character (0..31 or 127).

    is_digit

    fn is_digit(b : Byte) -> Bool

    ASCII digit '0'..'9'.

    is_ht

    fn is_ht(b : Byte) -> Bool

    ASCII horizontal tab (9).

    is_lf

    fn is_lf(b : Byte) -> Bool

    ASCII line feed (10).

    is_lower_ascii

    fn is_lower_ascii(b : Byte) -> Bool

    US-ASCII lower-case letter.

    is_separator

    fn is_separator(b : Byte) -> Bool

    Whether b is an RFC 2616 separator. Separators terminate a token and can never appear inside a field name.

    is_sp

    fn is_sp(b : Byte) -> Bool

    ASCII space (32).

    is_token_byte

    fn is_token_byte(b : Byte) -> Bool

    Whether b is a legal token byte: US-ASCII CHAR (0..127), not a control character, not a separator. Bytes above 127 are not tokens.

    is_upper_ascii

    fn is_upper_ascii(b : Byte) -> Bool

    US-ASCII upper-case letter.

    kind_of_code

    fn kind_of_code(code : String) -> WarcErrorKind?

    Resolve a machine-readable kind name back to a kind.

    lower_ascii

    fn lower_ascii(b : Byte) -> Byte

    Lower-case an US-ASCII letter; other bytes are returned unchanged.

    parse_archive

    fn parse_archive(data : Bytes, limits : Limits) -> Result[WarcArchive, WarcError]

    Parse a complete archive from a byte buffer.

    An empty buffer parses to an archive with no records. Bytes that follow the last complete record but cannot form another complete record — a truncated record or a version line that is not WARC/1.1 — are reported as TrailingGarbage; failures inside the first record keep their original, more specific diagnosis. Enforces max_archive_bytes and max_records.

    parse_content_length

    fn parse_content_length(fields : Array[WarcField], limits : Limits, record_index : Int64) -> Result[Int64, WarcError]

    Extract and parse the mandatory Content-Length field of a parsed record header.

    Fails with MissingRequiredField when absent, DuplicateField when repeated (both spellings being case-insensitive matches), InvalidContentLength on a non-decimal value, IntegerOverflow when the digit count or value overflows, and LimitExceeded when the declared block size exceeds max_block_bytes.

    parse_decimal

    fn parse_decimal(data : Bytes, start : Int, end : Int, max_digits : Int, context : String) -> Result[Int64, WarcError]

    Parse data[start:end], which must consist entirely of ASCII digits, as an Int64.

    • an empty span fails with InvalidContentLength (a mandatory numeric value is missing);
    • more than max_digits digits or a value beyond Int64 fails with IntegerOverflow.

    context names the field for error reporting (e.g. "Content-Length").

    parse_digest

    fn parse_digest(text : String, record_index : Int64) -> Result[WarcDigest, WarcError]

    Parse a algorithm:value digest field value.

    The algorithm must be a legal token, the separator a single colon, and the value non-empty and made only of base-encoding alphabet characters.

    parse_header

    fn parse_header(data : Bytes, start : Int, limits : Limits, record_index : Int64) -> Result[(Array[WarcField], Int), WarcError]

    Parse named fields from start up to and including the blank line that terminates the header.

    Returns the ordered fields and the byte offset just past the blank line, i.e. the start of the content block. Enforces the header size, field count, name length and value length limits.

    parse_record

    fn parse_record(data : Bytes, start : Int, limits : Limits, record_index : Int64) -> Result[(WarcRecord, Int), WarcError]

    Parse one complete record starting at start.

    Returns the record and the offset of the next record (just past the trailing CRLF CRLF). Fails with a structured WarcError on malformed framing: unterminated lines, bad version, bad header, missing/duplicate/overflowing Content-Length, a truncated block or a missing/invalid trailing CRLF CRLF separator.

    parse_uri_ref

    fn parse_uri_ref(s : String, record_index : Int64) -> Result[String, WarcError]

    Parse a <uri> field value and return the URI inside the angle brackets.

    Fails with InvalidUri when the value is not exactly <...> with a non-empty interior, or when the interior contains whitespace or control characters.

    parse_version_line

    fn parse_version_line(data : Bytes, start : Int, end : Int, record_index : Int64) -> Result[String, WarcError]

    Parse a raw version line (without its trailing CRLF) as the WARC version string.

    Any other spelling — including WARC/1.0, which differs in its framing rules — yields a structured InvalidVersion error.

    parse_warc_date

    fn parse_warc_date(s : String, record_index : Int64) -> Result[WarcDate, WarcError]

    Parse a WARC-Date value (without the field name or line ending).

    parse_warc_date_bytes

    fn parse_warc_date_bytes(data : Bytes, start : Int, end : Int, record_index : Int64) -> Result[WarcDate, WarcError]

    Parse a WARC-Date over a byte span. Shared with the streaming decoder, which works on raw bytes rather than field strings.

    record_type_of_code

    fn record_type_of_code(code : String) -> WarcRecordType?

    Resolve a lowercase type code back to a record type.

    scan_line

    fn scan_line(data : Bytes, from : Int, record_index : Int64) -> Result[(Int, Int), WarcError]

    Scan one CRLF-terminated line starting at from.

    Returns (line_start, line_end) where [line_start, line_end) excludes the trailing CRLF. Fails with UnexpectedEof when the buffer ends without a line terminator and with InvalidSeparator on a bare CR or bare LF.

    segment_errors

    fn segment_errors(rec : WarcRecord, index : Int64) -> Array[WarcError]

    Segmentation field syntax errors of one record, if any.

    severity_of_code

    fn severity_of_code(code : String) -> FindingSeverity?

    Resolve a severity code back to a severity.

    stage_of_code

    fn stage_of_code(code : String) -> WarcErrorStage?

    Resolve a machine-readable stage name back to a stage.

    starts_with

    fn starts_with(data : Bytes, prefix : Bytes) -> Bool

    Whether data starts with the byte sequence prefix.

    valid_field_name

    fn valid_field_name(data : Bytes, start : Int, end : Int) -> Bool

    Whether data[start:end] forms a valid field name (1*token).

    valid_ip_address

    fn valid_ip_address(s : String) -> Bool

    Validate an IP address value: a strict dotted-quad IPv4 or a permissive IPv6 form. Anything else is invalid.

    validate_archive

    fn validate_archive(a : WarcArchive) -> Array[WarcError]

    Validate a whole archive: per-record semantic validation, unique WARC-Record-IDs, and segment-sequence consistency. Errors are reported in file order.

    validate_record

    fn validate_record(rec : WarcRecord, index : Int64) -> Array[WarcError]

    Validate one record and return every semantic violation found, in discovery order.