moon-warc

Streaming WARC 1.1 parser, writer, validator, indexer and audit toolkit for MoonBit.

warc
web-archive
archive
parser
streaming
iso28500
webarchiving
moonbit
moon add xiaojing012/moon-warc@0.1.0
Download zip
Version
0.1.0
License
Apache-2.0
Last updated
6 hours ago
Downloads
1
README

#moon-warc

A streaming WARC 1.1 (ISO 28500) parser, writer, validator, indexer and audit toolkit for MoonBit.

  • Module: xiaojing012/moon-warc
  • Version: 0.1.0
  • Repository: https://github.com/xiaojing012/moon-warc
  • Applicant / Maintainer: 宋晓静 / xiaojing012
  • Mooncakes: xiaojing012/moon-warc

#Overview

WARC (Web ARChive) is the ISO-standard container format (ISO 28500:2017) used by web archives, crawlers and digital-preservation systems to store billions of harvested resources. A WARC file concatenates any number of WARC records; each record carries a set of named fields plus an arbitrary binary content block.

moon-warc implements the record framing and semantic layer of WARC 1.1 in pure MoonBit: binary-safe record framing, ordered named fields, buffered and incremental streaming parsing, deterministic writing, semantic validation, segmentation support, in-memory indexing, and an advisory audit engine — with structured errors, resource limits and cross-target tests along the way.

#Install

moon add xiaojing012/moon-warc

Add the package alias to the consumer's moon.pkg:

import {
"xiaojing012/moon-warc" @warc,
}

Parse an in-memory WARC archive with structured limits and errors:

match @warc.parse_archive(data, @warc.Limits::default()) {
Ok(archive) => println("records: \{archive.record_count()}")
Err(error) => println(error.to_string())
}

#Why WARC

  • Web archive interoperability — read and write the format used by the Internet Archive, national libraries and Common Crawl;
  • Binary-safe streaming — blocks are arbitrary bytes; parsing never depends on scanning for version strings;
  • Archival data processing — record framing, validation, querying and statistics for offline analysis pipelines;
  • Reproducible data pipelines — a deterministic writer/builder with no hidden clock or network access;
  • MoonBit ecosystem infrastructure — a foundation for crawler storage layers, migration tools and archive inspection utilities.

#Record Structure

warc-record = header CRLF block CRLF CRLF header = version warc-fields version = "WARC/1.1" CRLF warc-fields = *named-field CRLF block = *OCTET

#CLI and Examples

A small command-line entry point (cli/) operates on a built-in demo archive (this build has no file-system I/O):

moon run cli help moon run cli parse | validate | inspect | stats | audit moon run cli query http://example.org/a moon run cli build

Runnable examples (examples/) demonstrate each slice of the library:

moon run examples parse | build | validate | index | audit

#Development Status

Released version 0.1.0. The complete record model, framing, buffered parser, streaming decoder, writer/builder, semantic validator, segmentation, indexing and statistics, and the audit engine are implemented and covered by 208 tests on all four MoonBit targets (wasm, wasm-gc, js, native); see docs/ for the specification map, architecture notes, limitations and the reproducibility guide. verify_all.ps1 runs the whole verification in one go.

#License

Apache License 2.0. See LICENSE.

#
ArchiveStats

pub struct ArchiveStats {
record_count : Int
total_record_bytes : Int64
total_block_bytes : Int64
type_counts : Array[(String, Int)]
distinct_targets : Array[String]
earliest : WarcDate?
latest : WarcDate?
}

Aggregate statistics over an archive.

#
ArchiveStats::distinct_target_count

fn ArchiveStats::distinct_target_count(self : ArchiveStats) -> Int

The number of distinct WARC-Target-URI values.

#
ArchiveStats::earliest_date

fn ArchiveStats::earliest_date(self : ArchiveStats) -> WarcDate?

The earliest WARC-Date seen, when any record carried a parseable one.

#
ArchiveStats::latest_date

fn ArchiveStats::latest_date(self : ArchiveStats) -> WarcDate?

The latest WARC-Date seen, when any record carried a parseable one.

#
ArchiveStats::record_count

fn ArchiveStats::record_count(self : ArchiveStats) -> Int

The number of records in the archive.

#
ArchiveStats::total_block_bytes

fn ArchiveStats::total_block_bytes(self : ArchiveStats) -> Int64

The total size of all content blocks in octets.

#
ArchiveStats::total_record_bytes

fn ArchiveStats::total_record_bytes(self : ArchiveStats) -> Int64

The total serialized size of all records in octets.

#
ArchiveStats::type_count

fn ArchiveStats::type_count(self : ArchiveStats, name : String) -> Int

The number of records of the given type name (lowercase); records of unknown WARC-Type count as unknown.

#
FindingSeverity

pub enum FindingSeverity {
Info
Warning
} derive(Eq,
Debug
)

The severity of an audit finding.

#
FindingSeverity::severity_name

fn FindingSeverity::severity_name(self : FindingSeverity) -> String

The lowercase machine-readable severity name.

#
Limits

pub struct Limits {
max_archive_bytes : Int64
max_record_bytes : Int64
max_header_bytes : Int64
max_header_count : Int
max_field_name_bytes : Int
max_field_value_bytes : Int
max_block_bytes : Int64
max_records : Int
max_content_length_digits : Int
}

#
Limits::default

fn Limits::default() -> Limits

Balanced defaults for general-purpose archive processing.

#
Limits::permissive

fn Limits::permissive() -> Limits

Near-unbounded ceilings for controlled offline processing.

#
Limits::strict

fn Limits::strict() -> Limits

Tight bounds for untrusted input and memory-constrained targets.

#
Limits::with_max_archive_bytes

fn Limits::with_max_archive_bytes(self : Limits, n : Int64) -> Limits

A copy of these limits with max_archive_bytes replaced.

#
Limits::with_max_block_bytes

fn Limits::with_max_block_bytes(self : Limits, n : Int64) -> Limits

A copy of these limits with max_block_bytes replaced.

#
Limits::with_max_content_length_digits

fn Limits::with_max_content_length_digits(self : Limits, n : Int) -> Limits

A copy of these limits with max_content_length_digits replaced.

#
Limits::with_max_field_name_bytes

fn Limits::with_max_field_name_bytes(self : Limits, n : Int) -> Limits

A copy of these limits with max_field_name_bytes replaced.

#
Limits::with_max_field_value_bytes

fn Limits::with_max_field_value_bytes(self : Limits, n : Int) -> Limits

A copy of these limits with max_field_value_bytes replaced.

#
Limits::with_max_header_bytes

fn Limits::with_max_header_bytes(self : Limits, n : Int64) -> Limits

A copy of these limits with max_header_bytes replaced.

#
Limits::with_max_header_count

fn Limits::with_max_header_count(self : Limits, n : Int) -> Limits

A copy of these limits with max_header_count replaced.

#
Limits::with_max_record_bytes

fn Limits::with_max_record_bytes(self : Limits, n : Int64) -> Limits

A copy of these limits with max_record_bytes replaced.

#
Limits::with_max_records

fn Limits::with_max_records(self : Limits, n : Int) -> Limits

A copy of these limits with max_records replaced.

#
PendingRecord

type PendingRecord

A partially parsed record whose header is complete but whose block and trailing CRLF CRLF have not fully arrived.

#
SegmentInfo

pub struct SegmentInfo {
number : Int64?
origin_id : String?
total_length : Int64?
}

The parsed segmentation fields of a record. Absent fields are None.

#
SegmentInfo::new

fn SegmentInfo::new(number : Int64?, origin_id : String?, total_length : Int64?) -> SegmentInfo

Construct segment info from optional parts.

#
SegmentInfo::number

fn SegmentInfo::number(self : SegmentInfo) -> Int64?

The WARC-Segment-Number value, when present.

#
SegmentInfo::origin_id

fn SegmentInfo::origin_id(self : SegmentInfo) -> String?

The interior URI of WARC-Segment-Origin-ID, when present.

#
SegmentInfo::total_length

fn SegmentInfo::total_length(self : SegmentInfo) -> Int64?

The WARC-Segment-Total-Length value, when present.

#
WarcArchive

pub struct WarcArchive {
records : Array[WarcRecord]
}

A fully parsed in-memory WARC archive: the records in file order.

#
WarcArchive::record

fn WarcArchive::record(self : WarcArchive, index : Int) -> WarcRecord?

The record at index, or None when out of range.

#
WarcArchive::record_count

fn WarcArchive::record_count(self : WarcArchive) -> Int

The number of records in the archive.

#
WarcArchive::total_block_bytes

fn WarcArchive::total_block_bytes(self : WarcArchive) -> Int64

The total number of octets across all content blocks.

#
WarcBuilder

pub struct WarcBuilder {
version : String
fields : Array[WarcField]
}

An incremental WARC record builder.

#
WarcBuilder::build

fn WarcBuilder::build(self : WarcBuilder, block : Bytes) -> Result[WarcRecord, WarcError]

Build the record around a content block.

A missing Content-Length is generated from the block size; a present one must match it exactly (and must not be repeated). Field names must be legal tokens and values must be single-line, so the result always serializes to a parseable record.

#
WarcBuilder::field

fn WarcBuilder::field(self : WarcBuilder, name : String, value : String) -> WarcBuilder

Append a named field. The original spelling is preserved; names are validated at build time and values must fit on one header line.

#
WarcBuilder::new

Create a builder for a WARC/1.1 record with no fields.

#
WarcBuilder::record_id

fn WarcBuilder::record_id(self : WarcBuilder, uri : String) -> WarcBuilder

Append a WARC-Record-ID field (value in <uri> form).

#
WarcBuilder::target_uri

fn WarcBuilder::target_uri(self : WarcBuilder, uri : String) -> WarcBuilder

Append a WARC-Target-URI field (value in <uri> form).

#
WarcBuilder::version

fn WarcBuilder::version(self : WarcBuilder, v : String) -> WarcBuilder

Set the version line. Only WARC/1.1 is accepted at build time.

#
WarcBuilder::warc_date

fn WarcBuilder::warc_date(self : WarcBuilder, date : String) -> WarcBuilder

Append a WARC-Date field (W3CDTF UTC timestamp).

#
WarcBuilder::with_type

fn WarcBuilder::with_type(self : WarcBuilder, t : String) -> WarcBuilder

Append a WARC-Type field.

#
WarcDate

pub struct WarcDate {
year : Int
month : Int?
day : Int?
hour : Int?
minute : Int?
second : Int?
fraction : String
} derive(Eq,
Debug
)

A parsed WARC-Date. Optional components are absent at coarser granularities; fraction is the exact fractional-second digit string ("" when there are no fractional seconds).

#
WarcDate::day

fn WarcDate::day(self : WarcDate) -> Int?

The day, present at YYYY-MM-DD granularity and finer.

#
WarcDate::has_fraction

fn WarcDate::has_fraction(self : WarcDate) -> Bool

Whether this is the finest granularity: date plus time plus fractional seconds.

#
WarcDate::has_time

fn WarcDate::has_time(self : WarcDate) -> Bool

Whether a time of day is present.

#
WarcDate::month

fn WarcDate::month(self : WarcDate) -> Int?

The month, present at YYYY-MM granularity and finer.

#
WarcDate::year

fn WarcDate::year(self : WarcDate) -> Int

The four-digit year.

#
WarcDecoder

pub struct WarcDecoder {
limits : Limits
bytes : Bytes
start : Int
pos : Int64
records_done : Int
total_fed : Int64
pending : PendingRecord?
error : WarcError?
}

A streaming WARC decoder.

#
WarcDecoder::feed

fn WarcDecoder::feed(self : WarcDecoder, chunk : Bytes) -> Result[Array[WarcRecord], WarcError]

Feed one chunk of the input stream.

Returns the records completed by this chunk (possibly none). The first error makes the decoder terminal: later feed/finish calls return the same error. Enforces max_archive_bytes across the whole stream, max_records at each record boundary and all per-record limits through the shared parsing functions.

#
WarcDecoder::finish

fn WarcDecoder::finish(self : WarcDecoder) -> Result[Unit, WarcError]

Signal end of input.

Returns Ok when the stream ended exactly at a record boundary and every record was complete. Bytes that remain at end of input — a truncated record or anything after the last complete record that cannot start a new one — are reported with the same trailing-garbage semantics as parse_archive.

#
WarcDecoder::new

fn WarcDecoder::new(limits : Limits) -> WarcDecoder

Create a decoder with the given limits.

#
WarcDigest

pub struct WarcDigest {
algorithm : String
value : String
}

A parsed labelled digest such as sha1:B2QYELGDQWXWAWJ4OA2GSCBXDAOB7UHR.

#
WarcDigest::algorithm

fn WarcDigest::algorithm(self : WarcDigest) -> String

The algorithm label (e.g. sha1).

#
WarcDigest::new

fn WarcDigest::new(algorithm : String, value : String) -> WarcDigest

Construct a labelled digest from its parts.

#
WarcDigest::to_string

fn WarcDigest::to_string(self : WarcDigest) -> String

The canonical algorithm:value rendering.

#
WarcDigest::value

fn WarcDigest::value(self : WarcDigest) -> String

The encoded digest value.

#
WarcError

pub struct WarcError {
stage : WarcErrorStage
kind : WarcErrorKind
byte_offset : Int64
record_index : Int64
context : String
}

A structured WARC error.

#
WarcError::new

fn WarcError::new(stage : WarcErrorStage, kind : WarcErrorKind, byte_offset : Int64, record_index : Int64, context : String) -> WarcError

Create an error at a byte offset within a specific record.

#
WarcError::to_string

fn WarcError::to_string(self : WarcError) -> String

A human-readable one-line description of the error.

#
WarcErrorKind

pub enum WarcErrorKind {
UnexpectedEof
InvalidVersion
MissingColon
InvalidFieldName
InvalidUtf8
InvalidContentLength
IntegerOverflow
ContentLengthMismatch
MissingRequiredField
DuplicateField
InvalidDate
InvalidUri
InvalidDigest
InvalidRecordType
InvalidSeparator
TrailingGarbage
LimitExceeded
InvalidFieldValue
MisplacedField
InvalidIpAddress
} derive(Eq,
Debug
)

The precise failure kind. Parse-time failures and semantic validation failures share this model; advisory findings are reported separately by the audit engine.

#
WarcErrorKind::kind_name

fn WarcErrorKind::kind_name(self : WarcErrorKind) -> String

The lowercase machine-readable kind name.

#
WarcErrorStage

pub enum WarcErrorStage {
Input
Version
Header
Field
Number
ContentLength
Block
Separator
Date
Uri
Digest
Record
Archive
Segment
Limit
Builder
} derive(Eq,
Debug
)

The processing stage in which an error occurred.

#
WarcErrorStage::stage_name

fn WarcErrorStage::stage_name(self : WarcErrorStage) -> String

The lowercase machine-readable stage name.

#
WarcField

pub struct WarcField {
name : String
value : String
}

One named field of a WARC record header.

Per ISO 28500 clause 5, field names are compared case-insensitively; the original spelling is kept so callers can inspect or reproduce it.

#
WarcField::new

fn WarcField::new(name : String, value : String) -> WarcField

Construct a named field from its raw spelling and value.

#
WarcFinding

pub struct WarcFinding {
severity : FindingSeverity
code : String
record_index : Int64
context : String
}

An advisory audit finding anchored to a record.

#
WarcFinding::code

fn WarcFinding::code(self : WarcFinding) -> String

The machine-readable finding code.

#
WarcFinding::context

fn WarcFinding::context(self : WarcFinding) -> String

A human-readable description of the finding.

#
WarcFinding::new

fn WarcFinding::new(severity : FindingSeverity, code : String, record_index : Int64, context : String) -> WarcFinding

Construct a finding.

#
WarcFinding::record_index

fn WarcFinding::record_index(self : WarcFinding) -> Int64

The record the finding is anchored to.

#
WarcFinding::severity

fn WarcFinding::severity(self : WarcFinding) -> FindingSeverity

The finding's severity.

#
WarcFinding::to_string

fn WarcFinding::to_string(self : WarcFinding) -> String

A one-line rendering of the finding.

#
WarcIndex

pub struct WarcIndex {
by_id : Array[(String, Int)]
by_target : Array[(String, Int)]
by_type : Array[(String, Int)]
}

An in-memory index over an archive's records.

#
WarcIndex::build

fn WarcIndex::build(a : WarcArchive) -> WarcIndex

Build an index over an archive. Records whose keys are absent or malformed are simply not indexed.

#
WarcIndex::by_record_id

fn WarcIndex::by_record_id(self : WarcIndex, id : String) -> Int?

The index of the first record whose WARC-Record-ID interior is id, or None when no record carries it.

#
WarcIndex::by_target_uri

fn WarcIndex::by_target_uri(self : WarcIndex, uri : String) -> Array[Int]

The indices of all records whose WARC-Target-URI interior is uri, in file order.

#
WarcIndex::by_type

fn WarcIndex::by_type(self : WarcIndex, t : WarcRecordType) -> Array[Int]

The indices of all records of the given type, in file order.

#
WarcIndex::size

fn WarcIndex::size(self : WarcIndex) -> Int

The total number of indexed keys across all three key kinds.

#
WarcRecord

pub struct WarcRecord {
version : String
fields : Array[WarcField]
block : Bytes
}

A fully framed WARC record: version line, ordered named fields and the raw binary content block.

#
WarcRecord::block_length

fn WarcRecord::block_length(self : WarcRecord) -> Int

The number of octets in the record's content block.

#
WarcRecord::declared_content_length

fn WarcRecord::declared_content_length(self : WarcRecord) -> String?

The raw Content-Length field value, when present.

#
WarcRecord::field_all

fn WarcRecord::field_all(self : WarcRecord, name : String) -> Array[String]

All values of fields with a case-insensitive name match, in input order (relevant for repeatable fields such as WARC-Concurrent-To).

#
WarcRecord::field_count

fn WarcRecord::field_count(self : WarcRecord, name : String) -> Int

How many fields carry a case-insensitive name match.

#
WarcRecord::field_first

fn WarcRecord::field_first(self : WarcRecord, name : String) -> String?

The value of the first field with a case-insensitive name match.

#
WarcRecord::field_has

fn WarcRecord::field_has(self : WarcRecord, name : String) -> Bool

Whether at least one field carries a case-insensitive name match.

#
WarcRecord::new

fn WarcRecord::new(version : String, fields : Array[WarcField], block : Bytes) -> WarcRecord

Construct a WARC record from its parts.

#
WarcRecord::record_id

fn WarcRecord::record_id(self : WarcRecord, record_index : Int64) -> Result[String, WarcError]

The WARC-Record-ID of the record: the mandatory field, parsed as a <uri> reference. Returns the URI inside the angle brackets.

#
WarcRecord::record_type

fn WarcRecord::record_type(self : WarcRecord) -> WarcRecordType?

The record type declared by WARC-Type, or None when the field is absent or carries an unknown type.

#
WarcRecord::require_field

fn WarcRecord::require_field(self : WarcRecord, name : String, record_index : Int64) -> Result[String, WarcError]

The value of a mandatory field, or a structured error naming it.

#
WarcRecord::segment_info

fn WarcRecord::segment_info(self : WarcRecord, index : Int64) -> Result[SegmentInfo, WarcError]

Parse the segmentation fields of a record.

Fails when a field is present but malformed: a non-decimal or non-positive WARC-Segment-Number, a WARC-Segment-Origin-ID that is not a <uri>, or a non-decimal WARC-Segment-Total-Length.

#
WarcRecord::target_uri

fn WarcRecord::target_uri(self : WarcRecord, record_index : Int64) -> Result[String, WarcError]

The WARC-Target-URI of the record, parsed as a <uri> reference. Callers decide whether the record type requires the field; this helper parses it whenever it is present.

#
WarcRecord::to_bytes

fn WarcRecord::to_bytes(self : WarcRecord) -> Bytes

Serialize a record to its canonical byte form.

The output is deterministic: field order and spelling are preserved, a single space follows each colon, and the content block is written byte-exact followed by CRLF CRLF. The block boundary is expressed solely by the Content-Length field, as the specification requires.

#
WarcRecord::warc_date

fn WarcRecord::warc_date(self : WarcRecord, record_index : Int64) -> Result[WarcDate, WarcError]

The WARC-Date of the record, parsed to a structured date.

#
WarcRecordType

pub enum WarcRecordType {
Warcinfo
Response
Resource
Request
Metadata
Revisit
Conversion
Continuation
} derive(Eq,
Debug
)

The eight standard WARC record types of clause 6.

#
WarcRecordType::parse

fn WarcRecordType::parse(s : String) -> WarcRecordType?

Parse a WARC-Type field value case-insensitively; unknown types yield None.

#
WarcRecordType::type_name

fn WarcRecordType::type_name(self : WarcRecordType) -> String

The canonical lowercase spelling of the type.

#
WARC_FIELDS_MIME_TYPE

let WARC_FIELDS_MIME_TYPE : String

The MIME type of the warcinfo field block (ISO 28500 clause 8).

#
WARC_MIME_TYPE

let WARC_MIME_TYPE : String

The MIME type of a whole archive (ISO 28500 clause 8).

#
WARC_VERSION_LINE

let WARC_VERSION_LINE : String

The version string that begins every WARC 1.1 record.

#
all_digits

fn all_digits(data : Bytes, start : Int, end : Int) -> Bool

Whether data[start:end] consists solely of US-ASCII digits.

#
all_kind_codes

fn all_kind_codes() -> Array[String]

Every kind code in declaration order. Stable across versions.

#
all_record_type_codes

fn all_record_type_codes() -> Array[String]

Every record type code in declaration order (lowercase, as written in the specification).

#
all_severity_codes

fn all_severity_codes() -> Array[String]

Every severity code in declaration order.

#
all_stage_codes

fn all_stage_codes() -> Array[String]

Every stage code in declaration order. Stable across versions so callers can render error documentation without matching enums.

#
append_decimal_digit

fn append_decimal_digit(value : Int64, d : Int) -> Int64?

Append one decimal digit to value, returning None on overflow. This is also the primitive used by the incremental streaming decoder when a Content-Length is split across input chunks.

#
archive_stats

fn archive_stats(a : WarcArchive) -> ArchiveStats

Compute aggregate statistics over an archive.

#
audit_archive

fn audit_archive(a : WarcArchive) -> Array[WarcFinding]

Run the audit over a whole archive and return every finding in file order.

#
digit_count

fn digit_count(data : Bytes, start : Int, end : Int) -> Int

Number of leading ASCII digits in data[start:end].

#
digit_value

fn digit_value(b : Byte) -> Int

The decimal value of an ASCII digit byte, or -1 if it is no digit.

#
eq_bytes

fn eq_bytes(data : Bytes, start : Int, end : Int, needle : String) -> Bool

Whether data[start:end] byte-equals the UTF-8 encoding of needle (exact case-sensitive comparison).

#
eq_ignore_ascii_case

fn eq_ignore_ascii_case(data : Bytes, start : Int, end : Int, needle : String) -> Bool

Whether data[start:end] byte-equals needle ignoring ASCII case.

#
find_crlf

fn find_crlf(data : Bytes, from : Int) -> Int

Index of the CR that starts a CRLF pair at or after from, or -1.

A trailing CR without a following LF is never reported: it cannot form a line ending and the caller must treat it as data (or EOF).

#
index_of_byte

fn index_of_byte(data : Bytes, from : Int, needle : Byte) -> Int

Index of the first needle byte at or after from, or -1.

#
is_cr

fn is_cr(b : Byte) -> Bool

ASCII carriage return (13).

#
is_ctl

fn is_ctl(b : Byte) -> Bool

US-ASCII control character (0..31 or 127).

#
is_digit

fn is_digit(b : Byte) -> Bool

ASCII digit '0'..'9'.

#
is_ht

fn is_ht(b : Byte) -> Bool

ASCII horizontal tab (9).

#
is_lf

fn is_lf(b : Byte) -> Bool

ASCII line feed (10).

#
is_lower_ascii

fn is_lower_ascii(b : Byte) -> Bool

US-ASCII lower-case letter.

#
is_separator

fn is_separator(b : Byte) -> Bool

Whether b is an RFC 2616 separator. Separators terminate a token and can never appear inside a field name.

#
is_sp

fn is_sp(b : Byte) -> Bool

ASCII space (32).

#
is_token_byte

fn is_token_byte(b : Byte) -> Bool

Whether b is a legal token byte: US-ASCII CHAR (0..127), not a control character, not a separator. Bytes above 127 are not tokens.

#
is_upper_ascii

fn is_upper_ascii(b : Byte) -> Bool

US-ASCII upper-case letter.

#
kind_of_code

fn kind_of_code(code : String) -> WarcErrorKind?

Resolve a machine-readable kind name back to a kind.

#
lower_ascii

fn lower_ascii(b : Byte) -> Byte

Lower-case an US-ASCII letter; other bytes are returned unchanged.

#
parse_archive

fn parse_archive(data : Bytes, limits : Limits) -> Result[WarcArchive, WarcError]

Parse a complete archive from a byte buffer.

An empty buffer parses to an archive with no records. Bytes that follow the last complete record but cannot form another complete record — a truncated record or a version line that is not WARC/1.1 — are reported as TrailingGarbage; failures inside the first record keep their original, more specific diagnosis. Enforces max_archive_bytes and max_records.

#
parse_content_length

fn parse_content_length(fields : Array[WarcField], limits : Limits, record_index : Int64) -> Result[Int64, WarcError]

Extract and parse the mandatory Content-Length field of a parsed record header.

Fails with MissingRequiredField when absent, DuplicateField when repeated (both spellings being case-insensitive matches), InvalidContentLength on a non-decimal value, IntegerOverflow when the digit count or value overflows, and LimitExceeded when the declared block size exceeds max_block_bytes.

#
parse_decimal

fn parse_decimal(data : Bytes, start : Int, end : Int, max_digits : Int, context : String) -> Result[Int64, WarcError]

Parse data[start:end], which must consist entirely of ASCII digits, as an Int64.

  • an empty span fails with InvalidContentLength (a mandatory numeric value is missing);
  • more than max_digits digits or a value beyond Int64 fails with IntegerOverflow.

context names the field for error reporting (e.g. "Content-Length").

#
parse_digest

fn parse_digest(text : String, record_index : Int64) -> Result[WarcDigest, WarcError]

Parse a algorithm:value digest field value.

The algorithm must be a legal token, the separator a single colon, and the value non-empty and made only of base-encoding alphabet characters.

#
parse_header

fn parse_header(data : Bytes, start : Int, limits : Limits, record_index : Int64) -> Result[(Array[WarcField], Int), WarcError]

Parse named fields from start up to and including the blank line that terminates the header.

Returns the ordered fields and the byte offset just past the blank line, i.e. the start of the content block. Enforces the header size, field count, name length and value length limits.

#
parse_record

fn parse_record(data : Bytes, start : Int, limits : Limits, record_index : Int64) -> Result[(WarcRecord, Int), WarcError]

Parse one complete record starting at start.

Returns the record and the offset of the next record (just past the trailing CRLF CRLF). Fails with a structured WarcError on malformed framing: unterminated lines, bad version, bad header, missing/duplicate/overflowing Content-Length, a truncated block or a missing/invalid trailing CRLF CRLF separator.

#
parse_uri_ref

fn parse_uri_ref(s : String, record_index : Int64) -> Result[String, WarcError]

Parse a <uri> field value and return the URI inside the angle brackets.

Fails with InvalidUri when the value is not exactly <...> with a non-empty interior, or when the interior contains whitespace or control characters.

#
parse_version_line

fn parse_version_line(data : Bytes, start : Int, end : Int, record_index : Int64) -> Result[String, WarcError]

Parse a raw version line (without its trailing CRLF) as the WARC version string.

Any other spelling — including WARC/1.0, which differs in its framing rules — yields a structured InvalidVersion error.

#
parse_warc_date

fn parse_warc_date(s : String, record_index : Int64) -> Result[WarcDate, WarcError]

Parse a WARC-Date value (without the field name or line ending).

#
parse_warc_date_bytes

fn parse_warc_date_bytes(data : Bytes, start : Int, end : Int, record_index : Int64) -> Result[WarcDate, WarcError]

Parse a WARC-Date over a byte span. Shared with the streaming decoder, which works on raw bytes rather than field strings.

#
record_type_of_code

fn record_type_of_code(code : String) -> WarcRecordType?

Resolve a lowercase type code back to a record type.

#
scan_line

fn scan_line(data : Bytes, from : Int, record_index : Int64) -> Result[(Int, Int), WarcError]

Scan one CRLF-terminated line starting at from.

Returns (line_start, line_end) where [line_start, line_end) excludes the trailing CRLF. Fails with UnexpectedEof when the buffer ends without a line terminator and with InvalidSeparator on a bare CR or bare LF.

#
segment_errors

fn segment_errors(rec : WarcRecord, index : Int64) -> Array[WarcError]

Segmentation field syntax errors of one record, if any.

#
severity_of_code

fn severity_of_code(code : String) -> FindingSeverity?

Resolve a severity code back to a severity.

#
stage_of_code

fn stage_of_code(code : String) -> WarcErrorStage?

Resolve a machine-readable stage name back to a stage.

#
starts_with

fn starts_with(data : Bytes, prefix : Bytes) -> Bool

Whether data starts with the byte sequence prefix.

#
valid_field_name

fn valid_field_name(data : Bytes, start : Int, end : Int) -> Bool

Whether data[start:end] forms a valid field name (1*token).

#
valid_ip_address

fn valid_ip_address(s : String) -> Bool

Validate an IP address value: a strict dotted-quad IPv4 or a permissive IPv6 form. Anything else is invalid.

#
validate_archive

fn validate_archive(a : WarcArchive) -> Array[WarcError]

Validate a whole archive: per-record semantic validation, unique WARC-Record-IDs, and segment-sequence consistency. Errors are reported in file order.

#
validate_record

fn validate_record(rec : WarcRecord, index : Int64) -> Array[WarcError]

Validate one record and return every semantic violation found, in discovery order.