multipart-stream

    Streaming multipart/form-data parser and writer for MoonBit.

    multipart
    form-data
    streaming
    http
    upload
    Download zip
    Version
    0.1.1
    License
    Apache-2.0
    Last updated
    2 months ago
    Downloads
    46

    #multipart-stream

    面向 MoonBit 的二进制安全 multipart/form-data 流式编解码与上传编排库。它把任意 分块的 HTTP body 转换为有序 part 事件,也能按显式生命周期生成 multipart wire;核心 以 Bytes 处理正文,统一约束 MIME 语法、boundary、资源上限、事务 sink 和诊断,避免 应用把整个上传请求或文件缓存到内存。

    #已实现的完整功能范围

    • 严格 Content-Type、Content-Disposition、quoted parameter、filename* 与 _charset_ 处理,拒绝 header 注入和编码后的控制字符;
    • 基于 KMP 的增量 boundary matcher,支持任意切块、regular/closing delimiter、 transport-padding、近似前缀和有界 retained 状态;
    • PartBegin、PartData、PartEnd、MessageEnd 有序事件、二进制正文、严格 EOF、 sticky failure、limits、脱敏 diagnostic 与 parser stats;
    • 增量 MultipartWriter、canonical headers、boundary collision 检查、精确 Content-Length plan 与长度校验 PlannedEmitter;
    • 有界字段 collector、callback source/sink、legacy multipart/mixed 增量展开;
    • 事务 UploadPipeline、field/file/aggregate quota、exactly-once abort、CRC-32 审计和 peak-chunk metrics;
    • Native mpstream CLI、basic/limits 示例、Go mime/multipart 双向互操作与真实 curl -F 客户端验证。

    库核心在 Wasm、Wasm-GC、JavaScript、Native target 上测试;CLI 仅支持 Native。

    #安装

    该库已发布到 mooncakes.io。在使用方项目根目录添加稳定的 v0.1.1:

    moon add longhuanlin/multipart-stream@0.1.1

    在 MoonBit 包的 moon.pkg 中导入根 facade:

    import {
    "longhuanlin/multipart-stream" @multipart,
    }

    #快速开始

    在项目根目录执行:

    moon run src/examples/basic --target native

    输出:

    Content-Type: multipart/form-data; boundary=basic-demo begin greeting data 13 bytes end #0 message complete

    使用 CLI 生成一个字段和文件 part:

    moon run src/cmd/mpstream --target native -- \ generate --boundary demo --field title=hello \ --file blob=@test/interop/curl_fixture.txt

    CLI 输出 content-type 和 multipart wire 的 hex;再把两者交给 inspect,可按小 chunk 解析并只输出 metadata、body size 和事件顺序,不输出正文。

    mpstream validate-boundary <value> mpstream generate --boundary <value> [--field name=value] [--file name=@path] ... mpstream inspect --content-type <value> (--hex <wire-hex> | --file <path>) [--chunk-size N] mpstream version

    #库 API

    let writer = MultipartWriter::new("demo", default_limits())
    let meta : PartMeta = {
    index: 0,
    name: "title",
    filename: None,
    content_type: Some("text/plain; charset=utf-8"),
    headers: [],
    }
    let wire = @buffer.Buffer()
    wire.write_bytes(writer.begin_part(meta))
    wire.write_bytes(writer.write(b"hello"))
    wire.write_bytes(writer.end_part())
    wire.write_bytes(writer.finish())

    let parser = MultipartParser::new("demo", default_limits())
    let events = parser.feed(wire.to_bytes())
    let completed = parser.finish() // [MessageEnd]

    大型上传可将 bytes chunk 依次送入 MultipartParser::feed 或 UploadPipeline::feed; 只有在 finish 成功后才将事务 sink 的结果视为可提交。FieldCollector 只收集受限的小 字段,文件正文应交给 callback sink 或 upload pipeline 立即消费。

    #验证

    moon fmt --check moon check --target all --warn-list +73 --deny-warn moon test --target all --warn-list -68-1+73 moon build src/cmd/mpstream --target native --warn-list +73 --deny-warn moon info --target all bash test/cli_smoke.sh bash test/interop/run.sh --required bash bench/run.sh

    当前每个 target 均通过 86 个 MoonBit 测试;覆盖 MIME/boundary、切块、EOF、limits、 writer/plan/emitter、form/mixed/pipeline、mutation、facade 和 CLI。Go oracle 比较两个 方向的 metadata、顺序和正文 bytes,curl -F 路径验证真实客户端请求。

    #设计边界

    • 协议范围是 multipart/form-data,另有受限的 legacy multipart/mixed 兼容层;不实现 完整邮件 MIME、Content-Transfer-Encoding、HTTP server/client、TCP 或 TLS;
    • parser、writer 和 pipeline 是同步单 owner 状态机;同一实例不能并发或重入调用;
    • PartData 当前是 owned Bytes;borrowed/zero-copy event 属于后续演进;
    • 真实文件系统、网络、对象存储、backpressure 和安全摘要由宿主实现;CRC-32 仅用于非安全 审计,filename 也始终是不可信 metadata;
    • CLI 的 --file 是诊断便利入口,会整体读取指定文件;生产大文件应使用增量 API;
    • 模块名为 longhuanlin/multipart-stream,当前稳定发布版为 v0.1.1。

    #文档

    #许可证

    本项目采用 Apache License 2.0。

    AuditSummary

    Public zero-dependency upload routing pipeline and transactional sink API.

    AuditedSink

    Public zero-dependency upload routing pipeline and transactional sink API.

    ChunkSourceAdapter

    Public small-field collection and transport-neutral streaming adapters.

    DecodedField

    Public small-field collection and transport-neutral streaming adapters.

    FieldCollector

    Public small-field collection and transport-neutral streaming adapters.

    FieldCollectorLimits

    Public small-field collection and transport-neutral streaming adapters.

    FormError

    Public small-field collection and transport-neutral streaming adapters.

    FormField

    Public small-field collection and transport-neutral streaming adapters.

    HeaderField

    Public protocol model and errors.

    LimitKind

    Public protocol model and errors.

    Limits

    Public protocol model and errors.

    MeasuredSink

    Public zero-dependency upload routing pipeline and transactional sink API.

    MediaType

    Public protocol model and errors.

    MixedError

    Bounded expansion for legacy nested multipart/mixed containers.

    MixedEvent

    Bounded expansion for legacy nested multipart/mixed containers.

    MixedLimits

    Bounded expansion for legacy nested multipart/mixed containers.

    MixedPart

    Bounded expansion for legacy nested multipart/mixed containers.

    MixedStream

    Bounded expansion for legacy nested multipart/mixed containers.

    MultipartDiagnostic

    Public protocol model and errors.

    MultipartError

    Public protocol model and errors.

    MultipartEvent

    Public protocol model and errors.

    MultipartParser

    Public streaming parser.

    MultipartPlan

    Public streaming writer.

    MultipartWriter

    Public streaming writer.

    Parameter

    Public protocol model and errors.

    ParserPhase

    Public protocol model and errors.

    ParserStats

    Public protocol model and errors.

    PartDigest

    Public zero-dependency upload routing pipeline and transactional sink API.

    PartMeta

    Public protocol model and errors.

    PartSinkAdapter

    Public small-field collection and transport-neutral streaming adapters.

    PlannedEmitter

    Public streaming writer.

    PlannedPart

    Public streaming writer.

    SinkMetrics

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadError

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadPart

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadPartKind

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadPipeline

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadPolicy

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadProgress

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadReceipt

    Public zero-dependency upload routing pipeline and transactional sink API.

    UploadSink

    Public zero-dependency upload routing pipeline and transactional sink API.

    body_contains_boundary

    fn body_contains_boundary(body : Bytes, boundary : String) -> Bool raise
    MultipartError

    boundary_from_entropy

    fn boundary_from_entropy(entropy : Bytes) -> String raise
    MultipartError

    crc32_finish

    fn crc32_finish(state : Int64) -> Int64

    Finalize an incremental CRC-32 state into its conventional unsigned value.

    crc32_initial

    fn crc32_initial() -> Int64

    Start an incremental CRC-32 state.

    crc32_update

    fn crc32_update(state : Int64, bytes : Bytes) -> Int64

    Advance a CRC-32 state with binary-safe data.

    decode_extended_filename

    fn decode_extended_filename(value : String) -> String raise
    MultipartError

    Decode RFC 5987 filename* values without guessing unsupported charsets. UTF-8 and ISO-8859-1 are the supported interoperable modes; callers receive a structured error for malformed percent escapes or unknown charsets.

    default_field_collector_limits

    default_limits

    Conservative defaults for examples and initial integrations.

    Applications are expected to choose limits appropriate for their threat model before accepting untrusted uploads.

    default_mixed_limits

    default_upload_policy

    expand_mixed

    Compatibility helper for callers that already own a complete container. New upload paths should use MixedStream directly to avoid buffering the nested body before parsing it.

    format_content_type

    fn format_content_type(boundary : String) -> String raise
    MultipartError

    parse_part_headers

    plan

    Calculate the exact output size for bodies that will be supplied later to MultipartWriter. No body bytes are copied or buffered by this operation.

    redact_error

    Convert a protocol error into log-safe structured data.

    Reasons, header values, field names, filenames, and body bytes are never copied into the result.

    validate_boundary

    fn validate_boundary(boundary : String) -> Unit raise
    MultipartError

    Source Files