🥮 mooncakes.io

    moonspectralprep

    Continuous laboratory spectrum preprocessing: penalized baselines, scatter calibration, local polynomial filters and bounded pipelines

    spectroscopy
    baseline
    chemometrics
    whittaker
    Download zip
    Author
    Version
    0.1.0
    License
    MIT
    Repository
    Last updated
    2 hours ago
    Downloads
    5

    No dependencies

    #MoonSpectralPrep

    纯 MoonBit 的连续实验光谱预处理库，模块 Qjoy127/moonspectralprep@0.1.0。面向需要把测量数组变成可比较、可积分数据的 Raman、红外/NIR 和仪器数据处理程序。提供带状惩罚基线、冻结的批次散射参考、物理坐标导数及可追踪处理管线。

    #能完成什么

    • 二阶差分 Whittaker 五对角 Cholesky 求解；AsLS、arPLS 给出基线、校正值、实际求解权重、迭代次数、相对变化和收敛状态。
    • SNIP 峰裁剪，可选择计数变换；显式背景区间/掩码多项式基线。
    • 局部多项式平滑与物理坐标导数；均匀轴对应 Savitzky–Golay mode=interp，非均匀轴重新局部拟合。提供中值滤波。
    • SNV、MSC、向量/范围归一化、透射率与吸光度转换；MSC 仅用训练数据建立一次参考，新样本不会触发重拟合。
    • 有界插值、公共轴批次对齐、带端点插值的裁剪、分段线性带积分、正面积重心和带面积比。
    • 批次逐通道均值/样本标准差、失败样本保留索引、描述性质量指标、版本化 JSON 数据/模型/管线报告。
    • 管线拥有配置数组，按固定顺序执行，检查估算工作量，默认拒绝未收敛基线。

    不提供 FFT、质谱 centroid/峰匹配、JCAMP 解码、化学鉴定、临床诊断、监督预测、仪器驱动或 GUI。所有演示数据均为本项目合成数据，尚无真实实验样本效能或化学定量准确率结论。

    #环境与安装

    验证固定工具链为 moonc v0.10.15+943464bfa / moon 0.1.20261012。CI 使用同版官方 SDK；不以 latest 的变化作为兼容性保证。库运行时只依赖 moonbitlang/core/math 与 moonbitlang/core/json，不需要 Python。

    支持 wasm-gc、wasm、JavaScript 和 native 构建目标。JavaScript 运行需要 Node.js；native 需要系统 C 编译器。本机 Windows 的前三个目标已实测，native 在配有 C 编译器的 Linux CI 中执行；最新真实状态见 Actions。

    在已有 MoonBit 模块中安装精确版本：

    moon add Qjoy127/moonspectralprep@0.1.0

    在使用库的 moon.pkg 中配置：

    import { "Qjoy127/moonspectralprep" @prep }
options(is_main: true)

    #快速开始

    把下面内容放入该包的 main.mbt 后运行 moon run .：

    fn main {
      try {
        let axis = @prep.regular_axis(0.0, 8.0, 9)
        let measured = @prep.spectrum(axis, [2.0, 2.1, 2.2, 3.3, 8.4, 3.5, 2.6, 2.7, 2.8])
        let result = @prep.asls(measured, options=@prep.iteration_options(lambda=100.0))
        let corrected = measured.with_values(result.corrected)
        let band = @prep.integrate_band(corrected, 2.0, 6.0)
        println("area=\{band.area}; converged=\{result.converged}")
      } catch {
        error => { println("preprocessing failed: \{error}"); panic() }
      }
    }

    轴单位由调用者保持一致；如果横轴为 cm⁻¹、强度为计数，面积单位就是计数·cm⁻¹。库不会自动识别或转换坐标单位。Spectrum 构造和访问均复制数组，外部修改不会改变已有光谱。

    #三个可运行工作流

    克隆仓库后，在根目录运行：

    moon run examples/raman
moon run examples/nir
moon run examples/alignment

    使用者与输入实际处理与输出
    Raman 算法开发者：含荧光背景与两个峰的合成连续光谱AsLS → 局部平滑 → 两个区间的弦线扣除积分及面积比；示例同时检查数值范围
    NIR/红外质控程序：带散射差异的训练批次和独立新样本训练均值参考 → JSON 保存/恢复 → MSC → SNV；输出斜率、截距、残差及标准差，参考保持冻结
    仪器/数据工程程序：两个不同测量网格显式公共轴插值 → 批次均值/样本标准差 → 裁剪和一阶导数 → 含逐步诊断的 JSON 报告；重复运行结果一致

    示例结果和单位见 验证记录。插值不会隐式外推；不重叠的测量域需要调用者先选择公共区间。

    #核心 API 与数据交换

    完整接口：pkg.generated.mbti。Spectrum、ScatterModel、Polynomial 和 Pipeline 的内部数组不公开；结果结构中的数组由调用者拥有。错误类型为 SpectralError::{Invalid, Numerical, Budget}。

    // 输入 JSON schema=1，仅包含 schema、axis、values 三个字段。
    let signal = @prep.parse_spectrum("{\"schema\":1,\"axis\":[0,1,2],\"values\":[1,3,1]}")
    let flow = @prep.pipeline([@prep.LocalPolynomial(3, 2, 0), @prep.Snv])
    let result = flow.run(signal)
    let report_json = result.to_json().stringify()

    上述片段放在能够处理 SpectralError 的函数中。背景掩码用 interval_mask(signal, [(left, right), ...]) 生成，再传给 masked_polynomial_baseline。fit_scatter(training) 需要完全相同的轴；model.transform(sample) 默认也严格比较轴。如确需坐标容差，可显式设置 axis_tolerance，其比较尺度为 tolerance * (1 + abs(reference_axis))。

    scatter_batch 为每个输入返回 Corrected(index, result) 或 Rejected(index, message)。align_batch 是整批成功或报错，不返回部分结果。Pipeline::run 默认要求基线收敛；明确传入 require_convergence=false 时仍保留未收敛标记，不可把该标记当作分析成功证明。

    #数值与资源边界

    • 单条光谱 3–65,536 点，有限严格递增轴，轴/强度绝对值不超过 1e100；不隐式排序、补缺失值或去除 NaN。
    • Whittaker 的二阶差分在采样索引上定义。跨仪器比较应先对齐到合适的均匀物理轴；它不是非均匀网格有限元平滑器。求解为 O(n) 时间与空间；大 λ 与极端权重对比可能损失精度或导致 Numerical，应降低 λ/权重对比。
    • λ 在 (0,1e12]，迭代 1–200 次；权重在 [1e-8,1e8]。arPLS 有 1e-8 权重下限，负残差不足/方差退化时可能返回未收敛；与 pybaselines 不保证逐位一致。
    • 多项式次数 0–8，局部窗口为 3–101 的奇数；局部拟合为 O(n·window·degree²)，无缓存卷积加速。SNIP 半窗口 1–2048，且小于样本数的一半；边缘缺少完整对称邻居时保留旧值，与采用 padding 的实现边缘不同。
    • SNV 使用 ddof=1；零方差归一化、无变化 MSC 参考、非正/过小散射斜率、零透射率和近零比值分母均显式失败。
    • 批次最多 256 条/1,048,576 个样本；摘要至少两条，MSC 拟合允许一条指定参考。管线最多 32 步，默认估算工作量上限 1e8，可设置至 1e9；它不是运行时超时。直接调用单算法受各自规模限制，不受管线总预算约束。
    • JSON 文本最多 8 Mi UTF-16 代码单元、嵌套不超过八层；严格检查字段集合与版本，不接受 null 强度。JSON 解析沿用标准库的重复键语义。模型只保存参考，加载时重算回归统计。
    • QualityMetrics 是描述统计；差分 MAD 和曲率不是已校准的噪声/异常判定。正面积重心只积分大于零的线性片段，不替代峰拟合。

    #检查、构建与独立数值对照

    moon info
moon fmt --check
moon check --deny-warn --target wasm-gc
moon test --deny-warn --target wasm-gc
moon build --deny-warn --target wasm-gc

    将目标改为 wasm、js 或 native 可执行对应检查；native 需要 C 编译器。仓库提供 29 个测试，包含方程残差、三点解析逆、非均匀物理导数、冻结参考、失败保留、所有权、退化数据和资源边界。

    独立对照在 Python 3.12 虚拟环境中运行，验证依赖不会进入库运行时：

    python -m venv .venv
# 激活环境；Windows: .venv\Scripts\Activate.ps1；Linux/macOS: source .venv/bin/activate
python -m pip install -r scripts/requirements-oracle.txt
python scripts/verify_numerics.py --target wasm-gc
python scripts/check_project.py

    对照 NumPy 密集求解、SciPy Savitzky–Golay、多项式拟合和 pybaselines 的 AsLS/arPLS/SNIP；固定 81 点合成数据、10 项比较，不构成所有参数/真实数据的精度保证。SNIP 仅比较不受 padding 影响的内部区间，arPLS 差异及容差有代码注释。具体误差见 验证记录。

    #许可、参考与 AI 使用

    本项目是原创 MoonBit 实现，采用 MIT。算法行为参考 pybaselines 与 SciPy savgol_filter 文档；没有复制其源码或测试数据。测试环境依赖 NumPy（BSD-3-Clause）、SciPy（BSD-3-Clause）、pybaselines（BSD-3-Clause）；MoonBit core 为 Apache-2.0，均不作为项目内 vendored 源码。详见 参考边界。

    项目的选题调研、实现、测试、中文说明和验证自动化使用了 AI 编程辅助。数值结果以可运行程序、参考库对照和真实 CI 为依据；不声称绝对唯一性或主办方验收。生态近邻与检索局限记录于 查重说明。

    SpectralError

    source
    pub(all) suberror SpectralError {
      Invalid(String)
      Numerical(String)
      Budget(String)
    }

    SpectralError::output

    source
    fn SpectralError::output(self : SpectralError, logger : &Logger) -> Unit

    SpectralError::to_string

    source
    fn SpectralError::to_string(self : SpectralError) -> String

    BandMetrics

    source
    pub(all) struct BandMetrics {
      area : Double
      positive_area : Double
      positive_centroid : Double?
      height : Double
      height_position : Double
      sampled_points : Int
    }

    BaselineResult

    source
    pub(all) struct BaselineResult {
      baseline : Array[Double]
      corrected : Array[Double]
      weights : Array[Double]
      iterations : Int
      converged : Bool
      relative_change : Double
    }

    BatchSummary

    source
    pub(all) struct BatchSummary {
      mean_spectrum : Spectrum
      sample_deviations : Array[Double]
      count : Int
    }

    IterationOptions

    source
    pub(all) struct IterationOptions {
      lambda : Double
      max_iterations : Int
      tolerance : Double
    }

    Pipeline

    source
    pub struct Pipeline {
      steps : Array[Step]
      maximum_work : Double
      require_convergence : Bool
    }

    Pipeline::run

    source
    fn Pipeline::run(self : Pipeline, input : Spectrum) -> PipelineResult raise SpectralError

    Run deterministically and return only a complete result. Original input and configuration remain owned; each successful step adds compact metrics.

    PipelineResult

    source
    pub(all) struct PipelineResult {
      spectrum : Spectrum
      trace : Array[StepReport]
      estimated_work : Double
    }

    PipelineResult::to_json

    source

    Compact deterministic report; no intermediate spectra, environment paths, account credentials, timestamps or private metadata are collected.

    Polynomial

    source
    pub struct Polynomial {
      coefficients : Array[Double]
      center : Double
      scale : Double
    }

    Polynomial coefficients refer to t=(x-center)/scale; arrays stay private.

    Polynomial::center

    source
    fn Polynomial::center(self : Polynomial) -> Double

    Polynomial::coefficients

    source
    fn Polynomial::coefficients(self : Polynomial) -> Array[Double]

    Polynomial::degree

    source
    fn Polynomial::degree(self : Polynomial) -> Int

    Polynomial::evaluate

    source
    fn Polynomial::evaluate(self : Polynomial, x : Double, derivative? : Int) -> Double raise SpectralError

    Evaluate a physical-coordinate derivative (order 0 returns fitted value).

    Polynomial::scale

    source
    fn Polynomial::scale(self : Polynomial) -> Double

    QualityMetrics

    source
    pub(all) struct QualityMetrics {
      samples : Int
      minimum : Double
      maximum : Double
      mean : Double
      deviation : Double
      rms : Double
      negative_fraction : Double
      first_difference_mad : Double
      curvature_rms : Double
      uniform_axis : Bool
    }

    ScatterModel

    source
    pub struct ScatterModel {
      reference : Spectrum
      reference_mean : Double
      reference_energy : Double
    }

    Frozen training reference. Neither transform nor public access mutates it.

    ScatterModel::reference

    source
    fn ScatterModel::reference(self : ScatterModel) -> Spectrum raise SpectralError

    ScatterModel::to_json

    source

    Persist only the frozen reference; derived regression statistics are recomputed and validated on load rather than trusting serialized numbers.

    ScatterModel::transform

    source
    fn ScatterModel::transform(self : ScatterModel, input : Spectrum, axis_tolerance? : Double) -> ScatterResult raise SpectralError

    Regress sample = intercept + slope*reference, then return (sample-intercept)/slope. Negative or almost-zero physical slopes fail.

    ScatterOutcome

    source
    pub(all) enum ScatterOutcome {
      Corrected(Int, ScatterResult)
      Rejected(Int, String)
    }

    ScatterResult

    source
    pub(all) struct ScatterResult {
      spectrum : Spectrum
      intercept : Double
      slope : Double
      residual_rms : Double
    }

    Spectrum

    source
    pub struct Spectrum {
      axis : Array[Double]
      values : Array[Double]
    }

    Owned arrays; construction rejects nonfinite samples and unordered axes.

    Spectrum::axis

    source
    fn Spectrum::axis(self : Spectrum) -> Array[Double]

    Spectrum::length

    source
    fn Spectrum::length(self : Spectrum) -> Int

    Spectrum::subtract_baseline

    source
    fn Spectrum::subtract_baseline(self : Spectrum, baseline : Array[Double]) -> Spectrum raise SpectralError

    Remove a fitted curve while retaining the original owned axis.

    Spectrum::to_json

    source

    Versioned data interchange. Units and provenance are caller-managed; this schema deliberately serializes measured coordinates and values only.

    Spectrum::values

    source
    fn Spectrum::values(self : Spectrum) -> Array[Double]

    Spectrum::with_values

    source
    fn Spectrum::with_values(self : Spectrum, values : Array[Double]) -> Spectrum raise SpectralError

    Step

    source
    pub(all) enum Step {
      Asls(Double, IterationOptions)
      Arpls(IterationOptions)
      Snip(Int, Bool, Bool)
      PolynomialBaseline(Array[Bool], Int)
      LocalPolynomial(Int, Int, Int)
      Median(Int)
      Snv
      UnitVector
      RangeNormalize
      Scatter(ScatterModel)
      Affine(Double, Double)
      Crop(Double, Double)
      Resample(Array[Double])
      Absorbance(Bool)
      Transmittance(Bool)
    }

    StepReport

    source
    pub(all) struct StepReport {
      name : String
      input_samples : Int
      output_samples : Int
      rms_before : Double
      rms_after : Double
      iterations : Int?
      converged : Bool?
      relative_change : Double?
    }

    absorbance

    source
    fn absorbance(input : Spectrum, percent? : Bool) -> Spectrum raise SpectralError

    Convert transmittance to absorbance. Values must lie in (0,1] (fraction) or (0,100] (percent). Zero is rejected instead of being silently clipped.

    affine_intensity

    source
    fn affine_intensity(input : Spectrum, scale : Double, offset : Double) -> Spectrum raise SpectralError

    Apply a finite affine intensity calibration; scale may be negative.

    align_batch

    source
    fn align_batch(inputs : Array[Spectrum], axis : Array[Double]) -> Array[Spectrum] raise SpectralError

    Resample multiple measured spectra onto one owned common axis. Any out-of-domain input fails the operation; no partial batch escapes.

    arpls

    source
    fn arpls(input : Spectrum, options? : IterationOptions) -> BaselineResult raise SpectralError

    Adaptive reweighted penalized least squares with sample deviation of negative residuals and a bounded logistic update. At least two distinct negative residuals are required for another statistically defined update.

    asls

    source
    fn asls(input : Spectrum, asymmetry? : Double, options? : IterationOptions) -> BaselineResult raise SpectralError

    Asymmetric least-squares. Weights in the result are those used for the returned baseline, rather than the next unsolved weight proposal.

    band_ratio

    source
    fn band_ratio(numerator : BandMetrics, denominator : BandMetrics, minimum_area? : Double) -> Double raise SpectralError

    Divide signed areas; reject a small denominator using a caller threshold.

    crop

    source
    fn crop(input : Spectrum, left : Double, right : Double) -> Spectrum raise SpectralError

    Crop to a closed interval, inserting interpolated endpoints when needed. At least three distinct coordinates must remain.

    fit_polynomial

    source
    fn fit_polynomial(axis : Array[Double], values : Array[Double], degree : Int) -> Polynomial raise SpectralError

    Least squares via twice-orthogonalized modified Gram-Schmidt, on a centered/scaled Vandermonde basis. Degree is limited to eight.

    fit_scatter

    source
    fn fit_scatter(training : Array[Spectrum]) -> ScatterModel raise SpectralError

    Fit a mean reference using training data only, on an exactly common axis. At most 256 spectra and 1,048,576 total samples are accepted per fit.

    integrate_band

    source
    fn integrate_band(input : Spectrum, left : Double, right : Double, remove_chord? : Bool) -> BandMetrics raise SpectralError

    Exact integral of the piecewise-linear interpolant, including partial endpoint segments. Optional chord removal is local to this band. Positive centroid uses only signal above zero, independently of signed area.

    interpolate

    source
    fn interpolate(input : Spectrum, x : Double) -> Double raise SpectralError

    Linear interpolation within the measured domain; extrapolation is refused.

    interval_mask

    source
    fn interval_mask(input : Spectrum, intervals : Array[(Double, Double)]) -> Array[Bool] raise SpectralError

    Build a mask by including closed physical-coordinate intervals. The union is evaluated once; overlapping intervals never duplicate points.

    is_uniform_axis

    source
    fn is_uniform_axis(input : Spectrum, tolerance? : Double) -> Bool raise SpectralError

    Relative spacing tolerance is measured against mean spacing.

    iteration_options

    source
    fn iteration_options(lambda? : Double, max_iterations? : Int, tolerance? : Double) -> IterationOptions

    local_polynomial

    source
    fn local_polynomial(input : Spectrum, window? : Int, degree? : Int, derivative? : Int) -> Spectrum raise SpectralError

    Local polynomial smoothing and derivatives on physical coordinates. Uniform grids reproduce Savitzky-Golay mode=interp; irregular grids use an actual local fit. At edges a complete shifted window is fitted.

    masked_polynomial_baseline

    source
    fn masked_polynomial_baseline(input : Spectrum, selected : Array[Bool], degree? : Int) -> BaselineResult raise SpectralError

    Fit only user-selected background samples, then evaluate across all peaks. True means include. No implicit automatic peak exclusion is performed.

    mean

    source
    fn mean(values : Array[Double]) -> Double raise SpectralError

    median_absolute_deviation

    source
    fn median_absolute_deviation(values : Array[Double]) -> Double raise SpectralError

    Median absolute deviation; no Gaussian scale factor is applied.

    median_filter

    source
    fn median_filter(input : Spectrum, window? : Int) -> Spectrum raise SpectralError

    Median filter with a shifted complete window at edges.

    parse_spectrum

    source
    fn parse_spectrum(text : String) -> Spectrum raise SpectralError

    Bounded parser for untrusted spectrum text; no I/O or filesystem access.

    pipeline

    source
    fn pipeline(steps : Array[Step], maximum_work? : Double, require_convergence? : Bool) -> Pipeline raise SpectralError

    Copy configuration arrays at construction. The budget is a conservative operation estimate, not wall-clock time or a cancellation guarantee.

    quality

    source
    fn quality(input : Spectrum) -> QualityMetrics raise SpectralError

    Descriptive QA metrics, not a detector of chemical identity or noise truth. First differences and curvature are in sample-index coordinates.

    quantile

    source
    fn quantile(values : Array[Double], probability : Double) -> Double raise SpectralError

    Linear interpolation quantile at p*(n-1), on an owned sorted copy.

    range_normalize

    source
    fn range_normalize(input : Spectrum) -> Spectrum raise SpectralError

    Map min/max onto [0,1]. A constant spectrum is rejected explicitly.

    regular_axis

    source
    fn regular_axis(start : Double, stop : Double, count : Int) -> Array[Double] raise SpectralError

    Make an inclusive regular grid; caller explicitly chooses axis units.

    relative_rms

    source
    fn relative_rms(next : Array[Double], previous : Array[Double]) -> Double raise SpectralError

    Relative RMS difference, with absolute RMS fallback for a zero reference.

    resample

    source
    fn resample(input : Spectrum, axis : Array[Double]) -> Spectrum raise SpectralError

    Two-pointer resampling in O(n+m), with owned axis and intensity arrays.

    root_mean_square

    source
    fn root_mean_square(values : Array[Double]) -> Double raise SpectralError

    sample_deviation

    source
    fn sample_deviation(values : Array[Double]) -> Double raise SpectralError

    Sample standard deviation, using a second pass and scaled differences.

    scatter_batch

    source
    fn scatter_batch(model : ScatterModel, inputs : Array[Spectrum]) -> Array[ScatterOutcome] raise SpectralError

    Transform independently against one frozen model; keep every input index. A bad sample is reported, never silently removed or used to refit the model.

    scatter_model_from_json

    source
    fn scatter_model_from_json(value : 
    Json    ) -> ScatterModel raise SpectralError

    scatter_reference

    source
    fn scatter_reference(reference : Spectrum) -> ScatterModel raise SpectralError

    Construct a model from an explicitly supplied reference spectrum.

    snip

    source
    fn snip(input : Spectrum, half_window? : Int, decreasing? : Bool, count_transform? : Bool) -> BaselineResult raise SpectralError

    Statistics-sensitive nonlinear iterative peak clipping. Uses synchronous neighbors, leaves unavailable edge pairs unchanged, and optionally applies the log(log(sqrt(y+1)+1)+1) count transform. Transformed input must be >= 0.

    snv

    source
    fn snv(input : Spectrum) -> Spectrum raise SpectralError

    Standard normal variate using sample standard deviation (n-1).

    spectral_angle

    source
    fn spectral_angle(left : Spectrum, right : Spectrum) -> Double raise SpectralError

    Spectral angle in radians; meaningful for nonzero aligned vectors.

    spectral_correlation

    source
    fn spectral_correlation(left : Spectrum, right : Spectrum) -> Double raise SpectralError

    Pearson correlation on exactly aligned axes; constant inputs are refused.

    spectrum

    source
    fn spectrum(axis : Array[Double], values : Array[Double]) -> Spectrum raise SpectralError

    spectrum_from_json

    source
    fn spectrum_from_json(value : 
    Json    ) -> Spectrum raise SpectralError

    Validate lengths before allocating a second numeric copy; then use the same spectrum constructor invariants as direct API input.

    stable_sum

    source
    fn stable_sum(values : Array[Double]) -> Double raise SpectralError

    Compensated sum reduces cancellation without changing sample order.

    summarize_batch

    source
    fn summarize_batch(batch : Array[Spectrum]) -> BatchSummary raise SpectralError

    Stable per-channel Welford statistics on an exactly common axis.

    transmittance

    source
    fn transmittance(input : Spectrum, percent? : Bool) -> Spectrum raise SpectralError

    Convert absorbance to fractional or percent transmittance.

    unit_vector

    source
    fn unit_vector(input : Spectrum) -> Spectrum raise SpectralError

    Normalize the full vector to unit Euclidean norm, retaining sign.

    validate_iteration_options

    source
    fn validate_iteration_options(o : IterationOptions) -> Unit raise SpectralError

    Validate bounded options before allocating iterative workspaces.

    whittaker

    source
    fn whittaker(values : Array[Double], weights : Array[Double], lambda : Double) -> Array[Double] raise SpectralError

    Solve (W + lambda D2' D2) z = W y using pentadiagonal Cholesky. The penalty is in sample-index coordinates; use a common uniform axis for physically comparable fits. Positive weights keep the system definite.

    whittaker_objective

    source
    fn whittaker_objective(values : Array[Double], baseline : Array[Double], weights : Array[Double], lambda : Double) -> Double raise SpectralError

    Evaluate the objective independently from the matrix factorization.

    Source Files