🥮 mooncakes.io

moontrajectory

Generic MoonBit trajectory, replay, sequence sampling, offline dataset, and reinforcement-learning training utilities.

moonbit
reinforcement-learning
replay-buffer
trajectory
moon add hmyhmyhmyss/moontrajectory@0.1.1
Download zip
Version
0.1.1
License
Apache-2.0
Repository
Last updated
19 hours ago
Downloads
4
README

#MoonTrajectory

MoonTrajectory 是一个面向强化学习训练基础设施的 MoonBit 轨迹与经验回放库。它把 transition、episode、经验回放、优先级采样、序列批处理、离线数据集和训练目标计算组织成可复用 API，可用于 DQN、PPO、行为克隆和离线强化学习原型。

#已交付范围

  • Transition[S, A]：泛型状态、动作、奖励、下一状态和终止标记。
  • Episode[S, A]：顺序存储、总回报、折扣回报、n-step、bootstrap、窗口切分、校验和奖励统计。
  • ReplayBuffer[S, A]：固定容量环形缓冲区、覆盖策略、边界访问、确定性均匀采样、序列采样、padding mask 和统计报告。
  • PrioritizedReplayBuffer[S, A]：比例优先级采样、priority 更新、重要性采样权重、优先级分布和边界统计。
  • TrajectoryDataset[S, A]：完整 episode 的离线数据集、校验拒绝、按 episode 切分、展平、抽样和统计。
  • 训练辅助：TD target、GAE、折扣前缀、target batch、奖励缩放、裁剪和批处理操作。
  • 评估与基准：回报/长度直方图、成功率、评估报告，以及 4 组确定性基准数据（sparse、dense、long-horizon、short-horizon）。

#工程边界

本库负责“轨迹数据层”和“训练批处理层”，不负责神经网络、环境执行器、自动求导、设备后端或 checkpoint。状态和动作保持泛型，可以接入字符串、数值、记录类型或上层张量表示。优先级缓冲区当前采用线性扫描采样，优点是实现透明、易审计；百万级样本场景应替换为 segment tree，同时保持公开数据结构和采样语义。

#快速开始

环境要求：MoonBit toolchain 0.10.3 或更高版本。

moon add hmyhmyhmyss/moontrajectory
moon test
moon run cmd/main

import hmyhmyhmyss/moontrajectory as traj

fn main {
  let episode = traj.Episode::new()
    .push(traj.Transition::non_terminal("s0", "left", 1.0, "s1"))
    .push(traj.Transition::terminal("s1", "right", 5.0, "s2"))
  let buffer : traj.ReplayBuffer[String, String] = traj.ReplayBuffer::new(1024)
  let _ = buffer.extend_episode(episode)
  let report = buffer.report()
  println("steps=\{report.size()}, reward=\{report.total_reward()}")
}

#完整本地验收命令

moon check --deny-warn
moon test --deny-warn
moon fmt --check
moon info
moon run cmd/main

当前仓库有 26 个测试，覆盖空容器、零容量、负索引、环形覆盖、终止/截断冲突、n-step、bootstrap、序列 padding、优先级归一化、优先级小于 1 的重算、负 priority 归一化、数据集分割、训练 targets、评估报告和确定性基准 fixture。

#项目结构

文件职责
trajectory.mbtTransition、Episode 和基础回报计算
trajectory_analysis.mbt校验、统计、bootstrap 与 episode 特征
replay_buffer.mbt均匀经验回放
prioritized_replay.mbt优先级经验回放
sequence_sampling.mbt固定长度序列、padding 和 mask
dataset.mbt离线轨迹数据集
training_targets.mbtTD、GAE、折扣 targets 和 batch
evaluation.mbt评估报告和直方图
episode_windows.mbtRNN/序列模型窗口
batch_ops.mbt通用批处理和数值操作
benchmarks.mbt确定性工作负载 fixture
replay_analytics.mbt回放统计和优先级诊断
cmd/main可直接运行的 smoke demo

#开源合规

项目采用 Apache-2.0，完整文本见根目录 LICENSE。当前源代码为独立 MoonBit 实现，不包含 vendored 第三方源文件、生成代码、商业代码或未授权测试数据；来源和设计边界见 docs/SOURCE.md。后续引入第三方算法或数据时，必须记录原项目链接、许可证、版权声明、改动范围和再分发说明。

#提交信息

GitHub、GitLink 和 mooncakes.io 的远端发布状态不能仅由本地工作区证明。推送前请确认两个代码托管平台的默认分支都显示最新提交，并在 mooncakes.io 查询模块版本；本地仓库不会自动推送任何远端。

#后续路线

当前版本已覆盖可复用的轨迹和 replay 基础闭环。后续可在不破坏现有 API 的前提下增加 segment tree、序列优先级采样、持久化格式、压缩存储和具体网络库适配层。

#
BatchCursor

source
pub struct BatchCursor {
  total : Int
  batch_size : Int
  position : Int
}

#
BatchCursor::batch_count

source
fn BatchCursor::batch_count(self : BatchCursor) -> Int

#
BatchCursor::batch_size

source
fn BatchCursor::batch_size(self : BatchCursor) -> Int

#
BatchCursor::has_next

source
fn BatchCursor::has_next(self : BatchCursor) -> Bool

#
BatchCursor::new

source
fn BatchCursor::new(total : Int, batch_size : Int) -> BatchCursor

#
BatchCursor::next_indices

source
fn BatchCursor::next_indices(self : BatchCursor) -> Array[Int]

#
BatchCursor::position

source
fn BatchCursor::position(self : BatchCursor) -> Int

#
BatchCursor::remaining

source
fn BatchCursor::remaining(self : BatchCursor) -> Int

#
BatchCursor::reset

source
fn BatchCursor::reset(self : BatchCursor) -> Unit

#
BatchCursor::total

source
fn BatchCursor::total(self : BatchCursor) -> Int

#
BenchmarkCase

source
pub struct BenchmarkCase {
  name : String
  episodes : Int
  steps_per_episode : Int
  reward_scale : Double
}

Deterministic benchmark fixtures for regression tests and demos.

#
BenchmarkCase::episodes

source
fn BenchmarkCase::episodes(self : BenchmarkCase) -> Int

#
BenchmarkCase::name

source
fn BenchmarkCase::name(self : BenchmarkCase) -> String

#
BenchmarkCase::new

source
fn BenchmarkCase::new(name : String, episodes : Int, steps_per_episode : Int, reward_scale : Double) -> BenchmarkCase

#
BenchmarkCase::reward_scale

source
fn BenchmarkCase::reward_scale(self : BenchmarkCase) -> Double

#
BenchmarkCase::steps_per_episode

source
fn BenchmarkCase::steps_per_episode(self : BenchmarkCase) -> Int

#
BenchmarkCase::total_steps

source
fn BenchmarkCase::total_steps(self : BenchmarkCase) -> Int

#
Episode

source
pub struct Episode[S, A] {
  transitions : Array[Transition[S, A]]
  terminated : Bool
  truncated : Bool
  total_reward : Double
}

A compact episode container for ordered transitions.

#
Episode::actions

source
fn[S, A] Episode::actions(self : Episode[S, A]) -> Array[A]

#
Episode::count_terminals

source
fn[S, A] Episode::count_terminals(self : Episode[S, A]) -> Int

#
Episode::discounted_returns

source
fn[S, A] Episode::discounted_returns(self : Episode[S, A], gamma : Double) -> Array[Double]

Return discounted returns in rollout order.

#
Episode::discounted_returns_with_bootstrap

source
fn[S, A] Episode::discounted_returns_with_bootstrap(self : Episode[S, A], gamma : Double, bootstrap : Double) -> Array[Double]

Compute returns while treating a value as the continuation value after the last transition. This is useful for truncated rollouts in bootstrapped RL.

#
Episode::first_transition

source
fn[S, A] Episode::first_transition(self : Episode[S, A]) -> Transition[S, A]?

#
Episode::from_transitions

source
fn[S, A] Episode::from_transitions(transitions : Array[Transition[S, A]]) -> Episode[S, A]

#
Episode::is_empty

source
fn[S, A] Episode::is_empty(self : Episode[S, A]) -> Bool

#
Episode::is_terminated

source
fn[S, A] Episode::is_terminated(self : Episode[S, A]) -> Bool

#
Episode::is_truncated

source
fn[S, A] Episode::is_truncated(self : Episode[S, A]) -> Bool

#
Episode::last_transition

source
fn[S, A] Episode::last_transition(self : Episode[S, A]) -> Transition[S, A]?

#
Episode::len

source
fn[S, A] Episode::len(self : Episode[S, A]) -> Int

#
Episode::mark_terminated

source
fn[S, A] Episode::mark_terminated(self : Episode[S, A]) -> Episode[S, A]

#
Episode::mark_truncated

source
fn[S, A] Episode::mark_truncated(self : Episode[S, A]) -> Episode[S, A]

#
Episode::n_step_transitions

source
fn[S, A] Episode::n_step_transitions(self : Episode[S, A], n : Int, gamma : Double) -> Array[Transition[S, A]]

Return N-step transitions. For each transition i, the reward is accumulated for N steps (or until done), and the next_state is the state after N steps (or the terminal state).

#
Episode::new

source
fn[S, A] Episode::new() -> Episode[S, A]

#
Episode::push

source
fn[S, A] Episode::push(self : Episode[S, A], transition : Transition[S, A]) -> Episode[S, A]

#
Episode::reward_stats

source
fn[S, A] Episode::reward_stats(self : Episode[S, A]) -> RewardStats

#
Episode::rewards

source
fn[S, A] Episode::rewards(self : Episode[S, A]) -> Array[Double]

#
Episode::states

source
fn[S, A] Episode::states(self : Episode[S, A]) -> Array[S]

#
Episode::terminal_index

source
fn[S, A] Episode::terminal_index(self : Episode[S, A]) -> Int?

#
Episode::total_reward

source
fn[S, A] Episode::total_reward(self : Episode[S, A]) -> Double

#
Episode::transitions

source
fn[S, A] Episode::transitions(self : Episode[S, A]) -> Array[Transition[S, A]]

#
Episode::validate

source
fn[S, A] Episode::validate(self : Episode[S, A]) -> Array[String]

Validate common data invariants before an episode enters a training buffer. The returned array is empty when the episode is safe to consume.

#
Episode::window

source
fn[S, A] Episode::window(self : Episode[S, A], offset : Int, length : Int) -> EpisodeWindow[S, A]

#
Episode::windows

source
fn[S, A] Episode::windows(self : Episode[S, A], length : Int, stride : Int) -> Array[EpisodeWindow[S, A]]

#
EpisodeWindow

source
pub struct EpisodeWindow[S, A] {
  offset : Int
  items : Array[Transition[S, A]]
  terminal : Bool
}

A contiguous view of an episode used by recurrent policies and sequence models.

#
EpisodeWindow::all_non_terminal

source
fn[S, A] EpisodeWindow::all_non_terminal(self : EpisodeWindow[S, A]) -> Bool

#
EpisodeWindow::contains_terminal

source
fn[S, A] EpisodeWindow::contains_terminal(self : EpisodeWindow[S, A]) -> Bool

#
EpisodeWindow::discounted_reward

source
fn[S, A] EpisodeWindow::discounted_reward(self : EpisodeWindow[S, A], gamma : Double) -> Double

#
EpisodeWindow::first

source
fn[S, A] EpisodeWindow::first(self : EpisodeWindow[S, A]) -> Transition[S, A]?

#
EpisodeWindow::is_empty

source
fn[S, A] EpisodeWindow::is_empty(self : EpisodeWindow[S, A]) -> Bool

#
EpisodeWindow::is_terminal

source
fn[S, A] EpisodeWindow::is_terminal(self : EpisodeWindow[S, A]) -> Bool

#
EpisodeWindow::items

source
fn[S, A] EpisodeWindow::items(self : EpisodeWindow[S, A]) -> Array[Transition[S, A]]

#
EpisodeWindow::last

source
fn[S, A] EpisodeWindow::last(self : EpisodeWindow[S, A]) -> Transition[S, A]?

#
EpisodeWindow::length

source
fn[S, A] EpisodeWindow::length(self : EpisodeWindow[S, A]) -> Int

#
EpisodeWindow::offset

source
fn[S, A] EpisodeWindow::offset(self : EpisodeWindow[S, A]) -> Int

#
EpisodeWindow::rewards

source
fn[S, A] EpisodeWindow::rewards(self : EpisodeWindow[S, A]) -> Array[Double]

#
EpisodeWindow::state_count

source
fn[S, A] EpisodeWindow::state_count(self : EpisodeWindow[S, A]) -> Int

#
EpisodeWindow::terminal_index

source
fn[S, A] EpisodeWindow::terminal_index(self : EpisodeWindow[S, A]) -> Int?

#
EpisodeWindow::total_reward

source
fn[S, A] EpisodeWindow::total_reward(self : EpisodeWindow[S, A]) -> Double

#
EvaluationReport

source
pub struct EvaluationReport {
  episodes : Int
  total_steps : Int
  total_reward : Double
  mean_reward : Double
  best_reward : Double
  worst_reward : Double
  terminated : Int
  truncated : Int
}

Aggregate evaluation metrics for a group of completed rollouts.

#
EvaluationReport::best_reward

source
fn EvaluationReport::best_reward(self : EvaluationReport) -> Double

#
EvaluationReport::episodes

source
fn EvaluationReport::episodes(self : EvaluationReport) -> Int

#
EvaluationReport::mean_reward

source
fn EvaluationReport::mean_reward(self : EvaluationReport) -> Double

#
EvaluationReport::terminated

source
fn EvaluationReport::terminated(self : EvaluationReport) -> Int

#
EvaluationReport::total_reward

source
fn EvaluationReport::total_reward(self : EvaluationReport) -> Double

#
EvaluationReport::total_steps

source
fn EvaluationReport::total_steps(self : EvaluationReport) -> Int

#
EvaluationReport::truncated

source
fn EvaluationReport::truncated(self : EvaluationReport) -> Int

#
EvaluationReport::worst_reward

source
fn EvaluationReport::worst_reward(self : EvaluationReport) -> Double

#
ExponentialSchedule

source
pub struct ExponentialSchedule {
  initial : Double
  final_value : Double
  decay_steps : Int
}

#
ExponentialSchedule::decay_steps

source
fn ExponentialSchedule::decay_steps(self : ExponentialSchedule) -> Int

#
ExponentialSchedule::final_value

source
fn ExponentialSchedule::final_value(self : ExponentialSchedule) -> Double

#
ExponentialSchedule::initial

source
fn ExponentialSchedule::initial(self : ExponentialSchedule) -> Double

#
ExponentialSchedule::new

source
fn ExponentialSchedule::new(initial : Double, final_value : Double, decay_steps : Int) -> ExponentialSchedule

#
ExponentialSchedule::value

source
fn ExponentialSchedule::value(self : ExponentialSchedule, step : Int) -> Double

#
LinearSchedule

source
pub struct LinearSchedule {
  start : Double
  end : Double
  duration : Int
}

A deterministic schedule for changing a training hyperparameter by step.

#
LinearSchedule::duration

source
fn LinearSchedule::duration(self : LinearSchedule) -> Int

#
LinearSchedule::end

source
fn LinearSchedule::end(self : LinearSchedule) -> Double

#
LinearSchedule::new

source
fn LinearSchedule::new(start : Double, end : Double, duration : Int) -> LinearSchedule

#
LinearSchedule::start

source
fn LinearSchedule::start(self : LinearSchedule) -> Double

#
LinearSchedule::value

source
fn LinearSchedule::value(self : LinearSchedule, step : Int) -> Double

#
PrioritizedReplayBuffer

source
pub struct PrioritizedReplayBuffer[S, A] {
  capacity : Int
  len : Int
  head : Int
  data : Array[Transition[S, A]?]
  priorities : Array[Double]
  max_priority : Double
  alpha : Double
  beta : Double
  epsilon : Double
}

Basic proportional prioritized replay.

#
PrioritizedReplayBuffer::capacity

source
fn[S, A] PrioritizedReplayBuffer::capacity(self : PrioritizedReplayBuffer[S, A]) -> Int

#
PrioritizedReplayBuffer::clear

source

#
PrioritizedReplayBuffer::count_priority_at_least

source
fn[S, A] PrioritizedReplayBuffer::count_priority_at_least(self : PrioritizedReplayBuffer[S, A], threshold : Double) -> Int

#
PrioritizedReplayBuffer::effective_priority_sum

source
fn[S, A] PrioritizedReplayBuffer::effective_priority_sum(self : PrioritizedReplayBuffer[S, A]) -> Double

#
PrioritizedReplayBuffer::extend_episode

source
fn[S, A] PrioritizedReplayBuffer::extend_episode(self : PrioritizedReplayBuffer[S, A], episode : Episode[S, A]) -> Int

#
PrioritizedReplayBuffer::get

source
fn[S, A] PrioritizedReplayBuffer::get(self : PrioritizedReplayBuffer[S, A], logical_index : Int) -> (Transition[S, A], Double)?

#
PrioritizedReplayBuffer::is_empty

source
fn[S, A] PrioritizedReplayBuffer::is_empty(self : PrioritizedReplayBuffer[S, A]) -> Bool

#
PrioritizedReplayBuffer::len

source
fn[S, A] PrioritizedReplayBuffer::len(self : PrioritizedReplayBuffer[S, A]) -> Int

#
PrioritizedReplayBuffer::new

source
fn[S, A] PrioritizedReplayBuffer::new(capacity : Int) -> PrioritizedReplayBuffer[S, A]

#
PrioritizedReplayBuffer::priority_at

source
fn[S, A] PrioritizedReplayBuffer::priority_at(self : PrioritizedReplayBuffer[S, A], index : Int) -> Double?

#
PrioritizedReplayBuffer::priority_distribution

source
fn[S, A] PrioritizedReplayBuffer::priority_distribution(self : PrioritizedReplayBuffer[S, A]) -> Array[Double]

#
PrioritizedReplayBuffer::priority_max

source
fn[S, A] PrioritizedReplayBuffer::priority_max(self : PrioritizedReplayBuffer[S, A]) -> Double

#
PrioritizedReplayBuffer::priority_sum

source
fn[S, A] PrioritizedReplayBuffer::priority_sum(self : PrioritizedReplayBuffer[S, A]) -> Double

#
PrioritizedReplayBuffer::push

source
fn[S, A] PrioritizedReplayBuffer::push(self : PrioritizedReplayBuffer[S, A], transition : Transition[S, A], priority? : Double) -> Int

#
PrioritizedReplayBuffer::sample_batch

source
fn[S, A] PrioritizedReplayBuffer::sample_batch(self : PrioritizedReplayBuffer[S, A], batch_size : Int, seed : Int) -> Array[PrioritizedSample[S, A]]

#
PrioritizedReplayBuffer::to_array

source
fn[S, A] PrioritizedReplayBuffer::to_array(self : PrioritizedReplayBuffer[S, A]) -> Array[Transition[S, A]]

#
PrioritizedReplayBuffer::update_priority

source
fn[S, A] PrioritizedReplayBuffer::update_priority(self : PrioritizedReplayBuffer[S, A], logical_index : Int, priority : Double) -> Bool

#
PrioritizedReplayBuffer::with_params

source
fn[S, A] PrioritizedReplayBuffer::with_params(capacity : Int, alpha : Double, beta : Double, epsilon : Double) -> PrioritizedReplayBuffer[S, A]

#
PrioritizedSample

source
pub struct PrioritizedSample[S, A] {
  index : Int
  priority : Double
  probability : Double
  weight : Double
  transition : Transition[S, A]
}

#
ReplayBuffer

source
pub struct ReplayBuffer[S, A] {
  capacity : Int
  len : Int
  head : Int
  data : Array[Transition[S, A]?]
}

#
ReplayBuffer::all_sequences

source
fn[S, A] ReplayBuffer::all_sequences(self : ReplayBuffer[S, A], sequence_length : Int) -> Array[SequenceSample[S, A]]

#
ReplayBuffer::capacity

source
fn[S, A] ReplayBuffer::capacity(self : ReplayBuffer[S, A]) -> Int

#
ReplayBuffer::clear

source
fn[S, A] ReplayBuffer::clear(self : ReplayBuffer[S, A]) -> ReplayBuffer[S, A]

#
ReplayBuffer::count_reward_at_least

source
fn[S, A] ReplayBuffer::count_reward_at_least(self : ReplayBuffer[S, A], threshold : Double) -> Int

#
ReplayBuffer::count_reward_below

source
fn[S, A] ReplayBuffer::count_reward_below(self : ReplayBuffer[S, A], threshold : Double) -> Int

#
ReplayBuffer::extend_episode

source
fn[S, A] ReplayBuffer::extend_episode(self : ReplayBuffer[S, A], episode : Episode[S, A]) -> Int

#
ReplayBuffer::get

source
fn[S, A] ReplayBuffer::get(self : ReplayBuffer[S, A], logical_index : Int) -> Transition[S, A]?

#
ReplayBuffer::is_empty

source
fn[S, A] ReplayBuffer::is_empty(self : ReplayBuffer[S, A]) -> Bool

#
ReplayBuffer::is_full

source
fn[S, A] ReplayBuffer::is_full(self : ReplayBuffer[S, A]) -> Bool

#
ReplayBuffer::len

source
fn[S, A] ReplayBuffer::len(self : ReplayBuffer[S, A]) -> Int

#
ReplayBuffer::new

source
fn[S, A] ReplayBuffer::new(capacity : Int) -> ReplayBuffer[S, A]

#
ReplayBuffer::newest

source
fn[S, A] ReplayBuffer::newest(self : ReplayBuffer[S, A]) -> Transition[S, A]?

#
ReplayBuffer::oldest

source
fn[S, A] ReplayBuffer::oldest(self : ReplayBuffer[S, A]) -> Transition[S, A]?

#
ReplayBuffer::push

source
fn[S, A] ReplayBuffer::push(self : ReplayBuffer[S, A], transition : Transition[S, A]) -> Int

#
ReplayBuffer::report

source
fn[S, A] ReplayBuffer::report(self : ReplayBuffer[S, A]) -> ReplayReport

#
ReplayBuffer::reward_histogram

source
fn[S, A] ReplayBuffer::reward_histogram(self : ReplayBuffer[S, A], lower : Double, upper : Double, buckets : Int) -> Array[Int]

#
ReplayBuffer::sample_all

source
fn[S, A] ReplayBuffer::sample_all(self : ReplayBuffer[S, A]) -> Array[ReplaySample[S, A]]

#
ReplayBuffer::sample_batch

source
fn[S, A] ReplayBuffer::sample_batch(self : ReplayBuffer[S, A], batch_size : Int, seed : Int) -> Array[ReplaySample[S, A]]

#
ReplayBuffer::sample_sequences

source
fn[S, A] ReplayBuffer::sample_sequences(self : ReplayBuffer[S, A], batch_size : Int, sequence_length : Int, seed : Int, allow_padding : Bool) -> Array[SequenceSample[S, A]]

#
ReplayBuffer::sequence

source
fn[S, A] ReplayBuffer::sequence(self : ReplayBuffer[S, A], start : Int, sequence_length : Int, pad : Bool) -> SequenceSample[S, A]

#
ReplayBuffer::terminal_indices

source
fn[S, A] ReplayBuffer::terminal_indices(self : ReplayBuffer[S, A]) -> Array[Int]

#
ReplayBuffer::to_array

source
fn[S, A] ReplayBuffer::to_array(self : ReplayBuffer[S, A]) -> Array[Transition[S, A]]

#
ReplayBuffer::valid_sequence_starts

source
fn[S, A] ReplayBuffer::valid_sequence_starts(self : ReplayBuffer[S, A], sequence_length : Int, allow_padding : Bool) -> Array[Int]

#
ReplayReport

source
pub struct ReplayReport {
  size : Int
  capacity : Int
  occupancy : Double
  terminal_count : Int
  total_reward : Double
  mean_reward : Double
  minimum_reward : Double
  maximum_reward : Double
}

Descriptive statistics for a replay buffer without exposing storage details.

#
ReplayReport::capacity

source
fn ReplayReport::capacity(self : ReplayReport) -> Int

#
ReplayReport::maximum_reward

source
fn ReplayReport::maximum_reward(self : ReplayReport) -> Double

#
ReplayReport::mean_reward

source
fn ReplayReport::mean_reward(self : ReplayReport) -> Double

#
ReplayReport::minimum_reward

source
fn ReplayReport::minimum_reward(self : ReplayReport) -> Double

#
ReplayReport::occupancy

source
fn ReplayReport::occupancy(self : ReplayReport) -> Double

#
ReplayReport::size

source
fn ReplayReport::size(self : ReplayReport) -> Int

#
ReplayReport::terminal_count

source
fn ReplayReport::terminal_count(self : ReplayReport) -> Int

#
ReplayReport::total_reward

source
fn ReplayReport::total_reward(self : ReplayReport) -> Double

#
ReplayRng

source
pub struct ReplayRng {
  state : Int
}

Tiny deterministic RNG used for sampling.

#
ReplayRng::new

source
fn ReplayRng::new(seed : Int) -> ReplayRng

#
ReplayRng::next_index

source
fn ReplayRng::next_index(self : ReplayRng, upper : Int) -> Int

#
ReplayRng::next_state

source
fn ReplayRng::next_state(self : ReplayRng) -> Int

#
ReplayRng::next_unit

source
fn ReplayRng::next_unit(self : ReplayRng) -> Double

#
ReplaySample

source
pub struct ReplaySample[S, A] {
  index : Int
  transition : Transition[S, A]
}

#
RewardStats

source
pub struct RewardStats {
  count : Int
  sum : Double
  mean : Double
  minimum : Double
  maximum : Double
  positive_count : Int
  negative_count : Int
  zero_count : Int
}

A compact summary of rewards in an episode or a collection of transitions. The summary is deliberately independent of the state and action types, so it can be used with strings, records, tensors, or application-specific values.

#
RewardStats::count

source
fn RewardStats::count(self : RewardStats) -> Int

#
RewardStats::empty

source
fn RewardStats::empty() -> RewardStats

#
RewardStats::maximum

source
fn RewardStats::maximum(self : RewardStats) -> Double

#
RewardStats::mean

source
fn RewardStats::mean(self : RewardStats) -> Double

#
RewardStats::minimum

source
fn RewardStats::minimum(self : RewardStats) -> Double

#
RewardStats::negative_count

source
fn RewardStats::negative_count(self : RewardStats) -> Int

#
RewardStats::positive_count

source
fn RewardStats::positive_count(self : RewardStats) -> Int

#
RewardStats::sum

source
fn RewardStats::sum(self : RewardStats) -> Double

#
RewardStats::zero_count

source
fn RewardStats::zero_count(self : RewardStats) -> Int

#
SequenceSample

source
pub struct SequenceSample[S, A] {
  start : Int
  transitions : Array[Transition[S, A]?]
  mask : Array[Bool]
}

A fixed-size rollout window. mask[i] is false for padding positions.

#
SequenceSample::is_padded

source
fn[S, A] SequenceSample::is_padded(self : SequenceSample[S, A]) -> Bool

#
SequenceSample::length

source
fn[S, A] SequenceSample::length(self : SequenceSample[S, A]) -> Int

#
SequenceSample::mask

source
fn[S, A] SequenceSample::mask(self : SequenceSample[S, A]) -> Array[Bool]

#
SequenceSample::rewards

source
fn[S, A] SequenceSample::rewards(self : SequenceSample[S, A]) -> Array[Double]

#
SequenceSample::start

source
fn[S, A] SequenceSample::start(self : SequenceSample[S, A]) -> Int

#
SequenceSample::terminals

source
fn[S, A] SequenceSample::terminals(self : SequenceSample[S, A]) -> Array[Bool]

#
SequenceSample::transitions

source
fn[S, A] SequenceSample::transitions(self : SequenceSample[S, A]) -> Array[Transition[S, A]?]

#
SequenceSample::valid_length

source
fn[S, A] SequenceSample::valid_length(self : SequenceSample[S, A]) -> Int

#
TargetBatch

source
pub struct TargetBatch {
  values : Array[Double]
  mask : Array[Bool]
}

Common scalar targets used by value-based and policy-gradient examples.

#
TargetBatch::length

source
fn TargetBatch::length(self : TargetBatch) -> Int

#
TargetBatch::mask

source
fn TargetBatch::mask(self : TargetBatch) -> Array[Bool]

#
TargetBatch::values

source
fn TargetBatch::values(self : TargetBatch) -> Array[Double]

#
TrajectoryDataset

source
pub struct TrajectoryDataset[S, A] {
  episodes : Array[Episode[S, A]]
  transition_count : Int
}

A lightweight offline-RL dataset made of complete episodes.

#
TrajectoryDataset::add

source
fn[S, A] TrajectoryDataset::add(self : TrajectoryDataset[S, A], episode : Episode[S, A]) -> Bool

#
TrajectoryDataset::add_unchecked

source
fn[S, A] TrajectoryDataset::add_unchecked(self : TrajectoryDataset[S, A], episode : Episode[S, A]) -> Unit

#
TrajectoryDataset::append

source
fn[S, A] TrajectoryDataset::append(self : TrajectoryDataset[S, A], other : TrajectoryDataset[S, A]) -> Unit

#
TrajectoryDataset::count_with_reward_at_least

source
fn[S, A] TrajectoryDataset::count_with_reward_at_least(self : TrajectoryDataset[S, A], threshold : Double) -> Int

#
TrajectoryDataset::count_with_terminal

source
fn[S, A] TrajectoryDataset::count_with_terminal(self : TrajectoryDataset[S, A]) -> Int

#
TrajectoryDataset::discounted_returns

source
fn[S, A] TrajectoryDataset::discounted_returns(self : TrajectoryDataset[S, A], gamma : Double) -> Array[Double]

#
TrajectoryDataset::drop_episodes

source
fn[S, A] TrajectoryDataset::drop_episodes(self : TrajectoryDataset[S, A], count : Int) -> TrajectoryDataset[S, A]

#
TrajectoryDataset::episode_count

source
fn[S, A] TrajectoryDataset::episode_count(self : TrajectoryDataset[S, A]) -> Int

#
TrajectoryDataset::episode_lengths

source
fn[S, A] TrajectoryDataset::episode_lengths(self : TrajectoryDataset[S, A]) -> Array[Int]

#
TrajectoryDataset::episode_reward_range

source
fn[S, A] TrajectoryDataset::episode_reward_range(self : TrajectoryDataset[S, A]) -> (Double, Double)?

#
TrajectoryDataset::episodes

source
fn[S, A] TrajectoryDataset::episodes(self : TrajectoryDataset[S, A]) -> Array[Episode[S, A]]

#
TrajectoryDataset::filter_by_length

source
fn[S, A] TrajectoryDataset::filter_by_length(self : TrajectoryDataset[S, A], minimum : Int, maximum : Int) -> TrajectoryDataset[S, A]

Dataset-level filtering and deterministic ordering helpers.

#
TrajectoryDataset::filter_by_reward

source
fn[S, A] TrajectoryDataset::filter_by_reward(self : TrajectoryDataset[S, A], minimum : Double, maximum : Double) -> TrajectoryDataset[S, A]

#
TrajectoryDataset::flatten

source
fn[S, A] TrajectoryDataset::flatten(self : TrajectoryDataset[S, A]) -> Array[Transition[S, A]]

#
TrajectoryDataset::from_episodes

source
fn[S, A] TrajectoryDataset::from_episodes(episodes : Array[Episode[S, A]]) -> TrajectoryDataset[S, A]

#
TrajectoryDataset::get

source
fn[S, A] TrajectoryDataset::get(self : TrajectoryDataset[S, A], index : Int) -> Episode[S, A]?

#
TrajectoryDataset::is_empty

source
fn[S, A] TrajectoryDataset::is_empty(self : TrajectoryDataset[S, A]) -> Bool

#
TrajectoryDataset::longest_episode

source
fn[S, A] TrajectoryDataset::longest_episode(self : TrajectoryDataset[S, A]) -> Episode[S, A]?

#
TrajectoryDataset::mean_episode_length

source
fn[S, A] TrajectoryDataset::mean_episode_length(self : TrajectoryDataset[S, A]) -> Double

#
TrajectoryDataset::new

source
fn[S, A] TrajectoryDataset::new() -> TrajectoryDataset[S, A]

#
TrajectoryDataset::reward_per_step

source
fn[S, A] TrajectoryDataset::reward_per_step(self : TrajectoryDataset[S, A]) -> Double

#
TrajectoryDataset::reward_stats

source
fn[S, A] TrajectoryDataset::reward_stats(self : TrajectoryDataset[S, A]) -> RewardStats

#
TrajectoryDataset::sample_episodes

source
fn[S, A] TrajectoryDataset::sample_episodes(self : TrajectoryDataset[S, A], count : Int, seed : Int) -> Array[Episode[S, A]]

#
TrajectoryDataset::shortest_episode

source
fn[S, A] TrajectoryDataset::shortest_episode(self : TrajectoryDataset[S, A]) -> Episode[S, A]?

#
TrajectoryDataset::split_at

source
fn[S, A] TrajectoryDataset::split_at(self : TrajectoryDataset[S, A], first_count : Int) -> (TrajectoryDataset[S, A], TrajectoryDataset[S, A])

#
TrajectoryDataset::take_episodes

source
fn[S, A] TrajectoryDataset::take_episodes(self : TrajectoryDataset[S, A], count : Int) -> Array[Episode[S, A]]

#
TrajectoryDataset::terminal_episode_count

source
fn[S, A] TrajectoryDataset::terminal_episode_count(self : TrajectoryDataset[S, A]) -> Int

#
TrajectoryDataset::terminal_rate

source
fn[S, A] TrajectoryDataset::terminal_rate(self : TrajectoryDataset[S, A]) -> Double

#
TrajectoryDataset::transition_count

source
fn[S, A] TrajectoryDataset::transition_count(self : TrajectoryDataset[S, A]) -> Int

#
TrajectoryDataset::truncated_episode_count

source
fn[S, A] TrajectoryDataset::truncated_episode_count(self : TrajectoryDataset[S, A]) -> Int

#
TrajectoryDataset::truncation_rate

source
fn[S, A] TrajectoryDataset::truncation_rate(self : TrajectoryDataset[S, A]) -> Double

#
Transition

source
pub struct Transition[S, A] {
  state : S
  action : A
  reward : Double
  next_state : S
  done : Bool
}

One transition in an RL rollout.

#
Transition::new

source
fn[S, A] Transition::new(state : S, action : A, reward : Double, next_state : S, done : Bool) -> Transition[S, A]

#
Transition::non_terminal

source
fn[S, A] Transition::non_terminal(state : S, action : A, reward : Double, next_state : S) -> Transition[S, A]

#
Transition::terminal

source
fn[S, A] Transition::terminal(state : S, action : A, reward : Double, next_state : S) -> Transition[S, A]

#
WarmupCosineSchedule

source
pub struct WarmupCosineSchedule {
  warmup_steps : Int
  total_steps : Int
  maximum : Double
  minimum : Double
}

#
WarmupCosineSchedule::maximum

source
fn WarmupCosineSchedule::maximum(self : WarmupCosineSchedule) -> Double

#
WarmupCosineSchedule::minimum

source
fn WarmupCosineSchedule::minimum(self : WarmupCosineSchedule) -> Double

#
WarmupCosineSchedule::new

source
fn WarmupCosineSchedule::new(warmup_steps : Int, total_steps : Int, maximum : Double, minimum : Double) -> WarmupCosineSchedule

#
WarmupCosineSchedule::total_steps

source
fn WarmupCosineSchedule::total_steps(self : WarmupCosineSchedule) -> Int

#
WarmupCosineSchedule::value

source
fn WarmupCosineSchedule::value(self : WarmupCosineSchedule, step : Int) -> Double

#
WarmupCosineSchedule::warmup_steps

source
fn WarmupCosineSchedule::warmup_steps(self : WarmupCosineSchedule) -> Int

#
array_cumulative

source
fn array_cumulative(values : Array[Double]) -> Array[Double]

#
array_difference

source
fn array_difference(values : Array[Double]) -> Array[Double]

#
array_drop

source
fn array_drop(values : Array[Double], count : Int) -> Array[Double]

#
array_l1_distance

source
fn array_l1_distance(left : Array[Double], right : Array[Double]) -> Double

#
array_l2_squared_distance

source
fn array_l2_squared_distance(left : Array[Double], right : Array[Double]) -> Double

#
array_mask

source
fn array_mask(values : Array[Double], mask : Array[Bool]) -> Array[Double]

#
array_max

source
fn array_max(values : Array[Double]) -> Double

#
array_mean

source
fn array_mean(values : Array[Double]) -> Double

#
array_min

source
fn array_min(values : Array[Double]) -> Double

#
array_pad

source
fn array_pad(values : Array[Double], length : Int, fill : Double) -> Array[Double]

#
array_scale_add

source
fn array_scale_add(left : Array[Double], scale : Double, right : Array[Double]) -> Array[Double]

#
array_sum

source
fn array_sum(values : Array[Double]) -> Double

Generic batch operations shared by offline and online training loops.

#
array_take

source
fn array_take(values : Array[Double], count : Int) -> Array[Double]

#
array_weighted_average

source
fn array_weighted_average(values : Array[Double], weights : Array[Double]) -> Double

#
benchmark_case_by_name

source
fn benchmark_case_by_name(cases : Array[BenchmarkCase], name : String) -> BenchmarkCase?

#
benchmark_cases

source
fn benchmark_cases() -> Array[BenchmarkCase]

#
benchmark_reward_scales

source
fn benchmark_reward_scales(cases : Array[BenchmarkCase]) -> Array[Double]

#
benchmark_summary

source
fn[S, A] benchmark_summary(case : BenchmarkCase, state : S, action : A) -> EvaluationReport

#
benchmark_workload_size

source
fn benchmark_workload_size(cases : Array[BenchmarkCase]) -> Int

#
clip_values

source
fn clip_values(values : Array[Double], minimum : Double, maximum : Double) -> Array[Double]

#
discounted_prefixes

source
fn discounted_prefixes(rewards : Array[Double], gamma : Double) -> Array[Double]

#
discounted_sum

source
fn discounted_sum(rewards : Array[Double], gamma : Double) -> Double

#
episode_lengths_histogram

source
fn[S, A] episode_lengths_histogram(episodes : Array[Episode[S, A]], buckets : Int) -> Array[Int]

#
episode_reward_histogram

source
fn[S, A] episode_reward_histogram(episodes : Array[Episode[S, A]], lower : Double, upper : Double, buckets : Int) -> Array[Int]

#
evaluate_episodes

source
fn[S, A] evaluate_episodes(episodes : Array[Episode[S, A]]) -> EvaluationReport

#
generalized_advantages

source
fn generalized_advantages(rewards : Array[Double], values : Array[Double], next_values : Array[Double], terminals : Array[Bool], gamma : Double, lambda : Double) -> Array[Double]

#
generate_benchmark_dataset

source
fn[S, A] generate_benchmark_dataset(case : BenchmarkCase, state : S, action : A) -> TrajectoryDataset[S, A]

#
normalize_values

source
fn normalize_values(values : Array[Double]) -> Array[Double]

#
scale_rewards

source
fn scale_rewards(rewards : Array[Double], scale : Double) -> Array[Double]

#
schedule_delta

source
fn schedule_delta(schedule : LinearSchedule, step : Int) -> Double

#
schedule_is_finished

source
fn schedule_is_finished(schedule : LinearSchedule, step : Int) -> Bool

#
schedule_progress

source
fn schedule_progress(schedule : LinearSchedule, step : Int) -> Double

#
schedule_values

source
fn schedule_values(schedule : LinearSchedule, steps : Int) -> Array[Double]

#
success_rate

source
fn[S, A] success_rate(episodes : Array[Episode[S, A]], reward_threshold : Double) -> Double

#
target_batches

source
fn target_batches(values : Array[Double], batch_size : Int) -> Array[TargetBatch]

#
td_targets

source
fn td_targets(rewards : Array[Double], next_values : Array[Double], terminals : Array[Bool], gamma : Double) -> Array[Double]

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io