🥮 mooncakes.io

moonrewardforge

Reward modeling, shaping, and debugging toolkit for MoonBit.

moonbit
reinforcement-learning
reward-shaping
analysis
moon add weidekais/moonrewardforge@0.1.1
Download zip
Author
Version
0.1.1
License
Apache-2.0
Repository
Last updated
17 hours ago
Downloads
4
README

#MoonRewardForge

MoonRewardForge is a MoonBit toolkit for modeling, normalizing, clipping, and debugging reward signals in reinforcement-learning style workflows.

Repository links:

It focuses on four practical questions:

  1. How much reward comes from the base objective, shaping terms, and penalties?
  2. Is the signal too sparse to learn from reliably?
  3. Is clipping hiding useful signal?
  4. How do different normalization modes change the final score?

#What it does

  • Composes reward terms into a per-step breakdown.
  • Computes trace-level totals and sparsity ratios.
  • Compares None, ScaleToAbsSum, and ZScore normalization modes.
  • Renders a plain-text report that is easy to inspect in CI or from the terminal.
  • Emits machine-readable quality alerts for sparse, clipped, and penalty-dominated signals.
  • Compares traces and runs a deterministic benchmark suite for regression checks.
  • Ships with a demo scenario matrix so the project runs out of the box.

#Why this topic

Reward shaping is a mature engineering problem in RL systems, and it has room to grow into a larger debugging workflow without becoming too narrow.

This project is intentionally framed as a reusable base:

  • today it is a scoring and reporting library,
  • later it can grow into log ingestion, YAML/JSON config parsing, visual dashboards, and experiment comparisons,
  • and it can be reused for robotics, game AI, recommendation, or agent-evaluation pipelines.

#Repository layout

  • moonrewardforge.mbt core reward model and analysis routines
  • rewardforge_report.mbt report rendering helpers
  • cmd/main/main.mbt runnable demo
  • moonrewardforge_test.mbt blackbox tests
  • moonrewardforge_wbtest.mbt whitebox tests
  • .github/workflows/moon-ci.yml minimal CI
  • LICENSE Apache-2.0
  • docs/OSC2026_CHECKLIST.md submission self-check

#Installation

Add this library to your project by running:

moon add weidekais/moonrewardforge

#Core API Example

Here is a quick example of how to evaluate a simple trace and print the summary:

import weidekais/moonrewardforge


fn main {
  let step = @moonrewardforge.RewardStep::{
    index: 0,
    terms: [
      @moonrewardforge.RewardTerm::{ name: "goal", raw: 1.0, weight: 1.0, kind: Base },
      @moonrewardforge.RewardTerm::{ name: "time_penalty", raw: -0.1, weight: 1.0, kind: Penalty },
    ],
    terminal: true,
  }
  let trace = [step]
  let config = @moonrewardforge.default_config()
  let (_breakdowns, summary) = @moonrewardforge.evaluate_trace(trace, config)

  println("Reward span: \{summary.reward_span}")
  println("Final clipped total: \{summary.total_clipped}")
}

#Run locally

moon check
moon build
moon test
moon run cmd/main

#Source and scope note

This repository is an original implementation.

The only external references used for design were:

  • the MoonBit language and toolchain documentation,
  • the OSC2026 submission guide,
  • and general reward shaping literature for the underlying RL concepts.

No upstream project was ported line-for-line. No other contributor should be introduced when you submit the project.

#License

Apache-2.0

#Production-oriented workflow

The library is deterministic and dependency-light, so it can run in an evaluator, simulator, or CI job without a service. Evaluate a trace, inspect the summary, and turn the same result into quality alerts with audit_trace(trace, config, sparse_threshold).

The benchmark API provides five deterministic cases covering dense progress, sparse goals, collision-heavy penalties, and control-loop oscillation. RewardAlert codes are stable: SPARSE_SIGNAL, CLIPPED_SIGNAL, PENALTY_DOMINATED, and SHAPING_DOMINATED.

For request validation at an API boundary, call validate_trace_issues before evaluate_trace. This reports empty traces, empty steps, invalid clipping ranges, invalid epsilon values, and invalid sparsity thresholds without panicking.

#Scope and limitations

This release analyzes in-memory reward traces. It does not claim to train an agent, ingest arbitrary files, or replace an environment's reward contract. Callers should validate their own step ordering and choose thresholds that fit their domain. Configuration errors are rejected early; an empty trace or empty step is treated as programmer misuse.

#
AlertLevel

source
pub(all) enum AlertLevel {
  Info
  Warning
  Critical
} derive(
Debug)

A stable, machine-readable severity used by integration quality gates.

#
AlertLevel::label

source
fn AlertLevel::label(self : AlertLevel) -> String

#
BenchmarkCase

source
pub(all) struct BenchmarkCase {
  name : String
  scenario : RewardScenario
  expected_min_total : Double
  expected_max_total : Double
} derive(
Debug)

A deterministic benchmark case for regression testing and demonstrations.

#
NormalizationMode

source
pub(all) enum NormalizationMode {
  None
  ScaleToAbsSum
  ZScore
} derive(
Debug)

#
NormalizationMode::label

source
fn NormalizationMode::label(self : NormalizationMode) -> String

#
RewardAlert

source
pub(all) struct RewardAlert {
  step_index : Int
  level : AlertLevel
  code : String
  message : String
  value : Double
} derive(
Debug)

A diagnostic produced from one evaluated reward step.

#
RewardAlert::render

source
fn RewardAlert::render(self : RewardAlert) -> String

#
RewardBreakdown

source
pub(all) struct RewardBreakdown {
  step_index : Int
  term_count : Int
  raw_total : Double
  weighted_total : Double
  normalized_total : Double
  clipped_total : Double
  mean : Double
  spread : Double
  sparse_terms : Int
  zero_ratio : Double
  shaping_share : Double
  penalty_share : Double
  dominant_name : String
  terminal : Bool
} derive(
Debug)

#
RewardConfig

source
pub(all) struct RewardConfig {
  mode : NormalizationMode
  clip_min : Double
  clip_max : Double
  epsilon : Double
} derive(
Debug)

#
RewardKind

source
pub(all) enum RewardKind {
  Base
  Shaping
  Penalty
} derive(
Debug)

#
RewardScenario

source
pub(all) struct RewardScenario {
  name : String
  steps : Array[RewardStep]
} derive(
Debug)

#
RewardStep

source
pub(all) struct RewardStep {
  index : Int
  terms : Array[RewardTerm]
  terminal : Bool
} derive(
Debug)

#
RewardTerm

source
pub(all) struct RewardTerm {
  name : String
  raw : Double
  weight : Double
  kind : RewardKind
} derive(
Debug)

#
RewardTerm::contribution

source
fn RewardTerm::contribution(self : RewardTerm) -> Double

#
RewardTerm::is_penalty

source
fn RewardTerm::is_penalty(self : RewardTerm) -> Bool

#
RewardTerm::is_shaping

source
fn RewardTerm::is_shaping(self : RewardTerm) -> Bool

#
RewardTerm::is_sparse

source
fn RewardTerm::is_sparse(self : RewardTerm) -> Bool

#
TraceSummary

source
pub(all) struct TraceSummary {
  step_count : Int
  sparse_steps : Int
  clipped_steps : Int
  avg_zero_ratio : Double
  total_raw : Double
  total_weighted : Double
  total_normalized : Double
  total_clipped : Double
  best_step : Int
  worst_step : Int
  best_score : Double
  worst_score : Double
  terminal_score : Double
  reward_span : Double
} derive(
Debug)

#
ValidationCode

source
pub(all) enum ValidationCode {
  EmptyTrace
  EmptyStep
  InvalidClipRange
  InvalidEpsilon
  InvalidSparseThreshold
} derive(
Debug)

Stable preflight diagnostics for callers that need to reject bad input without triggering the fail-fast evaluation API.

#
ValidationCode::label

source
fn ValidationCode::label(self : ValidationCode) -> String

#
ValidationIssue

source
pub(all) struct ValidationIssue {
  code : ValidationCode
  step_index : Int
  message : String
} derive(
Debug)

#
ValidationIssue::render

source
fn ValidationIssue::render(self : ValidationIssue) -> String

#
abs_value

source
fn abs_value(value : Double) -> Double

#
audit_trace

source
fn audit_trace(steps : Array[RewardStep], config : RewardConfig, sparse_threshold : Double) -> Array[RewardAlert]

#
benchmark_cases

source
fn benchmark_cases() -> Array[BenchmarkCase]

#
benchmark_catalog

source
fn benchmark_catalog() -> Array[BenchmarkCase]

A reproducible catalog of domain-neutral stress samples. Values are intentionally varied to exercise sign, sparsity, and terminal behavior without depending on random number generators.

#
benchmark_report

source
fn benchmark_report(config : RewardConfig) -> String

#
catalog_report

source
fn catalog_report(config : RewardConfig) -> String

#
clamp

source
fn clamp(value : Double, low : Double, high : Double) -> Double

#
compare_traces

source
fn compare_traces(left : Array[RewardStep], right : Array[RewardStep], config : RewardConfig) -> (TraceSummary, TraceSummary, Double)

#
count_sparse_terms

source
fn count_sparse_terms(terms : Array[RewardTerm]) -> Int

#
default_config

source
fn default_config() -> RewardConfig

#
evaluate_step

source
fn evaluate_step(step : RewardStep, config : RewardConfig) -> RewardBreakdown

#
evaluate_trace

source
fn evaluate_trace(steps : Array[RewardStep], config : RewardConfig) -> (Array[RewardBreakdown], TraceSummary)

#
inspect_breakdown

source
fn inspect_breakdown(breakdown : RewardBreakdown, sparse_threshold : Double, clip_threshold : Double) -> Array[RewardAlert]

#
is_valid_trace

source
fn is_valid_trace(steps : Array[RewardStep], config : RewardConfig, sparse_threshold : Double) -> Bool

#
make_benchmark_case

source
fn make_benchmark_case(name : String, steps : Array[RewardStep], expected_min_total : Double, expected_max_total : Double) -> BenchmarkCase

#
max_double

source
fn max_double(left : Double, right : Double) -> Double

#
min_double

source
fn min_double(left : Double, right : Double) -> Double

#
oscillating_terms

source
fn oscillating_terms() -> Array[RewardTerm]

#
render_alerts

source
fn render_alerts(alerts : Array[RewardAlert]) -> String

#
render_breakdown

source
fn render_breakdown(breakdown : RewardBreakdown) -> String

#
render_scenario_matrix

source
fn render_scenario_matrix() -> String

#
render_summary

source
fn render_summary(summary : TraceSummary) -> String

#
render_trace

source
fn render_trace(steps : Array[RewardStep], config : RewardConfig) -> String

#
render_validation_issues

source
fn render_validation_issues(issues : Array[ValidationIssue]) -> String

#
run_benchmarks

source
fn run_benchmarks(config : RewardConfig) -> Array[TraceSummary]

#
run_catalog_benchmarks

source
fn run_catalog_benchmarks(config : RewardConfig) -> Array[TraceSummary]

#
safe_div

source
fn safe_div(numerator : Double, denominator : Double, epsilon : Double) -> Double

#
sample_configs

source
fn sample_configs() -> Array[RewardConfig]

#
sample_scenarios

source
fn sample_scenarios() -> Array[RewardScenario]

#
sample_terms

source
fn sample_terms() -> Array[RewardTerm]

#
sample_trace

source
fn sample_trace() -> Array[RewardStep]

#
sparse_terms

source
fn sparse_terms() -> Array[RewardTerm]

#
sum_contributions

source
fn sum_contributions(terms : Array[RewardTerm]) -> Double

#
sum_raw

source
fn sum_raw(terms : Array[RewardTerm]) -> Double

#
validate_config

source
fn validate_config(config : RewardConfig) -> Unit

#
validate_config_issues

source
fn validate_config_issues(config : RewardConfig) -> Array[ValidationIssue]

#
validate_trace_issues

source
fn validate_trace_issues(steps : Array[RewardStep], config : RewardConfig, sparse_threshold : Double) -> Array[ValidationIssue]

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io