#faceoff

#Get it running

• the MoonBit toolchain (step 1)
• a C compiler — gcc or clang, usually already present

#1. Install MoonBit

• English: https://www.moonbitlang.com/download/
• 中文: https://www.moonbitlang.cn/download/

moon version

#2. Build it and try it — without any API key

git clone <this repo> && cd moonbit-llm-faceoff
moon run --target native scripts/demo.mbtx

==> 1/3 one-shot
航空母舰是一种以舰载机为主要作战武器的大型水面舰艇。

==> 2/3 streaming (fragments arrive one by one)
侧风掠过甲板，把雨线吹成斜的。

==> 3/3 comparing two models
model mock-a
  runs 1   failures 0   truncated 0   retried 0
    first token (ms)       1          0 ...
...

#3. Run the web page — also without a key

make serve          # builds the page, then starts the server from web/
# → http://127.0.0.1:8137/

moon run --target native scripts/build-web.mbtx
cd web
../_build/native/debug/build/web/cmd/server/server.exe

#4. Point it at a real endpoint

export MOONLLM_BASE_URL="https://api.deepseek.com/v1"   # or any compatible base URL
export MOONLLM_MODEL="deepseek-chat"
export MOONLLM_API_KEY="sk-..."

client=./_build/native/debug/build/cmd/faceoff/faceoff.exe
$client "用一句话说明什么是航空母舰"
$client --stream "写一首关于侧风的短诗"

#5. Zero to a real comparison

# 1. build once (the demo in step 2 builds as well)
make deps
MOON_CC=gcc moon build --target native

# 2. point at your gateway — export it, never commit it
#    any OpenAI-compatible service will do
bench=./_build/native/debug/build/cmd/bench/bench.exe
export MOONLLM_BASE_URL="https://<your-gateway>/v1"
export MOONLLM_API_KEY="sk-..."

# 3. write the suite: JSON Lines, one case per line, `prompt` is the only
#    required field (bench/cases.example.jsonl is a fuller example)
mkdir -p my-run
cat > my-run/cases.jsonl <<'JSONL'
{"id": "math-short", "prompt": "计算 17 × 23。只输出数字。", "max_tokens": 512, "temperature": 0.0}
{"id": "fact-zh", "prompt": "用一句话说明什么是航空母舰。"}
JSONL

# 4. compare — the same suite, one model at a time
$bench \
  --models <model-a>,<model-b> \
  --cases my-run/cases.jsonl \
  --repeats 3 --max-tokens 2048 --temperature 0.0 \
  --pace-ms 1000 --retry 2 \
  --json my-run/runs.jsonl

# 5. render it as a self-contained page — no server, no key
moon run --target native scripts/build-web.mbtx my-run/runs.jsonl        # → web/out/report.html

# 6. the same loop again, with assertions and a written record
moon run --target native scripts/real-gateway.mbtx   # → docs/real-gateway-run.md

• Send --json somewhere of your own. bench/results-example.jsonl is a committed sample of a real run, not scratch space.
• Read the counters before the latency. failures, retried and truncated head each model's block. A reply cut off by --max-tokens is not a slow model, it is a model that ran out of budget: raise the budget and run again before comparing tok/s. The [truncated] marker on the progress line and attempts > 1 in runs.jsonl are the same information one layer down.

#New to MoonBit?

#If the build fails

#Contents

• 1. One-shot and streaming
• 2. Comparing models
• 3. The web page
• 4. Using the library
• 5. Architecture notes
• 6. Testing
• 7. Known limits
• 8. Layout
• 9. Further reading
• 10. Contributing
• 11. Provenance and dependencies

#1. One-shot and streaming

# the build puts the binaries under _build/; name them once per shell
faceoff=./_build/native/debug/build/cmd/faceoff/faceoff.exe
bench=./_build/native/debug/build/cmd/bench/bench.exe

# one-shot
$faceoff "用一句话说明什么是航空母舰"

# streamed, printed as fragments arrive
$faceoff --stream "写一首关于侧风的短诗"

# prompt from stdin
echo "总结一下这段日志" | $faceoff --stream

#Flags

$ faceoff --max-tokens 64 "用一句话说明什么是甲板风。"
-> POST https://api.example.com/v1/chat/completions  model=some-model  max_tokens=64
<- 3.2s  content 41 chars  reasoning 512 chars  tokens 21+64(reasoning 64)  finish_reason=length

$ faceoff --max-tokens 64 "..." >answer.txt
$ echo $?
1

#Environment variables

$ $faceoff --api-key wrong "hi"
error: http 401: {"error":{"message":"invalid api key"}}

#2. Comparing models

$bench \
  --base-url https://api.modelbest.cn/v1 --api-key "$MB_KEY" \
  --models MiniCPM5-1B,MiniCPM5-2B \
  --cases bench/cases.example.jsonl \
  --repeats 3 --max-tokens 2048 --temperature 0.0 \
  --pace-ms 3000 --retry 3 \
  --json my-run/runs.jsonl

$bench --model MiniCPM5-1B --prompt "用一句话说明什么是航空母舰" --show-cot

#Suite format

{"id": "math-short", "prompt": "计算 17 × 23。只输出数字。", "max_tokens": 512, "temperature": 0.0}

#What it measures

#Rate limits

#Replaying a run

# no network, no API key needed
$bench --from-json bench/results-example.jsonl --no-key --web-data web/data.json

#Outputs

jq -r 'select(.case_id=="code-python") | "\(.model)\t\(.completion_tokens)\t\(.content)"' \
  bench/results-example.jsonl

#3. The web page

moon run --target native scripts/build-web.mbtx   # from the repository root

cd web                     # the server resolves out/, runs/ and the case file
                           # relative to its working directory — run it here,
                           # not from the repository root
MOONLLM_API_KEY=... \
MOONLLM_BASE_URL=https://api.modelbest.cn/v1 \
LLM_WEB_MODELS=MiniCPM5-1B,MiniCPM5-2B \
  ../_build/native/debug/build/web/cmd/server/server.exe
# → http://127.0.0.1:8137/

• pick models from the server's menu or type model ids directly, set repeats / max_tokens / temperature / pacing / retries, type a one-off prompt, and start a run;
• set the gateway address and API key for that one run (they sit at the top of the form, above the model picker), so a local ollama or a second provider does not need a server restart. Both fields are optional and fall back to the server's environment;
• write one system prompt for the run and let the cases be nothing but user prompts — the fixed half and the varying half in separate places. A case that carries its own system overrides it, and 请求上下文 marks which ones did;
• choose what to run in one panel with two tabs: a test set (tick the cases) or a single 临时 First User Prompt. Typing anything into the prompt box is what selects it — there is no checkbox to forget — and both tabs say so out loud, because "I filled in a prompt and got results for 33 other questions" is a confusing thing to debug;
• create a test set from a plain text file (one prompt per line, blank lines and # skipped); a .jsonl case set works too. The server reads the path you give it — it is running on your machine anyway;
• presets live in the sidebar next to the run history, where they belong: both are "how a run was configured", one already run and one ready to go. A preset covers models, test set, system prompt, gateway and the parameters, and the panel says exactly that before you apply one;
• watch progress, the live failure/retry/truncation counts, and a collapsible tail of the run's stderr;
• go back to any earlier run: the left column lists every run with its time, models, scale and counters. Open one to read its results (read-only, exports follow it), rerun it with exactly the parameters it ran with, or delete it;
• paste text in a line at a time: the case editor has an import box — one line becomes one case, blank lines and # comments are skipped, and 追加导入 / 替换为这些行 decide whether the new lines follow the old ones or replace them;
• keep your suites in the page: the case panel switches between case sets (web/cases/<name>.jsonl), ticks the cases to run, and edits them in place — prompt, id, and per-case system / max_tokens / temperature under a folded "more fields". Saving writes the file and does not touch fields it never showed;
• save a model + parameter combination as a preset and apply it with one click, so "the usual two models, three repeats, 2048 tokens" stops being four form fields you retype;
• compare two runs: tick two entries in the history and the main area becomes a comparison — what parameters actually differ (so "I changed one thing" is visible rather than remembered), the median of every metric side by side with a Δ column, and the two runs' answers for the same case next to each other, chain of thought included. The Δ column is deliberately not coloured: the same direction is good for throughput and bad for latency, and this layer has no per-metric polarity to colour by;
• compare the answers case by case — every answer carries its model's chain of thought underneath, folded into a <details> (思考过程 · N token · M 字). Comparing two reasoning models means comparing that text, and a token count cannot stand in for it;
• judge an answer by hand: ✅ 通过 / ❌ 不行 / 🤔 拿不准 plus a note saying what was wrong. The verdict is saved the moment you click it, the note when you press 存备注, and both live in that run's directory — so they show up in the Markdown copy and the static report too, and deleting a run takes its verdicts with it;
• see what was actually asked: every case has a 请求上下文 button that opens the request as the model receives it — the parameters, and then the system and user messages that were in effect for that case. A system written into the case set overrides the global one, and when that happens the dialog says so; "why did it answer that" is usually answered here rather than in the answer. The endpoint returns the scrubbed request document, so no key is ever in it;
• read a distribution, not a single number: the model cards put P50 next to P10 / P20 / P99 for every metric, and the comparison table has a percentile switch (P10 / P20 / P50 / P99). They answer different questions — P50 is "how fast is it usually", P99 is "does it occasionally hang", P10 is "how fast when things go well" — and a median alone answers only the first. Percentiles interpolate between order statistics, the same rule numpy and R's default use; with few samples the tails sit near the extremes, and the n beside them is what says whether to believe it;
• read two long answers side by side: the results section switches between one block per model, columns (one per model, equal width, each with its own scrollbar and a pinned header), and a diff. Long answers are the case that needed it: stacked, they are two walls of text that you have to hold in your head at once. The diff aligns lines, collapses the identical stretches, and pairs a line that only changed a little — highlighting the few characters that actually differ, which for Chinese prose is usually one comma;
• come back to a run at any address: the run you are looking at is in the URL fragment (#run=<id>), so a refresh — or a link sent to someone else — lands on the same run. A run still in progress shows up in the sidebar with a 看进度 button, which is the only way back if you opened something else while it ran; deleting a run asks twice, and an id that does not exist says so instead of showing an empty page;
• watch a run while it happens: the progress card carries a bar over the whole suite plus a live line — which model, which case, whether it is still waiting for the first token, how many characters have arrived, and the rate over the last fraction of a second (≈ 61 tok/s). It is derived from the arrival times of the stream fragments, so it moves at the speed of the model rather than at the speed of the suite: a progress line per finished case leaves the page dead for the seconds in between, which is the part a person actually stares at. Two seconds without a fragment stops the pulse and says how long it has been quiet;
• export the result — copy the report as Markdown, copy a shareable URL, or download runs.jsonl, data.json, or a self-contained report.html.

#The URL is the configuration

http://127.0.0.1:8137/?models=mock-a,mock-b&cases=math-short,fact-zh&repeats=3

#API

web/runs/<id>/
  request.json     what was asked for
  cases.jsonl      the filtered suite (when cases were selected)
  runs.jsonl       bench's raw output, grows as it runs — progress is its line count
  stdout.log       bench's stdout (the rendered report)
  stderr.log       bench's progress log
  exit_code        written when the bench process is reaped; its presence means "finished"
  data.json        the final page-data document

#Server environment

#Static report

moon run --target native scripts/build-web.mbtx path/to/other-runs.jsonl   # → web/out/report.html

#Styling

#4. Using the library

import {
  "conglinyizhi/moonbit-llm-faceoff" @faceoff,
}

// one-shot
let settings = @faceoff.Settings::from_env(env)
let reply = @faceoff.ask(settings, "用一句话说明什么是航空母舰")

// the same request, keeping what the reply says about itself: an empty answer
// is not a bug report, and the stop reason and token counts explain it
let outcome = @faceoff.ask_outcome(settings, prompt)
// outcome.content, outcome.reasoning, outcome.usage, outcome.finish_reason

// streaming, with reasoning fragments separated from the answer
let outcome = @faceoff.stream_parts(settings, prompt, async fn(part) {
  match part {
    Content(text) => handle_answer(text)
    Reasoning(thought) => handle_thought(thought)
  }
})
// outcome.content, outcome.reasoning, outcome.usage, outcome.finish_reason

// benchmarking
let cases = @bench.parse_cases(text)
let results = @bench.run_bench(settings, models, cases, options, on_start, on_part, on_result)
let summaries = @bench.summarize_all(models, results)
println(@bench.format_summaries(summaries))

#5. Architecture notes

#6. Testing

make ci        # two phases: check + unit tests, then smoke ∥ api ∥ web
make e2e       # the browser test as well — needs chromium, and it is slow
make           # list every target

moon test --target native      # 85 unit tests, no network
moon run --target native scripts/smoke.mbtx   # CLI end-to-end against a local mock endpoint
moon run --target native scripts/server-api.mbtx   # server HTTP contract, incl. key handling
bash scripts/web-e2e.sh        # browser end-to-end (headless chromium)

• Incremental delivery. The mock sleeps between fragments, and the test measures when the first byte arrived relative to process exit. Comparing final output alone cannot tell a streaming client from a buffering one.
• Non-ASCII error bodies. The mock sends a Chinese 429 body; the test asserts it decodes. Reinterpreting the bytes as UTF-16 instead of decoding UTF-8 is a mistake that produces mojibake only on non-ASCII payloads.
• Concurrent run creation. Eight parallel POST /api/runs must come back with eight distinct ids. A single-request test passes even while the id allocation is a read-modify-write counter — it only breaks when two requests arrive together, which is exactly what a hand-run test never does.
• Retry and truncation counters. The mock can turn a 429 off after the first request (RATE_LIMIT_ONCE), which is the only shape in which a successful retry is observable: the test asserts attempts: 2 and retried: 1 in the same run. A client that never actually retried fails outright instead. And a reply whose finish_reason is length has to land in the truncation counter — counting it as a plain success would quietly average a cut-off answer into the speed numbers.

#7. Known limits

• --timeout-ms does not apply to the streaming path. A total-duration timeout would cut off legitimately long replies, and an idle timeout would need a timer around each read. The one-shot path does honor it.
• The web server binds 127.0.0.1 and has no authentication. It is a local dev tool. Do not expose it.
• Only the OpenAI-compatible wire format is implemented. No Anthropic or Gemini translation; the endpoint must accept /chat/completions.
• Targets. The library and its two CLIs declare native only; in web/, cmd/app is js, cmd/server and cmd/ssg are native, and shared builds for js+native+wasm.
• Pacing and retry defaults are heuristics. They were tuned against one gateway's rate limiter. Check your own with --pace-ms 0 and see what happens.

#8. Layout

moon.pkg            library package imports (native only)
faceoff.mbt         package documentation
settings.mbt        Settings + ConfigError, environment resolution
cli.mbt             Cli::parse, usage text
api.mbt             request building, response/SSE decoding
runner.mbt          ask / stream_chat / stream_parts / stream_to_stdout
*_test.mbt          blackbox unit tests
*_wbtest.mbt        whitebox unit tests (internal helpers)

bench/              the measurement harness
  bench.mbt         entry point
  case.mbt          suite parsing
  runner.mbt        run_case / run_bench, retry, JSON round-trip
  metrics.mbt       Stats, summarize
  report.mbt        the human-readable report
  pagedata.mbt      the document the page consumes
  cli.mbt           bench flag parsing

cmd/faceoff/        the faceoff executable
cmd/bench/          the bench executable

web/                the page, its server and the report (Rabbita + precss)
  shared/           data model + result components (js + native)
  cmd/app/          interactive page (js, Rabbita TEA)
    main.mbt        the form, the run in progress, the results
    history.mbt     the run-history rail
    manage.mbt      case sets and presets
    compare.mbt     two runs side by side
  cmd/server/       static + API server (native)
    main.mbt        routing and handlers
    store.mbt       case sets, presets, run history — files and JSON
  cmd/ssg/          static report (native)
  cmd/build/        the page build entry point (native)
  styles/           site.scss → precss → site.css

web/cases/          your case sets (one <name>.jsonl each) — gitignored
web/presets.json    your model + parameter combinations — gitignored
web/runs/           one directory per run — gitignored
web/data.json       page data exported from a run — gitignored
web/out/            built page + static report, from scripts/build-web.mbtx
web/shell/          index.html shell for the interactive page

scripts/
  demo.mbtx             offline, no-API-key demo of all three paths
  mock_openai.mbtx      offline OpenAI-compatible endpoint, as a MoonBit script
  check_incremental.mbtx  measures that --stream really streams
  smoke.mbtx            CLI end-to-end
  web-e2e.sh            browser end-to-end
  cdp-dump.mjs          drives headless chromium over the DevTools protocol
  lib.sh                helpers shared by the scripts above
  cdp-dump.mjs          DevTools-protocol DOM dump helper

docs/               library survey, benchmark notes

#9. Further reading

• docs/library-survey.md — the Mooncakes survey of LLM client libraries, the gaps found while integrating one of them, and why the dependency was eventually removed.
• docs/benchmark-notes.md — a worked MiniCPM5-1B vs MiniCPM5-2B comparison, including the result that contradicted the obvious reading of the throughput numbers.
• MoonBit: https://www.moonbitlang.com/ · docs https://docs.moonbitlang.com/ · packages https://mooncakes.io/

#10. Contributing

• CONTRIBUTING.md — build requirements, the house rules (no Python, why the page consumes bench's data.json), and what to run before a pull request.
• SECURITY.md — how to report a vulnerability, and what happens to your API key.
• CHANGELOG.md — what changed between versions, including the rename from llm_client.

#11. Provenance and dependencies

#License