🥮 mooncakes.io

    moonbit-llm-faceoff

    OpenAI-compatible LLM client, a harness that compares multiple models on the same case set, and a browser UI for the results — all in MoonBit.

    moonbit
    llm
    openai
    benchmark
    comparison
    rabbita
    Download zip
    Version
    0.2.0
    License
    MIT
    Repository
    Last updated
    7 days ago
    Downloads
    9

    Dependencies

    #faceoff

    English · 中文

    A MoonBit client for OpenAI-compatible chat endpoints, plus a harness for comparing models on the same prompt suite.

    binarywhat it doessource
    faceoffask one prompt, once or streamedcmd/faceoff
    benchrun a suite against several models and compare themcmd/bench
    web serverinteractive page: pick models/params, run, watch resultsweb/cmd/server

    Every test in this repo runs offline against a bundled mock, so no API key is needed to see it work.

    Three top-level pieces: bench/ (case parsing, running, statistics, report, page data), cmd/ (entry points for the two executables) and web/ (the interactive page, its server and the static report). scripts/ holds the offline mock endpoint, the demos and the test suites, and docs/ holds the surveys and the notes.

    #Quick start

    You do not need to know MoonBit to run this. You need the MoonBit toolchain (step 1) and a C compiler: gcc or clang, usually already present. That is the whole list, because the fake endpoint the demo and the tests run against is itself a MoonBit script (scripts/mock_openai.mbtx).

    #1. Install MoonBit

    Follow the official instructions, they are authoritative and stay current: https://www.moonbitlang.com/download/ · https://www.moonbitlang.cn/download/. For convenience, the three official methods are:

    platformcommand
    macOS / Linuxcurl -fsSL https://cli.moonbitlang.com/install/unix.sh \| bash
    Windows (PowerShell)Set-ExecutionPolicy RemoteSigned -Scope CurrentUser; irm https://cli.moonbitlang.com/install/powershell.ps1 \| iex
    VS Codecommand palette → MoonBit:install latest moonbit toolchain

    Then make sure ~/.moon/bin is on your PATH and verify with moon version. If any of the above fails or has changed, use the official page rather than this table. New to MoonBit, or building on Linux? CONTRIBUTING.md has a five-line orientation to the file types here and the three usual build failures.

    #2. Build it and try it, no API key

    git clone <this repo> && cd moonbit-llm-faceoff
moon run --target native scripts/demo.mbtx    # the same as make demo

    It builds everything, starts a local fake OpenAI-compatible endpoint, and walks the three main paths: one-shot, streaming, and a two-model comparison. Entirely offline, and it starts like this:

    ==> 1/3 one-shot
航空母舰是一种以舰载机为主要作战武器的大型水面舰艇。
==> 2/3 streaming (fragments arrive one by one)
侧风掠过甲板，把雨线吹成斜的。
==> 3/3 comparing two models
  runs 1   failures 0   truncated 0   retried 0

    #3. Run the web page, also without a key

    make serve          # builds the page, then serves it from web/ → http://127.0.0.1:8137/
make dev            # the same, but watching the sources: a page change rebuilds the page, a server change restarts it
make serve-demo     # mock endpoint + demo data + page: the quickest look at the whole thing

    Either way, remember one thing: the server must run with web/ as its working directory. That requirement is now down to the two paths it still resolves against the cwd, the static directory out/ and the seed file ../bench/cases.example.jsonl; runs, case sets and presets resolve to absolute paths under your user directories. Started from the repository root it answers /api/meta and then 404s on every page request. make serve does that cd for you, and without make it is moon run --target native scripts/build-web.mbtx, moon build web/cmd/server --target native, then the server binary from inside web/. Your data lives in your user directory, and moon run --target native web/cmd/server -- --print-dirs prints the four resolved paths and exits. Against a real gateway, either export MOONLLM_BASE_URL / MOONLLM_API_KEY before starting it, or start it without a key and type the key into the page: it is used for that one run, and it is not written to disk.

    #4. Point it at a real endpoint, and run a comparison

    Any OpenAI-compatible service works. One block covers the whole loop: a one-shot question, a streamed one, a suite of cases against two models, a report, and the one probe script here that talks to a real endpoint.

    export MOONLLM_BASE_URL="https://api.deepseek.com/v1"   # or any compatible base URL
export MOONLLM_MODEL="deepseek-chat"
export MOONLLM_API_KEY="sk-..."
make deps && MOON_CC=gcc moon build --target native     # build once
faceoff=./_build/native/debug/build/cmd/faceoff/faceoff.exe
bench=./_build/native/debug/build/cmd/bench/bench.exe
$faceoff "用一句话说明什么是航空母舰"                    # one-shot
$faceoff --stream "写一首关于侧风的短诗"                 # streamed, printed as fragments arrive
echo "总结一下这段日志" | $faceoff --stream              # prompt from stdin
mkdir -p my-run && printf '%s\n' '{"id":"math-short","prompt":"计算 17 × 23。只输出数字。"}' > my-run/cases.jsonl
$bench --models <model-a>,<model-b> --cases my-run/cases.jsonl --repeats 3 --max-tokens 2048 \
  --temperature 0.0 --pace-ms 1000 --retry 2 --json my-run/runs.jsonl
moon run --target native scripts/build-web.mbtx my-run/runs.jsonl     # → web/out/report.html
moon run --target native scripts/real-gateway.mbtx                    # → docs/real-gateway-run.md

    Two things to get right the first time: send --json somewhere of your own (bench/results-example.jsonl is a committed sample, not scratch space), and read the counters before the latency. If the gateway rate-limits you, --pace-ms spaces the attempts out and --retry re-sends 429/5xx with exponential backoff. The key lives in the environment (or in the page's memory for one run); it is never written to a run directory or a commit, see SECURITY.md. docs/cli.md has the flags, the environment variables, the suite format, the metrics and the two things in full.

    #5. Where the files land

    Runs, case sets and presets live outside the repository, in your user directories. On Linux that is ~/.local/share/faceoff/ and ~/.config/faceoff/; on Windows all three are under %LOCALAPPDATA%\faceoff\; macOS uses the XDG defaults. moon run --target native web/cmd/server -- --print-dirs prints the four paths as the server resolved them and exits, without creating a directory or starting anything.

    pathwhat
    $XDG_DATA_HOME/faceoff/cases/default.jsonlthe generated case set: seeded from bench/cases.example.jsonl the first time the server starts with an empty cases directory. Your own sets live here too
    $XDG_CONFIG_HOME/faceoff/presets.jsonpresets: models plus parameters
    $XDG_DATA_HOME/faceoff/runs/run history, one directory per run
    web/data.jsonpage data exported from a run; a build artifact inside the repository, gitignored
    web/out/built page and static report, produced by scripts/build-web.mbtx; a build artifact inside the repository, gitignored

    web/cases/, web/presets.json and web/runs/ are the pre-move positions. A checkout that still has one of them keeps using it until the new location exists; the server says so on stderr at startup and prints the mv command to run, and it never moves your files itself.

    #Known limits

    • --timeout-ms does not apply to the streaming path. A total-duration timeout would cut off legitimately long replies, and an idle timeout would need a timer around each read. The one-shot path does honor it.
    • The web server binds 127.0.0.1 and has no authentication. It is a local dev tool. Do not expose it.
    • Only the OpenAI-compatible wire format is implemented. No Anthropic or Gemini translation; the endpoint must accept /chat/completions.
    • Targets. The library and its two CLIs declare native only; in web/, cmd/app is js, cmd/server and cmd/ssg are native, and shared builds for js+native+wasm.
    • Pacing and retry defaults are heuristics. They were tuned against one gateway's rate limiter. Check your own with --pace-ms 0 and see what happens.

    #Usage index

    Four documents carry the detail; this section is the map, so the rest of this page stays a front page.

    documentcovers
    docs/cli.mdfaceoff and bench: installing them, every flag, the environment variables, the suite format, the metrics, rate limits and replay, the outputs, the loop end to end, and the probes against a real gateway
    docs/web.mdthe page and its server: what every panel does, the URL parameters, the HTTP API, the server environment, the static report, and the web/ layout
    docs/library.mdusing bench and the client as MoonBit packages: the import alias, one-shot, streaming, benchmarking, and the error type
    docs/testing.mdwhat make ci runs, the five suites (85 unit tests plus the end-to-end ones) and what each covers, why four of them are shaped that way, and the two suites CI does not run

    The commands you are most likely to want, with the document that explains them:

    commandwhat it doesdoc
    make demozero-API-key demo of all three pathsdocs/cli.md
    make serve / make devbuild the page, then serve it from web/; dev rebuilds on changedocs/web.md
    make serve-demomock endpoint + demo data + pagedocs/web.md
    make install / make uninstallinstall both CLIs into ~/.moon/bin / remove them againdocs/cli.md
    make ci / make e2ethe deterministic suite CI runs / that plus the browser testdocs/testing.md
    make real-gatewayfour probes against a real endpoint; needs a key, not in CIdocs/testing.md
    make webbuild the page into web/out/ and the static reportdocs/web.md

    #Provenance and dependencies

    faceoff is an original project, with no port and no vendored third-party code: the HTTP client, the streaming reader, the statistics, the comparison harness and the page are written here. There is no LLM client dependency either, and docs/library-survey.md records the Mooncakes library that was surveyed and tried, the gaps it had, and why it was removed again.

    packagelicenseused for
    moonbitlang/asyncApache-2.0HTTP client, streaming reads, the local server behind the page, and the concurrency the harness runs on
    moonbit-community/rabbitaApache-2.0the page app and the static report generator
    conglinyizhi/precssApache-2.0compiling web/styles/site.scss

    The wire format is the OpenAI-compatible POST /v1/chat/completions, the one interface every model in a comparison has to speak, and no OpenAI SDK or code is involved. No case set ships with the repository beyond bench/cases.example.jsonl: prompts belong to whoever wrote them.

    MIT. See LICENSE.

    #Further reading

    Thinking of contributing? These three will get you up to speed:

    • CONTRIBUTING.md: getting set up, the house rules (no Python, why the page consumes bench's data.json), and what to run before a pull request.
    • SECURITY.md: how to report a vulnerability, and what happens to your API key.
    • CHANGELOG.md: what changed between versions, including the rename from llm_client.

    Further reading:

    ClientError

    source
    pub(all) suberror ClientError {
      Transport(String)
      Status(code~ : Int, message~ : String)
      Decode(String)
    }

    Errors raised by this client.

    Both paths flatten their lower-level errors into this type so callers do not need to import the transport packages.

    ClientError::to_string

    source
    fn ClientError::to_string(self : ClientError) -> String

    ConfigError

    source
    pub(all) suberror ConfigError {
      MissingApiKey
      MissingValue(flag~ : String)
      BadNumber(flag~ : String, value~ : String)
      UnknownFlag(flag~ : String)
    }

    Errors raised while resolving settings or parsing arguments.

    ConfigError::to_string

    source
    fn ConfigError::to_string(self : ConfigError) -> String

    AskOutcome

    source
    pub(all) struct AskOutcome {
      content : String
      reasoning : String
      usage : TokenUsage?
      finish_reason : String?
    }

    Everything a non-streaming reply carries.

    The same information the streaming path assembles, and for a reason: without the stop reason and the token accounting, a reply that spent its whole budget thinking is indistinguishable from a broken client — the caller gets an empty string and no explanation for it.

    Cli

    source
    pub(all) struct Cli {
      settings : Settings
      prompt : String?
      stream : Bool
      show_cot : Bool
      quiet : Bool
      show_help : Bool
    }

    A parsed command line.

    Cli::parse

    source
    fn Cli::parse(env : Map[String, String], argv : Array[String]) -> Cli raise ConfigError

    Parse argv (without the program name) into a Cli.

    Settings

    source
    pub(all) struct Settings {
      api_key : String
      base_url : String
      model : String
      system : String
      temperature : Double?
      max_tokens : Int?
      timeout_ms : Int
      enable_thinking : Bool
      reasoning_effort : String?
    } derive(Eq)

    Connection and generation settings for an OpenAI-compatible endpoint.

    Settings::chat_completions_url

    source
    fn Settings::chat_completions_url(self : Settings) -> String

    The chat/completions URL for this base URL.

    Settings::equal

    source
    fn Settings::equal(Settings, Settings) -> Bool

    Settings::from_env

    source
    fn Settings::from_env(env : Map[String, String]) -> Settings

    Resolve settings from environment variables, applying defaults.

    Per-field lookup order:

    • API key: MOONLLM_API_KEY, OPENAI_API_KEY, LLM_API_KEY
    • base URL: MOONLLM_BASE_URL, OPENAI_BASE_URL
    • model: MOONLLM_MODEL, OPENAI_MODEL
    • system: MOONLLM_SYSTEM

    Settings::not_equal

    source
    fn Settings::not_equal(x : Settings, y : Settings) -> Bool

    SseEvent

    source
    pub(all) enum SseEvent {
      Content(String)
      Reasoning(String)
      Usage(TokenUsage)
      Finish(String)
      Done
      Ignore
    } derive(Eq, 
    Debug    )

    A meaningful line of an OpenAI-compatible server-sent event stream.

    SseEvent::equal

    source
    fn SseEvent::equal(SseEvent, SseEvent) -> Bool

    SseEvent::not_equal

    source
    fn SseEvent::not_equal(x : SseEvent, y : SseEvent) -> Bool

    SseEvent::to_repr

    source

    StreamOutcome

    source
    pub(all) struct StreamOutcome {
      content : String
      reasoning : String
      usage : TokenUsage?
      finish_reason : String?
    }

    Everything collected from one streamed reply.

    StreamPart

    source
    pub(all) enum StreamPart {
      Reasoning(String)
      Content(String)
    } derive(Eq, 
    Debug    )

    One incremental piece of a streamed reply.

    StreamPart::equal

    source
    fn StreamPart::equal(StreamPart, StreamPart) -> Bool

    StreamPart::not_equal

    source
    fn StreamPart::not_equal(x : StreamPart, y : StreamPart) -> Bool

    StreamPart::to_repr

    source

    TokenUsage

    source
    pub(all) struct TokenUsage {
      prompt_tokens : Int
      completion_tokens : Int
      reasoning_tokens : Int
    } derive(Eq, 
    Debug    )

    Token accounting from a usage block.

    Reasoning models report reasoning_tokens separately; on those, the completion count is thinking plus answer, which is why it is tracked here rather than collapsed into one number.

    TokenUsage::equal

    source
    fn TokenUsage::equal(TokenUsage, TokenUsage) -> Bool

    TokenUsage::new

    source
    fn TokenUsage::new(prompt_tokens : Int, completion_tokens : Int, reasoning_tokens : Int) -> TokenUsage

    Build a token accounting record.

    TokenUsage::not_equal

    source
    fn TokenUsage::not_equal(x : TokenUsage, y : TokenUsage) -> Bool

    TokenUsage::reasoning_ratio

    source
    fn TokenUsage::reasoning_ratio(self : TokenUsage) -> Double

    The share of the completion budget spent thinking, in [0, 1].

    Returns 0 when the model reported no completion tokens at all.

    TokenUsage::to_repr

    source

    ask

    source
    async fn ask(settings : Settings, prompt : String) -> String raise ClientError

    Send prompt and return the complete reply text, without streaming.

    ask_outcome

    source
    async fn ask_outcome(settings : Settings, prompt : String) -> AskOutcome raise ClientError

    Send prompt and return the complete reply, with the stop reason and the token accounting alongside it.

    Talks to the endpoint directly, same as the streaming path, so this module depends on nothing but the standard library and the async runtime. Settings::timeout_ms is honored here (the streaming path deliberately leaves it to the OS).

    default_base_url

    source
    let default_base_url : String

    default_model

    source
    let default_model : String

    default_system

    source
    let default_system : String

    default_timeout_ms

    source
    let default_timeout_ms : Int

    parse_sse_line

    source
    fn parse_sse_line(line : String) -> SseEvent?

    Parse one SSE line from a chat completions stream.

    None means "skip": a blank line, a comment, or a non-data: field.

    read_prompt_from_stdin

    source
    async fn read_prompt_from_stdin() -> String

    Read the whole of standard input, trimmed, as a prompt.

    request_body

    source
    fn request_body(settings : Settings, prompt : String, stream? : Bool) -> Json

    Build the chat completion request body for prompt.

    A plain Json value on purpose: the wire format is the contract, so nothing from a client library shows up in this module's public API.

    response_outcome

    source
    fn response_outcome(json : Json) -> AskOutcome raise ClientError

    Decode a non-streaming chat response body into everything it carries.

    response_text

    source
    fn response_text(json : Json) -> String raise ClientError

    Decode a non-streaming chat response body and return its first choice text.

    Just the text. response_outcome is what a caller wants when an empty answer needs an explanation.

    stream_chat

    source
    async fn stream_chat(settings : Settings, prompt : String, on_delta : async (String) -> Unit) -> String raise ClientError

    Stream a reply, forwarding only the visible answer fragments.

    stream_parts

    source
    async fn stream_parts(settings : Settings, prompt : String, on_part : async (StreamPart) -> Unit) -> StreamOutcome raise ClientError

    Stream a chat completion, forwarding every fragment to on_part and returning the assembled reply.

    This is the low-level streaming entry point; it reports reasoning fragments as well as answer fragments, which is what a benchmark needs. For a plain text stream, use stream_chat.

    Talk to the endpoint directly and frame the SSE stream here. Delegating this to a client library would mean handing fragments to a synchronous callback, and a synchronous callback cannot write to async stdout as they arrive — which is exactly what streaming is for.

    Settings::timeout_ms is not applied here: a total-duration timeout would cut off legitimately long streams, and an idle timeout would need a timer around each read. The one-shot path does honor it.

    stream_to_stdout

    source
    async fn stream_to_stdout(settings : Settings, prompt : String) -> String raise ClientError

    Stream a reply straight to standard output.

    Write failures on stdout (a closed pipe, for example) are dropped: there is nothing useful left to do with them, and the caller still gets the text.

    usage

    source
    let usage : String

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io