🥮 mooncakes.io
    Sign in

    llm

    A dialect-agnostic LLM IR, one wire mapping per provider protocol, a data-driven provider registry, and a transport a browser can implement

    llm
    ai
    anthropic
    openai
    gemini
    moonbit
    Download zip
    Version
    0.1.3
    License
    Apache-2.0
    Repository
    Last updated
    last month
    Downloads
    425

    #marianoguerra/llm

    A dialect-agnostic LLM IR, one wire mapping per provider protocol, a registry that makes a fifth provider a struct literal, and a transport a browser can implement.

    Modelled on pi.dev's packages/ai, which draws the same three lines: an api is a wire protocol, a provider is a vendor with credentials and a model catalog, and a collection routes between them.

    moon add marianoguerra/llm

    let registry = @catalog.default_registry()
    guard registry.dialect_of("anthropic:claude-sonnet-5") is Some(dialect)
    let body = dialect.lower_context(context)   // -> provider request JSON
    let reply = dialect.lift_message(json)      // -> ContextMessage

    Switching model mid-session is passing the same LlmContext to a different Dialect. No session state migrates, because none exists.

    #Packages

    PackageWhat it is
    llmLlmContext and the messages, blocks, usage and per-turn params it is made of; the Dialect trait; StreamEvent; ModelInfo; the Registry; LlmTransport and RelayTransport
    llm/anthropicanthropic-messages
    llm/openaiopenai-responses and openai-completions, plus a probed capability table
    llm/geminigoogle-generative-ai — the one whose endpoint carries the model
    llm/openrouteropenrouter, which borrows the completions mapping
    llm/catalogThe four providers as data. Credential-free
    llm/wireReading a key out of an environment, resolving an endpoint with it, and an HTTP transport. The only native package here

    Everything except llm/wire is target-neutral, so a browser page can lower a context, choose a dialect and lift a reply with no server involved — the page does both stages and a relay only forwards bytes, never learning which provider is on the other end. preferred_target is wasm-gc so that stays true by default; just check covers native too.

    #What is NOT here

    No agent loop, no tool executor, no session store, no prompt templating. This is the layer under those: what a turn's input is, how it becomes one provider's JSON and back, and where the bytes go. It grew out of a hypermedia agent that needed exactly this much and nothing above it.

    #Adding a provider

    If it speaks a protocol that already has an ApiSpec, it is a value:

    let registry = @catalog.default_registry().with_provider({
      id: "lmstudio",
      name: "LM Studio",
      api: "openai-completions",
      endpoint: "http://localhost:1234/v1/chat/completions",
      auth: NoAuth,
      key_env: "",
      model_env: "",
      default_model: "qwen3-coder",
      models: [@llm.ModelInfo::permissive("qwen3-coder")],
      headers: { "content-type": "application/json" },
      telemetry_name: "lmstudio",
      server_address: "localhost",
    })

    {model} in endpoint is substituted where a provider puts the model in the URL instead of the body. auth is BearerHeader | KeyHeader(name) |KeyQuery(name) | NoAuth — the credential itself is never in a ProviderSpec, only the name of the variable holding it, which is what lets one compile into a page.

    #Adding a wire protocol

    A new Dialect impl plus an ApiSpec naming it, handed to Registry::with_api. Nothing in this package has a match over provider names to edit.

    pub fn api_spec() -> @llm.ApiSpec {
      { id: "my-api", make: info => MyDialect::make(model=info.id), caps: id => @llm.ModelInfo::permissive(id) }
    }

    The trait is six methods, three of which have defaults that buffer the body and lift it whole — so a dialect is id, lower_context and lift_message until it wants real streaming.

    #Two things that look odd and are not

    stream_feed returns events, not a message. Providers differ in how a response is framed — event names, delta shapes, where usage and the stop reason arrive — and decoding that anywhere but inside the dialect would mean adding a provider in two places. lift_message is the non-streaming special case of stream_init ▸ stream_feed ▸ stream_finish.

    LlmTransport is in continuations, not async. A browser cannot have async: moonbitlang/async's event loop is unimplemented for wasm-gc, and an async function cannot be called from the plain exported functions a page starts from. A trait only a server could implement would not be a seam, so the native side parks on a semaphore instead.

    Dialect

    source
    pub(open) trait Dialect {
      fn id(Self) -> String
      fn model(Self) -> String = _
      fn lower_context(Self, LlmContext) -> Json
      fn lift_message(Self, Json) -> ContextMessage raise DialectError
      fn stream_init(Self) -> StreamState = _
      fn stream_feed(Self, StreamState, chunk~ : String) -> (StreamState, Array[StreamEvent]) raise DialectError = _
      fn stream_finish(Self, StreamState) -> ContextMessage raise DialectError = _
    }

    One provider protocol, both directions.

    Both are pure: switching LLMs mid-session is passing the same LlmContext to a different Dialect. Streaming decode (stream_*) keeps SSE framing knowledge inside the dialect, so a deployment can put the dialect on either side of a network boundary — the transport stays a byte pump that never learns which provider is on the other end.

    LlmTransport

    source
    pub(open) trait LlmTransport {
      fn send(Self, handle~ : String, target~ : String, body~ : Json, headers~ : Map[String, String], on_chunk~ : (String) -> Unit, on_done~ : () -> Unit, on_error~ : (String) -> Unit) -> Unit
      fn abort(Self, handle~ : String) -> Unit = _
    }

    Sends an already-lowered provider body to a NAMED target and pumps raw chunks back. It never inspects or rewrites the body and never learns the dialect.

    target is a name, not an endpoint — whoever holds the credential resolves it in their own table. That is what lets the same trait be an HTTP client with a key in it and a page posting to a relay that has one.

    CONTINUATIONS, not async, and that is not a style choice: a browser cannot have async — moonbitlang/async's event loop is unimplemented for wasm-gc, and an async function cannot be called from the plain exported functions a page starts from. A trait only one of the two sides could implement would not be a seam. The native side pays for this by parking on a semaphore, which is the cheaper half of the trade.

    on_error is for a call that never happened. A provider answering 429 or 400 answers with a BODY, and that body is a chunk: errors are data, and the dialect is what reads them.

    DialectError

    source
    pub(all) suberror DialectError {
      MalformedReply(String)
    } derive(ToJson, 
    Debug    )

    ApiSpec

    source
    pub(all) struct ApiSpec {
      id : String
      make : (ModelInfo) -> &Dialect
      caps : (String) -> ModelInfo
    }

    One wire protocol, and how to build a dialect that speaks it.

    make is what replaces a match provider somewhere: a new API is a value handed to a registry, not an arm somebody has to find. caps is what this API assumes about a model nobody wrote a row for — it lives with the dialect that reads it, because that is the code a wrong assumption breaks.

    Closures, so this is the one type here that derives nothing. Everything a proxy might want to hand a page — ProviderSpec, ModelInfo — is data and does.

    AssistantBlock

    source
    pub(all) enum AssistantBlock {
      AssistantText(text~ : String, signature~ : String?)
      Thinking(text~ : String, signature~ : String?, redacted~ : Bool)
      ToolCall(id~ : String, name~ : String, arguments~ : Json, signature~ : String?)
    } derive(ToJson, 
    Debug    )

    signature on all three is the provider's opaque proof that this block is the model's own output, replayed verbatim on the next turn or dropped. Anthropic signs thinking; Gemini 3 signs whatever part carried the reasoning — including tool calls, where it is mandatory: a functionCall replayed without its thoughtSignature is a 400. It survives whatever round-trip the embedding application puts the message through, and a dialect only replays a signature its own model produced.

    AuthStyle

    source
    pub(all) enum AuthStyle {
      BearerHeader
      KeyHeader(String)
      KeyQuery(String)
      NoAuth
    } derive(Eq, ToJson, 
    Debug    )

    How a provider is authenticated.

    The credential is never here. A registry is DATA — it compiles into a page — and data that names an environment variable is safe to ship where its value is not.

    CacheRetention

    source
    pub(all) enum CacheRetention {
      NoCache
      ShortCache
      LongCache
    } derive(Eq, ToJson, 
    Debug    )

    How long a provider may keep the cached prompt prefix: not at all, its own default (minutes, in memory), or the extended tier.

    CacheRetention::parse

    source
    fn CacheRetention::parse(s : String) -> CacheRetention?

    ContextMessage

    source
    pub(all) enum ContextMessage {
      UserMsg(content~ : Array[UserBlock])
      AssistantMsg(content~ : Array[AssistantBlock], stop~ : StopReason, model~ : String, response_id~ : String?, usage~ : Usage?, error~ : String?)
      ToolResultMsg(tool_call_id~ : String, tool_name~ : String, content~ : Array[UserBlock], is_error~ : Bool)
    } derive(ToJson, 
    Debug    )

    Cost

    source
    pub(all) struct Cost {
      input : Double
      output : Double
      cache_read : Double
      cache_write : Double
    } derive(Eq, ToJson, 
    Debug    )

    USD per million tokens. Separate from Usage on purpose: usage is what a call actually spent, this is what the provider charges, and the two are updated by completely different people.

    LlmContext

    source
    pub(all) struct LlmContext {
      system : String?
      messages : Array[ContextMessage]
      tools : Array[ToolDecl]
      params : LlmParams
    } derive(ToJson, 
    Debug    )

    One turn's whole input: what the model is told, what it has said, what it may call, and how hard it should think.

    LlmParams

    source
    pub(all) struct LlmParams {
      effort : ThinkingLevel?
      pro_mode : Bool
      reasoning_context : ReasoningContext?
      reasoning_summary : Bool
      max_output_tokens : Int?
      cache_key : String?
      cache_retention : CacheRetention
    } derive(ToJson, 
    Debug    )

    The knobs a turn is asked for, dialect-agnostic on purpose: they travel with the LlmContext so that switching model mid-session keeps meaning "pass the same context to a different Dialect". A dialect maps what its API can express and silently drops the rest — the same rule as a signature it cannot replay.

    LlmParams::default

    source
    fn LlmParams::default() -> LlmParams

    Modality

    source
    pub(all) enum Modality {
      TextIn
      ImageIn
    } derive(Eq, ToJson, 
    Debug    )

    What a turn may carry INTO a model. Nothing reads this yet; it is here because a catalog that cannot say "this one has no vision" makes the caller find out from a 400.

    ModelInfo

    source
    pub(all) struct ModelInfo {
      id : String
      api : String?
      reasoning : Bool
      thinking_levels : Array[(ThinkingLevel, String)]
      supports_pro_mode : Bool
      reasoning_contexts : Array[ReasoningContext]
      input : Array[Modality]
      context_window : Int
      max_output_tokens : Int?
      cost : Cost?
    } derive(ToJson, 
    Debug    )

    One model's request-side capabilities (pi's Model).

    thinking_levels maps the portable dial to this model's own spelling (pi's thinkingLevelMap); a level absent from the list does not exist here and clamps to the nearest one that does — asking for one a model lacks is a 400, not a downgrade. reasoning_contexts and supports_pro_mode are pass/drop rather than clamp: there is no nearly-pro.

    Data, not code, so a new model is a row. That is how pi.dev does it and it is why Registry::with_provider can add a vendor without a code path.

    ModelInfo::accepts

    source
    fn ModelInfo::accepts(self : ModelInfo, modality : Modality) -> Bool

    ModelInfo::allows_context

    source
    fn ModelInfo::allows_context(self : ModelInfo, context : ReasoningContext) -> Bool

    ModelInfo::clamp_effort

    source
    fn ModelInfo::clamp_effort(self : ModelInfo, level : ThinkingLevel) -> String?

    ModelInfo::permissive

    source
    fn ModelInfo::permissive(id : String) -> ModelInfo

    ProviderSpec

    source
    pub(all) struct ProviderSpec {
      id : String
      name : String
      api : String
      endpoint : String
      auth : AuthStyle
      key_env : String
      model_env : String
      default_model : String
      models : Array[ModelInfo]
      headers : Map[String, String]
      telemetry_name : String
      server_address : String
    } derive(
    Debug    )

    One provider: an endpoint, a credential shape, and the models it offers.

    All data. That is the point — pi's registerProvider({baseUrl, apiKey:"$MY_KEY", api: "openai-completions", models: [...]}) adds an OpenAI-compatible vendor with no code at all, and this is the same bet: a new provider is a row.

    models is both the offer and the capability table. Two arrays that had to agree is what a guard test used to be for.

    ReasoningContext

    source
    pub(all) enum ReasoningContext {
      AutoContext
      CurrentTurn
      AllTurns
    } derive(Eq, ToJson, 
    Debug    )

    How much of the earlier turns' reasoning the provider may reuse (OpenAI's reasoning.context, new with the gpt-5.6 family). Distinct from replaying reasoning items ourselves, which every turn already does: this asks the provider to render its own retained chain into the context.

    ReasoningContext::parse

    source
    fn ReasoningContext::parse(s : String) -> ReasoningContext?

    Registry

    source
    pub(all) struct Registry {
      apis : Map[String, ApiSpec]
      providers : Array[ProviderSpec]
    }

    Every API and provider a build knows.

    A VALUE threaded by whoever holds it, not a mutable global: two tests can hold different registries, a caller can add a provider without touching the package that ships the defaults, and nothing is initialized behind anybody's back.

    Registry::api

    source
    fn Registry::api(self : Registry, id : String) -> ApiSpec?

    Registry::dialect_of

    source
    fn Registry::dialect_of(self : Registry, label : String) -> &Dialect?

    Registry::endpoint

    source
    fn Registry::endpoint(self : Registry, provider~ : String, model~ : String, key~ : String) -> Result[(String, Map[String, String]), String]

    Registry::labels

    source
    fn Registry::labels(self : Registry) -> Array[String]

    Registry::model_info

    source
    fn Registry::model_info(self : Registry, provider~ : String, model~ : String) -> ModelInfo?

    Registry::of

    source
    fn Registry::of(apis : Array[ApiSpec], providers : Array[ProviderSpec]) -> Registry

    Registry::provider

    source
    fn Registry::provider(self : Registry, id : String) -> ProviderSpec?

    Registry::provider_for_api

    source
    fn Registry::provider_for_api(self : Registry, api : String) -> ProviderSpec?

    Registry::with_api

    source
    fn Registry::with_api(self : Registry, spec : ApiSpec) -> Registry

    Registry::with_provider

    source
    fn Registry::with_provider(self : Registry, spec : ProviderSpec) -> Registry

    RelayTransport

    source
    pub(all) struct RelayTransport {
      post : (String, String, String, (String) -> Unit, (String) -> Unit) -> Unit
      stop : (String) -> Unit
    }

    A transport that hands the call to something else holding the credential — a page posting to its own server, a worker posting to a gateway.

    post is (handle, target, body as text, ok, err): the relay is told which target to resolve and given bytes to forward, and answers with the whole reply. That it takes TEXT rather than Json is the point — a relay is not supposed to parse what it is relaying.

    ok delivering the whole body is what a buffering relay does. When one streams, this is the line that changes and nothing above it does: the dialect already decodes incrementally.

    RelayTransport::make

    source
    fn RelayTransport::make(post~ : (String, String, String, (String) -> Unit, (String) -> Unit) -> Unit, stop? : (String) -> Unit) -> RelayTransport

    StopReason

    source
    pub(all) enum StopReason {
      EndTurn
      MaxTokens
      ToolUse
      Errored
      Aborted
    } derive(Eq, ToJson, 
    Debug    )

    pi: stop | length | toolUse | error | aborted — errors are data, never exceptions.

    StreamEvent

    source
    pub(all) enum StreamEvent {
      Start(model~ : String)
      TextStart(content_index~ : Int)
      TextDelta(content_index~ : Int, delta~ : String)
      TextEnd(content_index~ : Int)
      ThinkingStart(content_index~ : Int)
      ThinkingDelta(content_index~ : Int, delta~ : String)
      ThinkingEnd(content_index~ : Int)
      ToolCallStart(content_index~ : Int, id~ : String, name~ : String)
      ToolCallDelta(content_index~ : Int, delta~ : String)
      ToolCallEnd(content_index~ : Int)
      Done(stop~ : StopReason, usage~ : Usage?)
      Error(message~ : String)
    } derive(Eq, ToJson, 
    Debug    )

    A step of the reply, as the dialect decoded it (pi's AssistantMessageEvent).

    content_index is the block's position in the assistant message being built — the same index stream_finish will hand back as content[i] — so a consumer can key a partial render by it without knowing which wire format produced it. Events for different blocks are not guaranteed contiguous: a provider may interleave a tool call's arguments with the text after it, which is exactly why the index is on every event and not implied by order.

    Nothing here mentions a transcript, a message id or a media type. Those belong to whoever is embedding this, and a dialect that had to know them could not be published on its own.

    StreamState

    source
    pub(all) struct StreamState {
      chunks : Array[String]
    } derive(
    Debug    )

    Accumulator threaded through incremental decoding. A dialect that decodes true per-chunk SSE keeps partial-block state here; the default encoding simply buffers raw chunks and lifts the whole reply at stream_finish.

    StreamState::new

    source

    StreamState::push

    source
    fn StreamState::push(self : StreamState, chunk : String) -> StreamState

    Append a chunk, returning a new state (pure — the shell owns the loop).

    StreamState::text

    source
    fn StreamState::text(self : StreamState) -> String

    The accumulated raw body so far.

    ThinkingLevel

    source
    pub(all) enum ThinkingLevel {
      Off
      Minimal
      Low
      Medium
      High
      XHigh
      Max
    } derive(Compare, Eq, ToJson, 
    Debug    )

    How hard the model should think, as a dial that means the same thing on every provider (pi's ThinkingLevel). Ordered low to high: a dialect maps it to whatever its own model spells, and clamps to the nearest level that model actually has — the levels are not the same everywhere, and a request for one a model lacks should land next to it rather than 400.

    ThinkingLevel::parse

    source
    fn ThinkingLevel::parse(s : String) -> ThinkingLevel?

    ToolDecl

    source
    pub(all) struct ToolDecl {
      name : String
      description : String
      input_schema : Json
    } derive(ToJson, 
    Debug    )

    Usage

    source
    pub(all) struct Usage {
      input : Int
      output : Int
      cache_read : Int
      cache_write : Int
    } derive(Eq, ToJson, 
    Debug    )

    Usage::from_json

    source
    fn Usage::from_json(value : Json) -> Usage?

    Usage::make

    source
    fn Usage::make(input~ : Int, output~ : Int, cache_read? : Int, cache_write? : Int) -> Usage

    UserBlock

    source
    pub(all) enum UserBlock {
      UserText(String)
      UserImage(media_type~ : String, data~ : Bytes)
    } derive(ToJson, 
    Debug    )

    api_error

    source
    fn api_error(reply : Json, model~ : String) -> ContextMessage?

    Provider API errors are data on the message, never exceptions (pi convention). Matches the {"error": {"message": ...}} shape all four providers answer with (Anthropic wraps it in {"type": "error", ...}, which this pattern also matches).

    reply_identity

    source
    fn reply_identity(reply : Json, fallback~ : String) -> (String, String?)

    Which model answered, and what the provider called the call.

    The tail of every lift_message, and identical in all of them: all four providers name the model in model and the call in id, at the top level of the reply. fallback is what the dialect was configured with, for a provider that echoes neither.

    This is here and the usage block above it is not, because usage is keyed by provider JSON key names — input_tokens against prompt_tokens — and this is not keyed by anything. That is the line the header draws.

    split_label

    source
    fn split_label(label : String) -> (String, String)

    user_text

    source
    fn user_text(blocks : Array[UserBlock]) -> String

    Flatten user blocks to one text: text passes through, images degrade to a marker (v1 wire formats are text-first).

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io