llm
A dialect-agnostic LLM IR, one wire mapping per provider protocol, a data-driven provider registry, and a transport a browser can implement
llm
ai
anthropic
openai
gemini
moonbit
Author
Version
0.3.0
License
Apache-2.0
Repository
Last updated
9 hours ago
Downloads
52
Dependencies
- moonbitlang/async0.21.2
#marianoguerra/llm
A dialect-agnostic LLM IR, one wire mapping per provider protocol, a registry
that makes a sixth provider a struct literal, and a transport a browser can
implement.
Modelled on pi.dev's packages/ai, which draws the same
three lines: an api is a wire protocol, a provider is a vendor with
credentials and a model catalog, and a collection routes between them.
moon add marianoguerra/llmlet registry = @catalog.default_registry()
guard registry.dialect_of("anthropic:claude-sonnet-5") is Some(dialect)
let body = dialect.lower_context(context) // -> provider request JSON
let reply = dialect.lift_message(json) // -> ContextMessageSwitching model mid-session is passing the same LlmContext to a different
Dialect. No session state migrates, because none exists.
#Packages
| Package | What it is |
|---|---|
| llm | LlmContext and the messages, blocks, usage and per-turn params it is made of; the Dialect trait; StreamEvent; ModelInfo; the Registry; LlmTransport and RelayTransport |
| llm/anthropic | anthropic-messages |
| llm/openai | openai-responses and openai-completions, plus a probed capability table |
| llm/gemini | google-generative-ai — the one whose endpoint carries the model |
| llm/openrouter | openrouter, which borrows the completions mapping |
| llm/catalog | The five providers as data — Cerebras among them, riding openai-completions with no dialect of its own. Credential-free |
| llm/wire | Reading a key out of an environment, resolving an endpoint with it, and an HTTP transport. The only native library here |
| cmd/smoke | A live check against every provider whose key is set. Native, and never run by just ci |
Every library here except llm/wire is target-neutral, so a browser page can lower a
context, choose a dialect and lift a reply with no server involved — the page
does both stages and a relay only forwards bytes, never learning which
provider is on the other end. preferred_target is wasm-gc so that stays
true by default; just check covers native too, and just targets-check
fails if any importable package but wire picks up a target restriction.
cmd/smoke is native and exempt, because nothing can import an executable.
#What is NOT here
No agent loop, no tool executor, no session store, no prompt templating. This
is the layer under those: what a turn's input is, how it becomes one
provider's JSON and back, and where the bytes go. It grew out of a hypermedia
agent that needed exactly this much and nothing above it.
#Adding a provider
If it speaks a protocol that already has an ApiSpec, it is a value:
let registry = @catalog.default_registry().with_provider({
id: "lmstudio",
name: "LM Studio",
api: "openai-completions",
endpoint: "http://localhost:1234/v1/chat/completions",
auth: NoAuth,
key_env: "",
model_env: "",
default_model: "qwen3-coder",
models: [@llm.ModelInfo::permissive("qwen3-coder")],
headers: { "content-type": "application/json" },
telemetry_name: "lmstudio",
server_address: "localhost",
})llm/catalog's cerebras_provider() is exactly this shape, shipped: Cerebras
speaks openai-completions, so it is a row with no dialect, no ApiSpec and
no code path — the same way pi.dev models it.
{model} in endpoint is substituted where a provider puts the model in the
URL instead of the body. auth is BearerHeader | KeyHeader(name) |KeyQuery(name) | NoAuth — the credential itself is never in a ProviderSpec,
only the name of the variable holding it, which is what lets one compile into a
page.
#Adding a wire protocol
A new Dialect impl plus an ApiSpec naming it, handed to
Registry::with_api. Nothing in this package has a match over provider names
to edit.
pub fn api_spec() -> @llm.ApiSpec {
{ id: "my-api", make: info => MyDialect::make(model=info.id), caps: id => @llm.ModelInfo::permissive(id) }
}The trait is six methods, three of which have defaults that buffer the body and
lift it whole — so a dialect is id, lower_context and lift_message until
it wants real streaming.
#Checking it against the real thing
moon test is offline and free — every dialect test lowers a context and lifts
a canned reply. To see whether the mapping still matches what the providers
actually do:
just smokeIt sends two real requests to every provider whose key is in the
environment, and skips the rest rather than failing them. .env.example
lists every variable the module reads, with what each one turns on:
cp .env.example .env # then fill in the keys you have
set -a; . ./.env; set +a
``` Each provider gets
its cheapest model, `effort: Off` clamped onto whatever the model's row says is
least, and a reply capped at a few dozen tokens — a full five-provider run is
fractions of a cent.
The two cases are the two halves of a mapping: one plain turn (system prompt,
user turn, text and usage back) and one with a tool declared (tool lowering,
and a `ToolCall` lifted with its arguments parsed back out of the JSON string
they travel as). Both go through `stream_init`/`stream_feed`/`stream_finish`,
which is the path a real caller drives.
It lives in `cmd/smoke` as an executable rather than a `_test.mbt`, because
`moon test` runs everything it finds and this costs money and needs
credentials. `just ci` does not run it.
## Two things that look odd and are not
**`stream_feed` returns events, not a message.** Providers differ in how a
response is *framed* — event names, delta shapes, where usage and the stop
reason arrive — and decoding that anywhere but inside the dialect would mean
adding a provider in two places. `lift_message` is the non-streaming special
case of `stream_init ▸ stream_feed ▸ stream_finish`.
**`LlmTransport` is in continuations, not `async`.** A browser cannot have
async: `moonbitlang/async`'s event loop is unimplemented for wasm-gc, and an
async function cannot be called from the plain exported functions a page starts
from. A trait only a server could implement would not be a seam, so the native
side parks on a semaphore instead.Dialect
pub(open) trait Dialect {
fn id(Self) -> String
fn model(Self) -> String = _
fn lower_context(Self, LlmContext) -> Json
fn lift_message(Self, Json) -> ContextMessage raise DialectError
fn stream_init(Self) -> StreamState = _
fn stream_feed(Self, StreamState, chunk~ : String) -> (StreamState, Array[StreamEvent]) raise DialectError = _
fn stream_finish(Self, StreamState) -> ContextMessage raise DialectError = _
}One provider protocol, both directions.
Both are pure: switching LLMs mid-session is passing the same LlmContext
to a different Dialect. Streaming decode (stream_*) keeps SSE framing
knowledge inside the dialect, so a deployment can put the dialect on
either side of a network boundary — the transport stays a byte pump that
never learns which provider is on the other end.
LlmTransport
Sends an already-lowered provider body to a NAMED target and pumps raw
chunks back. It never inspects or rewrites the body and never learns the
dialect.
target is a name, not an endpoint — whoever holds the credential
resolves it in their own table. That is what lets the same trait be an
HTTP client with a key in it and a page posting to a relay that has one.
CONTINUATIONS, not async, and that is not a style choice: a browser
cannot have async — moonbitlang/async's event loop is unimplemented for
wasm-gc, and an async function cannot be called from the plain exported
functions a page starts from. A trait only one of the two sides could
implement would not be a seam. The native side pays for this by parking
on a semaphore, which is the cheaper half of the trade.
on_error is for a call that never happened. A provider answering 429 or
400 answers with a BODY, and that body is a chunk: errors are data, and
the dialect is what reads them.
DialectError
impl Show for DialectErrorApiSpec
One wire protocol, and how to build a dialect that speaks it.
make is what replaces a match provider somewhere: a new API is a value
handed to a registry, not an arm somebody has to find. caps is what this
API assumes about a model nobody wrote a row for — it lives with the
dialect that reads it, because that is the code a wrong assumption breaks.
Closures, so this is the one type here that derives nothing. Everything a
proxy might want to hand a page — ProviderSpec, ModelInfo — is data
and does.
AssistantBlock
signature on all three is the provider's opaque proof that this block is
the model's own output, replayed verbatim on the next turn or dropped.
Anthropic signs thinking; Gemini 3 signs whatever part carried the
reasoning — including tool calls, where it is mandatory: a functionCall
replayed without its thoughtSignature is a 400. It survives whatever
round-trip the embedding application puts the message through, and a
dialect only replays a signature its own model produced.
AuthStyle
How a provider is authenticated.
The credential is never here. A registry is DATA — it compiles into a page
— and data that names an environment variable is safe to ship where its
value is not.
CacheRetention
How long a provider may keep the cached prompt prefix: not at all, its
own default (minutes, in memory), or the extended tier.
ContextMessage
pub(all) enum ContextMessage {
UserMsg(content~ : Array[UserBlock])
AssistantMsg(content~ : Array[AssistantBlock], stop~ : StopReason, model~ : String, response_id~ : String?, usage~ : Usage?, error~ : String?)
ToolResultMsg(tool_call_id~ : String, tool_name~ : String, content~ : Array[UserBlock], is_error~ : Bool)
} derive(ToJson, Debug)Cost
USD per million tokens. Separate from Usage on purpose: usage is what a
call actually spent, this is what the provider charges, and the two are
updated by completely different people.
LlmContext
One turn's whole input: what the model is told, what it has said, what it
may call, and how hard it should think.
LlmParams
pub(all) struct LlmParams {
effort : ThinkingLevel?
pro_mode : Bool
reasoning_context : ReasoningContext?
reasoning_summary : Bool
max_output_tokens : Int?
cache_key : String?
cache_retention : CacheRetention
} derive(ToJson, Debug)The knobs a turn is asked for, dialect-agnostic on purpose: they travel
with the LlmContext so that switching model mid-session keeps meaning
"pass the same context to a different Dialect". A dialect maps what its
API can express and silently drops the rest — the same rule as a
signature it cannot replay.
Modality
What a turn may carry INTO a model. Nothing reads this yet; it is here
because a catalog that cannot say "this one has no vision" makes the
caller find out from a 400.
ModelInfo
pub(all) struct ModelInfo {
id : String
api : String?
reasoning : Bool
thinking_levels : Array[(ThinkingLevel, String)]
supports_pro_mode : Bool
reasoning_contexts : Array[ReasoningContext]
input : Array[Modality]
context_window : Int
max_output_tokens : Int?
cost : Cost?
} derive(ToJson, Debug)One model's request-side capabilities (pi's Model).
thinking_levels maps the portable dial to this model's own spelling
(pi's thinkingLevelMap); a level absent from the list does not exist
here and clamps to the nearest one that does — asking for one a model
lacks is a 400, not a downgrade. reasoning_contexts and
supports_pro_mode are pass/drop rather than clamp: there is no nearly-pro.
Data, not code, so a new model is a row. That is how pi.dev does it and
it is why Registry::with_provider can add a vendor without a code path.
ModelInfo::allows_context
ModelInfo::clamp_effort
ProviderSpec
One provider: an endpoint, a credential shape, and the models it offers.
All data. That is the point — pi's registerProvider({baseUrl, apiKey:"$MY_KEY", api: "openai-completions", models: [...]}) adds an
OpenAI-compatible vendor with no code at all, and this is the same bet:
a new provider is a row.
models is both the offer and the capability table. Two arrays that had
to agree is what a guard test used to be for.
ReasoningContext
How much of the earlier turns' reasoning the provider may reuse
(OpenAI's reasoning.context, new with the gpt-5.6 family). Distinct
from replaying reasoning items ourselves, which every turn already does:
this asks the provider to render its own retained chain into the context.
Registry
Every API and provider a build knows.
A VALUE threaded by whoever holds it, not a mutable global: two tests can
hold different registries, a caller can add a provider without touching
the package that ships the defaults, and nothing is initialized behind
anybody's back.
Registry::endpoint
Registry::model_info
Registry::of
Registry::provider_for_api
Registry::with_provider
RelayTransport
pub(all) struct RelayTransport {
post : (String, String, String, (String) -> Unit, (String) -> Unit) -> Unit
stop : (String) -> Unit
}A transport that hands the call to something else holding the credential
— a page posting to its own server, a worker posting to a gateway.
post is (handle, target, body as text, ok, err): the relay is told
which target to resolve and given bytes to forward, and answers with the
whole reply. That it takes TEXT rather than Json is the point — a relay
is not supposed to parse what it is relaying.
ok delivering the whole body is what a buffering relay does. When one
streams, this is the line that changes and nothing above it does: the
dialect already decodes incrementally.
impl LlmTransport for RelayTransportfn send(self : RelayTransport, handle~ : String, target~ : String, body~ : Json, headers~ : Map[String, String], on_chunk~ : (String) -> Unit, on_done~ : () -> Unit, on_error~ : (String) -> Unit) -> UnitRelayTransport::make
fn RelayTransport::make(post~ : (String, String, String, (String) -> Unit, (String) -> Unit) -> Unit, stop? : (String) -> Unit) -> RelayTransportStopReason
pi: stop | length | toolUse | error | aborted — errors are data, never
exceptions.
StreamEvent
pub(all) enum StreamEvent {
Start(model~ : String)
TextStart(content_index~ : Int)
TextDelta(content_index~ : Int, delta~ : String)
TextEnd(content_index~ : Int)
ThinkingStart(content_index~ : Int)
ThinkingDelta(content_index~ : Int, delta~ : String)
ThinkingEnd(content_index~ : Int)
ToolCallStart(content_index~ : Int, id~ : String, name~ : String)
ToolCallDelta(content_index~ : Int, delta~ : String)
ToolCallEnd(content_index~ : Int)
Done(stop~ : StopReason, usage~ : Usage?)
Error(message~ : String)
} derive(Eq, ToJson, Debug)A step of the reply, as the dialect decoded it (pi's
AssistantMessageEvent).
content_index is the block's position in the assistant message being
built — the same index stream_finish will hand back as content[i] — so
a consumer can key a partial render by it without knowing which wire
format produced it. Events for different blocks are not guaranteed
contiguous: a provider may interleave a tool call's arguments with the
text after it, which is exactly why the index is on every event and not
implied by order.
Nothing here mentions a transcript, a message id or a media type. Those
belong to whoever is embedding this, and a dialect that had to know them
could not be published on its own.
StreamState
Accumulator threaded through incremental decoding. A dialect that decodes
true per-chunk SSE keeps partial-block state here; the default encoding
simply buffers raw chunks and lifts the whole reply at stream_finish.
StreamState::push
Append a chunk, returning a new state (pure — the shell owns the loop).
StreamState::text
The accumulated raw body so far.
ThinkingLevel
How hard the model should think, as a dial that means the same thing on
every provider (pi's ThinkingLevel). Ordered low to high: a dialect maps
it to whatever its own model spells, and clamps to the nearest level that
model actually has — the levels are not the same everywhere, and a
request for one a model lacks should land next to it rather than 400.
ToolDecl
Usage
UserBlock
api_error
Provider API errors are data on the message, never exceptions (pi
convention).
Two shapes, because "OpenAI-compatible" is not compatible about this one.
The wrapped {"error": {"message": ...}} is what the vendors with their
own protocol answer with (Anthropic wraps it again in {"type": "error",...}, which the same pattern matches). Cerebras answers with the error
object BARE — {"message": ..., "type": "payment_required_error","code": ...}, no envelope — and reading that as a malformed reply told
an operator their mapping was broken when their billing was.
Requiring a top-level STRING message beside an error marker is what
keeps the bare arm from shadowing a good reply: a success is keyed by
choices, content, candidates or output on the protocols here, and
none of them carries one. {"message": {...}} as an object — which some
compatible servers do return — does not match either.
reply_identity
Which model answered, and what the provider called the call.
The tail of every lift_message, and identical in all of them: all four
providers name the model in model and the call in id, at the top level
of the reply. fallback is what the dialect was configured with, for a
provider that echoes neither.
This is here and the usage block above it is not, because usage is keyed
by provider JSON key names — input_tokens against prompt_tokens — and
this is not keyed by anything. That is the line the header draws.
Install
Download zipA dialect-agnostic LLM IR, one wire mapping per provider protocol, a data-driven provider registry, and a transport a browser can implement
llmaianthropicopenaigeminimoonbit
0.3.0
Apache-2.0
Updated 9 hours ago
52 downloadsDependencies
- moonbitlang/async0.21.2