llm
A dialect-agnostic LLM IR, one wire mapping per provider protocol, a data-driven provider registry, and a transport a browser can implement
llm
ai
anthropic
openai
gemini
moonbit
moon add marianoguerra/llm@0.1.1
Download zipAuthor
Version
0.1.1
License
Apache-2.0
Repository
Last updated
4 days ago
Downloads
18
Dependencies
- moonbitlang/async0.20.3
- moonbitlang/x0.4.46
README#marianoguerra/llm
A dialect-agnostic LLM IR, one wire mapping per provider protocol, a registry
that makes a fifth provider a struct literal, and a transport a browser can
implement.
Modelled on pi.dev's packages/ai, which draws the same
three lines: an api is a wire protocol, a provider is a vendor with
credentials and a model catalog, and a collection routes between them.
moon add marianoguerra/llmlet registry = @catalog.default_registry()
guard registry.dialect_of("anthropic:claude-sonnet-5") is Some(dialect)
let body = dialect.lower_context(context) // -> provider request JSON
let reply = dialect.lift_message(json) // -> ContextMessageSwitching model mid-session is passing the same LlmContext to a different
Dialect. No session state migrates, because none exists.
#Packages
| Package | What it is |
|---|---|
| llm | LlmContext and the messages, blocks, usage and per-turn params it is made of; the Dialect trait; StreamEvent; ModelInfo; the Registry; LlmTransport and RelayTransport |
| llm/anthropic | anthropic-messages |
| llm/openai | openai-responses and openai-completions, plus a probed capability table |
| llm/gemini | google-generative-ai — the one whose endpoint carries the model |
| llm/openrouter | openrouter, which borrows the completions mapping |
| llm/catalog | The four providers as data. Credential-free |
| llm/wire | Reading a key out of an environment, resolving an endpoint with it, and an HTTP transport. The only native package here |
Everything except llm/wire is target-neutral, so a browser page can lower a
context, choose a dialect and lift a reply with no server involved — the page
does both stages and a relay only forwards bytes, never learning which
provider is on the other end. preferred_target is wasm-gc so that stays
true by default; just check covers native too.
#What is NOT here
No agent loop, no tool executor, no session store, no prompt templating. This
is the layer under those: what a turn's input is, how it becomes one
provider's JSON and back, and where the bytes go. It grew out of a hypermedia
agent that needed exactly this much and nothing above it.
#Adding a provider
If it speaks a protocol that already has an ApiSpec, it is a value:
let registry = @catalog.default_registry().with_provider({
id: "lmstudio",
name: "LM Studio",
api: "openai-completions",
endpoint: "http://localhost:1234/v1/chat/completions",
auth: NoAuth,
key_env: "",
model_env: "",
default_model: "qwen3-coder",
models: [@llm.ModelInfo::permissive("qwen3-coder")],
headers: { "content-type": "application/json" },
telemetry_name: "lmstudio",
server_address: "localhost",
}){model} in endpoint is substituted where a provider puts the model in the
URL instead of the body. auth is BearerHeader | KeyHeader(name) |KeyQuery(name) | NoAuth — the credential itself is never in a ProviderSpec,
only the name of the variable holding it, which is what lets one compile into a
page.
#Adding a wire protocol
A new Dialect impl plus an ApiSpec naming it, handed to
Registry::with_api. Nothing in this package has a match over provider names
to edit.
pub fn api_spec() -> @llm.ApiSpec {
{ id: "my-api", make: info => MyDialect::make(model=info.id), caps: id => @llm.ModelInfo::permissive(id) }
}The trait is six methods, three of which have defaults that buffer the body and
lift it whole — so a dialect is id, lower_context and lift_message until
it wants real streaming.
#Two things that look odd and are not
stream_feed returns events, not a message. Providers differ in how a
response is framed — event names, delta shapes, where usage and the stop
reason arrive — and decoding that anywhere but inside the dialect would mean
adding a provider in two places. lift_message is the non-streaming special
case of stream_init ▸ stream_feed ▸ stream_finish.
LlmTransport is in continuations, not async. A browser cannot have
async: moonbitlang/async's event loop is unimplemented for wasm-gc, and an
async function cannot be called from the plain exported functions a page starts
from. A trait only a server could implement would not be a seam, so the native
side parks on a semaphore instead.
#Dialect
pub(open) trait Dialect {
fn id(Self) -> String
fn model(Self) -> String = _
fn lower_context(Self, LlmContext) -> Json
fn lift_message(Self, Json) -> ContextMessage raise DialectError
fn stream_init(Self) -> StreamState = _
fn stream_feed(Self, StreamState, chunk~ : String) -> (StreamState, Array[StreamEvent]) raise DialectError = _
fn stream_finish(Self, StreamState) -> ContextMessage raise DialectError = _
}One provider protocol, both directions.
Both are pure: switching LLMs mid-session is passing the same LlmContext
to a different Dialect. Streaming decode (stream_*) keeps SSE framing
knowledge inside the dialect, so a deployment can put the dialect on
either side of a network boundary — the transport stays a byte pump that
never learns which provider is on the other end.
#LlmTransport
Sends an already-lowered provider body to a NAMED target and pumps raw
chunks back. It never inspects or rewrites the body and never learns the
dialect.
target is a name, not an endpoint — whoever holds the credential
resolves it in their own table. That is what lets the same trait be an
HTTP client with a key in it and a page posting to a relay that has one.
CONTINUATIONS, not async, and that is not a style choice: a browser
cannot have async — moonbitlang/async's event loop is unimplemented for
wasm-gc, and an async function cannot be called from the plain exported
functions a page starts from. A trait only one of the two sides could
implement would not be a seam. The native side pays for this by parking
on a semaphore, which is the cheaper half of the trade.
on_error is for a call that never happened. A provider answering 429 or
400 answers with a BODY, and that body is a chunk: errors are data, and
the dialect is what reads them.
#DialectError
impl Show for DialectError
#ApiSpec
One wire protocol, and how to build a dialect that speaks it.
make is what replaces a match provider somewhere: a new API is a value
handed to a registry, not an arm somebody has to find. caps is what this
API assumes about a model nobody wrote a row for — it lives with the
dialect that reads it, because that is the code a wrong assumption breaks.
Closures, so this is the one type here that derives nothing. Everything a
proxy might want to hand a page — ProviderSpec, ModelInfo — is data
and does.
#AssistantBlock
signature on all three is the provider's opaque proof that this block is
the model's own output, replayed verbatim on the next turn or dropped.
Anthropic signs thinking; Gemini 3 signs whatever part carried the
reasoning — including tool calls, where it is mandatory: a functionCall
replayed without its thoughtSignature is a 400. It survives whatever
round-trip the embedding application puts the message through, and a
dialect only replays a signature its own model produced.
#AuthStyle
How a provider is authenticated.
The credential is never here. A registry is DATA — it compiles into a page
— and data that names an environment variable is safe to ship where its
value is not.
#CacheRetention
How long a provider may keep the cached prompt prefix: not at all, its
own default (minutes, in memory), or the extended tier.
#ContextMessage
pub(all) enum ContextMessage {
UserMsg(content~ : Array[UserBlock])
AssistantMsg(content~ : Array[AssistantBlock], stop~ : StopReason, model~ : String, response_id~ : String?, usage~ : Usage?, error~ : String?)
ToolResultMsg(tool_call_id~ : String, tool_name~ : String, content~ : Array[UserBlock], is_error~ : Bool)
} derive(ToJson, Debug)#Cost
USD per million tokens. Separate from Usage on purpose: usage is what a
call actually spent, this is what the provider charges, and the two are
updated by completely different people.
#LlmContext
One turn's whole input: what the model is told, what it has said, what it
may call, and how hard it should think.
#LlmParams
pub(all) struct LlmParams {
effort : ThinkingLevel?
pro_mode : Bool
reasoning_context : ReasoningContext?
reasoning_summary : Bool
max_output_tokens : Int?
cache_key : String?
cache_retention : CacheRetention
} derive(ToJson, Debug)The knobs a turn is asked for, dialect-agnostic on purpose: they travel
with the LlmContext so that switching model mid-session keeps meaning
"pass the same context to a different Dialect". A dialect maps what its
API can express and silently drops the rest — the same rule as a
signature it cannot replay.
#Modality
What a turn may carry INTO a model. Nothing reads this yet; it is here
because a catalog that cannot say "this one has no vision" makes the
caller find out from a 400.
#ModelInfo
pub(all) struct ModelInfo {
id : String
api : String?
reasoning : Bool
thinking_levels : Array[(ThinkingLevel, String)]
supports_pro_mode : Bool
reasoning_contexts : Array[ReasoningContext]
input : Array[Modality]
context_window : Int
max_output_tokens : Int?
cost : Cost?
} derive(ToJson, Debug)One model's request-side capabilities (pi's Model).
thinking_levels maps the portable dial to this model's own spelling
(pi's thinkingLevelMap); a level absent from the list does not exist
here and clamps to the nearest one that does — asking for one a model
lacks is a 400, not a downgrade. reasoning_contexts and
supports_pro_mode are pass/drop rather than clamp: there is no nearly-pro.
Data, not code, so a new model is a row. That is how pi.dev does it and
it is why Registry::with_provider can add a vendor without a code path.
#ModelInfo::allows_context
#ModelInfo::clamp_effort
#ProviderSpec
One provider: an endpoint, a credential shape, and the models it offers.
All data. That is the point — pi's registerProvider({baseUrl, apiKey:"$MY_KEY", api: "openai-completions", models: [...]}) adds an
OpenAI-compatible vendor with no code at all, and this is the same bet:
a new provider is a row.
models is both the offer and the capability table. Two arrays that had
to agree is what a guard test used to be for.
#ReasoningContext
How much of the earlier turns' reasoning the provider may reuse
(OpenAI's reasoning.context, new with the gpt-5.6 family). Distinct
from replaying reasoning items ourselves, which every turn already does:
this asks the provider to render its own retained chain into the context.
#Registry
Every API and provider a build knows.
A VALUE threaded by whoever holds it, not a mutable global: two tests can
hold different registries, a caller can add a provider without touching
the package that ships the defaults, and nothing is initialized behind
anybody's back.
#Registry::endpoint
#Registry::model_info
#Registry::of
#Registry::provider_for_api
#Registry::with_provider
#RelayTransport
pub(all) struct RelayTransport {
post : (String, String, String, (String) -> Unit, (String) -> Unit) -> Unit
stop : (String) -> Unit
}A transport that hands the call to something else holding the credential
— a page posting to its own server, a worker posting to a gateway.
post is (handle, target, body as text, ok, err): the relay is told
which target to resolve and given bytes to forward, and answers with the
whole reply. That it takes TEXT rather than Json is the point — a relay
is not supposed to parse what it is relaying.
ok delivering the whole body is what a buffering relay does. When one
streams, this is the line that changes and nothing above it does: the
dialect already decodes incrementally.
impl LlmTransport for RelayTransportfn send(self : RelayTransport, handle~ : String, target~ : String, body~ : Json, headers~ : Map[String, String], on_chunk~ : (String) -> Unit, on_done~ : () -> Unit, on_error~ : (String) -> Unit) -> Unit#RelayTransport::make
fn RelayTransport::make(post~ : (String, String, String, (String) -> Unit, (String) -> Unit) -> Unit, stop? : (String) -> Unit) -> RelayTransport#StopReason
pi: stop | length | toolUse | error | aborted — errors are data, never
exceptions.
#StreamEvent
pub(all) enum StreamEvent {
Start(model~ : String)
TextStart(content_index~ : Int)
TextDelta(content_index~ : Int, delta~ : String)
TextEnd(content_index~ : Int)
ThinkingStart(content_index~ : Int)
ThinkingDelta(content_index~ : Int, delta~ : String)
ThinkingEnd(content_index~ : Int)
ToolCallStart(content_index~ : Int, id~ : String, name~ : String)
ToolCallDelta(content_index~ : Int, delta~ : String)
ToolCallEnd(content_index~ : Int)
Done(stop~ : StopReason, usage~ : Usage?)
Error(message~ : String)
} derive(Eq, ToJson, Debug)A step of the reply, as the dialect decoded it (pi's
AssistantMessageEvent).
content_index is the block's position in the assistant message being
built — the same index stream_finish will hand back as content[i] — so
a consumer can key a partial render by it without knowing which wire
format produced it. Events for different blocks are not guaranteed
contiguous: a provider may interleave a tool call's arguments with the
text after it, which is exactly why the index is on every event and not
implied by order.
Nothing here mentions a transcript, a message id or a media type. Those
belong to whoever is embedding this, and a dialect that had to know them
could not be published on its own.
#StreamState
Accumulator threaded through incremental decoding. A dialect that decodes
true per-chunk SSE keeps partial-block state here; the default encoding
simply buffers raw chunks and lifts the whole reply at stream_finish.
#StreamState::push
Append a chunk, returning a new state (pure — the shell owns the loop).
#StreamState::text
The accumulated raw body so far.
#ThinkingLevel
How hard the model should think, as a dial that means the same thing on
every provider (pi's ThinkingLevel). Ordered low to high: a dialect maps
it to whatever its own model spells, and clamps to the nearest level that
model actually has — the levels are not the same everywhere, and a
request for one a model lacks should land next to it rather than 400.
#ToolDecl
#Usage
#UserBlock
#api_error
Provider API errors are data on the message, never exceptions (pi
convention). Matches the {"error": {"message": ...}} shape all four
providers answer with (Anthropic wraps it in {"type": "error", ...},
which this pattern also matches).
#reply_identity
Which model answered, and what the provider called the call.
The tail of every lift_message, and identical in all of them: all four
providers name the model in model and the call in id, at the top level
of the reply. fallback is what the dialect was configured with, for a
provider that echoes neither.
This is here and the usage block above it is not, because usage is keyed
by provider JSON key names — input_tokens against prompt_tokens — and
this is not keyed by anything. That is the line the header draws.
A dialect-agnostic LLM IR, one wire mapping per provider protocol, a data-driven provider registry, and a transport a browser can implement
llmaianthropicopenaigeminimoonbit
0.1.1
Apache-2.0
Updated 4 days ago
18 downloadsDependencies
- moonbitlang/async0.20.3
- moonbitlang/x0.4.46