workflow

    Engine-agnostic multi-agent workflow orchestration with journaled replay/resume

    Download zip
    Version
    0.7.1
    License
    Apache-2.0
    Last updated
    1 hour ago
    Downloads
    4K

    Dependencies

    #moonbitlang/workflow

    Write a multi-agent workflow as an ordinary async MoonBit program. Share concurrency and call limits across its agents, choose how failures affect each stage, and resume completed work from a journal. The program's loops and branches determine what runs; there is no separate graph to declare.

    The core package never spawns anything. Its one seam is Runner: a single async function from AgentCall to AgentOutcome. Engines plug in from the outside — through the spawn sub-package's child-contract implementation for out-of-process engines, or any in-process function; tests plug in fakes, which is why everything below runs hermetically.

    #Setup and package map

    Add the module to a consuming project:

    moon add moonbitlang/workflow

    Import the packages you use in that project's moon.pkg. For the examples below, which also use the async runtime:

    import {
    "moonbitlang/workflow",
    "moonbitlang/async",
    }

    supported_targets = "native+wasm"

    Run with moon run --target native <package> or --target wasm. Every package in this module supports these two targets. Blocks marked mbt check in these READMEs are compiled and run by moon test; shell commands and nocheck snippets are usage instructions, not automatic agent launches.

    PackageUse it for
    workflowCalls, typed outcomes, concurrency, retries, replay, and accounting.
    workflow/spawnRunning a process that speaks the child contract.
    workflow/hostedRunning under a host that supplies child ids, launch limits, and journal paths.
    workflow/shimBuilding adapters from a foreign CLI's output to the child contract.
    workflow/shim/claudeExecutable adapter for Claude Code.
    workflow/shim/codexExecutable adapter for Codex.
    workflow/vizRendering a journal as a standalone HTML report.
    workflow/examples/scoutA small executable demonstrating process calls and journal replay.

    flowchart TD Program["Your async program"] --> Core["workflow<br/>calls and policies"] Core <--> Journal["Journal<br/>resolved outcomes"] Core --> Runner["Runner"] Runner --> Fake["In-process engine<br/>or test fake"] Runner --> Spawn["spawn<br/>child contract"] Host["Host handoff"] --> Hosted["hosted<br/>ids and sidecar"] Hosted --> Runner Spawn --> Engine["Contract-speaking<br/>engine"] Spawn --> Shim["shim/claude<br/>or shim/codex"] Shim --> CLI["Agent CLI"] Journal --> Viz["viz: HTML report"]

    The arrows show runtime handoffs, not package imports. hosted and spawn depend on the core; the core has no dependency on a particular engine.

    #Calls and limits

    Construct Workflow(runner=...) once and share it across concurrent tasks.

    Setting or methodMeaning
    max_concurrentMaximum simultaneous runner invocations; default 8, minimum 1.
    max_callsMaximum live runner invocations in this workflow; omitted means unlimited, 0 allows replay only.
    replay_scopeExtra identity string, default empty. Set it when work depends on a repository revision or configuration absent from the input.
    agent(prompt~, kind~)Sends {query, hints?}; the label defaults to a shortened prompt.
    agent_call(kind~, input~, label~)Sends the exact JSON shape a kind requires.
    agent_as[T : FromJson](...)Calls agent, then decodes the complete report as T.
    calls_made() / calls_replayed()Live runner invocations / outcomes served from prior journal entries.
    tokens_spent() / cost_usd()Usage from resolved live outcomes in this run. Historical replay usage is excluded.

    The call allowance counts entry into the runner, even if it refuses work or cannot spawn a process. Queue cancellation and replay consume no allowance. max_steps and schema are requests to the runner; the core does not enforce an engine's step limit or JSON Schema. agent_as independently decodes the returned value. There is no core token budget or wall deadline; process deadlines belong to spawn and hosted.

    #The failure model

    Everything else follows from three decisions:

    • Failure is data, in a typed channel. An agent that produced no usable report raises WorkflowError::AgentFailed with a typed AgentFailure cause; attempt folds it into a Result for fan-out sites. A failure is never a silent null: it either raises AgentFailed or lands in attempt's Result.
    • Resolved failures retain their usage. AgentOutcome carries the attempt's spend on BOTH arms — a timed-out child's tokens land in tokens_spent() just like a success's. attempt=None means no launch was ever tried. Engines that price their work report it too: cost_usd() sums their figures, and stays a floor when an engine in the mix prices nothing.
    • Cancellation propagates. Engine bugs and cancellation raise through and cancel the task group. They do not produce a journalled outcome. Usage from an interrupted call is not added to the core counters; adapters that need it during teardown can use spawn.ContractProgress.

    #A workflow, end to end

    One phase fans three verifiers out over a finding, a quorum policy requires at least 2 of the 3 CALLS to succeed (agreement between the returned verdicts is then the script's own judgment, as the filter below shows), and every launch is capped by the shared semaphore:

    ///|
    async fn verdict_runner(call : @workflow.AgentCall) -> @workflow.AgentOutcome {
    @async.pause()
    let verdict : Json = if call.input is { "query": String(q), .. } &&
    q.contains("lens=repro") {
    { "confirmed": false }
    } else {
    { "confirmed": true }
    }
    Finished(
    value=verdict,
    attempt=@workflow.AgentAttempt(
    attempt_id="sr-\{call.label}",
    steps_used=3,
    prompt_tokens=70,
    completion_tokens=30,
    ),
    )
    }

    ///|
    async test "fan out three lenses, gate on a 2-of-3 quorum" {
    let wf = @workflow.Workflow(
    runner=@workflow.Runner(verdict_runner),
    max_concurrent=4,
    max_calls=16,
    )
    wf.phase("Verify")
    let results = @workflow.fan_out(["correctness", "security", "repro"], lens => {
    @workflow.attempt(() => {
    wf.agent(
    kind="judge",
    prompt="Judge the finding through lens=\{lens}: real?",
    label="verify:\{lens}",
    )
    })
    })
    let confirmed = @workflow.quorum(results, need=2)
    .filter(v => v is { "confirmed": True, .. })
    .length()
    assert_eq(confirmed, 2)
    assert_eq(wf.calls_made(), 3)
    assert_eq(wf.tokens_spent(), 300)
    }

    fan_out gives every item its own Result slot — one lost verifier never poisons its siblings, and the tokens they spent stay spent. When later work is worthless without ALL of a stage, use parallel_all instead: the first failure cancels every sibling still in flight. The policies are one identifier each: all_ok, collect_ok(min_ok?), quorum(need~). Both fan-outs are one task group: a raise inside one cancels the rest and unwinds the whole stage, so nothing outlives it.

    PolicyWhen to use it
    parallel / fan_out with attemptKeep a Result for each input, in input order. Typed workflow failures stay in their own slots.
    parallel_allEvery branch is required; a raise cancels siblings still running.
    all_okRequire all already-collected results to succeed; raises the first error in array order.
    collect_ok(min_ok=0)Keep successes, optionally requiring a minimum count.
    quorum(need~)Require a count of successful calls; it does not compare answers for agreement.

    all_ok, collect_ok, and quorum inspect completed results; they do not cancel work early. attempt captures only WorkflowError, including AgentFailed, CallBudgetExhausted, and QuorumNotReached.

    A model that answered off-schema or ran out of steps can be worth asking again. retry wraps a raising step with max_retry extra attempts (default 1), spaced by backoff (default Immediate). Its default worth_retrying policy retries AgentFailed except Skipped; it does not retry a spent call budget or unmet quorum. A custom retriable callback can change that typed-error policy. Engine bugs and cancellation always propagate.

    Put attempt around retry, rather than passing retry a step that already returns Result. Each live retry uses a slot and call allowance, and its resolved outcome is journalled. On resume, an earlier successful report can be replayed:

    ///|
    async test "retry until the engine actually answers" {
    let flaky : Ref[Int] = { val: 0, }
    let wf = @workflow.Workflow(
    runner=@workflow.Runner(call => {
    @async.pause()
    flaky.val 1
    if flaky.val < 3 {
    DidNotFinish(failure=NoReport, attempt=None)
    } else {
    Finished(
    value={ "attempt": flaky.val },
    attempt=@workflow.AgentAttempt(
    attempt_id="sr-\{call.label}",
    steps_used=1,
    prompt_tokens=40,
    completion_tokens=10,
    ),
    )
    }
    }),
    )
    let report = @workflow.retry(
    () => wf.agent(prompt="Name the worst bug in spawn/", kind="judge"),
    max_retry=2,
    )
    assert_eq(report, { "attempt": 3 })
    assert_eq(wf.calls_made(), 3)
    }

    agent builds the explore input shape — {query, hints?} — because that is what most calls are. A kind with its own shape (the contract's review and worker) goes through agent_call, which sends the exact JSON it is given; that input IS the replay identity, so an engine encoding its own kinds gets replay for free. attempt folds the typed error channel around ANY raising step, so those calls reach a fan-out without a try_ twin per entry point:

    ///|
    async fn slice_runner(call : @workflow.AgentCall) -> @workflow.AgentOutcome {
    @async.pause()
    guard call.input is { "task": String(task), .. } else {
    return DidNotFinish(failure=Failed("not a worker input"), attempt=None)
    }
    Finished(
    value={ "status": "done", "task": task },
    attempt=@workflow.AgentAttempt(
    attempt_id="sr-\{call.label}",
    steps_used=1,
    prompt_tokens=10,
    completion_tokens=5,
    ),
    )
    }

    ///|
    async test "fan a worker kind out through its own input shape" {
    let wf = @workflow.Workflow(runner=@workflow.Runner(slice_runner))
    let slices = ["rename the seam", "widen the ceiling"]
    let done = @workflow.fan_out(slices, task => {
    @workflow.attempt(() => {
    wf.agent_call(
    kind="worker",
    input={ "task": task, "worker_root": "/w" },
    label="worker:\{task}",
    )
    })
    })
    assert_eq(@workflow.all_ok(done).length(), 2)
    }

    When the script wants a TYPE rather than JSON, decode at the boundary: agent_as runs the same call and turns a report that does not satisfy the type into a typed AgentFailed(Failed("report rejected: …")) carrying the decoder's path. A schema rides the request envelope so an engine that can constrain its model to the shape does (the Claude and Codex shims do); decoding still runs, because the engine is not trusted to validate, and the schema is part of the call's replay identity:

    ///|
    struct Confirmation {
    confirmed : Bool
    } derive(FromJson)

    ///|
    async test "decode the report at the boundary" {
    let wf = @workflow.Workflow(runner=@workflow.Runner(verdict_runner))
    let verdict : Confirmation = wf.agent_as(
    prompt="Judge the finding through lens=security: real?",
    kind="judge",
    schema={
    "type": "object",
    "properties": { "confirmed": { "type": "boolean" } },
    "required": ["confirmed"],
    },
    )
    assert_true(verdict.confirmed)
    }

    Multi-stage pipelines are just function composition inside the fan-out — stages need no barrier between them, so composing them per-item IS the pipeline:

    ///|
    async test "find then verify, with no barrier between the stages" {
    let wf = @workflow.Workflow(runner=@workflow.Runner(verdict_runner))
    let verified = @workflow.fan_out(["pkg/a", "pkg/b"], target => {
    let finding = @workflow.attempt(() => {
    wf.agent(prompt="Find the worst bug in \{target}", kind="judge")
    })
    match finding {
    // Each finding proceeds to verification the moment ITS finder
    // returns — b's finder may still be running while a verifies.
    Ok(_) =>
    @workflow.attempt(() => {
    wf.agent(
    prompt="Adversarially verify the finding in \{target}",
    kind="judge",
    )
    })
    Err(error) => Err(error)
    }
    })
    assert_eq(@workflow.all_ok(verified).length(), 2)
    }

    #Replay: crash, resume, and pay only for new work

    Every live outcome is appended to the Journal — the call's WORK identity (kind, input, max_steps, scope, schema — never the display label) plus the lossless outcome, plus attribution for readers: the phase the call was issued under and the wall-clock window it ran in, as milliseconds since the epoch. Attribution never takes part in replay matching. Re-running the same program against the same journal replays successes for free, keeps a human's Skipped refusal standing, and re-attempts other failures — getting past those is what resume is for:

    ///|
    async test "the second generation replays instead of re-paying" {
    let journal = @workflow.Journal::in_memory()
    let wf1 = @workflow.Workflow(
    runner=@workflow.Runner(verdict_runner),
    journal~,
    )
    let first = wf1.agent(
    prompt="Judge the finding through lens=security: real?",
    kind="judge",
    )
    assert_eq(wf1.tokens_spent(), 100)

    // Same program, next generation: served from the journal — no launch,
    // no slot, no fresh spend.
    let wf2 = @workflow.Workflow(
    runner=@workflow.Runner(verdict_runner),
    journal=@workflow.Journal::in_memory(prior=journal.recorded()),
    )
    assert_eq(
    wf2.agent(
    prompt="Judge the finding through lens=security: real?",
    kind="judge",
    ),
    first,
    )
    assert_eq(wf2.calls_made(), 0)
    assert_eq(wf2.calls_replayed(), 1)
    assert_eq(wf2.tokens_spent(), 0)
    }

    File-backed journals (@workflow.Journal::load(path)) are append-only JSONL, accumulated across generations. A malformed final line is dropped and the file repaired; corruption anywhere else raises JournalCorrupted. Repair preserves the healthy bytes without re-encoding them. Identical concurrent calls are treated as intentional samples, such as three identical verifiers, and consume separate prior entries.

    flowchart TD Call["Call identity: kind, input, max_steps, scope, schema"] --> Candidate{"Prior replay candidate?"} Candidate -->|Success first, then Skipped| Validate{"Runner accepts candidate?"} Validate -->|Serve| Replay["Return or raise without a launch"] Validate -->|Rerun| Candidate Candidate -->|None left| Slot["Acquire concurrency slot and check call budget"] Slot --> Live["Invoke runner"] Live -->|Resolved outcome| Record["Account usage and append journal entry"] Live -->|Cancellation or infrastructure error| Raise["Re-raise; no outcome entry"] Record --> Result["Return report or raise AgentFailed"]

    Replay is from prior entries only: two identical calls in the same live run still execute twice. Among matching prior entries, successes are considered in file order before recorded Skipped refusals. Other failures are not replay candidates. Labels, phases, and timestamps do not affect identity.

    Use one writing workflow process per journal path. Appends are serialized within a journal instance, not across processes. Journal::load is suitable for resuming a stopped writer, not watching an active one; use viz for a reader that never repairs the source file. A complete final entry missing only its newline is retained and the newline is restored. A malformed tail is removed. Each stored entry has the versioned envelope {"v":1,"e":...}. prior() and recorded() return copies of the loaded entries and this instance's new outcomes respectively.

    agent_as decodes after the raw report has been recorded. If decoding fails, the journal still holds a Finished raw report, and resume can reproduce the same decode error. Change the input, schema, or scope when the task's meaning changes, or reject an unsuitable candidate in validate_replay. The journal does not make arbitrary side effects exactly once: a process can finish work and crash before its outcome is durably recorded.

    A v1 line that carries no entry is metadata: replay skips it and repair preserves it, so a tool can annotate a journal, and a declared plan has a place to live when one arrives. Labels follow the convention stage:instance (verify:claude:3, survey:journal): readers group the instances of one stage by the prefix before the first colon.

    #Observability

    @workflow.Workflow(on_event=...) narrates the run: PhaseStarted, Log, and an AgentStarted/AgentFinished bracket that balances on EVERY path — success, typed failure, cancellation (Interrupted), and infrastructure error (Errored) — plus AgentReplayed for journal hits. Purely observational: no control flow rides on events.

    #Plugging in an engine

    A Runner wraps one async AgentCall -> AgentOutcome function. Return Finished(value~, attempt~) for a report or DidNotFinish(failure~, attempt~) for an expected unsuccessful call. Use attempt=None when there is no attempt to account for. Propagate cancellation and unexpected infrastructure errors.

    The checked examples above use in-process runners. To run a process, use spawn.contract_runner: its launch callback returns an executable and argv, and the package handles the child contract, accounting, deadlines, and teardown. A Runner::invoke call is useful for routing one runner to another; it does not itself add workflow limits, journalling, or replay.

    For reports that name mutable resources, supply validate_replay when constructing the runner. The validator receives the current call and a candidate outcome. Serve(outcome) accepts it, possibly with refreshed resource handles. Rerun consumes that candidate and tries the next match; if none is acceptable, the workflow runs live. A validation error rolls back the candidate claim and propagates. Set replay_scope to distinguish work whose meaning depends on a revision, model, or configuration outside its input; the workflow does not add those coordinates automatically.

    #Hosting and CLI adapters

    Choose hosted when a parent supplies the executable, reserved child ids, and output paths in WORKFLOW_HOST. ctx.run creates the workflow and attaches its journal and launch sidecar. The host owns reservation allocation; this handoff does not implement a sandbox. Host handoff defines the configuration protocol.

    The two executable adapters make existing agent CLIs speak the child contract:

    AdapterDefault tool policyStep definitionReport
    shim/claudeDisallows listed editing and shell tools.Distinct assistant message ids.{answer, engine, session_id?, num_turns?, cost_usd?}
    shim/codexRequests the read-only sandbox.Completed items excluding reasoning and notices.{answer, engine, thread_id?, notices?}

    Both select their write-capable policy for kind="worker" or --writable. Their max_steps enforcement is reactive: an observed step over the limit is counted before the CLI is stopped. Both accept a schema for the inner answer; agent_as decodes the complete report wrapper, so its result type must include that wrapper. The adapter READMEs describe the exact options, usage limitations, and recorded CLI versions.

    The shim library provides shared framing and process lifetime helpers for adding another CLI dialect. No framework package provisions worktrees or integrates worker edits; a controller that needs those operations must implement them.

    #Examples and development

    examples/scout is a small local executable for keyless probes and journal replay. The .mbtx scripts in examples/ are larger standalone programs:

    • simplify.mbtx runs a simplification sweep, choosing a hosted runner when configured or a standalone OpenSeek runner.
    • compose.mbtx routes calls across multiple engines.

    Those scripts use the published module without a version pin. They are not part of moon test; only moon run <script.mbtx> compiles and executes one. Running a model-backed example requires its engines and credentials.

    From the repository root, the normal validation sequence is:

    just check just test just tidy

    check type-checks native and wasm with warnings denied and checks formatting. test runs unit, process-contract, and checked README examples on both targets. tidy regenerates package interfaces and formats source; review the pkg.generated.mbti diff when changing APIs.

    The core implementation is split by responsibility: workflow.mbt handles calls and accounting, combinators.mbt supplies concurrency and failure policies, runner.mbt defines the engine boundary, journal.mbt implements persistence and replay matching, and types.mbt defines calls, attempts, and failures. The generated package interface lists all public signatures.

    JournalCorrupted

    pub suberror JournalCorrupted {
    JournalCorrupted(path~ : String, line~ : Int)
    }

    The journal file is damaged somewhere other than its final line. A torn TAIL is expected crash damage and is repaired on load; mid-file corruption means the append-only contract was broken by something else and must not be silently dropped.

    WorkflowError

    pub suberror WorkflowError {
    AgentFailed(label~ : String, failure~ : AgentFailure)
    CallBudgetExhausted(label~ : String)
    QuorumNotReached(need~ : Int, ok~ : Int, of~ : Int)
    }

    The typed error channel of the workflow layer. Every call site either propagates one of these (via raise) or catches it into attempt's Result — there is no silent null to forget.

    AgentAttempt

    pub struct AgentAttempt {
    attempt_id : String
    steps_used : Int
    prompt_tokens : Int
    completion_tokens : Int
    cost_usd : Double?
    } derive(Eq, ToJson,
    FromJson
    )

    The cost accounting of one attempt to run an agent, captured whether or not the attempt produced a report: a timed-out child spent real tokens, and losing that spend would understate every budget built on top.

    Engines BUILD one through AgentAttempt(...); everyone else READS the fields. That split is what lets a later accounting figure arrive as one more optional argument instead of breaking every engine that ever wrote the record out by hand.

    AgentAttempt::AgentAttempt

    fn AgentAttempt::AgentAttempt(attempt_id~ : String, steps_used~ : Int, prompt_tokens~ : Int, completion_tokens~ : Int, cost_usd? : Double) -> AgentAttempt

    An attempt's accounting from its parts. Everything an engine always knows is named; cost_usd is optional because an engine that prices nothing simply leaves it out — which is what None means on the field: unknown, never free.

    AgentCall

    pub struct AgentCall {
    kind : String
    input : Json
    label : String
    max_steps : Int?
    scope : String
    schema : Json?
    } derive(Eq, ToJson,
    FromJson
    )

    One request to run a sub-agent: the workflow-side view of a call before any engine specifics (binary path, model, endpoint) attach to it. kind names the child mode the engine dispatches on (explore, echo, later worker); input is the EXACT JSON the child receives as its input line — the persisted replay identity IS the child input, so an encoder change is an identity change and can never replay stale work. label is display metadata: journal replay matches on (kind, input, max_steps, schema, scope), never on the label. By convention a label reads stage:instance (verify:claude:3, survey:journal), so a reader can group the instances of one stage by the prefix before the first colon.

    The workflow BUILDS one and engines READ it; a caller seeding a journal by hand goes through AgentCall(...), so the replay identity can grow a field without invalidating every hand-written call.

    AgentCall::AgentCall

    fn AgentCall::AgentCall(kind~ : String, input~ : Json, label~ : String, max_steps? : Int, scope? : String, schema? : Json) -> AgentCall

    A call from its parts: the work (kind, input) and how a reader names it (label). The rest is optional because the common call has no ceiling, no shape, and no namespace — scope defaults to unscoped, which is what "" means on the field.

    AgentDisposition

    pub enum AgentDisposition {
    Succeeded
    FailedAgent
    Interrupted
    Errored
    } derive(Eq)

    How one agent's lifetime ended, from the event stream's point of view. Interrupted is cancellation tearing the call down; Errored is an infrastructure failure escaping the call — an engine bug in the runner, or the journal append failing after the outcome resolved — both re-raise after the event fires, so AgentStarted/AgentFinished brackets always balance.

    AgentFailure

    pub(all) enum AgentFailure {
    TimedOut
    NoReport
    MaxSteps
    ContextYield
    Skipped
    Failed(String)
    } derive(Eq, ToJson,
    FromJson
    )

    Why an agent produced no usable report — the workflow-facing vocabulary a script can meaningfully react to. Transient transport errors never appear here: retrying those is the engine's job, below this seam. What a script retries is a whole AGENT that came back with nothing usable — retry, whose default policy is written against this vocabulary. Mirrors @agent_subrun.SubrunTerminal minus Captured (that one is the success path), plus Skipped (a human declined the call — replay keeps it declined rather than overriding the decision).

    AgentOutcome

    pub(all) enum AgentOutcome {
    Finished(value~ : Json, attempt~ : AgentAttempt)
    DidNotFinish(failure~ : AgentFailure, attempt~ : AgentAttempt?)
    } derive(Eq, ToJson,
    FromJson
    )

    The lossless envelope one agent run resolves to: either a report value with its attempt accounting, or a failure that STILL carries the attempt's cost. attempt=None means no launch was ever TRIED — a Skipped call, or a refusal before spawn; a child that failed to spawn or died early is an attempt that observed zero cost, not an absent one. This is the unit the journal persists: replaying it must lose nothing the live run knew.

    Journal

    type Journal

    The replay journal: prior-run entries consumed by WORK-IDENTITY match as this run re-executes, plus an append-only record of this run's live outcomes. Replay rules:

    • a matching Finished entry replays its value — the work is not re-paid, and neither the launch allowance nor tokens_spent is charged (historical spend lives here, not in the fresh account);
    • a matching Skipped entry replays as the same refusal — a human declined this call once, and resume must not override that decision;
    • any other failure does NOT replay: getting past it is what resume is FOR, so the call runs live again (and consumes fresh launch allowance — size max_calls with re-attempts in mind).

    Identity is the call's work content — kind, input, max_steps, schema, and the workflow's replay scope — never the display label, and never the phase or time an entry records. Duplicate identical calls are intentional samples (three identical verifiers), so matching consumes entries as a multiset: each replay consumes one entry, in file order, and only LIVE outcomes are recorded — replays are never re-appended, so entry counts stay honest across any number of resume generations.

    A file-backed journal assumes ONE writing workflow process per path: appends are serialized within the process, not across processes.

    Journal::in_memory

    fn Journal::in_memory(prior? : Array[JournalEntry]) -> Journal

    A journal with no backing file: prior seeds replay (empty by default, copied — later mutation of the argument cannot skew the consumption bookkeeping), live outcomes accumulate in recorded. The test-and-embed constructor.

    Journal::load

    async fn Journal::load(path : String) -> Journal

    Open (or start) a file-backed journal: an append-only JSONL file, one entry per line, accumulated across every generation of the run. A missing file is an empty journal.

    Crash honesty: a torn TAIL — an unparsable or undecodable final line, or a final line missing its newline — is the signature of dying mid-append. The damaged suffix is dropped from replay AND repaired on disk (atomic rewrite-and-rename), so the next generation's appends land on a clean boundary instead of concatenating onto torn bytes. Damage anywhere else raises JournalCorrupted. Lines are split at the byte level before UTF-8 decoding, so a tail torn mid-codepoint cannot poison the healthy prefix.

    Journal::prior

    fn Journal::prior(self : Journal) -> Array[JournalEntry]

    The prior-run entries replay draws from, in file order (a copy).

    Journal::recorded

    fn Journal::recorded(self : Journal) -> Array[JournalEntry]

    This run's live outcomes, in completion order (a copy — the journal's own bookkeeping cannot be skewed through it).

    JournalEntry

    pub struct JournalEntry {
    call : AgentCall
    outcome : AgentOutcome
    phase : String?
    started_at : Int64?
    finished_at : Int64?
    } derive(Eq)

    One journal record: an agent call's work identity, what it resolved to, and the attribution a reader wants — the phase the call was issued under and the wall-clock window it ran in. The journal is the workflow's durability seam: everything a resumed run needs to skip re-paying for work already done rides these entries, so the entry must lose nothing the live run knew (hence AgentOutcome, the lossless envelope, and not just the value). Attribution is metadata: replay matches on the call's work identity alone, never on phase or time.

    JournalEntry::JournalEntry

    fn JournalEntry::JournalEntry(call~ : AgentCall, outcome~ : AgentOutcome, phase? : String, started_at? : Int64, finished_at? : Int64) -> JournalEntry

    An entry from its parts. Attribution is optional, so a caller seeding a journal by hand (tests, migrations) names only what it knows.

    ReplayDecision

    pub(all) enum ReplayDecision {
    Serve(AgentOutcome)
    Rerun
    }

    What the engine decided about a journal candidate at replay time: serve it (possibly REHYDRATED — the validator may substitute an outcome whose physical handles it refreshed), or veto it and run live. Raising out of the validator means validation itself failed — the claim rolls back and stays available.

    Runner

    type Runner

    The engine seam: HOW one agent call actually runs. The real runner spawns child processes; unit tests inject a fake that never leaves the process. Everything the run resolved to — success or failure, WITH its cost — rides the returned AgentOutcome; raise carries only what the workflow must not absorb: cancellation and engine bugs.

    validate_replay guards STATEFUL outcomes at resume time: when a journal hit is about to be served, the engine may inspect it against live state (does the branch this outcome names still exist?) and decide. Engines whose outcomes are pure values need no validator: replay of a plain report can never lie.

    Runner::Runner

    fn Runner::Runner(run : async (AgentCall) -> AgentOutcome, validate_replay? : async (AgentCall, AgentOutcome) -> ReplayDecision) -> Runner

    Wrap a run function (and optionally a replay validator) as a Runner.

    Runner::invoke

    async fn Runner::invoke(self : Runner, call : AgentCall) -> AgentOutcome

    Run one call through this runner. A COMPOSING runner — a dispatcher that picks an engine by kind and delegates — goes through here, so that composition never reaches for the private field.

    Workflow

    type Workflow

    One workflow run: the runner seam, the concurrency gate every agent launch passes through, the launch allowance, and the running token account. Combinators (parallel, fan_out, policies) are free functions — this context only owns what must be shared state.

    ABSTRACT, like Runner and Journal: the layout is not the contract. Every field a caller has business reading has a reader that says what the number MEANS — calls_made, calls_replayed, tokens_spent, cost_usd — and the rest are the bookkeeping those readers protect.

    Workflow::Workflow

    fn Workflow::Workflow(runner~ : Runner, max_concurrent? : Int, max_calls? : Int, journal? : Journal, replay_scope? : String, on_event? : (WorkflowEvent) -> Unit) -> Workflow raise

    A workflow over runner, running at most max_concurrent agents at once and LAUNCHING at most max_calls agents over its lifetime (no cap when omitted — the caller's loop is trusted to terminate).

    Workflow::agent

    async fn Workflow::agent(self : Workflow, prompt~ : String, kind~ : String, hints? : String, label? : String, max_steps? : Int, schema? : Json) -> Json

    Run ONE sub-agent and return its report value, from the standard scout input shape ({query, hints?} — contract v1). Raises AgentFailed when the child produced no usable report and CallBudgetExhausted when the launch allowance was spent — catch into a Result at fan-out sites (attempt folds it into one), propagate at load-bearing ones.

    Workflow::agent_as

    async fn[T :
    FromJson
    ] Workflow::agent_as(self : Workflow, prompt~ : String, kind~ : String, hints? : String, label? : String, max_steps? : Int, schema? : Json) -> T

    agent, decoded at the boundary: the report must satisfy T's FromJson, or the call fails as AgentFailed with Failed("report rejected: …") carrying the decoder's message and path — the feedback a retry wants. schema (a JSON Schema) additionally rides the envelope so an engine that can constrain the model to the shape does; decoding still runs, because the engine is not trusted to validate. The raw report is journaled as returned, so a rejection replays as the same rejection.

    Workflow::agent_call

    async fn Workflow::agent_call(self : Workflow, kind~ : String, input~ : Json, label~ : String, max_steps? : Int, schema? : Json) -> Json

    The kind-agnostic core agent wraps: run one sub-agent from its exact child input. input doubles as the journal's replay identity, so callers encoding their own kinds get replay for free.

    Workflow::calls_made

    fn Workflow::calls_made(self : Workflow) -> Int

    Agents actually launched so far (including ones still running); calls cancelled while queued and journal replays are not in this count.

    Workflow::calls_replayed

    fn Workflow::calls_replayed(self : Workflow) -> Int

    Calls served from the journal instead of launching.

    Workflow::cost_usd

    fn Workflow::cost_usd(self : Workflow) -> Double

    Money spent by THIS run, from the engines' own figures: the sum of every live attempt's cost_usd, on both outcome arms. Replayed calls cost nothing, and attempts without a figure add nothing — so a workflow that mixes a pricing engine with a non-pricing one sees a floor.

    Workflow::log

    fn Workflow::log(self : Workflow, message : String) -> Unit

    Emit one narrator line.

    Workflow::phase

    fn Workflow::phase(self : Workflow, title : String) -> Unit

    Start a new phase: agent calls ISSUED from now on carry this title in their AgentStarted events until the next phase call.

    Workflow::tokens_spent

    fn Workflow::tokens_spent(self : Workflow) -> Int

    Total tokens spent by finished attempts this run — successes AND failures that ran a child. Replayed results will not re-charge here: historical spend lives in the journal, this counts fresh execution.

    WorkflowEvent

    pub enum WorkflowEvent {
    PhaseStarted(String)
    Log(String)
    AgentStarted(label~ : String, kind~ : String, phase~ : String?)
    AgentFinished(label~ : String, disposition~ : AgentDisposition)
    AgentReplayed(label~ : String, disposition~ : AgentDisposition)
    }

    Progress the workflow narrates as it runs: phases group agents in a display, logs are one-line narrator messages, and the started/finished pair brackets each agent's lifetime — on every path, including cancellation. Purely observational: no control flow rides on these.

    all_ok

    fn[T] all_ok(results : Array[Result[T, WorkflowError]]) -> Array[T] raise WorkflowError

    Policy: every slot must have succeeded; re-raise the first failure otherwise. Post-hoc fail-fast — siblings have already run to completion; use parallel_all when failure should cancel them instead.

    attempt

    async fn[T] attempt(step : async () -> T) -> Result[T, WorkflowError]

    Fold the typed error channel into the return value: a workflow-level failure lands in Err, while cancellation and engine bugs still propagate — the distinction the whole failure model rests on.

    It folds ANY raising step: agent, agent_call carrying an engine's own input shape, an agent_as whose decode may reject, a retry around any of them, or a whole multi-stage per-item pipeline. ONE free function rather than a try_ twin per entry point — the shape a caller wants is a combinator, never another method.

    The right shape at fan-out call sites, where one lost agent must not poison its siblings.

    collect_ok

    fn[T] collect_ok(results : Array[Result[T, WorkflowError]], min_ok? : Int) -> Array[T] raise WorkflowError

    Policy: keep the successes, requiring at least min_ok of them (QuorumNotReached otherwise). min_ok=0 is pure best-effort collection; a negative min_ok is treated as 0.

    fan_out

    async fn[A, B] fan_out(items : Array[A], run : async (A) -> Result[B, WorkflowError]) -> Array[Result[B, WorkflowError]]

    Fan items out through one async stage, one Result slot per item. Multi-stage pipelines are function composition inside run — stages need no barrier between them, so composing them per-item IS the pipeline.

    parallel

    async fn[T] parallel(thunks : Array[async () -> Result[T, WorkflowError]]) -> Array[Result[T, WorkflowError]]

    Run thunks concurrently, each already carrying its failure policy in its type: a thunk resolves to Result[T, WorkflowError] (wrap a call in attempt), so one agent's failure lands in its own slot without disturbing siblings — tokens already spent on the others stay spent. Anything a thunk RAISES is deliberately not caught: engine bugs and cancellation fail the task group and cancel every peer, never demoted to a slot a policy could quietly discard. Pair with all_ok / collect_ok / quorum to choose the policy in one identifier.

    parallel_all

    async fn[T] parallel_all(thunks : Array[async () -> T]) -> Array[T]

    Fail-fast fan-out: the first thunk to RAISE (an agent call propagating its AgentFailed, an engine bug, cancellation) cancels every sibling still in flight and propagates. The right shape when later work is worthless without ALL of this stage — otherwise prefer parallel, which keeps what succeeded.

    quorum

    fn[T] quorum(results : Array[Result[T, WorkflowError]], need~ : Int) -> Array[T] raise WorkflowError

    Policy: k-of-n agreement gates (adversarial verification, vote panels): at least need slots must have succeeded.

    retry

    async fn[T] retry(attempt : async () -> T, max_retry? : Int, backoff? :
    RetryMethod
    , retriable? : (WorkflowError) -> Bool) -> T

    Re-run one agent step while it keeps failing retriably: max_retry extra attempts at most, spaced by backoff, raising the LAST attempt's error when the budget runs out. Wraps a RAISING step (agent, agent_as, or a whole per-item pipeline). Wrap it in attempt, not the other way round: a step already folded into a Result raises nothing there is left to retry on.

    What counts as retriable is worth_retrying unless retriable says otherwise; cancellation and engine bugs are never retried, they propagate on the first raise.

    backoff is the async runtime's own RetryMethod, and NAMING one (FixedDelay(500)) needs moonbitlang/async in the caller's moon.pkg — a re-export cannot lend it, since an aliased type still resolves through the package that owns it. The default needs nothing.

    Every attempt is a full LAUNCH: it queues for a slot, debits the launch allowance, and charges its tokens — so max_calls must be sized with re-attempts in mind, exactly as it must for resume. Journal replay is unaffected: a failed attempt is recorded and never replayed, so a resumed run replays the attempt that finally succeeded.

    worth_retrying

    fn worth_retrying(error : WorkflowError) -> Bool

    Is this failure worth ANOTHER launch? The default retry policy, and the vocabulary a caller NARROWS rather than rebuilds (retriable=e => worth_retrying(e) && …) — which is why it is public though retry is its only caller here:

    • an agent that produced no usable report is retriable — a model that timed out, blew its step ceiling, or answered off-schema may well answer on the next attempt;
    • Skipped is NOT: a human declined this call, and retrying overrides that decision exactly as replaying it would (the journal keeps a recorded skip standing for the same reason);
    • CallBudgetExhausted and QuorumNotReached are not: the allowance is already gone, and a quorum is the caller's verdict over a whole fan-out, not one call's misfortune.