workflow

    Engine-agnostic multi-agent workflow orchestration with journaled replay/resume

    Download zip
    Version
    0.2.0
    License
    Apache-2.0
    Last updated
    2 days ago
    Downloads
    1K

    Dependencies

    #moonbitlang/workflow

    Engine-agnostic multi-agent workflow orchestration with journaled replay/resume — a MoonBit take on the workflow-as-code model: a workflow is an ordinary async program, the "graph" unfolds as it runs, and durability comes from replaying a journal of typed outcomes rather than from a static DAG.

    The core package never spawns anything. Its one seam is Runner: a single async function from AgentCall to AgentOutcome. Engines plug in from the outside — through the spawn sub-package's child-contract implementation for out-of-process engines, or any in-process function; tests plug in fakes, which is why everything below runs hermetically.

    #The failure model

    Everything else follows from three decisions:

    • Failure is data, in a typed channel. An agent that produced no usable report raises WorkflowError::AgentFailed with a typed AgentFailure cause; try_agent folds it into a Result for fan-out sites. Forgetting to choose a failure policy is a compile error, never a silent null.
    • Cost is never lost. AgentOutcome carries the attempt's spend on BOTH arms — a timed-out child's tokens land in tokens_spent() just like a success's. attempt=None means no launch was ever tried.
    • Cancellation is not failure. Engine bugs and cancellation raise through and cancel the task group; the launch allowance only counts agents that actually launched — a call cancelled while queued costs nothing.

    #A workflow, end to end

    One phase fans three verifiers out over a finding, a quorum policy requires at least 2 of the 3 CALLS to succeed (agreement between the returned verdicts is then the script's own judgment, as the filter below shows), and every launch is capped by the shared semaphore:

    ///|
    async fn verdict_runner(call : @workflow.AgentCall) -> @workflow.AgentOutcome {
    @async.pause()
    let verdict : Json = if call.input is { "query": String(q), .. } &&
    q.contains("lens=repro") {
    { "confirmed": false }
    } else {
    { "confirmed": true }
    }
    Finished(value=verdict, attempt={
    attempt_id: "sr-\{call.label}",
    steps_used: 3,
    prompt_tokens: 70,
    completion_tokens: 30,
    })
    }

    ///|
    async test "fan out three lenses, gate on a 2-of-3 quorum" {
    let wf = @workflow.Workflow(
    runner=@workflow.Runner(verdict_runner),
    max_concurrent=4,
    max_calls=16,
    )
    wf.phase("Verify")
    let results = @workflow.fan_out(["correctness", "security", "repro"], lens => {
    wf.try_agent(
    kind="judge",
    "Judge the finding through lens=\{lens}: real?",
    label="verify:\{lens}",
    )
    })
    let confirmed = @workflow.quorum(results, need=2)
    .filter(v => v is { "confirmed": True, .. })
    .length()
    assert_eq(confirmed, 2)
    assert_eq(wf.calls_made(), 3)
    assert_eq(wf.tokens_spent(), 300)
    }

    fan_out gives every item its own Result slot — one lost verifier never poisons its siblings, and the tokens they spent stay spent. When later work is worthless without ALL of a stage, use parallel_all instead: the first failure cancels every sibling still in flight. The policies are one identifier each: all_ok, collect_ok(min_ok~), quorum(need~).

    Multi-stage pipelines are just function composition inside the fan-out — stages need no barrier between them, so composing them per-item IS the pipeline:

    ///|
    async test "find then verify, with no barrier between the stages" {
    let wf = @workflow.Workflow(runner=@workflow.Runner(verdict_runner))
    let verified = @workflow.fan_out(["pkg/a", "pkg/b"], target => {
    let finding = wf.try_agent("Find the worst bug in \{target}", kind="judge")
    match finding {
    // Each finding proceeds to verification the moment ITS finder
    // returns — b's finder may still be running while a verifies.
    Ok(_) =>
    wf.try_agent(
    "Adversarially verify the finding in \{target}",
    kind="judge",
    )
    Err(error) => Err(error)
    }
    })
    assert_eq(@workflow.all_ok(verified).length(), 2)
    }

    #Replay: crash, resume, and pay only for new work

    Every live outcome is appended to the Journal — the call's WORK identity (kind, input, max_steps — never the display label) plus the lossless outcome. Re-running the same program against the same journal replays successes for free, keeps a human's Skipped refusal standing, and re-attempts other failures — getting past those is what resume is for:

    ///|
    async test "the second generation replays instead of re-paying" {
    let journal = @workflow.Journal::in_memory()
    let wf1 = @workflow.Workflow(
    runner=@workflow.Runner(verdict_runner),
    journal~,
    )
    let first = wf1.agent(
    "Judge the finding through lens=security: real?",
    kind="judge",
    )
    assert_eq(wf1.tokens_spent(), 100)

    // Same program, next generation: served from the journal — no launch,
    // no slot, no fresh spend.
    let wf2 = @workflow.Workflow(
    runner=@workflow.Runner(verdict_runner),
    journal=@workflow.Journal::in_memory(prior=journal.recorded()),
    )
    assert_eq(
    wf2.agent("Judge the finding through lens=security: real?", kind="judge"),
    first,
    )
    assert_eq(wf2.calls_made(), 0)
    assert_eq(wf2.calls_replayed(), 1)
    assert_eq(wf2.tokens_spent(), 0)
    }

    File-backed journals (@workflow.Journal::load(path)) are append-only JSONL, accumulated across generations. A torn final line — the signature of crashing mid-append — is dropped AND repaired on disk; corruption anywhere else raises JournalCorrupted. Identical concurrent calls are intentional samples (three identical verifiers) and consume entries as a multiset.

    #Observability

    @workflow.Workflow(on_event=...) narrates the run: PhaseStarted, Log, and an AgentStarted/AgentFinished bracket that balances on EVERY path — success, typed failure, cancellation (Interrupted), and infrastructure error (Errored) — plus AgentReplayed for journal hits. Purely observational: no control flow rides on events.

    #Plugging in an engine

    Any process that speaks the CHILD CONTRACT is already an engine: one JSON line on stdin — the VERSIONED request envelope {"workflow_contract": 1, id, kind, max_steps?, input}, with the pipe held open (EOF is graceful cancel) — JSONL events on stdout (usage/agent_step are accounted exactly), and one final {"subrun_report": ...} line. The spawn sub-package is the contract's one implementation:

    ///|
    let runner = @spawn.contract_runner(launch=_ => {
    command: "my-engine",
    args: [],
    cwd: None,
    extra_env: None,
    deadline_ms: None,
    })

    For an engine whose reports are pure values, that is the whole adapter — a shim around a Rust CLI needs only to translate framing, and gets journal replay, budgets, and cancellation for free. An engine whose reports name stateful resources should also pass validate_replay.

    The wire contract in full — framing, cost accounting, terminal precedence, and what the reference engine reads from the envelope versus argv — is docs/child-contract.md.

    For in-process engines (and tests), implement one async function and wrap it:

    ///|
    let runner = @workflow.Runner(call => {
    // spawn something, await it, and account honestly:
    Finished(value=report_json, attempt={
    subrun_id,
    steps_used,
    prompt_tokens,
    completion_tokens,
    })
    })

    openseek's production adapter (its agent_workflow package) maps explore/review/echo kinds onto openseek subrun child processes and adds write-capable worker slices — confined git worktrees whose outcomes are captured from git evidence, replayed by their LOGICAL identity, and re-validated against the live registry at replay time (a stale outcome runs live instead of lying). The dependency points engine → framework: this module never learns openseek exists.

    JournalCorrupted

    pub suberror JournalCorrupted {
    JournalCorrupted(path~ : String, line~ : Int)
    }

    The journal file is damaged somewhere other than its final line. A torn TAIL is expected crash damage and is repaired on load; mid-file corruption means the append-only contract was broken by something else and must not be silently dropped.

    JournalCorrupted::output

    fn JournalCorrupted::output(self : JournalCorrupted, logger : &Logger) -> Unit

    JournalCorrupted::to_string

    fn JournalCorrupted::to_string(self : JournalCorrupted) -> String

    WorkflowError

    pub suberror WorkflowError {
    AgentFailed(label~ : String, failure~ : AgentFailure)
    CallBudgetExhausted(label~ : String)
    QuorumNotReached(need~ : Int, ok~ : Int, of~ : Int)
    }

    The typed error channel of the workflow layer. Every call site must either propagate one of these or catch it into a Result — forgetting to choose a failure policy is a compile error, not a silent null.

    WorkflowError::output

    fn WorkflowError::output(self : WorkflowError, logger : &Logger) -> Unit

    WorkflowError::to_string

    fn WorkflowError::to_string(self : WorkflowError) -> String

    AgentAttempt

    pub(all) struct AgentAttempt {
    attempt_id : String
    steps_used : Int
    prompt_tokens : Int
    completion_tokens : Int
    } derive(Eq, ToJson,
    FromJson
    )

    The cost accounting of one attempt to run an agent, captured whether or not the attempt produced a report: a timed-out child spent real tokens, and losing that spend would understate every budget built on top.

    AgentAttempt::equal

    AgentAttempt::not_equal

    fn AgentAttempt::not_equal(x : AgentAttempt, y : AgentAttempt) -> Bool

    AgentAttempt::to_json

    AgentCall

    pub(all) struct AgentCall {
    kind : String
    input : Json
    label : String
    max_steps : Int?
    scope : String
    } derive(Eq, ToJson,
    FromJson
    )

    One request to run a sub-agent: the workflow-side view of a call before any engine specifics (binary path, model, endpoint) attach to it. kind names the child mode the engine dispatches on (explore, echo, later worker); input is the EXACT JSON the child receives as its input line — the persisted replay identity IS the child input, so an encoder change is an identity change and can never replay stale work. label is display metadata: journal replay matches on (kind, input, max_steps), never on the label.

    AgentCall::equal

    fn AgentCall::equal(AgentCall, AgentCall) -> Bool

    AgentCall::not_equal

    fn AgentCall::not_equal(x : AgentCall, y : AgentCall) -> Bool

    AgentCall::to_json

    fn AgentCall::to_json(AgentCall) -> Json

    AgentDisposition

    pub(all) enum AgentDisposition {
    Succeeded
    FailedAgent
    Interrupted
    Errored
    } derive(Eq)

    How one agent's lifetime ended, from the event stream's point of view. Interrupted is cancellation tearing the call down; Errored is an infrastructure failure escaping the call — an engine bug in the runner, or the journal append failing after the outcome resolved — both re-raise after the event fires, so AgentStarted/AgentFinished brackets always balance.

    AgentDisposition::equal

    AgentDisposition::not_equal

    fn AgentDisposition::not_equal(x : AgentDisposition, y : AgentDisposition) -> Bool

    AgentFailure

    pub(all) enum AgentFailure {
    TimedOut
    NoReport
    MaxSteps
    ContextYield
    Skipped
    Failed(String)
    } derive(Eq, ToJson,
    FromJson
    )

    Why an agent produced no usable report — the workflow-facing vocabulary a script can meaningfully react to. Transient transport errors never appear here: retrying those is the engine's job, below this seam. Mirrors @agent_subrun.SubrunTerminal minus Captured (that one is the success path), plus Skipped (a human declined the call — replay keeps it declined rather than overriding the decision).

    AgentFailure::equal

    AgentFailure::not_equal

    fn AgentFailure::not_equal(x : AgentFailure, y : AgentFailure) -> Bool

    AgentFailure::output

    fn AgentFailure::output(self : AgentFailure, logger : &Logger) -> Unit

    AgentFailure::to_json

    AgentFailure::to_string

    fn AgentFailure::to_string(self : AgentFailure) -> String

    AgentOutcome

    pub(all) enum AgentOutcome {
    Finished(value~ : Json, attempt~ : AgentAttempt)
    DidNotFinish(failure~ : AgentFailure, attempt~ : AgentAttempt?)
    } derive(Eq, ToJson,
    FromJson
    )

    The lossless envelope one agent run resolves to: either a report value with its attempt accounting, or a failure that STILL carries the attempt's cost. attempt=None means no launch was ever TRIED — a Skipped call, or a refusal before spawn; a child that failed to spawn or died early is an attempt that observed zero cost, not an absent one. This is the unit the journal persists: replaying it must lose nothing the live run knew.

    AgentOutcome::equal

    AgentOutcome::not_equal

    fn AgentOutcome::not_equal(x : AgentOutcome, y : AgentOutcome) -> Bool

    AgentOutcome::to_json

    Journal

    pub struct Journal {
    // private fields
    }

    The replay journal: prior-run entries consumed by WORK-IDENTITY match as this run re-executes, plus an append-only record of this run's live outcomes. Replay rules:

    • a matching Finished entry replays its value — the work is not re-paid, and neither the launch allowance nor tokens_spent is charged (historical spend lives here, not in the fresh account);
    • a matching Skipped entry replays as the same refusal — a human declined this call once, and resume must not override that decision;
    • any other failure does NOT replay: getting past it is what resume is FOR, so the call runs live again (and consumes fresh launch allowance — size max_calls with re-attempts in mind).

    Identity is the call's work content — kind, input, max_steps, and the workflow's replay scope — never the display label. Duplicate identical calls are intentional samples (three identical verifiers), so matching consumes entries as a multiset: each replay consumes one entry, in file order, and only LIVE outcomes are recorded — replays are never re-appended, so entry counts stay honest across any number of resume generations.

    A file-backed journal assumes ONE writing workflow process per path: appends are serialized within the process, not across processes.

    Journal::in_memory

    fn Journal::in_memory(prior? : Array[JournalEntry]) -> Journal

    A journal with no backing file: prior seeds replay (empty by default, copied — later mutation of the argument cannot skew the consumption bookkeeping), live outcomes accumulate in recorded. The test-and-embed constructor.

    Journal::load

    async fn Journal::load(path : String) -> Journal

    Open (or start) a file-backed journal: an append-only JSONL file, one entry per line, accumulated across every generation of the run. A missing file is an empty journal.

    Crash honesty: a torn TAIL — an unparsable or undecodable final line, or a final line missing its newline — is the signature of dying mid-append. The damaged suffix is dropped from replay AND repaired on disk (atomic rewrite-and-rename), so the next generation's appends land on a clean boundary instead of concatenating onto torn bytes. Damage anywhere else raises JournalCorrupted. Lines are split at the byte level before UTF-8 decoding, so a tail torn mid-codepoint cannot poison the healthy prefix.

    Journal::prior

    fn Journal::prior(self : Journal) -> Array[JournalEntry]

    The prior-run entries replay draws from, in file order (a copy).

    Journal::recorded

    fn Journal::recorded(self : Journal) -> Array[JournalEntry]

    This run's live outcomes, in completion order (a copy — the journal's own bookkeeping cannot be skewed through it).

    JournalEntry

    pub(all) struct JournalEntry {
    call : AgentCall
    outcome : AgentOutcome
    } derive(Eq, ToJson,
    FromJson
    )

    One journal record: an agent call's work identity and what it resolved to. The journal is the workflow's durability seam — everything a resumed run needs to skip re-paying for work already done rides these entries, so the entry must lose nothing the live run knew (hence AgentOutcome, the lossless envelope, and not just the value).

    JournalEntry::equal

    JournalEntry::not_equal

    fn JournalEntry::not_equal(x : JournalEntry, y : JournalEntry) -> Bool

    JournalEntry::to_json

    ReplayDecision

    pub(all) enum ReplayDecision {
    Serve(AgentOutcome)
    Rerun
    }

    What the engine decided about a journal candidate at replay time: serve it (possibly REHYDRATED — the validator may substitute an outcome whose physical handles it refreshed), or veto it and run live. Raising out of the validator means validation itself failed — the claim rolls back and stays available.

    Runner

    pub struct Runner {
    // private fields
    }

    The engine seam: HOW one agent call actually runs. The real runner spawns child processes; unit tests inject a fake that never leaves the process. Everything the run resolved to — success or failure, WITH its cost — rides the returned AgentOutcome; raise carries only what the workflow must not absorb: cancellation and engine bugs.

    validate_replay guards STATEFUL outcomes at resume time: when a journal hit is about to be served, the engine may inspect it against live state (does the branch this outcome names still exist?) and decide. Engines whose outcomes are pure values need no validator: replay of a plain report can never lie.

    Runner::Runner

    fn Runner::Runner(run : async (AgentCall) -> AgentOutcome, validate_replay? : async (AgentCall, AgentOutcome) -> ReplayDecision) -> Runner

    Wrap a run function (and optionally a replay validator) as a Runner.

    Runner::invoke

    async fn Runner::invoke(self : Runner, call : AgentCall) -> AgentOutcome

    Run one call through this runner. Composing runners (a dispatcher delegating by kind) goes through here, never through the fields.

    Workflow

    pub struct Workflow {
    runner : Runner
    slots :
    Semaphore

    on_event : (WorkflowEvent) -> Unit
    journal : Journal?
    replay_scope : String
    max_calls : Int?
    calls_made : Int
    replays : Int
    prompt_tokens : Int
    completion_tokens : Int
    current_phase : String?
    }

    One workflow run: the runner seam, the concurrency gate every agent launch passes through, the launch allowance, and the running token account. Combinators (parallel, fan_out, policies) are free functions — this context only owns what must be shared state.

    Workflow::Workflow

    fn Workflow::Workflow(runner~ : Runner, max_concurrent? : Int, max_calls? : Int, journal? : Journal, replay_scope? : String, on_event? : (WorkflowEvent) -> Unit) -> Workflow raise

    A workflow over runner, running at most max_concurrent agents at once and LAUNCHING at most max_calls agents over its lifetime (no cap when omitted — the caller's loop is trusted to terminate).

    Workflow::agent

    async fn Workflow::agent(self : Workflow, prompt : String, kind~ : String, hints? : String, label? : String, max_steps? : Int) -> Json

    Run ONE sub-agent and return its report value, from the standard scout input shape ({query, hints?} — contract v1). Raises AgentFailed when the child produced no usable report and CallBudgetExhausted when the launch allowance was spent — catch into a Result at fan-out sites (try_agent is the shorthand), propagate at load-bearing ones.

    Workflow::agent_call

    async fn Workflow::agent_call(self : Workflow, kind~ : String, input~ : Json, label~ : String, max_steps? : Int) -> Json

    The kind-agnostic core agent wraps: run one sub-agent from its exact child input. input doubles as the journal's replay identity, so callers encoding their own kinds get replay for free.

    Workflow::calls_made

    fn Workflow::calls_made(self : Workflow) -> Int

    Agents actually launched so far (including ones still running); calls cancelled while queued and journal replays are not in this count.

    Workflow::calls_replayed

    fn Workflow::calls_replayed(self : Workflow) -> Int

    Calls served from the journal instead of launching.

    Workflow::log

    fn Workflow::log(self : Workflow, message : String) -> Unit

    Emit one narrator line.

    Workflow::phase

    fn Workflow::phase(self : Workflow, title : String) -> Unit

    Start a new phase: agent calls ISSUED from now on carry this title in their AgentStarted events until the next phase call.

    Workflow::tokens_spent

    fn Workflow::tokens_spent(self : Workflow) -> Int

    Total tokens spent by finished attempts this run — successes AND failures that ran a child. Replayed results will not re-charge here: historical spend lives in the journal, this counts fresh execution.

    Workflow::try_agent

    async fn Workflow::try_agent(self : Workflow, prompt : String, kind~ : String, hints? : String, label? : String, max_steps? : Int) -> Result[Json, WorkflowError]

    agent with the typed error channel folded into the return value: workflow-level failures land in Err, cancellation and engine errors still propagate. The right shape at fan-out call sites, where one lost agent must not poison its siblings.

    WorkflowEvent

    pub(all) enum WorkflowEvent {
    PhaseStarted(String)
    Log(String)
    AgentStarted(label~ : String, kind~ : String, phase~ : String?)
    AgentFinished(label~ : String, disposition~ : AgentDisposition)
    AgentReplayed(label~ : String, disposition~ : AgentDisposition)
    }

    Progress the workflow narrates as it runs: phases group agents in a display, logs are one-line narrator messages, and the started/finished pair brackets each agent's lifetime — on every path, including cancellation. Purely observational: no control flow rides on these.

    all_ok

    fn[T] all_ok(results : Array[Result[T, WorkflowError]]) -> Array[T] raise WorkflowError

    Policy: every slot must have succeeded; re-raise the first failure otherwise. Post-hoc fail-fast — siblings have already run to completion; use parallel_all when failure should cancel them instead.

    collect_ok

    fn[T] collect_ok(results : Array[Result[T, WorkflowError]], min_ok? : Int) -> Array[T] raise WorkflowError

    Policy: keep the successes, requiring at least min_ok of them (QuorumNotReached otherwise). min_ok=0 is pure best-effort collection; a negative min_ok is treated as 0.

    fan_out

    async fn[A, B] fan_out(items : Array[A], run : async (A) -> Result[B, WorkflowError]) -> Array[Result[B, WorkflowError]]

    Fan items out through one async stage, one Result slot per item. Multi-stage pipelines are function composition inside run — stages need no barrier between them, so composing them per-item IS the pipeline.

    parallel

    async fn[T] parallel(thunks : Array[async () -> Result[T, WorkflowError]]) -> Array[Result[T, WorkflowError]]

    Run thunks concurrently, each already carrying its failure policy in its type: a thunk resolves to Result[T, WorkflowError] (build them from try_agent), so one agent's failure lands in its own slot without disturbing siblings — tokens already spent on the others stay spent. Anything a thunk RAISES is deliberately not caught: engine bugs and cancellation fail the task group and cancel every peer, never demoted to a slot a policy could quietly discard. Pair with all_ok / collect_ok / quorum to choose the policy in one identifier.

    parallel_all

    async fn[T] parallel_all(thunks : Array[async () -> T]) -> Array[T]

    Fail-fast fan-out: the first thunk to RAISE (an agent call propagating its AgentFailed, an engine bug, cancellation) cancels every sibling still in flight and propagates. The right shape when later work is worthless without ALL of this stage — otherwise prefer parallel, which keeps what succeeded.

    quorum

    fn[T] quorum(results : Array[Result[T, WorkflowError]], need~ : Int) -> Array[T] raise WorkflowError

    Policy: k-of-n agreement gates (adversarial verification, vote panels): at least need slots must have succeeded.