workflow

Engine-agnostic multi-agent workflow orchestration with journaled replay/resume

Download zip
Author
Version
0.1.1
License
Apache-2.0
Last updated
2 hours ago
Downloads
6

Dependencies

#bobzhang/workflow

Engine-agnostic multi-agent workflow orchestration with journaled replay/resume — a MoonBit take on the workflow-as-code model: a workflow is an ordinary async program, the "graph" unfolds as it runs, and durability comes from replaying a journal of typed outcomes rather than from a static DAG.

The core package never spawns anything. Its one seam is Runner: a single async function from AgentCall to AgentOutcome. Engines plug in from the outside — through the spawn sub-package's child-contract implementation for out-of-process engines, or any in-process function; tests plug in fakes, which is why everything below runs hermetically.

#The failure model

Everything else follows from three decisions:

  • Failure is data, in a typed channel. An agent that produced no usable report raises WorkflowError::AgentFailed with a typed AgentFailure cause; try_agent folds it into a Result for fan-out sites. Forgetting to choose a failure policy is a compile error, never a silent null.
  • Cost is never lost. AgentOutcome carries the attempt's spend on BOTH arms — a timed-out child's tokens land in tokens_spent() just like a success's. attempt=None means no launch was ever tried.
  • Cancellation is not failure. Engine bugs and cancellation raise through and cancel the task group; the launch allowance only counts agents that actually launched — a call cancelled while queued costs nothing.

#A workflow, end to end

One phase fans three verifiers out over a finding, a quorum policy requires at least 2 of the 3 CALLS to succeed (agreement between the returned verdicts is then the script's own judgment, as the filter below shows), and every launch is capped by the shared semaphore:

///|
async fn verdict_runner(call : @workflow.AgentCall) -> @workflow.AgentOutcome {
@async.pause()
let verdict : Json = if call.input is { "query": String(q), .. } &&
q.contains("lens=repro") {
{ "confirmed": false }
} else {
{ "confirmed": true }
}
Finished(value=verdict, attempt={
attempt_id: "sr-\{call.label}",
steps_used: 3,
prompt_tokens: 70,
completion_tokens: 30,
})
}

///|
async test "fan out three lenses, gate on a 2-of-3 quorum" {
let wf = @workflow.Workflow(
runner=@workflow.Runner(verdict_runner),
max_concurrent=4,
max_calls=16,
)
wf.phase("Verify")
let results = @workflow.fan_out(["correctness", "security", "repro"], lens => {
wf.try_agent(
kind="judge",
"Judge the finding through lens=\{lens}: real?",
label="verify:\{lens}",
)
})
let confirmed = @workflow.quorum(results, need=2)
.filter(v => v is { "confirmed": True, .. })
.length()
assert_eq(confirmed, 2)
assert_eq(wf.calls_made(), 3)
assert_eq(wf.tokens_spent(), 300)
}

fan_out gives every item its own Result slot — one lost verifier never poisons its siblings, and the tokens they spent stay spent. When later work is worthless without ALL of a stage, use parallel_all instead: the first failure cancels every sibling still in flight. The policies are one identifier each: all_ok, collect_ok(min_ok~), quorum(need~).

Multi-stage pipelines are just function composition inside the fan-out — stages need no barrier between them, so composing them per-item IS the pipeline:

///|
async test "find then verify, with no barrier between the stages" {
let wf = @workflow.Workflow(runner=@workflow.Runner(verdict_runner))
let verified = @workflow.fan_out(["pkg/a", "pkg/b"], target => {
let finding = wf.try_agent("Find the worst bug in \{target}", kind="judge")
match finding {
// Each finding proceeds to verification the moment ITS finder
// returns — b's finder may still be running while a verifies.
Ok(_) =>
wf.try_agent(
"Adversarially verify the finding in \{target}",
kind="judge",
)
Err(error) => Err(error)
}
})
assert_eq(@workflow.all_ok(verified).length(), 2)
}

#Replay: crash, resume, and pay only for new work

Every live outcome is appended to the Journal — the call's WORK identity (kind, input, max_steps — never the display label) plus the lossless outcome. Re-running the same program against the same journal replays successes for free, keeps a human's Skipped refusal standing, and re-attempts other failures — getting past those is what resume is for:

///|
async test "the second generation replays instead of re-paying" {
let journal = @workflow.Journal::in_memory()
let wf1 = @workflow.Workflow(
runner=@workflow.Runner(verdict_runner),
journal~,
)
let first = wf1.agent(
"Judge the finding through lens=security: real?",
kind="judge",
)
assert_eq(wf1.tokens_spent(), 100)

// Same program, next generation: served from the journal — no launch,
// no slot, no fresh spend.
let wf2 = @workflow.Workflow(
runner=@workflow.Runner(verdict_runner),
journal=@workflow.Journal::in_memory(prior=journal.recorded()),
)
assert_eq(
wf2.agent("Judge the finding through lens=security: real?", kind="judge"),
first,
)
assert_eq(wf2.calls_made(), 0)
assert_eq(wf2.calls_replayed(), 1)
assert_eq(wf2.tokens_spent(), 0)
}

File-backed journals (@workflow.Journal::load(path)) are append-only JSONL, accumulated across generations. A torn final line — the signature of crashing mid-append — is dropped AND repaired on disk; corruption anywhere else raises JournalCorrupted. Identical concurrent calls are intentional samples (three identical verifiers) and consume entries as a multiset.

#Observability

@workflow.Workflow(on_event=...) narrates the run: PhaseStarted, Log, and an AgentStarted/AgentFinished bracket that balances on EVERY path — success, typed failure, cancellation (Interrupted), and infrastructure error (Errored) — plus AgentReplayed for journal hits. Purely observational: no control flow rides on events.

#Plugging in an engine

Any process that speaks the CHILD CONTRACT is already an engine: one JSON line on stdin — the VERSIONED request envelope {"workflow_contract": 1, id, kind, max_steps?, input}, with the pipe held open (EOF is graceful cancel) — JSONL events on stdout (usage/agent_step are accounted exactly), and one final {"subrun_report": ...} line. The spawn sub-package is the contract's one implementation:

///|
let runner = @spawn.contract_runner(launch=_ => {
command: "my-engine",
args: [],
cwd: None,
extra_env: None,
deadline_ms: None,
})

For an engine whose reports are pure values, that is the whole adapter — a shim around a Rust CLI needs only to translate framing, and gets journal replay, budgets, and cancellation for free. An engine whose reports name stateful resources should also pass validate_replay.

For in-process engines (and tests), implement one async function and wrap it:

///|
let runner = @workflow.Runner(call => {
// spawn something, await it, and account honestly:
Finished(value=report_json, attempt={
subrun_id,
steps_used,
prompt_tokens,
completion_tokens,
})
})

openseek's production adapter (agent_workflow in the openseek module) maps explore/review/echo kinds onto openseek subrun child processes and adds write-capable worker slices — confined git worktrees whose outcomes are captured from git evidence, replayed by their LOGICAL identity, and re-validated against the live registry at replay time (a stale outcome runs live instead of lying). The dependency points engine → framework: this module never learns openseek exists.

JournalCorrupted

pub suberror JournalCorrupted {
JournalCorrupted(path~ : String, line~ : Int)
}

The journal file is damaged somewhere other than its final line. A torn TAIL is expected crash damage and is repaired on load; mid-file corruption means the append-only contract was broken by something else and must not be silently dropped.

JournalCorrupted::output

fn JournalCorrupted::output(self : JournalCorrupted, logger : &Logger) -> Unit

JournalCorrupted::to_string

fn JournalCorrupted::to_string(self : JournalCorrupted) -> String

WorkflowError

pub suberror WorkflowError {
AgentFailed(label~ : String, failure~ : AgentFailure)
CallBudgetExhausted(label~ : String)
QuorumNotReached(need~ : Int, ok~ : Int, of~ : Int)
}

The typed error channel of the workflow layer. Every call site must either propagate one of these or catch it into a Result — forgetting to choose a failure policy is a compile error, not a silent null.

WorkflowError::output

fn WorkflowError::output(self : WorkflowError, logger : &Logger) -> Unit

WorkflowError::to_string

fn WorkflowError::to_string(self : WorkflowError) -> String

AgentAttempt

pub(all) struct AgentAttempt {
attempt_id : String
steps_used : Int
prompt_tokens : Int
completion_tokens : Int
} derive(Eq, ToJson,
FromJson
)

The cost accounting of one attempt to run an agent, captured whether or not the attempt produced a report: a timed-out child spent real tokens, and losing that spend would understate every budget built on top.

AgentAttempt::equal

AgentAttempt::not_equal

fn AgentAttempt::not_equal(x : AgentAttempt, y : AgentAttempt) -> Bool

AgentAttempt::to_json

AgentCall

pub(all) struct AgentCall {
kind : String
input : Json
label : String
max_steps : Int?
scope : String
} derive(Eq, ToJson,
FromJson
)

One request to run a sub-agent: the workflow-side view of a call before any engine specifics (binary path, model, endpoint) attach to it. kind names the child mode the engine dispatches on (explore, echo, later worker); input is the EXACT JSON the child receives as its input line — the persisted replay identity IS the child input, so an encoder change is an identity change and can never replay stale work. label is display metadata: journal replay matches on (kind, input, max_steps), never on the label.

AgentCall::equal

fn AgentCall::equal(AgentCall, AgentCall) -> Bool

AgentCall::not_equal

fn AgentCall::not_equal(x : AgentCall, y : AgentCall) -> Bool

AgentCall::to_json

fn AgentCall::to_json(AgentCall) -> Json

AgentDisposition

pub(all) enum AgentDisposition {
Succeeded
FailedAgent
Interrupted
Errored
} derive(Eq)

How one agent's lifetime ended, from the event stream's point of view. Interrupted is cancellation tearing the call down; Errored is an infrastructure failure escaping the call — an engine bug in the runner, or the journal append failing after the outcome resolved — both re-raise after the event fires, so AgentStarted/AgentFinished brackets always balance.

AgentDisposition::equal

AgentDisposition::not_equal

fn AgentDisposition::not_equal(x : AgentDisposition, y : AgentDisposition) -> Bool

AgentFailure

pub(all) enum AgentFailure {
TimedOut
NoReport
MaxSteps
ContextYield
Skipped
Failed(String)
} derive(Eq, ToJson,
FromJson
)

Why an agent produced no usable report — the workflow-facing vocabulary a script can meaningfully react to. Transient transport errors never appear here: retrying those is the engine's job, below this seam. Mirrors @agent_subrun.SubrunTerminal minus Captured (that one is the success path), plus Skipped (a human declined the call — replay keeps it declined rather than overriding the decision).

AgentFailure::equal

AgentFailure::not_equal

fn AgentFailure::not_equal(x : AgentFailure, y : AgentFailure) -> Bool

AgentFailure::output

fn AgentFailure::output(self : AgentFailure, logger : &Logger) -> Unit

AgentFailure::to_json

AgentFailure::to_string

fn AgentFailure::to_string(self : AgentFailure) -> String

AgentOutcome

pub(all) enum AgentOutcome {
Finished(value~ : Json, attempt~ : AgentAttempt)
DidNotFinish(failure~ : AgentFailure, attempt~ : AgentAttempt?)
} derive(Eq, ToJson,
FromJson
)

The lossless envelope one agent run resolves to: either a report value with its attempt accounting, or a failure that STILL carries the attempt's cost. attempt=None means no launch was ever TRIED — a Skipped call, or a refusal before spawn; a child that failed to spawn or died early is an attempt that observed zero cost, not an absent one. This is the unit the journal persists: replaying it must lose nothing the live run knew.

AgentOutcome::equal

AgentOutcome::not_equal

fn AgentOutcome::not_equal(x : AgentOutcome, y : AgentOutcome) -> Bool

AgentOutcome::to_json

Journal

pub struct Journal {
// private fields
}

The replay journal: prior-run entries consumed by WORK-IDENTITY match as this run re-executes, plus an append-only record of this run's live outcomes. Replay rules:

  • a matching Finished entry replays its value — the work is not re-paid, and neither the launch allowance nor tokens_spent is charged (historical spend lives here, not in the fresh account);
  • a matching Skipped entry replays as the same refusal — a human declined this call once, and resume must not override that decision;
  • any other failure does NOT replay: getting past it is what resume is FOR, so the call runs live again (and consumes fresh launch allowance — size max_calls with re-attempts in mind).

Identity is the call's work content — kind, input, max_steps, and the workflow's replay scope — never the display label. Duplicate identical calls are intentional samples (three identical verifiers), so matching consumes entries as a multiset: each replay consumes one entry, in file order, and only LIVE outcomes are recorded — replays are never re-appended, so entry counts stay honest across any number of resume generations.

A file-backed journal assumes ONE writing workflow process per path: appends are serialized within the process, not across processes.

Journal::in_memory

fn Journal::in_memory(prior? : Array[JournalEntry]) -> Journal

A journal with no backing file: prior seeds replay (empty by default, copied — later mutation of the argument cannot skew the consumption bookkeeping), live outcomes accumulate in recorded. The test-and-embed constructor.

Journal::load

async fn Journal::load(path : String) -> Journal

Open (or start) a file-backed journal: an append-only JSONL file, one entry per line, accumulated across every generation of the run. A missing file is an empty journal.

Crash honesty: a torn TAIL — an unparsable or undecodable final line, or a final line missing its newline — is the signature of dying mid-append. The damaged suffix is dropped from replay AND repaired on disk (atomic rewrite-and-rename), so the next generation's appends land on a clean boundary instead of concatenating onto torn bytes. Damage anywhere else raises JournalCorrupted. Lines are split at the byte level before UTF-8 decoding, so a tail torn mid-codepoint cannot poison the healthy prefix.

Journal::prior

fn Journal::prior(self : Journal) -> Array[JournalEntry]

The prior-run entries replay draws from, in file order (a copy).

Journal::recorded

fn Journal::recorded(self : Journal) -> Array[JournalEntry]

This run's live outcomes, in completion order (a copy — the journal's own bookkeeping cannot be skewed through it).

JournalEntry

pub(all) struct JournalEntry {
call : AgentCall
outcome : AgentOutcome
} derive(Eq, ToJson,
FromJson
)

One journal record: an agent call's work identity and what it resolved to. The journal is the workflow's durability seam — everything a resumed run needs to skip re-paying for work already done rides these entries, so the entry must lose nothing the live run knew (hence AgentOutcome, the lossless envelope, and not just the value).

JournalEntry::equal

JournalEntry::not_equal

fn JournalEntry::not_equal(x : JournalEntry, y : JournalEntry) -> Bool

JournalEntry::to_json

ReplayDecision

pub(all) enum ReplayDecision {
Serve(AgentOutcome)
Rerun
}

What the engine decided about a journal candidate at replay time: serve it (possibly REHYDRATED — the validator may substitute an outcome whose physical handles it refreshed), or veto it and run live. Raising out of the validator means validation itself failed — the claim rolls back and stays available.

Runner

pub struct Runner {
// private fields
}

The engine seam: HOW one agent call actually runs. The real runner spawns child processes; unit tests inject a fake that never leaves the process. Everything the run resolved to — success or failure, WITH its cost — rides the returned AgentOutcome; raise carries only what the workflow must not absorb: cancellation and engine bugs.

validate_replay guards STATEFUL outcomes at resume time: when a journal hit is about to be served, the engine may inspect it against live state (does the branch this outcome names still exist?) and decide. Engines whose outcomes are pure values need no validator: replay of a plain report can never lie.

Runner::Runner

fn Runner::Runner(run : async (AgentCall) -> AgentOutcome, validate_replay? : async (AgentCall, AgentOutcome) -> ReplayDecision) -> Runner

Wrap a run function (and optionally a replay validator) as a Runner.

Runner::invoke

async fn Runner::invoke(self : Runner, call : AgentCall) -> AgentOutcome

Run one call through this runner. Composing runners (a dispatcher delegating by kind) goes through here, never through the fields.

Workflow

pub struct Workflow {
runner : Runner
slots :
Semaphore

on_event : (WorkflowEvent) -> Unit
journal : Journal?
replay_scope : String
max_calls : Int?
calls_made : Int
replays : Int
prompt_tokens : Int
completion_tokens : Int
current_phase : String?
}

One workflow run: the runner seam, the concurrency gate every agent launch passes through, the launch allowance, and the running token account. Combinators (parallel, fan_out, policies) are free functions — this context only owns what must be shared state.

Workflow::Workflow

fn Workflow::Workflow(runner~ : Runner, max_concurrent? : Int, max_calls? : Int, journal? : Journal, replay_scope? : String, on_event? : (WorkflowEvent) -> Unit) -> Workflow raise

A workflow over runner, running at most max_concurrent agents at once and LAUNCHING at most max_calls agents over its lifetime (no cap when omitted — the caller's loop is trusted to terminate).

Workflow::agent

async fn Workflow::agent(self : Workflow, prompt : String, kind~ : String, hints? : String, label? : String, max_steps? : Int) -> Json

Run ONE sub-agent and return its report value, from the standard scout input shape ({query, hints?} — contract v1). Raises AgentFailed when the child produced no usable report and CallBudgetExhausted when the launch allowance was spent — catch into a Result at fan-out sites (try_agent is the shorthand), propagate at load-bearing ones.

Workflow::agent_call

async fn Workflow::agent_call(self : Workflow, kind~ : String, input~ : Json, label~ : String, max_steps? : Int) -> Json

The kind-agnostic core agent wraps: run one sub-agent from its exact child input. input doubles as the journal's replay identity, so callers encoding their own kinds get replay for free.

Workflow::calls_made

fn Workflow::calls_made(self : Workflow) -> Int

Agents actually launched so far (including ones still running); calls cancelled while queued and journal replays are not in this count.

Workflow::calls_replayed

fn Workflow::calls_replayed(self : Workflow) -> Int

Calls served from the journal instead of launching.

Workflow::log

fn Workflow::log(self : Workflow, message : String) -> Unit

Emit one narrator line.

Workflow::phase

fn Workflow::phase(self : Workflow, title : String) -> Unit

Start a new phase: agent calls ISSUED from now on carry this title in their AgentStarted events until the next phase call.

Workflow::tokens_spent

fn Workflow::tokens_spent(self : Workflow) -> Int

Total tokens spent by finished attempts this run — successes AND failures that ran a child. Replayed results will not re-charge here: historical spend lives in the journal, this counts fresh execution.

Workflow::try_agent

async fn Workflow::try_agent(self : Workflow, prompt : String, kind~ : String, hints? : String, label? : String, max_steps? : Int) -> Result[Json, WorkflowError]

agent with the typed error channel folded into the return value: workflow-level failures land in Err, cancellation and engine errors still propagate. The right shape at fan-out call sites, where one lost agent must not poison its siblings.

WorkflowEvent

pub(all) enum WorkflowEvent {
PhaseStarted(String)
Log(String)
AgentStarted(label~ : String, kind~ : String, phase~ : String?)
AgentFinished(label~ : String, disposition~ : AgentDisposition)
AgentReplayed(label~ : String, disposition~ : AgentDisposition)
}

Progress the workflow narrates as it runs: phases group agents in a display, logs are one-line narrator messages, and the started/finished pair brackets each agent's lifetime — on every path, including cancellation. Purely observational: no control flow rides on these.

all_ok

fn[T] all_ok(results : Array[Result[T, WorkflowError]]) -> Array[T] raise WorkflowError

Policy: every slot must have succeeded; re-raise the first failure otherwise. Post-hoc fail-fast — siblings have already run to completion; use parallel_all when failure should cancel them instead.

collect_ok

fn[T] collect_ok(results : Array[Result[T, WorkflowError]], min_ok? : Int) -> Array[T] raise WorkflowError

Policy: keep the successes, requiring at least min_ok of them (QuorumNotReached otherwise). min_ok=0 is pure best-effort collection; a negative min_ok is treated as 0.

fan_out

async fn[A, B] fan_out(items : Array[A], run : async (A) -> Result[B, WorkflowError]) -> Array[Result[B, WorkflowError]]

Fan items out through one async stage, one Result slot per item. Multi-stage pipelines are function composition inside run — stages need no barrier between them, so composing them per-item IS the pipeline.

parallel

async fn[T] parallel(thunks : Array[async () -> Result[T, WorkflowError]]) -> Array[Result[T, WorkflowError]]

Run thunks concurrently, each already carrying its failure policy in its type: a thunk resolves to Result[T, WorkflowError] (build them from try_agent), so one agent's failure lands in its own slot without disturbing siblings — tokens already spent on the others stay spent. Anything a thunk RAISES is deliberately not caught: engine bugs and cancellation fail the task group and cancel every peer, never demoted to a slot a policy could quietly discard. Pair with all_ok / collect_ok / quorum to choose the policy in one identifier.

parallel_all

async fn[T] parallel_all(thunks : Array[async () -> T]) -> Array[T]

Fail-fast fan-out: the first thunk to RAISE (an agent call propagating its AgentFailed, an engine bug, cancellation) cancels every sibling still in flight and propagates. The right shape when later work is worthless without ALL of this stage — otherwise prefer parallel, which keeps what succeeded.

quorum

fn[T] quorum(results : Array[Result[T, WorkflowError]], need~ : Int) -> Array[T] raise WorkflowError

Policy: k-of-n agreement gates (adversarial verification, vote panels): at least need slots must have succeeded.