Skip to content

How It Works

                    ┌─────────────┐
                    │ Human adds  │
                    │ "ready"     │
                    │ label       │
                    └──────┬──────┘
                           │
                           ▼
               ┌───────────────────────┐
               │     kew run          │
               └───────────┬───────────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
         ┌─────────┐ ┌─────────┐ ┌──────────┐
         │ Select  │ │ Create  │ │ Build    │
         │ highest │ │ git     │ │ prompt   │
         │ priority│ │ worktree│ │ from     │
         │ issue   │ │         │ │ template │
         └────┬────┘ └────┬────┘ └────┬─────┘
              │            │           │
              └────────────┼───────────┘
                           ▼
               ┌───────────────────────┐
               │  Configured runner    │
               │                       │
               │  Read CLAUDE.md       │
               │  Implement (TDD)      │
               │  Run CI               │
               │  Commit + Push        │
               │  Create PR            │
               └───────────┬───────────┘
                           │
              ┌────────────┴────────────┐
              ▼                         ▼
       ┌─────────────┐          ┌─────────────┐
       │  Success    │          │  Failure    │
       │             │          │             │
       │ Labels:     │          │ Labels:     │
       │ pr-pending  │          │ agent-      │
       │ -review     │          │ failed      │
       │             │          │             │
       │ Worktree    │          │ Worktree    │
       │ cleaned up  │          │ preserved   │
       └─────────────┘          └─────────────┘

The dispatch loop

In --loop mode with multiple workers, the full cycle runs continuously:

  1. Select -- query GitHub for ready-labeled issues, ordered by priority (p0 > p1 > p2 > p3), excluding any currently in-flight
  2. Isolate -- create a git worktree per issue on a dedicated branch (agent/issue-{N})
  3. Prompt -- build a Jinja2 prompt from the issue body, complexity labels, and your config
  4. Execute -- spawn the resolved runner with its governed command, timeout, budget controls, and security posture. Before this, the approval gate estimates cost and either confirms or holds the issue.
  5. Stream -- parse the runner's structured events in real time, showing tool use, token counts, and progress
  6. Retry -- if the agent exits with a retriable code, back off and retry (with --resume to reuse session context)
  7. Complete -- on success, push the branch, create a PR, label it pr-pending-review, and clean up the worktree. On failure, label agent-failed and preserve the worktree for debugging. For managed adaptive Switchyard runs, Kew snapshots the owned proxy before teardown, drains classifier records during teardown, reconciles the served model/tier before pricing, and marks incomplete terminal accounting unreconciled in both audit evidence and spend reports.

Dispatch outcomes

Each dispatch ends in one of the following outcomes:

  • Completed -- the agent called finish and committed coherent work. kew pushes the branch, creates a PR labeled pr-pending-review, and cleans up the worktree.
  • No changes -- the agent ran but made no changes and called finish. kew closes the issue without creating a PR.
  • Failed -- the agent crashed, hit a timeout, or exhausted its budget without finishing. kew labels the issue agent-failed and preserves the worktree for debugging.
  • Awaiting publication -- the agent completed, but the host could not publish its branch or PR. kew commits the work before remote publication, records the run as awaiting_publication, preserves the worktree, and records a continuation handle for kew epic resume. The epic driver does not dispatch implementation again. Host-publication PR discovery and creation use GitHub REST, so an exhausted GraphQL budget does not block this recovery path. Each tick reconciles the preserved branch against GitHub: an open PR moves the node to in_review; a merged PR finalizes the run and closes the Linear item (or observes GitHub's issue close). A successful run corroborated by an open PR also stays in_review if an older continuation handle is stale. A provider item already marked completed similarly wins over a stale run overlay. Older residue can be repaired explicitly with kew run finalize <run-id>, which is chain-attested. The publication_deferred audit event and dispatch log identify the failed step (commit, push, or pr_create) and retain a bounded, secret-free error detail.
  • Rescued (incomplete run) -- the agent exhausted its step budget (max_steps) without calling finish but left real worktree changes. kew commits and pushes them, opens a draft PR labeled incomplete-run, flags the issue agent-needs-help, and preserves the worktree. The issue is not closed and dependents are not unblocked — a human reviews the draft, then marks it ready and merges, or re-labels ready to retry from scratch. Only max-steps exhaustion triggers this; crashes, timeouts, and budget stops remain hard failures.

Troubleshooting deferred publication

publication_deferred means the implementation completed, but the host publication broker could not finish its commit, push, or pr_create step. It is a publication problem, not an agent failure: the run is recorded as awaiting_publication, the worktree is preserved, and the implementation is not dispatched again.

Read the publication_deferred audit event together with the dispatch log. Both identify the typed step (step=commit, step=push, or step=pr_create) and include a bounded, sanitized error value. The error is diagnostic evidence, not raw provider output; it should be safe to inspect without exposing credentials. When the epic frontier reports this as a waiting cause, its operator hint is: “restore GitHub access or open the preserved branch PR.”

There are three recovery exits, in order of preference:

  1. External PR, then driver reconciliation. Restore GitHub access or open an external PR from the preserved branch. Run the epic driver again; its next tick discovers the branch PR, moves the node to in_review, and later reconciles an external merge.
  2. Explicit finalization. If publication was completed out of band but the run still has older awaiting_publication residue, use kew run finalize <run-id>. This chain-attested repair marks the run complete; it does not publish a branch or create a PR.
  3. Preserved-worktree continuation (last resort). If no PR can be opened, continue the preserved checkout explicitly with kew epic resume <name> <node>. This validates the recorded worktree and branch, reuses the existing bytes, and hands publication back to the normal broker.

Label lifecycle

State Labels Meaning
Queued ready + priority Waiting for dispatch
Working in-progress + agent-working The configured runner is implementing
Held needs-approval Approval gate fired — agent paused until a human approves
Done pr-pending-review PR created, needs human review
Failed agent-failed Something went wrong (comment explains why)
Rescued issue: agent-needs-help + incomplete-run; draft PR: agent-pr + incomplete-run Max-steps exhausted; draft PR opened, awaiting human decision

The Held state occurs when the approval gate (if configured) estimates the dispatch cost at or above your threshold. kew removes in-progress/agent-working and adds needs-approval so the issue sits inert in the queue. Release it via kew approve --issue N, the TUI A key, a /approve GitHub comment, or the web inbox (kew serve).

Loop guardrails

Mid-flight budget watchdog. A delegated loop is stopped as soon as its running cost — prior spend plus this session's incremental, usage-derived cost (a list-price estimate) — crosses budget_usd, transitioning to budget_limited (resume with a bumped budget via kew loop resume <id> --budget <n> --guidance "…"). The check is on that live token-derived running cost, not the reported spent_usd aggregate — spent_usd is the persisted total and may lag the live figure depending on the runner. Enforcement is one cost event late (post-hoc): the loop is killed after the event that reports the overshoot, so the figure is a near-cap, not an exact cap. Runners that emit no incremental usage cannot be enforced mid-flight and fall back to the wall-clock cap (the current native runners — claude-code, opencode, pi — all emit per-turn usage).

Governance layer

The dispatch loop sits inside a control plane designed to keep autonomous runs observable and stoppable:

  • Tamper-evident audit log — every dispatch event is written to a hash-chained log; kew audit verify checks integrity and exits non-zero on any gap.
  • Fleet kill switch — kew stop kills all in-flight runs and halts dispatch immediately; kew resume re-enables it. Issues are reset to ready so no work is lost.
  • Approval gates — configure a cost threshold and kew holds issues under needs-approval before the agent runs. Approve via kew approve, the TUI A key, a /approve GitHub comment, or the web inbox (kew serve).
  • Outcome-linked spend — kew report correlates API spend to merged PRs, failed runs, and loop steps so you can see what each outcome actually cost.
  • First-class loops — kew dispatch <issue> --loop runs an issue as a supervised multi-step loop (distinct from kew run, which runs the whole queue). Pause or resume mid-flight with kew loop pause / kew loop resume.

The epic driver tick {#the-epic-driver-tick}

kew epic advance/run (see kew epic) drive a whole epic:<name>-labeled dependency graph, one issue-per-node, instead of a flat queue. Because it dispatches and merges without a human driving each step, it owns a few primitives the flat fleet doesn't need:

  • Single-driver lock + atomic reservation — the whole tick runs under a non-blocking advisory lock (a file under logging.dir); a second advance/run that can't acquire it exits cleanly instead of racing. Before spawning any subprocess, the driver also reserves the node with a run-store write guarded by a uniqueness constraint, so the in-flight count is authoritative the instant a dispatch is decided — not moments later, once the subprocess has started.
  • Driver-owned status overlay — the driver computes a richer status per node than the provider adapter alone can: IN_PROGRESS/FAILED are derived from the run-store (a live pending run, or the newest terminal run being a failure kind), DONE requires the issue closed as COMPLETED (a not-planned close reads as CANCELLED, a terminal state that does not satisfy dependents), and HELD is an overlay-only pseudo-status for a node that's approved but waiting on CI (see the merge gate below).
  • Bounded GitHub PR reads for Linear — one fresh gh pr list batch supplies the PR facts for every active Linear node in a tick and is reused by the merge pass. Completed Linear nodes skip GitHub entirely, and a failed batch does not fan out into one fallback query per node.
  • Positive epic:* fleet exclusion — the fleet's issue selector skips any issue carrying an epic label outright, regardless of ready, so the flat fleet can never grab a node the driver owns mid-flight; the fleet's conflict auto-resolver excludes epic PRs the same way.
  • Approval-gated, CI-green merge — the driver merges a node's PR only when a verified, non-author human approval is on record, a fresh GitHub lookup confirms the reviewer still has write authority, and the PR's required checks are green. Bot identities, missing permission evidence, and lookup failures fail closed. See the merge gate for the exact rule and why there is deliberately no autonomous --auto-merge.
  • Failure containment — a FAILED node is terminal for the tick (never auto-redispatched); its transitive_dependents are reported as its blast radius so an operator can see exactly what a failed, cancelled, or stuck node is holding up. kew epic retry <name> <node> is the only way back in.
  • Preserved dirty-work recovery — retrying a failed node whose existing worktree still contains uncommitted changes does not recreate or reset that worktree. The driver records the exact branch, worktree, and original run as an awaiting_continuation and leaves the bytes untouched. Resume that work explicitly with kew epic resume <name>; a normal retry never launches a second implementation writer against the preserved checkout.
  • External merge reconciliation — awaiting_merges remains in the poll roster. A hand-merged PR runs through the same hold and configured-label cleanup as a driver merge, advances dependents on that tick, and is never attested as a kew-performed merge.
  • Provider-aware node dispatch — each node is dispatched by a detached child (kew _dispatch-node) that adapts to the configured tracker. For GitHub the node id is the issue number and the classic issue → worktree → PR flow runs unchanged. For a non-GitHub tracker (Linear), the child instead fetches the item's title/description/labels from the tracker and claims it (assigns the viewer, moves it to the started state) before the agent spawns; the opaque item id (e.g. SFD-133) threads through the branch (agent/issue-sfd-133), worktree, run-store row, and PR body while the GitHub-only issue number stays a 0 sentinel. GitHub-issue label edits are skipped for non-GitHub trackers (their state is owned by the claim and the merge pass, not labels). A missing key, a fetch failure, or a claim failure is a loud exit that writes a terminal run record against the node's reservation rather than a silent skip, and a refusal before the agent spawns (fleet-stop, worker-cap, budget) rolls the Linear claim back so the item is never stranded In Progress with nothing running.
  • Reliable child resolution + dead-spawn diagnostic — the detached child is spawned via the kew console script next to the driver's own interpreter (the active venv's bin/), preferred over an ambient PATH kew that could be a stale or mismatched install, and the resolved path is logged at spawn. A child that crashes before writing its first run row (a bad import, a config failure) fails silently under DEVNULL stdio; once its reservation ages past the startup grace with no run row, the driver fails the node with a terminal dispatch_child_died diagnostic that names the spawned executable — instead of silently re-spawning the same broken child every grace window.
  • Empty graph is an error, not success — an epic whose tracker query resolves to zero nodes (a typo'd epic:<name>, or an advance/run fired from a cwd wired to the wrong tracker) raises before any tick is journaled and exits 2 with a diagnostic naming the empty query (the GitHub label or the Linear project) — it no longer prints COMPLETE done=0 remaining=0. A driver never declares success for work that does not exist; complete stays reachable only for a non-empty graph whose nodes are all DONE. kew epic run --all surfaces the empty epic as needing an operator (contributing exit 1 to the aggregate) instead of silently dropping it or reading it as complete.
  • Post-action tick summary — the merge pass runs against the snapshot taken at tick start, so a node the tick itself merges or closes still reads IN_REVIEW in that snapshot. Before reporting, the tick folds its own actions — dispatched, merged, and out-of-band-closed nodes — back onto the overlay and derives every printed field (waiting/done/remaining and the tri-state signal) from that post-action view. A close that unblocks the frontier this tick reads progressing, not stalled, and stdout agrees with the journal on what the tick just did — the new state no longer only appears on the next tick.
  • Tracker-aware id rendering — every item id the driver prints (status lines, waiting/blocked on reasons, a failed node's blast radius) routes through one item_ref rule: a GitHub numeric id renders #123 (which GitHub turns into a live cross-link), while any other tracker's id renders bare (SFD-133). So a Linear node reads blocked on SFD-133 while a GitHub node stays byte-identically blocked on #123.

Each tick reports complete/progressing/awaiting_merges/stalled. awaiting_merges keeps polling for asynchronous human merges; stalled exits for repair. Full detail lives in kew epic.

Governed adversarial review

Every substantive defect that reached the merge surface across the delegation-hardening waves was caught not by the implementing agent's own tests (honest, green) but by a fresh-context adversarial reviewer run by hand: same-context self-review structurally cannot see interactions with actors outside its context. The governed adversarial-review stage makes that manual loop a pipeline stage the control plane owns, between "an agent opened a PR" and the human approval surface. It is opt-in ([review].adversarial = true) and off by default.

Wiring status

The driver now drives the full per-round lifecycle end-to-end: on each tick a review-enabled node due a reviewer has one dispatched against the recorded head SHA, its host-stored sidecar is parsed, the round is recorded, and the verdict is attested — then the node transitions (READY → in_review, NEEDS_CHANGES → remediating, round cap / unparseable → HELD). The reviewer launcher (the sandbox that actually runs the fresh-context reviewer and writes the sidecar) is injected as a seam: a read-only, non-pushing reviewer sandbox is the one remaining production piece, so until a launcher is wired at the driver call site the drive pass logs loudly and takes no action rather than silently clearing the gate. Do not enable blocking = true in production until a reviewer launcher is wired — the merge-pass invariant is live and a policy-covered node with no verdict blocks forever by design. Advisory mode is fully operational with a launcher injected; the out-of-band-merge race (attesting MERGED_UNREVIEWED) is live regardless.

How the stage works. When enabled, a node that opens a PR enters adversarial_review instead of going straight to in_review. The driver records the PR head SHA and dispatches a fresh-context, sandboxed reviewer against that recorded SHA — never the implementing session, and with the head-ref auto-load files (CLAUDE.md, .claude/, .mcp.json, …) masked to their base-ref copies so code under review can't rewrite the reviewer's own instructions. Kew resolves both commits inside the reviewer clone and pins them as refs/kew/review/base and refs/kew/review/head. The reviewer prompt names git diff --no-ext-diff --no-color refs/kew/review/base...refs/kew/review/head -- as the authoritative PR boundary; working-tree status can include masking artifacts and does not define PR scope. The reviewer writes a structured verdict to a broker-collected sidecar (stored on the host, not a deletable PR comment); kew parses only the sidecar. A READY verdict carries executed acceptance evidence; a NEEDS_CHANGES verdict carries evidenced file:line findings, which feed an adapted kew review remediation pass (remediating) that loops back for round N+1 against a new head SHA. Non-convergence within [review].max_rounds (default 2) HELDs the node for a human — never a silent loop. Reviewer and remediation runs draw from a separate attested budget pool ([review].max_review_usd), so review work can never starve implementation retries or vice versa.

Remediation publication is bound to the head that was reviewed. If a reviewer has already pushed its clean review commit before the host broker runs, the broker verifies that commit (or a fast-forward containing it), records that SHA, and completes without a second push. An unrelated branch move remains a publication failure and is reported as “review completed but was not published because the branch moved.”

Advisory → blocking adoption path. The stage has two modes, and the recommended path is to adopt it advisory-first:

  • Advisory (blocking = false) — the verdict (or a DEGRADED marker if the reviewer crashed) is attested and the node proceeds under the pre-existing approval + CI rules. A non-READY verdict remains visible: DEGRADED is published as adversarial review: DEGRADED (advisory) with a successful kew-review status, so it does not make an otherwise-green PR ineligible for delivery. Unrelated failed checks remain blocking. This is the on-ramp: turn it on, watch the verdicts against your real PRs, and tune the pinned prompt artifact before you let it gate anything.
  • Blocking (blocking = true, the default once adversarial is on) — a non-READY outcome (reviewer crash after one retry, unparseable sidecar, review-budget refusal, a post-verdict force-push) all land in the same place: HELD: needs-human, with an attested reason. There is no advisory-degrade path inside blocking mode, so a PR-controlled reviewer crash cannot bypass the gate.

THE invariant (enforced at the merge pass, not the status label). HELD is a live merge candidate — approval + green CI would merge it — so gating only the status transition would leave a hole. Instead the merge pass gains one invariant, checked at merge time for every policy-covered candidate of a review-enabled epic:

merge only if the latest verdict is READY ∧ verdict.head_sha == the current head SHA ∧ the approval timestamp postdates the verdict.

This is what makes blocking mode real: it closes the HELD-is-mergeable hole, enforces merged-SHA == reviewed-SHA at the merge itself (a post-verdict force-push can no longer merge unreviewed code), and is the natural home of the approval-postdates-verdict rule (a stale, pre-existing approval surfaces as "re-approve" rather than auto-merging). A candidate that fails the invariant is skipped with a loud driver line and an attested reason — never a silent skip. Nodes already at/past in_review when the flag flips are grandfathered (PRE_POLICY) and merge under the pre-existing rules; flipping the flag never throws an approved PR back into review.

The GitHub-native gate (kew-review status check). The merge-pass invariant enforces kew's own merge — it cannot reach the GitHub merge button. A human (or an autonomous flow) can click "Merge" on GitHub before the reviewer finishes, and the verdict then lands on an already-merged PR (incident #1013). To close that race, kew mirrors the same verdict onto the PR head SHA as a kew-review commit status ([review].publish_status = true, on by default): pending the moment a reviewer round is dispatched, success on a READY bound to the current head, failure on NEEDS_CHANGES or any crash-path HELD, and back to pending when a post-verdict force-push moves the head (keyed to the exact reviewed SHA, mirroring the invariant's head-sha binding). Posting is best-effort — a gh/GitHub failure is logged and swallowed, never crashing the tick. Publishing the kew-review status alone does not block merges — the check only enforces the merge button once a branch-protection rule on main requires it. To make it block the merge button, enable a branch-protection rule on main requiring the kew-review check; that is the piece the code cannot enforce. Apply it once the check first publishes:

gh api -X PUT repos/OWNER/REPO/branches/main/protection/required_status_checks \
  -f strict=true -f 'contexts[]=kew-review'

This is an additional GitHub-native gate that mirrors the verdict; it does not change the verdict contract or the kew-driven merge-pass invariant.

The verdict is an input to human approval — never a replacement for it. Human review has repeatedly caught what the full AI chain missed, and nothing here dilutes that gate: there is no auto-merge on READY. Approval stays human; the chain-attested verdict is evidence for that decision — which exact commit was reviewed, under which policy, at what spend — never a reason to rubber-stamp. A reviewer can still run tests and lazily approve; that residual is bounded by the human gate (and a v2 sampled second-reviewer audit), not hidden.

Scope (v1). The stage proves itself on the epic path, where a human is already watching the approval surface. The unattended delegation surface (kew delegate watch) — the surface that arguably needs adversarial review most — is deliberately out of scope until v1.1: extending the stage there is mechanical once the shared reservation reconciler is proven, but no unattended operation is advertised as review-gated until it actually is.

Sandboxed dispatch

When [sandbox] enabled = true, kew replaces the host-subprocess agent launch with an ephemeral container per dispatch. The flow diverges from the standard dispatch loop at step 4 (Execute):

  1. Standalone checkout — kew clones the repository into a temporary directory at the base commit of the issue branch. This self-contained checkout is the container's only volume mount — host ~/.claude, dispatch.db, and GitHub credentials are outside the mount and invisible to the container.
  2. Container run — kew builds the docker run argv with hardening flags (--rm, --read-only, --cap-drop ALL, --security-opt no-new-privileges, --pids-limit) and launches the agent image. The resolved runner's own provider credential is forwarded by name (-e ANTHROPIC_API_KEY for claude-code, -e OPENROUTER_API_KEY for an OpenRouter-backed runner) — never the value, and never another runner's key; see the per-runner credential allowlist. If that credential is absent from the host environment, the dispatch is refused before the container starts. No GitHub token is passed.
  3. Agent commits — the agent works entirely within the mounted checkout, commits the branch, and exits. The container is removed (--rm).
  4. Host-brokered push and PR — after the container exits cleanly, kew pushes the branch and creates the PR through GitHub REST on the host, using the host's GitHub credentials. For epic-driver children, kew resolves stored gh auth before detaching and carries it as a host-only capability; authenticated HTTPS publication therefore does not depend on a controlling terminal, and the credential is removed from the agent subprocess environment. The agent never touches these paths.

The interface between kew and the image — mounted checkout, prompt-file→stdin handoff, "$@" extra args, one JSON event per line on stdout, and the reviewer verdict sidecar — is fixed and documented in the container entrypoint contract. Per-runner variation lives inside the image; kew's host-side contract does not change.

The security guard (.claude/hooks/security-guard.sh) is still active inside the container as best-effort layer 2 defence-in-depth. It is known-bypassable in-process (#502); the mount boundary is the primary containment. See Security and Configuration for details.

Official Codex follows a separate host path: Codex-native workspace-write governs the agent process, command network is off, and kew records external container posture as disabled. On exit 0, kew's host broker owns commit, push, PR creation, and the dispatch report. Failure of that ceremony remains a failed dispatch. Usage is reported as API-equivalent included-plan value, not a native bill.

Prompt system

The dispatch prompt is built from a Jinja2 template with configurable sections:

  • Safety rules -- branch protection, no secrets
  • Execution strategy -- TDD workflow, subagent delegation for plans
  • Token budget -- complexity-aware guidance (simple/medium/complex)
  • Verification -- your CI and format commands
  • CodeRabbit -- optional pre-PR review (2 rounds)
  • Dispatch report -- structured PR summary template. If the agent finishes without posting one, kew posts a fail-soft fallback report (cost, tokens, duration, attempts) on the PR after dispatch, so the review surface stays consistent (#664)

Override the template entirely with prompt.template, or append extra instructions with prompt.append.