Skip to content

Tuning the governance dials

kew's governance dials interact: a budget cap can silently starve a review cycle, a concurrency cap can make a healthy epic look stalled, and a breaker floor tuned for one model class is wrong for another. This page maps each dial to the failure mode you'll actually observe when it's mis-set — every entry here was hit for real during dogfooding, with the incident issue linked.

The short version: when an epic looks stuck, check holds, then run rows, then slots — in that order.

kew epic status <name>          # holds + stuck reasons, ready frontier
sqlite3 logs/dispatch/dispatch.db \
  "select issue_num, run_kind, status from runs where status='pending'"

[parallel] max_workers — the fleet slot pool

What it governs. The global cap on concurrently live runs — and "live" means every pending run row: implementation dispatches, adversarial reviewers, remediations, and operator-invoked kew dispatch / kew review runs all draw from the same pool.

Failure mode: silent review starvation. Reviewer dispatch under a full fleet is refused with a fleet_full result. The refusal is recorded as a refused run row, but the driver's stdout keeps printing signal=progressing — nothing tells you reviews are being starved. Observed: two operator dispatches plus one reviewer saturated a max_workers = 3 fleet, and an epic's due review was refused every tick for ~10 minutes.

Tuning. Size it as: expected concurrent epic nodes + 1–2 reviewer slots + headroom for operator dispatches you run by hand alongside a drive. If you drive epics while also dispatching ad hoc, 3 is too small; 6 is a sane single-operator default. Watch for refused run rows as the signal it's still too tight.

[review] max_review_usd — the per-node review pool

What it governs. Total spend per epic node across ALL review-stage activity: every verdict round and every governed remediation draws from the same pool.

Failure mode: mid-cycle exhaustion → HELD. A full adversarial cycle on a frontier model costs real money (observed: verdict rounds ~$2.50, governed remediations $3–5 each). Two rounds + remediations ran $11–13 per node — through a $10 default pool. The node HELDs with review_budget: review pool exhausted after the work that found the bugs is already paid for, stranding an almost-done node. (#993's original misdiagnosis was exactly this dial; drilled again live 2026-07-26.)

Tuning. Budget for max_rounds full cycles: roughly max_rounds × (verdict cost + remediation cost) for your reviewer model. For an Opus-class reviewer with max_rounds = 2, $10 is structurally too small; $25–40 matches observed costs. The pool is the ceiling on rigor — an underfunded pool converts caught-bugs into operator toil.

[review] max_rounds — the remediation-cycle cap

What it governs. How many verdict→remediate cycles a node gets before the driver HELDs it (review_rounds_exhausted) for human judgment.

Failure mode: legitimate convergence cut short. A review that finds different real bugs each round (not thrashing) hits the cap while converging — observed: round 1 caught a pricing bug, round 2 caught a deeper resume-bookkeeping bug, both real, node HELD anyway. The operator then owns the tail: remediate by hand (kew review <PR>), raise the cap, or merge over the findings.

Tuning. 2 is right for keeping cost bounded on routine nodes. For governance-critical or cross-cutting nodes, expect 3. Read the held node's round history before deciding: distinct findings each round → grant another round; the same finding rephrased → the reviewer and implementer disagree, and that's a human call, not another round.

Interaction warning. max_rounds × cost-per-cycle must fit inside max_review_usd or the pool HELDs you before the round cap ever applies.

[agent] max_budget_usd vs [budget] per_issue_usd

What they govern. The single-run spend ceiling vs the lifetime per-issue ceiling (which includes spend from killed/failed prior runs).

Failure mode: retry death-spiral. If per_issue_usd doesn't cover sunk spend from killed runs plus one fresh run at the runtime cap, a retried node fails preflight forever (#445). Separately, a runtime cap larger than the remaining per-issue headroom means a run can spend past the per-issue budget before the preflight gate ever sees it — kew warns about this at dispatch.

Tuning. Keep per_issue_usd ≥ 1.5–2× max_budget_usd so one kill plus one retry fits. Heed the dispatch-banner warning when overrides skew the pair.

[breaker] — cost-anomaly factor and floor

What it governs. Runaway-spend kill. Since #1116 the floor is bucket-relative, and the threshold is the largest of three terms:

Term Value When it wins
median_factor bucket median × cost_anomaly_factor established buckets (≥ min_samples)
relative_floor bucket median × anomaly_floor_multiple under-sampled buckets, or when the multiple exceeds the factor
abs_floor min_anomaly_threshold_usd near-zero buckets only

Every term is proportionate to the bucket's own median — including on the under-sampled path, where the bucket has samples but fewer than min_samples.

An under-sampled bucket arms only when both conditions hold:

  1. It has at least three of its own samples. One or two observations are not a median — arming off them makes the threshold a coin flip on the first run's size — so anything thinner stays dormant, as it did before #1116.
  2. That median has reached the route reference: the same baseline query with the run-kind predicate inverted, i.e. the median over every other kind the route has run. It carries the same exclusions, so the two can never disagree about which rows count as normal.

The reference gates arming and never sets the threshold. It measures other kinds of work, so a review_remediation bucket seen only cheaper than the rest of its route tells you nothing about what a normal remediation costs there, and 3× one cheap sample would SIGTERM a legitimate one — the #992 failure again. Excluding the bucket's own kind is what makes the comparison mean anything: read across all kinds, the reference is a superset of the bucket, and on a route that has only ever run one kind it equals the bucket median exactly, so the gate would pass for free.

Dormant, likewise, for a bucket with no samples at all, one on a route with no other kinds (no reference to clear), and one where anomaly_floor_multiple = 0. In each, the only thing left to arm on is the flat min_anomaly_threshold_usd — the "floor is the whole threshold on a fresh bucket" mode that killed #992. The absolute floor never arms a bucket on its own; it only raises a threshold some relative term already set.

Why it's shaped this way (#1116). A single absolute floor failed in both directions once routing spanned Opus and OpenRouter flash models in one fleet:

  • Floor too high → breaker decorative. With cheap OpenRouter routes (~$0.08 medians), a $50 floor is 69× the median — a runaway needs ~600× overshoot before the breaker notices.
  • Floor too low for fresh buckets → kills legitimate work. A new run-kind bucket has no median, so the floor IS the threshold; a legitimately large run gets SIGTERM-killed at exactly the floor (the #992/#993 incident, 3×, ~$20 lost).

Tuning. Leave min_anomaly_threshold_usd small ($1–2): its only job is to absorb noise in near-zero buckets. Note that lowering it is not behaviour-preserving — every established bucket whose median sits below floor ÷ factor was governed by the floor and is now governed by median × factor, which is tighter. Reach for anomaly_floor_multiple when a bucket's spread is wide — raising it above cost_anomaly_factor loosens every bucket proportionately instead of flattening them onto one dollar value. Each dispatch logs the armed threshold and which term won (Circuit-breaker cost-anomaly threshold $X … via median_factor (…)), so a drill can verify arming without tripping it.

max_budget_usd enforcement — runner coverage

Mid-run budget kills are enforced natively by the claude-code runner and by kew's own watchdog for other runners (#1108, fixed 2026-07-25). If you run a kew older than that fix with non-claude runners, the per-run cap is preflight-only — verify with a low-cap drill run before trusting it.

Runtime overrides from the web surface

Budget and routing knobs set through the web config manager persist in dispatch.db (config_overrides) and win over the committed kew.toml — the dispatch banner marks them (override). A forgotten override from a heavy week quietly quadruples your caps. Audit with:

sqlite3 logs/dispatch/dispatch.db "select * from config_overrides"

Operational failure modes (not dials, but you'll hit them)

  • Don't kill the terminal to pause work. Killing the shell/driver can take in-flight governed runs down with it — inconsistently (SIGTERM to the process vs the tree). Use kew stop / kew resume. (#1124)
  • A dead fleet is currently silent. A driver whose lock-holder PID is dead plus pending runs with dead process groups produces no alarm — check kew epic status and the lock file when things go quiet. (#1124)
  • kew epic retry needs a fresh driver. A long-running poller that performed an orphan-reap can wedge: it prints progressing but ignores retry markers. Bounce the driver; the fresh one dispatches re-admitted nodes on its first tick. (#1119)
  • Operator-invoked runs are invisible to the epic reaper. A killed kew review <PR> or ad hoc dispatch leaves a pending corpse that holds a fleet slot until cleaned by hand. (#1124)
  • Contained runs need the credential in the driver's environment. Sandbox resolution is credential-driven: with the agent token exported everything goes contained — including runners whose CLI isn't in the sandbox image (fails closed at $0); without it, everything downgrades to guard-only. Mixed-runner epics are all-or-nothing until per-runner images land. (#779)